Content extraction method, apparatus, device, storage medium, and product

Abstract

The application relates to the technical field of illegal website detection, and discloses a content extraction method, device and equipment, a storage medium and a product, which comprise the following steps: in response to a content extraction request of a target page, loading the target page based on a preset browser instance to obtain a rendered target page; performing text recognition on a visual content image of the rendered target page to generate a text recognition result; and determining structured text content of the target page according to the text recognition result. The application shifts the entry point of content extraction from a document object model structure to a visual presentation level, presents the page to a final visual state through browser dynamic rendering, and then extracts text from the corresponding image. As long as the content is visually visible in the browser, it can be effectively extracted, and all interference contents injected through a visually invisible mode are naturally excluded because they do not exist in the visual content image, so that the purity and accuracy of the extracted content are significantly improved.

Description

Technical Field

[0001] This application relates to the field of illegal website detection technology, and in particular to a content extraction method, apparatus, device, storage medium and product. Background Technology

[0002] Currently, in order to combat automated content scraping, illegal websites commonly inject a large amount of visually hidden interference content into the page's DOM (Document Object Model) structure. This interference content exists mixed with normal content in the DOM structure, making it difficult for DOM-based content extraction schemes to effectively distinguish between visually visible real content and visually invisible interference content, resulting in low-quality extracted content. Summary of the Invention

[0003] The main objective of this application is to provide a content extraction method, apparatus, device, storage medium, and product, which aims to solve the technical problem that DOM-based content extraction schemes have difficulty distinguishing visually invisible interference content, resulting in low-quality extracted content.

[0004] To achieve the above objectives, this application proposes a content extraction method, which includes: In response to a content extraction request for the target page, the target page is loaded based on a preset browser instance to obtain the rendered target page; Text recognition is performed on the visual content image of the rendered target page to generate text recognition results; The structured text content of the target page is determined based on the text recognition results.

[0005] Optionally, the step of responding to a content extraction request for the target page and loading the target page based on a preset browser instance to obtain a rendered target page includes: In response to a content extraction request for the target page, the target address corresponding to the target page is input into the browser instance to obtain the target page; Based on the browser instance, a multi-rendering waiting strategy is executed on the target page to obtain the rendered target page.

[0006] Optionally, the step of performing a multi-rendering waiting strategy on the target page based on the browser instance to obtain the rendered target page includes: Based on the browser instance, the target page is monitored for network idle status to obtain the first waiting completion result; Based on the browser instance, key page elements of the target page are monitored to obtain a second pending completion result; Based on the browser instance, a simulated user scrolling operation is performed on the target page to obtain a third waiting completion result; The rendering wait time is determined based on the first wait completion result, the second wait completion result, and the third wait completion result. The target page is rendered based on the rendering wait time to obtain the rendered target page.

[0007] Optionally, before loading the target page based on a preset browser instance in response to the content extraction request of the target page to obtain the rendered target page, the method further includes: Launch a headless browser instance based on a pre-defined browser automation framework; The headless browser instance that has been launched is subjected to anti-automatic detection configuration to obtain the configured browser instance.

[0008] Optionally, the step of performing anti-automation detection configuration on the launched headless browser instance to obtain a configured browser instance includes: An anti-detection script is injected into the page context of the headless browser instance, and the headless browser instance is adjusted according to the anti-detection script; Configure the adjusted headless browser instance to obtain the configured browser instance.

[0009] Optionally, the step of performing text recognition on the visual content image of the rendered target page to generate text recognition results includes: Take a screenshot of the rendered target page to obtain a visual content image; The visual content image is subjected to optical character recognition by a preset recognition engine to obtain text recognition results, which include text content, confidence level and location information.

[0010] Optionally, taking a screenshot of the rendered target page to obtain a visual content image includes: Based on the page height of the rendered target page, a screenshot is taken of the rendered target page to obtain an initial image fragment; Feature matching is performed on the overlapping regions between the initial image segments, and a visual content image is generated based on the matching results.

[0011] Optionally, determining the structured text content of the target page based on the text recognition result includes: The identified text content is filtered based on the confidence level in the text recognition results to obtain the filtered text content. Based on the location information in the text recognition results, the filtered text content is structured to generate the structured text content of the target page.

[0012] Optionally, the step of performing structuring processing on the filtered text content based on the location information in the text recognition result to generate the structured text content of the target page includes: The filtered text content is merged based on the horizontal coordinates in the location information to obtain text lines; The text lines are clustered into paragraphs based on the vertical coordinates in the location information to obtain paragraph segmentation results; The structured text content of the target page is generated based on the paragraph segmentation results.

[0013] Optionally, generating the structured text content of the target page based on the paragraph segmentation results includes: Each paragraph is labeled based on the paragraph division results to obtain the labeled paragraphs; The marked paragraphs are sorted according to a preset reading order to obtain the sorted text; The sorted text is corrected to obtain the structured text content of the target page.

[0014] Optionally, correcting the sorted text to obtain the structured text content of the target page includes: Obtain the confidence score of each character in the sorted text; Characters with a confidence level lower than the preset character confidence threshold are marked as characters to be corrected; According to the preset character obfuscation mapping rules, the characters to be corrected are replaced to obtain the structured text content of the target page.

[0015] Optionally, after determining the structured text content of the target page based on the text recognition result, the method further includes: The structured text content is subjected to violation feature matching to obtain the detection result of the target page; The detection results will be used to determine whether the target page violates regulations.

[0016] Furthermore, to achieve the above objectives, this application also proposes a content extraction device, which includes: The page loading module is used to respond to the content extraction request of the target page, load the target page based on a preset browser instance, and obtain the rendered target page; The character recognition module is used to perform text recognition on the visual content image of the rendered target page and generate text recognition results; The content determination module is used to determine the structured text content of the target page based on the text recognition results.

[0017] Optionally, the page loading module is further configured to respond to the content extraction request of the target page by inputting the target address corresponding to the target page into the browser instance to obtain the target page; and to perform a multi-rendering waiting strategy on the target page based on the browser instance to obtain the rendered target page.

[0018] Optionally, the page loading module is further configured to: monitor the network idle status of the target page based on the browser instance to obtain a first waiting completion result; monitor key page elements of the target page based on the browser instance to obtain a second waiting completion result; perform simulated user scrolling operations on the target page based on the browser instance to obtain a third waiting completion result; determine a rendering waiting time based on the first waiting completion result, the second waiting completion result, and the third waiting completion result; and render the target page based on the rendering waiting time to obtain the rendered target page.

[0019] Optionally, the page loading module is further configured to launch a headless browser instance based on a preset browser automation framework; and to perform anti-automation detection configuration on the launched headless browser instance to obtain a configured browser instance.

[0020] Optionally, the page loading module is further configured to inject an anti-detection script into the page context of the headless browser instance, and adjust the headless browser instance according to the anti-detection script; configure the adjusted headless browser instance to obtain a configured browser instance.

[0021] In addition, to achieve the above objectives, this application also proposes a content extraction device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the content extraction method as described above.

[0022] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the content extraction method described above.

[0023] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the content extraction method described above.

[0024] This application discloses a method for responding to a content extraction request from a target page, loading the target page based on a preset browser instance to obtain a rendered target page; performing text recognition on the visual content image of the rendered target page to generate a text recognition result; and determining the structured text content of the target page based on the text recognition result. This method shifts the focus of content extraction from the DOM structure to the visual presentation level, dynamically rendering the page to its final visual state through the browser, and then extracting text from the corresponding image. As long as the content is visually visible in the browser, it can be effectively extracted. Furthermore, all interfering content injected through methods that are visually invisible, such as hiding, moving out of the visible area, or embedding in extremely small fonts, is naturally excluded because it does not exist in the visual content image, significantly improving the purity and accuracy of the extracted content. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the first embodiment of the content extraction method of this application; Figure 2 This is a schematic diagram illustrating the operation of one embodiment of the content extraction method of this application; Figure 3 This is a flowchart illustrating the second embodiment of the content extraction method of this application; Figure 4 This is a flowchart illustrating the third embodiment of the content extraction method of this application; Figure 5 This is a flowchart illustrating the fourth embodiment of the content extraction method of this application; Figure 6 This is a schematic diagram of the structure of the target webpage according to an embodiment of the content extraction method of this application; Figure 7 This is a schematic diagram of structured text content according to an embodiment of the content extraction method of this application; Figure 8 This is a schematic diagram of the module structure of the content extraction device according to an embodiment of this application; Figure 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the content extraction method in the embodiments of this application.

[0028] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0029] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0030] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0031] This application provides a content extraction method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the content extraction method of this application.

[0032] In this embodiment, the content extraction method includes: Step S10: In response to the content extraction request of the target page, the target page is loaded based on a preset browser instance to obtain the rendered target page.

[0033] It should be noted that the executing entity in this embodiment can be a website page content extraction device with data processing, network communication and program running functions, such as a server, computer, etc., or other electronic devices that can achieve the same or similar functions. This embodiment does not limit this.

[0034] It should be understood that a target page can refer to a webpage whose content needs to be determined to violate laws and regulations, such as a fake website, a violation page, or another webpage with anti-scraping measures. A content extraction request can refer to an instruction or signal that triggers the content extraction operation on the target page. A preset browser instance can refer to a headless browser process that is pre-built and configured with anti-automation detection, capable of simulating ordinary user browsing behavior. A rendered target page can refer to a page that has reached its final visual presentation state after the browser has fully loaded all dynamic content, asynchronous resources, and lazy-loaded elements.

[0035] In practice, when it is necessary to extract content from a target page (such as a fake bank website), the content extraction device responds to the content extraction request, calls a pre-configured browser instance to navigate to the address of the target page, and executes multiple waiting strategies during the page loading process to ensure that all scripts are executed and dynamic content is loaded, and finally obtains the rendered target page for subsequent screenshotting and recognition processing.

[0036] Step S20: Perform text recognition on the visual content image of the rendered target page to generate text recognition results.

[0037] It should be understood that visual content images can refer to images obtained by taking screenshots of the rendered target page. Visual content images reflect the final visual state of the page in the browser; all visually invisible interference content is naturally absent in the image. Text recognition can refer to the detection and recognition of text regions in an image using an optical character recognition engine. Text recognition results can refer to the structured data output after recognition, including the recognized text content, the position coordinates of each text block in the image, and a confidence score representing the reliability of the recognition.

[0038] It is understood that this embodiment does not rely on parsing HTML source code or JavaScript code, and does not require cracking the obfuscation logic or decryption algorithm of the illegal website. Regardless of the code obfuscation methods used by the illegal website (such as variable name randomization, control flow flattening, string encryption), as long as the final content is rendered and visible in the browser, this embodiment can capture it through screenshots, shifting the cost of countermeasures from code cracking to visual recognition, significantly improving the efficiency and success rate of anti-scraping breakthroughs.

[0039] In practice, after obtaining the rendered target page, the content extraction device acquires the complete visual content image of the page by segmenting and stitching. Then, the visual content image is input into a preset OCR (Optical Character Recognition) engine to recognize the characters in each area and output a structured text recognition result containing text content, position coordinates and confidence level for subsequent content purification and structuring processing.

[0040] Step S30: Determine the structured text content of the target page based on the text recognition result.

[0041] It should be understood that structured text content can be the final output text content with a clear structural hierarchy and logical order. The presentation of structured text content can be consistent with the display interface of the original target page.

[0042] In its implementation, after obtaining the text recognition results, the content extraction device first filters the recognized text content based on the confidence score, removing interfering text whose confidence score is lower than a preset threshold due to factors such as blurry images or small fonts; then rearranges the text according to the reading order from top to bottom and from left to right; finally, it corrects common characters that are difficult to recognize and outputs content with a clear paragraph structure as the structured text content of the target page.

[0043] Furthermore, to ensure that the extracted high-quality structured text content can be directly used for violation feature matching without additional manual cleaning or preprocessing, thus improving the processing efficiency of the entire violation website monitoring business chain, after step S30, the method further includes: performing violation feature matching on the structured text content to obtain the detection result of the target page; and determining whether the target page violates regulations based on the detection result.

[0044] It should be understood that violation feature matching can refer to comparing structured text content with a pre-set violation feature database, which may contain illegal or non-compliant vocabulary. The detection result can refer to the matching degree information output after violation feature matching, representing the degree of similarity between the target page content and various violation features in the violation feature database.

[0045] In its implementation, after obtaining the structured text content of the target page (such as a fake bank website), the content extraction device inputs the structured text content into a preset illegal content detection system. The illegal content detection system matches the text content with the counterfeit features (such as the counterfeit bank name, counterfeit login prompts, etc.) in the illegal feature database, counts the number and weight of the matched features, calculates the illegal risk score as the detection result, and determines whether the target page is an illegal page based on the comparison of the detection result with the preset judgment threshold, thus completing the fully automated process from content extraction to illegal judgment.

[0046] For ease of understanding, the following examples are provided, but they are not intended to limit this application. In one example, reference is made to... Figure 2 , Figure 2 This is a schematic diagram illustrating an embodiment of the content extraction method of this application. The device's workflow is as follows: Step 1: Browser Environment Setup and Anti-Fingerprinting Configuration. Start a Chromium-based browser using the Puppeteer or Playwright framework. Inject anti-detection scripts to rewrite the `navigator.webdriver` property, remove headless mode signature fingerprints, integrate a proxy IP pool for dynamic rotation, and configure a User-Agent string that matches the operating system. This ensures the browser instance can stably load the target black market website without being blocked by early anti-scraping mechanisms.

[0047] Step Two: Dynamic Page Loading and Full Rendering Waiting. The target URL is loaded via a browser instance, employing multiple waiting mechanisms to ensure complete page rendering, including network idle waiting, critical element waiting, fixed delay waiting, and scroll-triggered waiting. This strategy ensures that all dynamically generated, encrypted, asynchronously loaded, and lazy-loaded content has been rendered, achieving the final visual state expected by the user during normal access.

[0048] Step 3: Visual Area Image Capture. After the page is fully rendered, capture screenshots of the webpage content. For long pages, use a scrolling capture strategy, taking screenshots in segments and then stitching them together to form a complete long image, ensuring that all visible content areas are covered.

[0049] Step 4: OCR Visual Refinement and Interference Filtering. The captured content area image is fed into the OCR recognition engine. After image preprocessing, the text is extracted, and a structured result containing text content, confidence level, and location coordinates is output. All visually invisible interference text is naturally excluded because it does not exist in the screenshot. At the same time, embedded spam text is filtered out based on the confidence threshold, thus achieving content refinement.

[0050] Step 5: Post-processing and High-Quality Output. The OCR-recognized text undergoes post-processing, utilizing coordinate information to reconstruct paragraph structure, merge text blocks within the same line, identify paragraph boundaries, and arrange them in reading order. Recognition errors are eliminated through confidence level filtering, and common OCR misidentifications are corrected. The final output is structured content with annotations for titles, body text, lists, etc., making it easy for downstream detection systems to use directly.

[0051] In this embodiment, the focus of content extraction shifts from the DOM structure to the visual presentation level. The page is rendered to its final visual state through dynamic browser rendering, and then text is extracted from the corresponding images. As long as the content is visually visible in the browser, it can be effectively extracted. Furthermore, all interfering content injected through visually invisible means such as hiding, moving out of the visible area, or embedding in extremely small fonts is naturally excluded because it does not exist in the visual content image, significantly improving the purity and accuracy of the extracted content.

[0052] Reference Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the content extraction method of this application. Based on the first embodiment described above, a second embodiment of the content extraction method of this application is proposed. In the second embodiment, step S10 includes: Step S101: In response to the content extraction request of the target page, the target address corresponding to the target page is input into the browser instance to obtain the target page.

[0053] Understandably, in this embodiment, after obtaining the content extraction request for the target page, the target address of the target page can be input into a pre-configured browser instance, allowing the target page to be loaded in an environment simulating normal user access, thereby avoiding early interception by the target website for automated tools. Here, the target address can refer to the Uniform Resource Locator (URL) corresponding to the target page on the Internet. The webpage obtained after the browser instance navigates to the target address is the target page.

[0054] In practice, when it is necessary to extract content from a website page with anti-scraping measures (such as a fake bank login page), the content extraction device responds to the content extraction request, calls a pre-configured browser instance, inputs the Uniform Resource Locator (URL) corresponding to the fake bank login page into the address bar of the browser instance, and the browser instance initiates a page navigation request to obtain the initial page resources of the webpage, thus obtaining the target page for subsequent rendering processing.

[0055] Step S102: Execute a multi-rendering waiting strategy on the target page based on the browser instance to obtain the rendered target page.

[0056] It should be noted that target pages typically employ various dynamic loading techniques, including asynchronous data requests and dynamic DOM construction based on JavaScript, lazy loading mechanisms based on scroll event listeners, and deferred rendering triggered by CSS animations or timers. These techniques result in uncertain page loading times. Simple fixed-time waiting cannot meet the loading needs of pages with varying complexity. If the waiting time is too short, some dynamic content will not be rendered, leading to missing content in the screenshot; if the waiting time is too long, extraction efficiency will be reduced. Therefore, in this embodiment, a multi-rendering waiting strategy can be used to determine the optimal waiting time in order to capture suitable page images.

[0057] It should be understood that the multi-rendering waiting strategy in this embodiment can be a page rendering completion determination strategy formed by a combination of network idle waiting, key page element appearance waiting, simulated user scrolling trigger waiting, and preset fixed delay waiting. After the rendered target page has been fully loaded by the browser, it reaches its final visual presentation state. By executing the multi-rendering waiting strategy, the rendering completion status of the page can be judged collaboratively from multiple dimensions such as network requests, key element presentation, and dynamic content triggering, thereby ensuring that all dynamic content, asynchronously loaded resources, and lazy-loaded elements on the target page have been fully presented, avoiding content omissions due to insufficient waiting.

[0058] In its implementation, after the content extraction device obtains the target page from the browser instance, it executes multiple rendering waiting strategies during the page loading process. These strategies include monitoring network request activity and determining that the main resources have been loaded when no new requests are initiated within a preset time period; detecting whether the preset key content elements in the document object model have been rendered in the viewport; controlling the browser viewport to scroll from the top to the bottom of the page in a step-by-step manner to trigger the loading of lazy-loaded content; and waiting for a preset delay after the above conditions are met to ensure that all asynchronous rendering operations are completely completed, ultimately obtaining the rendered target page that can be used for subsequent screenshot processing.

[0059] Furthermore, to ensure the complete acquisition of the target page content, step S102 may include: monitoring the network idle status of the target page based on the browser instance to obtain a first waiting completion result; monitoring key page elements of the target page based on the browser instance to obtain a second waiting completion result; performing a simulated user scrolling operation on the target page based on the browser instance to obtain a third waiting completion result; determining a rendering waiting time based on the first waiting completion result, the second waiting completion result, and the third waiting completion result; and rendering the target page based on the rendering waiting time to obtain the rendered target page.

[0060] It should be understood that in this embodiment, by decomposing the multiple rendering waiting strategy into three independent waiting dimensions—network idle state monitoring, key page element monitoring, and simulated user scrolling operation—and comprehensively determining the rendering waiting time based on the completion results of the three, the determination of the rendering completion time can cover different types of dynamic loading scenarios.

[0061] It is understandable that the first waiting completion result refers to the completion signal of network idle state monitoring, indicating that the target page has completed the request and loading of all network resources. The second waiting completion result refers to the completion signal of key page element monitoring, indicating that the preset key content containers in the document object model of the target page have been loaded and rendered. The third waiting completion result refers to the completion signal of simulated user scrolling operation, indicating that the deferred loading content triggered by the scrolling event on the target page has been activated and rendered. The rendering wait time can refer to the subsequent delay duration determined based on the waiting completion results of the above three dimensions.

[0062] It's important to note that network idle state monitoring addresses the timing of asynchronous data requests and resource loading completion, critical page element monitoring addresses the timing of DOM dynamic construction completion, and simulated user scrolling addresses the timing of lazy-loaded content triggering and rendering. These three waiting conditions each target a typical dynamic loading mechanism; they are not interchangeable but complementary. Only when all three dimensions are satisfied can it be ensured that the page has completed rendering from the data layer, structural layer, to the visual layer, thus avoiding misjudgments caused by a single waiting condition.

[0063] In its implementation, the content extraction device performs the following operations synchronously or sequentially during the loading of the target page: continuously monitors the number of network requests to the target page; when the number of requests drops below a preset threshold, it determines that the network is idle and obtains the first completion result; continuously polls the document object model (DOM) of the target page; when a preset key content element (such as the main page container) appears in the DOM and contains substantial text content, it determines that the key element has finished loading and obtains the second completion result; controls the browser viewport to scroll down from the top of the page in a step-by-step manner, waiting for the page layout to stabilize after each scroll until the bottom of the page is reached, then determines that the scrolling trigger has finished loading and obtains the third completion result. After obtaining the completion results in all three dimensions, the content extraction device determines a preset rendering wait time and confirms that the page has been fully rendered after waiting for this duration, thus obtaining the rendered target page.

[0064] For ease of understanding, examples are given below, but this does not limit the scope of this application. During the page loading phase, a combined waiting strategy can be adopted to ensure that the page is fully rendered. Specifically, this includes: (1) network idle waiting, that is, waiting for the number of network connections to drop to a preset threshold (such as below 2) to ensure that the main resources are loaded; (2) key element waiting, that is, waiting for the main container of the page to appear and contain substantial content; (3) fixed delay waiting, for pages with complex animations or lazy loading logic, a fixed delay time of 1 to 3 seconds is set; (4) scroll trigger waiting, that is, simulating user scrolling operations to trigger the rendering of lazy-loaded content, and waiting for the new content to be loaded after each scroll. The above mechanism shifts the entry point for content extraction from the code level to the visual level, fundamentally avoiding anti-crawling obstacles such as JavaScript dynamic generation, content encryption, and code obfuscation.

[0065] In the specific implementation process, a headless browser instance can be launched and configured with anti-detection features using a preset browser automation framework. This allows the browser instance to simulate normal user access behavior and circumvent early blocking by the target website for automation tools. Prior to step S10, the process includes: launching a headless browser instance based on the preset browser automation framework; and configuring anti-automation detection on the launched headless browser instance to obtain the configured browser instance.

[0066] It should be noted that the target page usually has an anti-scraping detection mechanism that can determine whether the visitor is an automated tool by detecting browser environment characteristics (such as the value of the navigator.webdriver property, the fingerprint of headless mode characteristics, the consistency between User-Agent and operating system, etc.) in the initial stage of page loading. If automated characteristics are detected, the service will be directly denied or false content will be returned.

[0067] It should be understood that in this embodiment, by pre-building and configuring a browser instance before loading the target page, subsequent page loading operations can be performed in an environment simulating ordinary user access, thereby circumventing the target website's early identification and interception of automation tools. The pre-defined browser automation framework can refer to an automation tool used for programmatically controlling the browser, capable of starting and managing the lifecycle of a headless browser instance. A headless browser instance can refer to a browser process that does not require a graphical user interface and can complete operations such as page loading, script execution, and page rendering in the background. Anti-automation detection configuration can refer to a series of processing operations that modify the attributes and environment parameters of the browser instance to hide its automation tool characteristics. After the configured browser instance undergoes anti-automation detection configuration, its external behavior is consistent with that of an ordinary user's browser.

[0068] In its implementation, before loading the target page, the content extraction device first launches a headless browser instance using a pre-defined browser automation framework (such as Puppeteer or Playwright). Then, it injects an anti-detection script into the page context of the headless browser instance. This script rewrites the attributes that identify the automation environment (such as the navigator.webdriver property), removes the headless mode's characteristic information, and configures the browser instance with proxy server information and a user agent string that matches the target operating system. After completing the above configuration operations, the headless browser instance becomes a configured browser instance, which can be called by subsequent target page loading steps.

[0069] Furthermore, to ensure that the browser instance exhibits behavior consistent with that of a regular user's browser, effectively circumventing the target website's automated detection mechanism based on browser fingerprinting, and ensuring that the browser instance can successfully load the target page without being blocked, the step of configuring the launched headless browser instance to perform anti-automatic detection includes: injecting an anti-detection script into the page context of the headless browser instance, and adjusting the headless browser instance according to the anti-detection script; configuring the adjusted headless browser instance to obtain the configured browser instance.

[0070] It's important to note that target websites' automated detection mechanisms typically detect both browser internal attribute fingerprints and external request characteristics simultaneously. Browser internal attribute fingerprints include JavaScript-readable attributes such as `navigator.webdriver`, which need to be modified within the page context via script injection. External request characteristics include the egress IP address of network requests and the User-Agent field in the HTTP request header; these parameters need to be set through the browser instance's external configuration interface. Performing script injection to adjust browser internal attributes first, followed by configuring external network and identification parameters, ensures that both levels of automated characteristics are eliminated, preventing the browser instance from being detected by the target website due to improper processing order or omission of any layer's configuration.

[0071] It should be understood that this embodiment decomposes the anti-automation detection configuration into two stages: script injection adjustment and instance parameter configuration. By adjusting the browser's internal attributes first and then configuring external parameters, the automated tool characteristics of the browser instance are systematically eliminated. Here, the page context can refer to the JavaScript execution environment of the current page in the headless browser instance. The anti-detection script can refer to a piece of script code used to modify the browser's automation identifier attributes and hide the headless mode characteristics. After the adjusted headless browser instance is modified by the anti-detection script, the browser process with the automation identifier has been removed from its internal attributes, but external parameters such as network and user agent have not yet been configured.

[0072] In its implementation, after launching the headless browser instance, the content extraction device first injects a pre-written anti-detection script into the page context of the browser instance through the script injection interface provided by the browser automation framework. After the anti-detection script is executed in the page context, it modifies the navigator.webdriver property to return a non-automation identifier value and removes or modifies browser characteristic attributes specific to headless mode, thus obtaining an adjusted headless browser instance. Subsequently, the content extraction device configures external parameters for the adjusted headless browser instance, including setting proxy server information to achieve dynamic rotation of network request addresses and configuring user agent strings that match mainstream desktop operating systems, ultimately obtaining the configured browser instance.

[0073] In this embodiment, in response to a content extraction request for a target page, the target address corresponding to the target page is input into the browser instance to obtain the target page; a multi-rendering waiting strategy is then executed on the target page based on the browser instance to obtain the rendered target page. By navigating the browser instance to the target address and then executing the multi-rendering waiting strategy, it is ensured that all dynamic content and asynchronously loaded content on the target page has been rendered, thus avoiding content omissions due to insufficient waiting.

[0074] Reference Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the content extraction method of this application. Based on the first embodiment described above, a third embodiment of the content extraction method of this application is proposed. In the third embodiment, step S20 includes: Step S201: Take a screenshot of the rendered target page to obtain a visual content image.

[0075] It's important to note that to combat automated content scraping, target pages typically inject a large amount of visually invisible distracting content into their Document Object Model (DOM). While this distracting content exists within the code structure, it doesn't appear in the browser's rendered visual output. Extracting text from the DOM becomes ineffective in distinguishing between visually visible content and this invisible distraction. However, by taking a screenshot, only the content ultimately rendered onto the visual canvas by the browser's rendering engine is captured in the image. This naturally filters out the invisible distracting content, thus solving the problem of distracting content removal at the data source.

[0076] It should be understood that visual content images reflect the page content that the human eye sees during normal browsing. All distracting text that is hidden by cascading style sheets, moved out of the visible area, or embedded in a very small font will not be included in the image because it is not visually visible.

[0077] Understandably, after the page is fully rendered, a scrolling capture strategy can be used for long pages, taking screenshots in segments and then stitching them together to form a complete long image, ensuring that all visible content areas are covered.

[0078] Furthermore, to avoid content loss or misalignment at the splicing points of segmented screenshots and to ensure the integrity and splicing accuracy of the long page content, step S201 may include: taking screenshots of the rendered target page based on its height to obtain initial image fragments; performing feature matching on the overlapping areas between the initial image fragments; and generating visual content images based on the matching results.

[0079] It should be understood that page height can refer to the total pixel height of the rendered target page when fully expanded in the browser. Initial image segments can refer to several partial images obtained by segmenting the target page according to a preset viewport height, with a preset pixel width of overlapping area maintained between adjacent initial image segments. The overlapping area can refer to the portion of content that overlaps between adjacent initial image segments, used to provide a reference for stitching alignment. Feature matching can refer to the detection and comparison of image feature points in the overlapping area of ​​adjacent initial image segments to determine the precise alignment position between the two image segments.

[0080] It's important to note that when the target page is segmented into screenshots, the browser needs to scroll to move the viewport. During scrolling, factors such as changes in page layout, unstable asynchronous rendering, or incomplete scrolling animation may cause content misalignment or breaks between adjacent screenshots. If the screenshots are directly joined end-to-end at a fixed step size, the misaligned content will cause text breaks or content repetition at the seams, affecting the accuracy of subsequent text recognition. By preserving overlapping areas between adjacent initial image segments and performing intelligent matching and alignment based on image feature points within the overlapping areas, the positional deviation caused by scrolling can be automatically corrected, ensuring the continuity and accuracy of the stitched visual content image and providing a high-quality image data foundation for subsequent optical character recognition.

[0081] For ease of understanding, examples are provided below, but these are not intended to limit this application. For the common long-page characteristics of illegal websites (such as lists of illegal betting records, aggregation pages of illegal content, etc.), a strategy of segmented screenshotting and preservation of overlapping areas can be adopted to ensure the complete capture of long-page content. Specific steps include: 1) Segmented screenshotting: The page is segmented according to the viewport height, with each segment overlapping the previous one by approximately 50 pixels to prevent content loss at seams; 2) Anti-jitter processing: During scrolling, the page is captured only after the scrolling animation is complete and the page has stabilized, avoiding image misalignment caused by the scrolling animation; 3) Intelligent stitching: Seamless stitching is performed based on feature matching of the overlapping areas to generate a continuous long image covering the entire page content. This method differs from the simple viewport screenshotting or fixed-step scrolling screenshotting in existing technologies, ensuring the integrity of long-page content and stitching accuracy.

[0082] Step S202: Optical character recognition is performed on the visual content image using a preset recognition engine to obtain text recognition results, which include text content, confidence level, and location information.

[0083] It should be understood that the preset recognition engine can refer to a pre-deployed OCR system capable of detecting text regions in an image and recognizing the characters within those regions. The text content can refer to the sequence of text characters output after recognition. Confidence score can refer to the reliability assessment value given by the recognition engine for each recognition result, reflecting the accuracy of that result. Location information can refer to the spatial coordinate data of each recognized text block within the visual content image, including horizontal and vertical coordinates.

[0084] In its implementation, after acquiring the visual content image, the content extraction device inputs the image into a preset recognition engine. The preset recognition engine first performs image preprocessing on the visual content image, including grayscale conversion, adaptive threshold binarization, and noise point removal, to improve the image's clarity and contrast. Subsequently, it detects the text regions in the image, identifies the characters in each text region one by one, and outputs the text content of each text block, the confidence score representing the recognition credibility, and the position coordinate information of the text block in the image. Finally, it integrates these into a structured text recognition result.

[0085] Understandably, using the confidence score of OCR recognition output can automatically filter out various visual interference texts injected by black market websites, achieving pure content extraction. Specific filtering mechanisms include: 1) Natural elimination of visually invisible interference: All CSS hidden text, text with 0 opacity, text off-screen, and text covered by other elements are automatically filtered out because they simply do not exist in the screenshot; 2) Removal of low-confidence interference: To pollute automated crawling results, illegal websites often embed a large number of extremely small fonts (such as pixel-level fonts) and low-contrast texts (such as light gray text on a white background). These texts have significantly low OCR confidence scores (usually below 60%) due to blurred image features, and are automatically removed by setting a confidence threshold (such as 60%); 3) Image preprocessing enhancement: Screenshots are grayscaled, adaptively thresholded binarized, noise points removed, and small font areas enlarged to improve the recognition rate of low-quality images. This mechanism fundamentally solves the problem that DOM-based solutions struggle to distinguish between real content and interference text.

[0086] In this embodiment, the visual content image of the rendered target page is obtained by taking a screenshot. All visually invisible interference text is naturally excluded because it does not exist in the screenshot. This shifts the cost of countermeasures from code cracking to visual recognition, significantly improving the efficiency and success rate of anti-scraping breakthroughs.

[0087] Reference Figure 5 , Figure 5 This is a flowchart illustrating the fourth embodiment of the content extraction method of this application. Based on the above embodiments, a fourth embodiment of the content extraction method of this application is proposed. In the fourth embodiment, step S30 includes: Step S301: Filter the identified text content based on the confidence level in the text recognition result to obtain the filtered text content.

[0088] It should be understood that filtering text recognition results through confidence levels can automatically eliminate low-quality recognition results caused by factors such as blurry images, small fonts, or low contrast, thereby improving the purity of the text content.

[0089] In practice, after obtaining the text recognition results, the content extraction device traverses each text block in the recognition results and compares the confidence value of each text block with the preset confidence threshold. Text blocks with a confidence value higher than or equal to the preset threshold (real text) are retained, while text blocks with a confidence value lower than the preset threshold (interference text) are removed, and finally the filtered text content is obtained.

[0090] Step S302: Based on the location information in the text recognition result, perform structuring processing on the filtered text content to generate the structured text content of the target page.

[0091] It should be noted that location information refers to the spatial coordinate data of each text block within the visual content image, including horizontal and vertical coordinates. By using location information to perform structured processing on the filtered text content, the natural paragraph structure and reading order of the document are restored from the visual coordinate level, rather than relying on the document object model structure of the target page. This avoids interference caused by the target page's document object model obfuscation in content extraction.

[0092] Furthermore, in order to use visual coordinate information for document structure reconstruction without relying on the DOM structure of the target page, and to avoid interference caused by the target page's DOM obfuscation techniques in content extraction, step S302 may include: merging the filtered text content according to the horizontal coordinates in the location information to obtain text lines; clustering the text lines into paragraphs according to the vertical coordinates in the location information to obtain paragraph segmentation results; and generating the structured text content of the target page based on the paragraph segmentation results.

[0093] It should be noted that the horizontal coordinate refers to the horizontal position of each text block in the visual content image, used to determine whether different text blocks are on the same text line. A text line can refer to a complete line of text units formed by merging text blocks on the same horizontal baseline. The vertical coordinate refers to the vertical position of each text line in the visual content image, used to calculate the spacing between adjacent text lines. Paragraph clustering refers to grouping text lines with small spacing into the same paragraph based on the vertical spacing between adjacent text lines, and dividing text lines with spacing exceeding a preset paragraph threshold into different paragraphs. The paragraph division result can refer to the set of text paragraphs with clear paragraph boundary markers obtained after clustering.

[0094] Understandably, the horizontal coordinate can merge text within the same line, merging text blocks within the same line based on the horizontal coordinate, thus avoiding text breaks caused by OCR segmentation; the vertical coordinate can cluster and identify paragraph boundaries, determining that adjacent text blocks are different paragraphs when the vertical distance between them exceeds a preset threshold.

[0095] It should be understood that optical character recognition engines typically detect and recognize text in visual content images on a region-by-region basis. Text within the same line may be segmented into multiple independent text blocks due to factors such as character spacing, font variations, or image noise. Directly clustering these scattered text blocks into paragraphs can lead to in-line fragments being misidentified as independent lines, disrupting the document's line and paragraph structure. Therefore, a better approach is to first merge text blocks belonging to the same line based on horizontal coordinates, restoring them to complete lines. Then, paragraph clustering can be performed on the text lines based on vertical coordinates, accurately restoring the document's two-level structure of lines and paragraphs. Furthermore, this process is entirely based on visual coordinate data, independent of the target page's document object model structure, enabling accurate restoration of the document's true structural layout even when the document object model is highly obfuscated.

[0096] In its implementation, after obtaining the filtered text content, the content extraction device first merges text blocks with similar vertical coordinates and continuous horizontal alignment into the same text line based on the horizontal coordinates in the position information of each text block. This solves the problem of the same line of text being split into multiple text blocks due to the segmentation caused by optical character recognition. Then, based on the vertical coordinates in the position information of each text line, it calculates the vertical spacing between adjacent text lines. When the vertical spacing between adjacent text lines exceeds a preset paragraph threshold, it marks the paragraph boundary at that position, dividing the text line into different paragraphs and obtaining the paragraph segmentation result. Finally, based on the paragraph segmentation result, it labels each paragraph with a structure type and rearranges the paragraphs according to the reading order to generate the structured text content of the target page.

[0097] Furthermore, to restore the natural logical order of the document, correct character misrecognition issues in the optical character recognition process, and improve the structured quality and text accuracy of the output content, the step of generating the structured text content of the target page based on the paragraph segmentation results includes: annotating each paragraph according to the paragraph segmentation results to obtain annotated paragraphs; sorting the annotated paragraphs according to a preset reading order to obtain sorted text; and correcting the sorted text to obtain the structured text content of the target page.

[0098] It should be noted that annotated paragraphs can refer to paragraphs that have been assigned structure type tags, including types such as headings, body text, and lists. Preset reading order can refer to the visual reading order from top to bottom or left to right. Sorted text can refer to the text sequence obtained by rearranging the annotated paragraphs according to the preset reading order.

[0099] In its implementation, after obtaining the paragraph segmentation results, the content extraction device first performs feature analysis on each paragraph. Based on features such as text length, position, and relative font size, it labels each paragraph with a corresponding structural type (e.g., short, centered paragraphs are labeled as headings, and continuous long paragraphs are labeled as body text), thus obtaining labeled paragraphs. Then, it sorts the labeled paragraphs according to the visual reading order from top to bottom and from left to right to ensure the logical coherence of the text content, thus obtaining the sorted text. Finally, it corrects misidentified characters in the sorted text using optical character recognition (such as confusing the number "0" with the letter "O", or the number "1" with the letter "l") according to rules, thus obtaining the structured text content of the target page.

[0100] Furthermore, in order to specifically correct common character confusion problems in optical character recognition (such as confusion between numbers and letters) and effectively improve the recognition accuracy of text content, the step of correcting the sorted text to obtain the structured text content of the target page includes: obtaining the confidence score of each character in the sorted text; marking characters with confidence scores lower than a preset character confidence threshold as characters to be corrected; and replacing the characters to be corrected according to a preset character confusion mapping rule to obtain the structured text content of the target page.

[0101] It should be noted that the preset character confidence threshold can refer to a critical confidence level used to determine whether a character has a recognition error. The character to be corrected can refer to a character with a confidence level lower than the preset character confidence threshold, which is considered potentially having a recognition error. The preset character confusion mapping rule can refer to pre-established rules for easily confused character correspondences, including confusion correspondences between numbers and letters (such as confusion between the number "0" and the letter "O", confusion between the number "1" and the letter "l") and confusion correspondences between letters.

[0102] In its implementation, after obtaining the sorted text, the content extraction device iterates through each character in the sorted text and obtains the character-level confidence score of each character output during the optical character recognition stage. It then compares the confidence score of each character with a preset character confidence threshold, marking characters with confidence scores below the threshold as characters to be corrected. For each character to be corrected, it combines its contextual semantic information and preset character obfuscation mapping rules to determine the target replacement character from the obfuscation correspondence. For example, when the character to be corrected is the letter "I" and the context is a sequence of numbers, it replaces it with the number "l" according to the obfuscation correspondence between numbers and letters. After completing the character replacement for each character to be corrected, the corrected text is obtained and output as the structured text content of the target page.

[0103] Understandably, by filtering low-quality identification results through confidence thresholds, restoring paragraphs based on coordinate clustering, correcting common misidentifications, and rearranging the reading order, the final output is structured, clean content. Downstream illegal and non-compliant content detection systems can directly use this output for analysis without additional manual cleaning or preprocessing, significantly improving the overall efficiency of detecting illegal websites.

[0104] For ease of understanding, the following examples are provided, but they are not intended to limit this application. In one example, reference is made to... Figure 6 and Figure 7 , Figure 6 This is a schematic diagram of the target webpage structure according to an embodiment of the content extraction method of this application. The left side is the target webpage, which employs various types of text interference countermeasures in its visual presentation. The target webpage displays normally displayed page content, such as "Detailed Rules for Upgrading Protection Settings of Collaborative Office Systems." Simultaneously, by opening the front-end HTML / CSS source code, the target webpage also embeds interfering text, such as pixel-level font sizes displayed in the corners of the page, including irrelevant phrases like "Community elderly care services are constantly improving" and "Be a sunflower generation, facing the sun, striving to grow." These interfering texts are mixed with normal content on the same page, causing the extraction results to simultaneously contain normal content and a large amount of invalid interfering information when using traditional document object model parsing methods.

[0105] Figure 7 This is a schematic diagram of structured text content according to an embodiment of the content extraction method of this application. The structured text content processed by the content extraction method of this application retains only the visually visible text content of the target webpage that meets the preset confidence level. For example, the extraction result retains "Detailed Rules for Upgrading Protection Settings of Collaborative Office System" and marks it as a title type, while "To improve the security of the collaborative office system, ..." is marked as body text type. All of the above content is text that appears normally in the image. Visually invisible interference content (such as content in obscured text layers) is naturally excluded because it was not captured during the screenshot operation, and embedded interference text (such as "Be a sunflower, face the sun, and strive to grow") is filtered out because the optical character recognition confidence level is lower than the preset threshold. The final output structured text content has a clear paragraph hierarchy, with each paragraph labeled with a corresponding structure type tag, and the text content is arranged in a top-to-bottom reading order. It can be directly used by downstream illegal content detection systems without additional manual cleaning or preprocessing.

[0106] In this embodiment, confidence filtering can effectively eliminate low-quality recognition results caused by factors such as blurry images and small fonts, while location information structuring processing can restore the spatial layout relationship of the text, so that the final generated structured text content has both high purity and good structural integrity.

[0107] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the method of extracting the content of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0108] This application also provides a content extraction device, please refer to... Figure 8 The content extraction device includes: The page loading module 10 is used to respond to the content extraction request of the target page, load the target page based on a preset browser instance, and obtain the rendered target page. The character recognition module 20 is used to perform text recognition on the visual content image of the rendered target page and generate text recognition results; The content determination module 30 is used to determine the structured text content of the target page based on the text recognition result.

[0109] The content extraction device provided in this application, employing the content extraction method described in the above embodiments, can solve the technical problem that DOM-based content extraction schemes struggle to distinguish visually invisible interference content, leading to low-quality extracted content. Compared with the prior art, the beneficial effects of the content extraction device provided in this application are the same as those of the content extraction method described in the above embodiments, and other technical features in the content extraction device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0110] This application provides a content extraction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the content extraction method in Embodiment 1 above.

[0111] The following is for reference. Figure 9 The diagram illustrates a structural schematic of a content extraction device suitable for implementing embodiments of this application. The content extraction device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9The content extraction device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0112] like Figure 9 As shown, the content extraction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the content extraction device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. The communication device 1009 allows the content extraction device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show content extraction devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0113] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0114] The content extraction device provided in this application, employing the content extraction method described in the above embodiments, can solve the technical problem that DOM-based content extraction schemes struggle to distinguish visually invisible interference content, leading to low-quality extracted content. Compared with the prior art, the beneficial effects of the content extraction device provided in this application are the same as those of the content extraction method described in the above embodiments, and other technical features of this content extraction device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0115] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0117] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the content extraction method described in the above embodiments.

[0118] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0119] The aforementioned computer-readable storage medium may be included in the content extraction device; or it may exist independently and not assembled into the content extraction device.

[0120] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the content extraction device, cause the content extraction device to perform the content extraction method described above.

[0121] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0124] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described content extraction method. This solves the technical problem that DOM-based content extraction schemes struggle to distinguish visually invisible interference content, leading to low-quality extracted content. Compared to existing technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the content extraction method provided in the above embodiments, and will not be elaborated upon here.

[0125] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the content extraction method described above.

[0126] The computer program product provided in this application can solve the technical problem that DOM-based content extraction schemes have difficulty distinguishing visually invisible interference content, resulting in low-quality extracted content. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the content extraction method provided in the above embodiments, and will not be repeated here.

[0127] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

[0128] It should be noted that the data collection, tag management, rule setting, and push decision-making processes involved in this application are designed to work with other technical features to solve technical problems. They do not involve or support any illegal activities. Any data processing that may violate laws and regulations (such as unauthorized collection of privacy data, generation of discriminatory tags, setting unfair rules, or pushing illegal information) is not within the scope of protection of this application's technical solution. Of course, the user data in this application will be encrypted, anonymized, or de-identified before storage to ensure user data security.

[0129] This application discloses A1, a content extraction method, the content extraction method comprising: In response to a content extraction request for the target page, the target page is loaded based on a preset browser instance to obtain the rendered target page; Text recognition is performed on the visual content image of the rendered target page to generate text recognition results; The structured text content of the target page is determined based on the text recognition results.

[0130] A2. The content extraction method as described in A1, wherein in response to a content extraction request for the target page, the target page is loaded based on a preset browser instance to obtain a rendered target page, including: In response to a content extraction request for the target page, the target address corresponding to the target page is input into the browser instance to obtain the target page; Based on the browser instance, a multi-rendering waiting strategy is executed on the target page to obtain the rendered target page.

[0131] A3. The content extraction method as described in A2, wherein the step of performing a multi-rendering waiting strategy on the target page based on the browser instance to obtain the rendered target page includes: Based on the browser instance, the target page is monitored for network idle status to obtain the first waiting completion result; Based on the browser instance, key page elements of the target page are monitored to obtain a second pending completion result; Based on the browser instance, a simulated user scrolling operation is performed on the target page to obtain a third waiting completion result; The rendering wait time is determined based on the first wait completion result, the second wait completion result, and the third wait completion result. The target page is rendered based on the rendering wait time to obtain the rendered target page.

[0132] A4. The content extraction method as described in A1, before loading the target page based on a preset browser instance in response to the content extraction request of the target page to obtain the rendered target page, further includes: Launch a headless browser instance based on a pre-defined browser automation framework; The headless browser instance that has been launched is subjected to anti-automatic detection configuration to obtain the configured browser instance.

[0133] A5. The content extraction method as described in A4, wherein performing anti-automatic detection configuration on the launched headless browser instance to obtain the configured browser instance includes: An anti-detection script is injected into the page context of the headless browser instance, and the headless browser instance is adjusted according to the anti-detection script; Configure the adjusted headless browser instance to obtain the configured browser instance.

[0134] A6. The content extraction method as described in A1, wherein performing optical character recognition on the visual content image of the rendered target page to generate a text recognition result includes: Take a screenshot of the rendered target page to obtain a visual content image; The visual content image is subjected to optical character recognition by a preset recognition engine to obtain text recognition results, which include text content, confidence level and location information.

[0135] A7. The content extraction method as described in A6, wherein taking a screenshot of the rendered target page to obtain a visual content image includes: Based on the page height of the rendered target page, a screenshot is taken of the rendered target page to obtain an initial image fragment; Feature matching is performed on the overlapping regions between the initial image segments, and a visual content image is generated based on the matching results.

[0136] A8. The content extraction method as described in any one of A1 to A5, wherein determining the structured text content of the target page based on the text recognition result includes: The identified text content is filtered based on the confidence level in the text recognition results to obtain the filtered text content. Based on the location information in the text recognition results, the filtered text content is structured to generate the structured text content of the target page.

[0137] A9. The content extraction method as described in A8, wherein the step of performing structured processing on the filtered text content based on the position information in the text recognition result to generate structured text content for the target page includes: The filtered text content is merged based on the horizontal coordinates in the location information to obtain text lines; The text lines are clustered into paragraphs based on the vertical coordinates in the location information to obtain paragraph segmentation results; The structured text content of the target page is generated based on the paragraph segmentation results.

[0138] A10. The content extraction method as described in A9, wherein generating the structured text content of the target page based on the paragraph segmentation results includes: Each paragraph is labeled based on the paragraph division results to obtain the labeled paragraphs; The marked paragraphs are sorted according to a preset reading order to obtain the sorted text; The sorted text is corrected to obtain the structured text content of the target page.

[0139] A11. The content extraction method as described in A10, wherein correcting the sorted text to obtain the structured text content of the target page includes: Obtain the confidence score of each character in the sorted text; Characters with a confidence level lower than the preset character confidence threshold are marked as characters to be corrected; According to the preset character obfuscation mapping rules, the characters to be corrected are replaced to obtain the structured text content of the target page.

[0140] A12. The content extraction method as described in any one of A1 to A5, after determining the structured text content of the target page based on the text recognition result, further includes: The structured text content is subjected to violation feature matching to obtain the detection result of the target page; The detection results will be used to determine whether the target page violates regulations.

[0141] This application also discloses B13, a content extraction device, the device comprising: The page loading module is used to respond to the content extraction request of the target page, load the target page based on a preset browser instance, and obtain the rendered target page; The character recognition module is used to perform text recognition on the visual content image of the rendered target page and generate text recognition results; The content determination module is used to determine the structured text content of the target page based on the text recognition results.

[0142] B14. The content extraction device as described in B13, wherein the page loading module is further configured to respond to the content extraction request of the target page by inputting the target address corresponding to the target page into the browser instance to obtain the target page; and to perform a multi-rendering waiting strategy on the target page based on the browser instance to obtain the rendered target page.

[0143] B15. The content extraction device as described in B14, wherein the page loading module is further configured to: monitor the network idle state of the target page based on the browser instance to obtain a first waiting completion result; monitor key page elements of the target page based on the browser instance to obtain a second waiting completion result; perform simulated user scrolling operations on the target page based on the browser instance to obtain a third waiting completion result; determine a rendering waiting time based on the first waiting completion result, the second waiting completion result, and the third waiting completion result; and render the target page based on the rendering waiting time to obtain the rendered target page.

[0144] B16. The content extraction device as described in B13, wherein the page loading module is further configured to launch a headless browser instance based on a preset browser automation framework; and to perform anti-automation detection configuration on the launched headless browser instance to obtain a configured browser instance.

[0145] B17. The content extraction device as described in B16, wherein the page loading module is further configured to inject an anti-detection script into the page context of the headless browser instance, and adjust the headless browser instance according to the anti-detection script; configure the adjusted headless browser instance to obtain a configured browser instance.

[0146] This application also discloses C18, a content extraction device, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the content extraction method as described above.

[0147] This application also discloses D19, a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the content extraction method described above.

[0148] This application also discloses E20, a computer program product comprising a computer program that, when executed by a processor, implements the steps of the content extraction method described above.

Claims

1. A content extraction method, characterized in that, The content extraction method includes: In response to a content extraction request for the target page, the target page is loaded based on a preset browser instance to obtain the rendered target page; Text recognition is performed on the visual content image of the rendered target page to generate text recognition results; The structured text content of the target page is determined based on the text recognition results.

2. The content extraction method as described in claim 1, characterized in that, The step of responding to a content extraction request for a target page and loading the target page based on a preset browser instance to obtain a rendered target page includes: In response to a content extraction request for the target page, the target address corresponding to the target page is input into the browser instance to obtain the target page; Based on the browser instance, a multi-rendering waiting strategy is executed on the target page to obtain the rendered target page.

3. The content extraction method as described in claim 2, characterized in that, The step of performing a multi-rendering waiting strategy on the target page based on the browser instance to obtain the rendered target page includes: Based on the browser instance, the target page is monitored for network idle status to obtain the first waiting completion result; Based on the browser instance, key page elements of the target page are monitored to obtain a second pending completion result; Based on the browser instance, a simulated user scrolling operation is performed on the target page to obtain a third waiting completion result; The rendering wait time is determined based on the first wait completion result, the second wait completion result, and the third wait completion result. The target page is rendered based on the rendering wait time to obtain the rendered target page.

4. The content extraction method as described in claim 1, characterized in that, Before loading the target page based on a preset browser instance and obtaining the rendered target page in response to the content extraction request of the target page, the process further includes: Launch a headless browser instance based on a pre-defined browser automation framework; The headless browser instance that has been launched is subjected to anti-automatic detection configuration to obtain the configured browser instance.

5. The content extraction method as described in claim 4, characterized in that, The step of performing anti-automatic detection configuration on the launched headless browser instance to obtain the configured browser instance includes: An anti-detection script is injected into the page context of the headless browser instance, and the headless browser instance is adjusted according to the anti-detection script; Configure the adjusted headless browser instance to obtain the configured browser instance.

6. The content extraction method as described in claim 1, characterized in that, The step of performing optical character recognition on the visual content image of the rendered target page to generate text recognition results includes: Take a screenshot of the rendered target page to obtain a visual content image; The visual content image is subjected to optical character recognition by a preset recognition engine to obtain text recognition results, which include text content, confidence level and location information.

7. A content extraction device, characterized in that, The device includes: The page loading module is used to respond to the content extraction request of the target page, load the target page based on a preset browser instance, and obtain the rendered target page; The character recognition module is used to perform text recognition on the visual content image of the rendered target page and generate text recognition results; The content determination module is used to determine the structured text content of the target page based on the text recognition results.

8. A content extraction device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the content extraction method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the content extraction method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the content extraction method as described in any one of claims 1 to 6.

Images