Page anti-crawling method and device and related equipment

Abstract

Embodiments of the present application disclose a page anti-crawling method, device and related equipment. The method scans identifiable elements in a page to generate a list of elements to be encrypted; generates a dynamic encryption rule according to a preset strategy; encrypts the identifiable elements in the list of elements to be encrypted according to the dynamic encryption rule to obtain encrypted page elements, inputs the encrypted page elements into a front-end rendering engine for page rendering, and stores a mapping relationship between the identifiable elements and the dynamic encryption rule; when an access request of an automated tool is received, the automated tool is authenticated, and a decryption key corresponding to the dynamic encryption rule is issued after authentication; the automated tool queries the mapping relationship based on the decryption key, restores the encrypted page elements to corresponding original elements, and performs an automated operation according to the original elements. The method makes it impossible for general AI tools and third-party crawlers to achieve malicious crawling and manipulation through one-time cracking or regular sampling.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a method, apparatus and related equipment for preventing web scraping. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially large language models, various general-purpose AI automation tools have gradually acquired the ability to parse web page front-end source code and identify page DOM structure. Online applications such as B / S architecture web pages and SaaS cloud platforms are widely used across various industries, and users have increasingly higher requirements for page data security and business process protection capabilities. Currently, common related technical solutions in the industry include: firstly, traditional web page automation frameworks based on DOM selectors (such as XPath and CSS Selectors), which locate page controls through fixed element identifiers; secondly, visual automation tools based on image recognition and OCR technology, which identify interactive areas through page screenshots.

[0003] In existing technologies, during the development phase of online application pages, the identifiers such as id, class, and name of page DOM elements and interactive controls are usually kept fixed. This allows AI tools and web crawlers to easily establish a mapping relationship between elements and page functions, thereby enabling attacks based on DOM structure / element identifiers / interface parameters for programmatic parsing and automated manipulation. Summary of the Invention

[0004] This invention provides a method, apparatus, and related equipment for preventing page scraping, aiming to solve the technical problem that existing anti-scraping technologies are unable to prevent AI intelligent tools from performing programmatic parsing and automated operations based on the page DOM structure.

[0005] In a first aspect, embodiments of the present invention provide a method for preventing web scraping, comprising: Scan the identifiable elements on the page to generate a list of elements to be encrypted. The identifiable elements include at least one of DOM tags, control attributes, interface call parameters, and event binding logic. According to a preset strategy, dynamic encryption rules are generated. The dynamic encryption rules support change granularities including random generation, periodic replacement, and session generation. According to the dynamic encryption rules, the identifiable elements in the list of elements to be encrypted are encrypted to obtain encrypted page elements. The encrypted page elements are then input into the front-end rendering engine for page rendering. The mapping relationship between the identifiable elements and the dynamic encryption rules is stored. The mapping relationship includes encryption granularity, effective scope and effective time. When an access request is received from an automation tool, the automation tool is authenticated. Once the authentication is successful, the decryption key corresponding to the dynamic encryption rule is issued. The automated tool queries the mapping relationship based on the decryption key, restores the encrypted page elements to their corresponding original elements, and performs automated operations based on the original elements.

[0006] Secondly, embodiments of the present invention provide a page anti-scraping device, comprising: The scanning module is used to scan identifiable elements in the page and generate a list of elements to be encrypted. The identifiable elements include at least one of DOM tags, control attributes, interface call parameters, and event binding logic. The rule generation module is used to generate dynamic encryption rules according to a preset strategy. The dynamic encryption rules support change granularities including random generation, periodic replacement, and session generation. An encryption module is used to encrypt the identifiable elements in the list of elements to be encrypted according to the dynamic encryption rules, to obtain encrypted page elements, and to input the encrypted page elements into the front-end rendering engine for page rendering. The module stores the mapping relationship between the identifiable elements and the dynamic encryption rules, and the mapping relationship includes encryption granularity, effective scope and effective time. The authentication module is used to authenticate the automation tool when it receives an access request from the automation tool, and to issue the decryption key corresponding to the dynamic encryption rule after successful authentication. The execution module is used by the automated tool to query the mapping relationship based on the decryption key, restore the encrypted page element to the corresponding original element, and perform automated operations based on the original element.

[0007] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the page anti-scraping method described in the first aspect.

[0008] Fourthly, embodiments of the present invention provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the page anti-scraping method described in the first aspect.

[0009] This invention provides a method, apparatus, and related equipment for preventing web scraping. The method scans identifiable elements on a webpage to generate a list of elements to be encrypted; generates dynamic encryption rules according to a preset strategy; encrypts the identifiable elements in the list of elements to be encrypted according to the dynamic encryption rules, obtaining encrypted page elements, and inputs these encrypted page elements into a front-end rendering engine for page rendering, storing the mapping relationship between identifiable elements and dynamic encryption rules; when an access request from an automated tool is received, the automated tool is authenticated, and upon successful authentication, a decryption key corresponding to the dynamic encryption rule is issued; the automated tool queries the mapping relationship based on the decryption key, restores the encrypted page elements to their corresponding original elements, and performs automated operations based on the original elements. This method effectively blocks malicious manipulation based on structured DOM identifiers and interface parameters by using dynamic encryption rules to encrypt and replace identifiable elements on the page. Since the encryption rules are not fixed and unpredictable, this method effectively blocks malicious manipulation based on structured DOM identifiers and interface parameters. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating an embodiment of the page anti-scraping method provided by the present invention. Figure 2 This is a schematic block diagram of a page anti-scraping device provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] Please see Figure 1 This is a flowchart illustrating a page anti-scraping method provided in an embodiment of the present invention, which includes steps S110 to S150.

[0017] Step S110: Scan the identifiable elements on the page to generate a list of elements to be encrypted; In this embodiment, the page anti-scraping method is applied to online application scenarios such as B / S architecture web pages and SaaS cloud platforms. During the application building phase, the page element scanning module automatically traverses the page source code to extract all identifiable elements from the identifier layer, including DOM tag names (such as div, span, button, etc.), control attribute values ​​(such as the specific values ​​of attributes such as id, class, name, etc.), API call parameters (such as parameter names and values ​​carried in API requests), and event binding logic (such as the identifiers of event handling functions such as onclick, onload, etc.), forming a list of elements to be encrypted. Among them, identifiable elements include at least one of DOM tags, control attributes, API call parameters, and event binding logic.

[0018] In one embodiment, the `id` attribute of a DOM tag is a unique identifier for a page element, typically used in JavaScript code to locate elements using the `getElementById` method; the `class` attribute identifies the style category of an element, and multiple elements can share the same `class` value; the `name` attribute identifies the name of a form field when a form is submitted. In existing technologies, these attributes typically use naming conventions with clear semantic features. For example, `id: login-btn` represents a login button, `class: submit-form` represents the style class of the submitted form, and `name: username` represents a username input box. These semantic names allow AI tools to easily establish a mapping relationship between elements and page functions. In this embodiment, the page element scanning module includes the `id`, `class`, and `name` attribute values ​​in the list of elements to be encrypted, which are then encrypted and replaced by the encryption processing module. For example, `id: login-btn` may become `id: a3f8b2e1` after encryption, `class: submit-form` becomes `class: c7d9f2a4`, and `name: username` becomes `name: x8b3e1f6`. The encrypted identifiers no longer possess any semantic information, and AI tools cannot identify element functions through semantic inference. In practical applications, identifiable elements can also include other attributes carrying semantic information, such as custom data attributes and aria accessibility attributes. The specific scanning range can be flexibly configured according to the security requirements of the application.

[0019] Step S120: Generate dynamic encryption rules according to the preset strategy; In this embodiment, the preset strategy is pre-configured by the system administrator or developer according to security requirements, and one or more combinations of three change granularities can be selected. The dynamic encryption rules support change granularities including random generation, periodic replacement, and session generation. When random generation is selected, the system generates a unique set of encryption rules in real time each time a user accesses a page or initiates a request. This rule is only valid during the current loading process, and there is no fixed mapping relationship between the encryption rules used in different loading orders, thus preventing attackers from deduce the encryption pattern through multiple samplings. When periodic replacement is selected, the system generates new encryption rules when a new version of the application is released, using a unified encryption rule within the same version's runtime cycle. This scheme is suitable for application scenarios with short version iteration cycles. When session generation is selected, the system generates an independent set of encryption keys when each user session is established. The encryption keys between different user sessions are independent of each other, so even if the key of one session is leaked, it will not affect the security of other sessions.

[0020] In one embodiment, when the granularity of the dynamic encryption rules is randomly generated, a random number generator or a cryptographic hash function is invoked each time a page load event is triggered to generate a completely new set of encryption rules. Specifically, symmetric encryption algorithms such as AES and DES can be used to generate a new key and initial vector each time; alternatively, a hash function can be used to hash a random salt value and the original element identifier to generate an encrypted identifier. Because different encryption rules are used for each load, even if an attacker obtains the encryption mapping relationship of a certain load, they cannot apply it to the next loaded page. Taking a login page as an example, on the first load, the id login-btn is encrypted as id a3f8b2e1; on the second load, the same id login-btn is encrypted as id f7c2d9e4; and on the third load, it becomes id b8e1a3f5. This irregular dynamic change makes it impossible for attackers to establish a stable element function mapping table, fundamentally blocking reverse engineering methods based on regular sampling. In practice, the generation of random encryption rules can combine multiple factors such as timestamps, session identifiers, and random seeds to further enhance the unpredictability of the rules. Using the above technical solution, a unique encryption rule is generated each time a page loads, causing the encryption identifier to exhibit irregular dynamic changes. Attackers cannot deduce the encryption pattern from historical data, achieving the highest level of dynamic protection.

[0021] In one embodiment, when the dynamic encryption rule change granularity adopts a periodic replacement type, a new set of encryption rules is generated each time a new version of the application is released. This set of rules is stored in the encryption mapping storage module, marked with its effective version number and effective time. Within the same version runtime cycle, all page loads use the same unified encryption rule. For example, application version V1.0.0 uses encryption rule R1; when the application is upgraded to V1.1.0, the system generates a new encryption rule R2 and replaces R1. This change granularity is suitable for application scenarios with short version iteration cycles (such as weekly or bi-weekly releases), achieving a balance between security and system performance. Compared to the method of randomly generating rules for each load, the method of periodic replacement by version reduces the overhead of encryption rule generation and key management, while still ensuring that encryption rules are refreshed during version updates, preventing security risks caused by long-term use of the same rule. In actual implementation, version updates can be automatically triggered by the CI / CD pipeline, automatically generating new encryption rules during the build phase and injecting them into the application deployment package, achieving synchronous updates of encryption rules and application versions. Furthermore, the system can be configured with a version transition period, during which both the old and new encryption rules can be decrypted and queried simultaneously, ensuring compatibility during version switching. By adopting this technical solution and periodically changing the encryption rules according to the application version, system overhead is reduced while maintaining a certain degree of dynamism. This approach is suitable for application scenarios with frequent version iterations, balancing security and operational efficiency.

[0022] In one embodiment, when the granularity of dynamic encryption rule changes is session-generated, a unique set of encryption keys is generated for each session when a user logs in or establishes a session. Specifically, when a user accesses the application through a browser, an encryption key bound to that session is generated simultaneously with the session creation, and the key identifier is stored in the session context. During subsequent page rendering, this session-specific key is used to encrypt and replace page elements. The encryption keys between different user sessions are completely independent; user A's session key is different from user B's session key, and the session key for the same user logging in at different times is also different. Taking a SaaS platform as an example, after user A logs in, the id "login-btn" on their page is encrypted as id "k3m7n2p9"; when user B logs in at the same time, the same id "login-btn" on their page is encrypted as id "q8w4e6r1". Even if an attacker obtains user A's encryption mapping relationship, they cannot apply it to user B's session. This level of granularity is particularly suitable for SaaS platform scenarios with multiple concurrent users, effectively preventing attackers from deducing page element mapping relationships for other accounts using the key of one account. In practice, session keys can be bound to session tokens or JSON Web Tokens (JWTs), and the corresponding encryption keys are destroyed synchronously when the session is terminated, ensuring that the key's lifecycle is consistent with the session. By adopting the above technical solution, independent keys are dynamically generated for each user session, achieving user-level encryption isolation. Even if the key of a single session is leaked, it will not affect the security of other users, making it suitable for online application scenarios with multiple concurrent users.

[0023] Step S130: According to the dynamic encryption rules, encrypt the identifiable elements in the list of elements to be encrypted to obtain encrypted page elements, and input the encrypted page elements into the front-end rendering engine for page rendering, and store the mapping relationship between the identifiable elements and the dynamic encryption rules. In this embodiment, the reference consistency synchronization update logic is as follows: After encrypting all identifiable elements in the list of elements to be encrypted according to the dynamic encryption rules, it automatically traverses all CSS style selectors, native JS event bindings, Vue / React framework refs, and data-* binding directives on the page, synchronously replacing all code referencing the original identifiers with the encrypted ciphertext strings. Throughout the process, only identifier names such as id / class / name are modified; the page DOM structure, text content, images, business style attribute values, and other content layer information are not altered. The mapping relationship includes the correspondence between the original element semantic identifiers before encryption and the encrypted ciphertext identifiers, as well as metadata information such as the encryption algorithm used in this encryption, key version, encryption granularity type, effective scope, and effective time.

[0024] In one embodiment, the front-end rendering engine renders encrypted page elements into a user-visible and operable page. Whether the page renders correctly depends solely on the integrity of the HTML structure, text, styles, images, and other content layer data. The browser does not recognize or rely on semantic strings in the identifier layer such as id / class / name. As long as all code within the page synchronously replaces references in CSS selectors, JS event bindings, and framework directives (such as Vue / React's ref and data-* bindings), maintaining internal reference consistency, encryption only applies to the identifiers themselves and does not affect tag structure, text content, or style attribute values ​​(unless the user-requested "interface parameter / event binding logic" is improperly encrypted, altering the business data itself). Encryption will not cause rendering anomalies. This mechanism is similar to Webpack CSS Modules class name hashing and JS... The code obfuscation and variable renaming logic are consistent and represent an extension of mature front-end technologies. Specifically, when the browser receives the HTML document returned by the server, the element identifiers in the document have been replaced with encrypted ciphertext identifiers. However, the CSS style rules and JavaScript event handling logic are also simultaneously adapted and updated, referencing the encrypted identifiers. For example, in the original code, the button's id is login-btn, the CSS selector is #login-btn, and the JavaScript code retrieves the login-btn element using the getElementById method; after encryption, the button's id becomes a3f8b2e1, and the CSS selector is simultaneously updated to #a The JavaScript code for element 3f8b2e1 has been updated to retrieve the element using the `getElementById` method. This synchronous adaptation ensures that the page rendering and interactive functionality remain unaffected. For ordinary end-users, the encryption process is completely transparent; users can browse page content, click buttons, fill out forms, and submit data normally without noticing any anomalies. The front-end rendering engine can use existing browser rendering engines without requiring additional plugins or extensions. Furthermore, to prevent the encrypted identifier from being viewed and analyzed through browser developer tools, the system can also deploy anti-debugging and code obfuscation mechanisms on the front end to further enhance protection.

[0025] Step S140: When an access request from an automation tool is received, the automation tool is authenticated. After successful authentication, the decryption key corresponding to the dynamic encryption rule is issued. Step S150: The automated tool queries the mapping relationship based on the decryption key, restores the encrypted page element to the corresponding original element, and performs automated operations based on the original element.

[0026] In this embodiment, the identity authentication mechanism is used to distinguish between compliant automation tools and unauthorized third-party crawlers. When a compliant automation tool initiates an access request, the system verifies its identity according to a preset authentication method. After successful verification, a corresponding decryption key is issued. The compliant tool uses this key to query the encrypted mapping storage module, accurately restoring the encrypted elements of the current page to their original semantic identifiers, thereby enabling the normal execution of automated operations. Automated operations include, but are not limited to, automated testing, data collection, form filling, and process scheduling.

[0027] In one embodiment, the key can be distributed using a pre-defined hard-coding method. This involves hard-coding the decryption key into the tool's source code or configuration file during the development phase of the compliance automation tool. This method is simple to implement and suitable for internal automated testing tools, but its security is relatively low, as there is a risk of the key being reverse-engineered. Alternatively, a server-side interface authentication-based dynamic key distribution method can be used. During runtime, the compliance automation tool calls the server's authentication interface, submitting credential information such as the application ID and application key. After successful server verification, the decryption key is dynamically distributed. This method eliminates the need to hard-code the key on the tool side; the key can be updated or revoked as needed, resulting in higher security.

[0028] In one embodiment, the authentication mechanism employs a dual authentication method using device fingerprint and developer signature. This means that both the device fingerprint information of the device hosting the automation tool and the developer signature information are verified simultaneously. Only when both are verified successfully is the decryption key issued. The device fingerprint can include a combined hash value of hardware identification information such as the device's MAC address, CPU serial number, and hard drive serial number. The developer signature can be verified using asymmetric encryption algorithms (such as RSA or ECDSA). This method provides the highest level of security; even if the key is leaked, unauthorized devices or unauthorized developers will still be unable to pass authentication.

[0029] In one embodiment, automated tools that fail authentication can only obtain encrypted page elements but cannot obtain the decryption key, thus preventing them from restoring the encrypted page elements to their original semantic identifiers. Automated tools that fail authentication include general AI crawlers, third-party crawlers, and unauthorized automated testing tools. When such tools access application pages, the server normally returns encrypted page elements, but because the tools fail authentication, the authentication and key distribution module refuses to issue decryption keys. Therefore, these tools can only obtain encrypted page elements, such as a button with id a3f8b2e1 or a form with class c7d9f2a4, but cannot determine the original functions corresponding to these encrypted identifiers. Taking an LLM-based intelligent agent tool as an example, after accessing a page, the tool obtains encrypted element identifiers by parsing the DOM structure. However, because the identifiers do not contain any semantic information, LLM cannot determine through semantic inference which element is the login button, which is the submit button, and which is the username input box. Even if LLM attempts to associate elements through visual information (such as the text "Login" displayed on a button), the encryption rules differ for each load or session, resulting in different encryption identifiers for the same button in different loads. Therefore, LLM cannot establish a stable element-function mapping table. Furthermore, because API call parameters and event binding identifiers are also encrypted, tools cannot indirectly reconstruct page functions and operational logic by analyzing API request parameters and event triggering patterns. This prevents unauthorized AI tools and crawlers from performing automated manipulation based on DOM identifiers. If a crawler does not rely on DOM identifiers and directly reads plaintext text (product names, prices, copy, etc.) displayed on the page through screenshot OCR or visual large-scale models, this solution cannot intercept such information reading behavior. This type of attack belongs to a separate protection category and is not within the scope of this patent.

[0030] This method scans identifiable elements on a webpage to generate a list of elements to be encrypted. Based on a preset strategy, it generates dynamic encryption rules. According to these rules, it encrypts the identifiable elements in the list, obtaining encrypted page elements. These encrypted elements are then input into a front-end rendering engine for page rendering, storing the mapping relationship between the identifiable elements and the dynamic encryption rules. When an access request from an automated tool is received, the tool is authenticated. Upon successful authentication, a decryption key corresponding to the dynamic encryption rule is issued. The automated tool uses the decryption key to query the mapping relationship, restores the encrypted page elements to their original forms, and performs automated operations based on these original elements. This method uses dynamic encryption rules to encrypt and replace identifiable elements on the page. These rules are not fixed and are unpredictable, making it impossible for general AI tools and third-party crawlers to maliciously crawl and manipulate the page through one-time cracking or regular sampling.

[0031] This invention also provides a page anti-scraping device, which is used to execute any of the aforementioned page anti-scraping methods. Specifically, please refer to... Figure 2 , Figure 2 This is a schematic block diagram of a page anti-scraping device provided in an embodiment of the present invention. The page anti-scraping device 100 can be configured in a server.

[0032] like Figure 2 As shown, the page anti-crawling device 100 includes a scanning module 110, a rule generation module 120, an encryption module 130, an authentication module 140, and an execution module 150.

[0033] The scanning module 110 is used to scan identifiable elements in the page and generate a list of elements to be encrypted. The identifiable elements include at least one of DOM tags, control attributes, interface call parameters, and event binding logic. The rule generation module 120 is used to generate dynamic encryption rules according to a preset strategy. The dynamic encryption rules support change granularities including random generation, periodic replacement and session generation. The encryption module 130 is used to encrypt the identifiable elements in the list of elements to be encrypted according to the dynamic encryption rules, to obtain encrypted page elements, and to input the encrypted page elements into the front-end rendering engine for page rendering, and to store the mapping relationship between the identifiable elements and the dynamic encryption rules, wherein the mapping relationship includes encryption granularity, effective scope and effective time. The authentication module 140 is used to authenticate the automation tool when it receives an access request from the automation tool, and to issue the decryption key corresponding to the dynamic encryption rule after successful authentication. The execution module 150 is used by the automation tool to query the mapping relationship based on the decryption key, restore the encrypted page element to the corresponding original element, and perform automated operations based on the original element.

[0034] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the page anti-scraping method described above.

[0035] In another embodiment of the invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the page anti-scraping method as described above.

[0036] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0037] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0038] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0039] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0040] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0041] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for preventing web scraping, characterized in that, include: Scan the identifiable elements on the page to generate a list of elements to be encrypted. The identifiable elements include at least one of DOM tags, control attributes, interface call parameters, and event binding logic. According to a preset strategy, dynamic encryption rules are generated. The dynamic encryption rules support change granularities including random generation, periodic replacement, and session generation. According to the dynamic encryption rules, the identifiable elements in the list of elements to be encrypted are encrypted to obtain encrypted page elements. The encrypted page elements are then input into the front-end rendering engine for page rendering. The mapping relationship between the identifiable elements and the dynamic encryption rules is stored. The mapping relationship includes encryption granularity, effective scope and effective time. When an access request is received from an automation tool, the automation tool is authenticated. Once the authentication is successful, the decryption key corresponding to the dynamic encryption rule is issued. The automated tool queries the mapping relationship based on the decryption key, restores the encrypted page elements to their corresponding original elements, and performs automated operations based on the original elements.

2. The page anti-scraping method as described in claim 1, characterized in that, The identifiable elements also include attribute value semantic content, and the identifiable elements in the scanned page include: Traverse the page source code of the page, extract the ID and class name of DOM tags, the name attribute of controls, the field name and parameter value of API call parameters, and the event handler identifier in the event binding logic.

3. The page anti-scraping method as described in claim 1, characterized in that, The dynamic encryption rules support three granularities: random generation, periodic replacement, and session generation, as detailed below: The random generation type is to randomly generate a unique set of encryption rules each time the page is loaded or requested; The periodic replacement type means that the encryption rule is changed according to the application version update; The session generator generates a set of independent encryption keys for each user session, and the encryption keys between different user sessions are independent of each other.

4. The page anti-scraping method as described in claim 1, characterized in that, The authentication method is a dual authentication of device fingerprint and developer signature.

5. The page anti-scraping method as described in claim 4, characterized in that, The dual authentication of device fingerprint and developer signature includes: Collect the device fingerprint information of the device where the automation tool is located, and verify the developer signature information. When both the device fingerprint and the developer signature are verified, the identity authentication is confirmed to be successful.

6. A webpage anti-scraping device, characterized in that, include: The scanning module is used to scan identifiable elements in the page and generate a list of elements to be encrypted. The identifiable elements include at least one of DOM tags, control attributes, interface call parameters, and event binding logic. The rule generation module is used to generate dynamic encryption rules according to a preset strategy. The dynamic encryption rules support change granularities including random generation, periodic replacement, and session generation. An encryption module is used to encrypt the identifiable elements in the list of elements to be encrypted according to the dynamic encryption rules, to obtain encrypted page elements, and to input the encrypted page elements into the front-end rendering engine for page rendering. The module stores the mapping relationship between the identifiable elements and the dynamic encryption rules, and the mapping relationship includes encryption granularity, effective scope and effective time. The authentication module is used to authenticate the automation tool when it receives an access request from the automation tool, and to issue the decryption key corresponding to the dynamic encryption rule after successful authentication. The execution module is used by the automated tool to query the mapping relationship based on the decryption key, restore the encrypted page element to the corresponding original element, and perform automated operations based on the original element.

7. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the page anti-scraping method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the page anti-scraping method as described in any one of claims 1 to 5.

Images