Dynamic webpage structure reconstruction anti-crawler method and system based on AI

Abstract

The invention provides an AI-based dynamic webpage structure reconstruction anti-crawler method and system, which are used for solving the problem that the existing anti-crawler technology is easy to crack. The method is characterized in that a pre-trained large language model is utilized to generate confused HTML elements in real time during each page request, and a DOM structure is dynamically reconstructed. The method specifically comprises the steps of receiving a webpage access request; generating confusion elements by using an AI model; a DOM structure is reconstructed; dynamically generating an element identifier; inserting interference elements; generating a dynamic CSS style; and returning the reconstructed webpage. According to the method, the page structure is reconstructed for each request, so that the anti-crawler effect is greatly improved, meanwhile, the consistency of user experience is kept, and a more effective solution is provided for website content protection.

Description

[0001] The present invention relates to the field of Internet security technology, and in particular to an anti-crawler technology for protecting website content. Specifically, the present invention provides an artificial intelligence-based dynamic web page structure reconstruction method and system for preventing web crawlers from illegally obtaining website content. Background Art

[0002] Web crawler technology has important applications in data collection, search engine optimization, market analysis and other fields. However, with the widespread use of crawler technology, the problem of illegal acquisition and abuse of website content has become increasingly serious, posing a huge threat to the interests of website operators. Therefore, anti-crawler technology plays an increasingly important role in the field of Internet security and has become a key technology for protecting website content and maintaining digital copyright.

[0003] At present, the main anti-crawler technologies include IP restriction, User-Agent verification, and verification code mechanism. IP restriction prevents crawlers by monitoring and limiting the access frequency from the same IP; User-Agent verification identifies and blocks access from abnormal browsers by checking the User-Agent field in the HTTP request header; and the verification code mechanism requires users to enter a graphic or text verification code to distinguish between human users and automatic programs. In addition, some websites also use technologies such as dynamic content loading and JavaScript obfuscation to increase the difficulty for crawlers to obtain data.

[0004] However, these traditional anti-crawler solutions have some obvious shortcomings. First, IP restrictions and User-Agent verification can be easily bypassed by advanced crawlers through proxy IP and forged User-Agent. Second, although the verification code mechanism is effective, it will affect the user experience, especially in frequent use scenarios. Furthermore, although technologies such as dynamic loading and JavaScript obfuscation increase the difficulty of crawling, they can still be cracked by simulating browser environments or reverse engineering. These problems have caused the existing anti-crawler technology to often be unsatisfactory in the face of increasingly complex and intelligent crawlers.

[0005] To solve the above problems, some studies have proposed anti-crawler methods based on machine learning, which identify crawlers by analyzing user behavior patterns. Although this method has improved the accuracy of anti-crawler to a certain extent, it still has problems such as high misjudgment rate and the need for a large amount of training data. In addition, with the advancement of crawler technology, crawlers that imitate human behavior are becoming increasingly difficult to distinguish, which poses new challenges to methods based on behavioral analysis.

[0006] Therefore, there is an urgent need for a more intelligent, dynamic and unpredictable anti-crawler technology that can effectively prevent various types of crawlers without affecting the access experience of normal users. The development of such a technology has become a research hotspot in the field of network security and is also the key to protecting website content and maintaining the healthy development of the digital ecosystem. Summary of the invention

[0007] The purpose of the present invention is to solve the problems that the existing anti-crawler technology is easy to be cracked, affects the user experience, and is difficult to deal with advanced crawlers. In addition, the present invention also aims to provide an intelligent, dynamic and unpredictable anti-crawler solution, which effectively prevents various types of web crawlers without affecting normal user access.

[0008] To achieve the above objectives, the present invention provides an AI-based dynamic web page structure reconstruction anti-crawler method and system, characterized in that: a pre-trained large language model is used to generate obfuscated HTML elements in real time each time a page is requested, and the DOM structure is dynamically reconstructed so that the page structure is different each time it is visited, while maintaining consistency in visual presentation and functionality. In some embodiments, the method can also be applied to various types of websites, such as news media, e-commerce platforms, etc., to protect their sensitive content.

[0009] Specifically, the method of the present invention includes receiving an access request for a web page, and using a pre-trained large language model to generate in real time obfuscated HTML elements that are similar to the original web page content but do not contain key information.

[0010] Furthermore, under the premise of maintaining consistency in visual presentation and functionality, the DOM structure of the web page is reconstructed, including randomly adjusting the nesting relationship and order of elements.

[0011] Preferably, identifiers of web page elements are dynamically generated and assigned so that the identifiers in each request are different, increasing the difficulty for the crawler to locate the elements.

[0012] Preferably, the pre-trained large language model adopts a GPT series model to provide higher quality obfuscated HTML element generation.

[0013] Optionally, structurally similar noise elements are dynamically inserted around the key elements to further increase the complexity of data extraction.

[0014] In one embodiment, a dynamic CSS style is generated to maintain consistency in visual presentation, ensuring that the page layout is not affected by changes in the DOM structure.

[0015] In some embodiments, the present invention may also use JavaScript to perform secondary adjustments to the web page structure on the client side, further increasing the unpredictability of the page structure.

[0016] In addition, the present invention can also use multi-layer nesting and redundant representation for key content to increase the difficulty of data extraction.

[0017] In some embodiments, the process of rebuilding the DOM structure is completed on the server side, using server-side rendering technology to improve page loading speed and security.

[0018] In addition, the dynamic CSS style of the present invention can use Flexbox or Grid layout technology to more flexibly maintain the consistency of the page layout while allowing dynamic changes in the position of elements.

[0019] In other embodiments, the method can be used in combination with traditional anti-crawler technology (such as IP restriction, User-Agent verification, etc.) to form a multi-level protection mechanism.

[0020] In a preferred embodiment, the system of the present invention includes a request receiving module, an AI generating module, a DOM reconstruction module, an identifier generating module, an interference element inserting module, a CSS generating module and a response sending module, which work together to realize the anti-crawler function of dynamic web page structure reconstruction.

[0021] In another preferred embodiment, the method of the present invention can be applied to websites that require highly protected content, such as financial data platforms, academic research databases, etc., to provide more powerful data protection capabilities.

[0022] In another preferred embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the above-mentioned anti-crawler method for dynamic web page structure reconstruction.

[0023] By adopting the above scheme, the present invention has the following beneficial effects: (1) Each request generates a different page structure, which greatly increases the complexity and cost of the crawler; (2) Dynamics and unpredictability make traditional crawling methods based on pattern recognition ineffective; (3) The obfuscation strategy combined with the server and client provides multi-level anti-crawler protection; (4) The introduction of AI technology makes the obfuscation process more intelligent and can generate more natural and more difficult to identify structures; (5) While providing powerful anti-crawler capabilities, it does not affect the browsing experience of normal users.

[0024] In summary, the present invention innovatively solves the limitations of existing anti-crawler technology by combining AI technology and dynamic web page reconstruction, and provides a new method for website content protection that is efficient, intelligent and difficult to crack, which is of great significance for maintaining the security and value of digital content. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1 This is a flowchart of the basic dynamic web page structure reconstruction anti-crawler method. It shows the complete process from receiving a web page request to returning the reconstructed web page, including steps such as using AI to generate obfuscated elements, reconstructing DOM, dynamically generating identifiers, inserting interference elements, and generating dynamic CSS.

[0027] Figure 2 This is a flowchart of the application of AI models in dynamic web page structure reconstruction. It describes how to use AI models to generate obfuscated elements and perform structure verification, security filtering, and style consistency checks.

[0028] Figure 3 This is a flowchart of a server-side dynamic web page structure reconstruction method based on asynchronous updates. It shows the server-side processing from receiving a request to returning a response, including steps such as parsing parameters, obtaining templates, generating dynamic content, AI obfuscation, reconstructing DOM, and generating dynamic CSS.

[0029] Figure 4 This is a conceptual diagram of the dynamic anti-crawler strategy of secondary adjustment by the client and coordination with the server. It shows how DynamicShield performs dynamic element rearrangement, attribute obfuscation, content encryption, and behavior analysis on the client, and communicates with the server to update the defense strategy.

[0030] Figure 5 This is the architecture diagram of DynamicShieldX in complex Web application scenarios. It describes how DynamicShieldX implements dynamic routing obfuscation, virtual DOM obfuscation, component lazy loading, state encryption, and event stream obfuscation in SPA / PWA applications. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0033] Embodiment 1: Basic dynamic web page structure reconstruction anti-crawler method. This embodiment describes in detail a basic dynamic web page structure reconstruction anti-crawler method. The method generates obfuscated elements in real time and dynamically reconstructs the DOM structure to create a unique page structure each time a page is requested, thereby effectively preventing illegal data acquisition by web crawlers.

[0034] like Figure 1 As shown, the method includes the following steps.

[0035] Step 101: receiving a request to access a web page, the system receives an HTTP request from a client, where the client may be a browser, a mobile application, or other terminal device capable of sending HTTP requests.

[0036] Step 102: Generate obfuscated HTML elements using a pre-trained large language model. In this step, the system calls a pre-trained large language model (such as GPT-3.5) to generate obfuscated HTML elements. These elements are similar in structure and semantics to the original web page content, but do not contain actual key information. For example, for an e-commerce web page containing product information, the obfuscated element may be a similar but fictitious product description. The generation process is as follows.

[0037] The system provides the original web page content as input to the large language model.

[0038] The model analyzes the structural and semantic features of the input content.

[0039] The model generates HTML snippets that are similar to the original content but do not contain key information.

[0040] The generated HTML snippet is validated to ensure that it meets predefined security and structural standards.

[0041] Step 103: Reconstruct the DOM structure of the web page. In this step, the system reconstructs the DOM structure of the original web page. The reconstruction process includes: Parse the original HTML document and generate a DOM tree; Randomly adjust the nesting relationship and order of elements in the DOM tree; The obfuscated elements generated in step 102 are inserted at key positions.

[0042] The refactoring algorithm ensures that the visual presentation and functionality remain consistent while significantly changing the HTML structure of the page. For example, the original nested In The element may be moved to another <section>But make sure its visual position remains unchanged by adjusting CSS.

[0043] Step 104: Dynamically generate and assign identifiers to web page elements. To further increase the difficulty for crawlers to locate elements, the system dynamically generates and assigns new identifiers (such as id and class attributes) to page elements. This process follows the following rules: The generated identifier is different in each request; Keep the naming style semantic, but use unpredictable combinations; Make sure new identifiers do not conflict with existing CSS and JavaScript selectors; For example, an element that originally had an id of "product-title" might be given a new id such as "item-heading-7493".

[0044] Step 105: Insert distracting elements around key elements. The system identifies key elements in the page (such as tables or lists containing important data) and dynamically inserts distracting elements with similar structures around them. The number and location of these distracting elements will change in each request, further increasing the complexity of data extraction.

[0045] The insertion of interference elements follows the following principles: maintain an HTML structure similar to that of key elements; include semantically related but meaningless content; and be controlled through CSS to ensure that they do not affect the visual layout of the page.

[0046] Step 106: Generate dynamic CSS styles. In order to maintain the consistency of the visual presentation of the page, the system needs to generate corresponding dynamic CSS styles. This process includes: analyzing the reconstructed DOM structure; generating corresponding CSS rules according to the new element identifiers and position relationships; using CSS Flexbox or Grid layout technology to ensure the flexibility and consistency of the page layout.

[0047] For example, for an element that is moved, the following CSS rules may be generated: .item-heading-7493 {position: absolute;top: 50px;left: 100px; / * Other style attributes* / } Step 107: Return the reconstructed web page. Finally, the system combines the reconstructed HTML, dynamically generated CSS, and necessary JavaScript scripts into a complete web page response and returns it to the client.

[0048] The innovation of this embodiment is to use AI technology to generate a highly personalized page structure in real time, rather than simply combining existing technologies. The page structure generated by this method is unique in each request, greatly improving the anti-crawler effect. At the same time, the user experience is ensured to be unaffected through a carefully designed algorithm.

[0049] It should be noted that the order of the steps described in this embodiment can be adjusted according to specific needs. For example, step 104 and step 105 can be exchanged or executed in parallel. In addition, this method can also be used in combination with other anti-crawler technologies (such as request frequency limitation, User-Agent verification, etc.) to form a more comprehensive protection strategy.

[0050] In subsequent embodiments, the present invention will discuss in detail advanced topics such as the selection and training of AI models, the application of server-side rendering technology, client-side secondary adjustment, and the specific implementation of this method in different application scenarios. These variants and extensions are based on the basic method described in this embodiment, aiming to further improve the performance and adaptability of the system.

[0051] Example 2: Application of AI model in dynamic web page structure reconstruction. This example describes in detail how to apply and optimize AI models in the anti-crawler method for dynamic web page structure reconstruction, especially the use of GPT series models.

[0052] like Figure 2 As shown, in this system, the present invention chooses to use GPT-3.5 as the basic model. GPT-3.5 has powerful natural language understanding and generation capabilities, supports a variety of programming languages ​​and markup languages, and is particularly suitable for generating HTML, CSS, and JavaScript codes. In addition, its model scale achieves a good balance between performance and resource consumption.

[0053] In order to better meet the specific needs of web page structure reconstruction, the present invention fine-tunes the GPT-3.5 model. First, the present invention collects a large number of HTML structures of real web pages and manually annotates the key elements and non-key elements therein. The goal of fine-tuning is to train the model to identify key elements in the web page structure and generate confusing elements with similar structures but different contents. Through iterative training, the present invention optimizes the model parameters, paying special attention to the diversity and unpredictability of the generated content.

[0054] In practical applications, in order to improve the quality and efficiency of the model generating obfuscated elements, the present invention performs special processing on the input. First, key structural information such as tag hierarchy, class name pattern, etc. is extracted from the original HTML. Then the theme information and style instructions of the page are added to the input to help the model generate obfuscated content that is more in line with the style of the original page. At the same time, the present invention clearly specifies the key elements that need to be retained and the non-key elements that can be replaced.

[0055] The output generated by the model needs to be post-processed to ensure its usability and security. The present invention implements an automated post-processing process, including structure verification, security filtering, and style consistency checking. This process ensures that the generated HTML structure complies with Web standards, does not contain security risks, and is visually consistent with the original page style.

[0056] Considering that each page request needs to call the AI ​​model, the present invention has taken a series of optimization measures to improve the response speed. First, the present invention uses model quantization technology to reduce the model size and speed up loading and reasoning. Secondly, the present invention implements a batch processing mechanism to package the generation requests of multiple page elements and send them to the model for processing at one time. In addition, the present invention also designs an intelligent caching system to cache common page structures and element combinations, effectively reducing repeated calculations.

[0057] In order to cope with the evolving crawler technology, the present invention establishes a continuous learning mechanism for the model. The system automatically collects performance data and failure cases during operation, and regularly retrains and updates the model based on these new data. The present invention also compares the effects of different versions of the model through A / B testing to select the best solution.

[0058] It is worth noting that the application of AI models is not limited to generating obfuscated HTML elements. In actual deployment, the present invention found that the AI ​​model can also be used to dynamically generate CSS styles, so that the page layout has structural unpredictability while maintaining visual consistency. In addition, it can also generate anti-crawler JavaScript code to further increase the security of the client. The present invention is also exploring the use of AI to analyze request patterns and identify possible crawler behaviors, thereby dynamically adjusting the intensity of the obfuscation strategy.

[0059] This AI-based dynamic web page structure reconstruction method has significant advantages over traditional static obfuscation technology. It can not only generate highly personalized and unpredictable page structures, but also make adaptive adjustments based on actual conditions. The pages generated by this method are unique in each request, greatly improving the anti-crawler effect while maintaining a good user experience.

[0060] In the next embodiment, the present invention will discuss in detail how to efficiently implement this dynamic reconstruction process on the server side and how to optimize performance to cope with high concurrency scenarios.

[0061] Example 3: Server-side dynamic web page structure reconstruction based on asynchronous update. This example describes a server-side dynamic web page structure reconstruction method based on asynchronous update, which can not only ensure the dynamic change of the page structure, but also significantly improve the system performance and user experience, while avoiding excessive cache overhead. The present invention introduces an asynchronous update mechanism based on the original server-side rendering architecture. The core idea of ​​this mechanism is to separate the reconstruction process of the page structure from the user request, so as to achieve faster response speed and higher concurrent processing capability.

[0062] When the system receives a page request, it immediately returns a basic version of the page. This basic version contains the necessary content and structure, but may not have been completely dynamically reconstructed. At the same time, the system triggers an asynchronous reconstruction task. This task includes operations such as using AI models to generate obfuscated elements, reorganizing DOM structures, and reconstructing styles. In order to efficiently manage these asynchronous tasks, the present invention uses a task queue system based on Redis. This system allows the present invention to finely control the priority, retry mechanism, and concurrency of tasks, so that the system can adapt to different load conditions.

[0063] After the reconstruction task is completed, the new page structure will be stored in the Redis cache. However, the present invention no longer associates it with a specific user, but maintains a small structure pool for each page. This structure pool is actually a circular buffer, and when a new structure is generated, it will replace the oldest one. This design allows the present invention to continuously update the page structure while keeping the cache size constant. For subsequent requests, the system will randomly select a version from the structure pool of the page and return it. If the structure pool is empty (probably because the reconstruction has not been completed), the base version of the page is returned.

[0064] The system will re-execute the reconstruction process periodically or under specific trigger conditions to update the page version in the structure pool. The present invention implements an adaptive reconstruction strategy. The system will analyze the overall access pattern and load conditions and dynamically adjust the reconstruction frequency and the size of the structure pool. The core of this strategy is an access frequency analysis algorithm based on a time window. The algorithm maintains a fixed-size time window and records all access timestamps within this window. When a new access occurs, the algorithm adds a new timestamp and removes the old timestamp outside the window. By calculating the number of accesses within the window, the algorithm can decide whether to trigger reconstruction and determine the time interval for the next reconstruction.

[0065] This algorithm dynamically adjusts the reconstruction interval based on the access frequency in the last hour. Under normal load, the structure pool may be updated every 10-15 minutes. When high-frequency access or suspicious patterns are detected, the update frequency will be increased to provide stronger protection. The reconstruction interval is calculated based on the inverse of the access rate, but is limited to between 5 minutes and 1 hour to balance the reaction speed and system load.

[0066] It is worth noting that this asynchronous update mechanism based on the structure pool not only reduces the cache pressure, but also provides a basis for implementing more complex anti-crawler strategies. For example, the present invention can select page structures of different complexities according to the characteristics of the request (such as User-Agent, IP address, etc.), or generate new structure variants in real time. This can be achieved through a decision tree algorithm, which selects or generates the most suitable page structure based on the combination of request characteristics.

[0067] The decision tree algorithm first analyzes the characteristics of the request, including but not limited to the User-Agent string, IP address, request frequency, access mode, etc. Then, it traverses a predefined rule set that defines the actions that should be taken under different combinations of characteristics. For example, if high-frequency access from the same IP is detected, the algorithm may select the most complex page version in the structure pool; if it is a normal access from a mobile device, a relatively simple version may be selected to optimize loading speed.

[0068] The advantage of this approach is that it can provide a potentially unique structure for each page request while maintaining high performance. Even consecutive requests from the same user may obtain different page structures, which greatly increases the difficulty of crawler analysis. At the same time, because the reconstruction process is performed asynchronously, the system can better allocate and utilize server resources and maintain stable performance under high load.

[0069] In the next embodiment, the present invention will explore how to combine this server-side asynchronous reconstruction mechanism with real-time adjustments on the client side to create a more dynamic and unpredictable page environment. This will include how to use JavaScript on the client side for secondary obfuscation, and how to achieve secure communication between the server and the client to coordinate the overall anti-crawler strategy. Through this server-side and client-side collaboration, the present invention expects to build a more powerful and flexible anti-crawler system to provide more comprehensive protection for website content.

[0070] Example 4: Dynamic anti-crawler strategy with secondary adjustment of the client and coordination with the server. This example describes an advanced anti-crawler strategy that combines asynchronous reconstruction on the server side and real-time adjustment on the client side. This method creates a more dynamic and unpredictable page environment through the collaborative work of the server and the client, further improving the anti-crawler effect.

[0071] After completing the initial page structure reconstruction on the server side, the present invention introduces a set of complex JavaScript mechanisms on the client side to perform secondary adjustments to the page. This client-side adjustment not only increases the dynamics of the page structure, but also enables real-time anti-crawler strategy adjustments based on the user's actual interactive behavior.

[0072] The core of the client-side adjustment mechanism is a lightweight JavaScript library, which we will call "DynamicShield". DynamicShield starts working immediately after the page is loaded, and its main functions include dynamic element rearrangement, attribute obfuscation, content encryption, and behavior analysis.

[0073] Dynamic element reflow uses a random seed-based shuffling algorithm. This algorithm first marks non-critical elements on the page, and then reorders these elements based on a random seed provided by the server. This process is progressive, that is, the reflow is not completed all at once, but slowly as the user browses the page to avoid affecting the user experience. The frequency and range of the reflow are dynamically adjusted based on the user's browsing behavior. For example, when fast, mechanical scrolling is detected, the reflow frequency will increase.

[0074] The attribute obfuscation function mainly targets the id and class attributes of DOM elements. DynamicShield will periodically change the values ​​of these attributes and generate new attribute values ​​using a hash algorithm based on time and random factors. This process is reversible, so it will not affect normal JavaScript functions, but it will greatly increase the difficulty for crawlers to locate and extract specific elements.

[0075] The content encryption feature uses a lightweight reversible encryption algorithm. Sensitive text content in the page is encrypted on the server side, and then decrypted and displayed by DynamicShield on the client side. The decryption process is progressive, and the decrypted content will remain in the DOM for a short time before being re-encrypted, which makes static analysis extremely difficult.

[0076] The behavior analysis function is a key component of DynamicShield. It collects user interaction data, such as mouse movement trajectory, click patterns, page dwell time, etc., and uses a machine learning-based anomaly detection algorithm to identify possible crawler behaviors. This algorithm uses an improved isolation forest method to quickly detect abnormal behaviors with low computational overhead.

[0077] DynamicShield maintains periodic communication with the server. It sends the collected behavior data and current page status to the server through an encrypted channel. The server dynamically adjusts the anti-crawler strategy based on this information, combined with global access statistics and threat intelligence. These adjustments are returned to DynamicShield in the form of instructions, guiding it to adjust the parameters of various functions, such as reordering frequency, obfuscation strength, etc.

[0078] In order to prevent DynamicShield from being analyzed and cracked, the present invention adopts code obfuscation and self-modification technology. The core logic of DynamicShield is divided into multiple small blocks, which are dynamically combined and executed at runtime. At the same time, DynamicShield will regularly obtain new code fragments from the server to update some of its own functions, which makes static analysis extremely difficult.

[0079] It is worth noting that this approach also brings some challenges, especially in terms of accessibility and search engine optimization (SEO). To address these issues, the present invention implements an intelligent judgment mechanism that can identify legitimate search engine crawlers and auxiliary technologies, provide special treatment for these visitors, and ensure that the website content can still be correctly indexed and accessed.

[0080] In addition, to further improve the robustness of the system, the present invention also implements an adaptive learning mechanism. The system continuously analyzes the crawler behaviors that are successfully blocked and uses this information to update and optimize the anti-crawler strategy. This continuous learning and evolution capability enables the system to quickly adapt to emerging crawler technologies.

[0081] In general, this method of combining server-side asynchronous reconstruction and client-side real-time adjustment creates a dynamically changing, unpredictable, and adaptive web page environment. It can not only effectively prevent automated data crawling, but also provide a smooth browsing experience for legitimate users. In the next embodiment, the present invention will explore how to extend this technology to more complex web application scenarios, such as single-page applications (SPA) and progressive web applications (PWA).

[0082] Example 5: Dynamic anti-crawler strategy in complex web application scenarios. This example explores how to extend the aforementioned dynamic anti-crawler strategy to more complex web application scenarios, especially single-page applications (SPA) and progressive web applications (PWA). These modern web application architectures bring new challenges and opportunities to anti-crawler technology.

[0083] In SPA and PWA, page content is usually loaded dynamically, and page jumps often do not involve a complete page refresh. This feature makes the traditional anti-crawler strategy based on page loading no longer applicable. To meet this challenge, a new anti-crawler framework was designed, called "DynamicShieldX".

[0084] The core idea of ​​DynamicShieldX is to tightly integrate anti-crawler strategies with the state management of the application. In SPA, state management libraries such as Redux or MobX are usually used. DynamicShieldX, as a middleware, intercepts all state update operations. Whenever the application state changes, DynamicShieldX triggers a series of dynamic protection measures.

[0085] These protections include dynamic routing obfuscation, virtual DOM obfuscation, component lazy loading strategy, state encryption, and event flow obfuscation. Dynamic routing obfuscation periodically changes the application's routes, making them unpredictable. For example, the original " / products" route may be temporarily changed to " / items-2468". This change is temporary and changes periodically. The actual routing information is stored encrypted and is only decrypted and used after the client is verified.

[0086] Virtual DOM obfuscation introduces additional obfuscation elements at the virtual DOM level. These elements will be filtered out during rendering and will not affect the actual user interface, but will make it extremely difficult to understand the application structure by analyzing the DOM structure. The component lazy loading strategy implements an intelligent component loading mechanism. Unlike traditional route-based lazy loading, this strategy dynamically determines the loading order and timing, making the application loading process unpredictable. At the same time, it will generate some fake loading requests to further obfuscate the real structure of the application.

[0087] State encryption ensures that the application's key state data is encrypted during storage and transmission. The encryption key is updated regularly and obtained from the server through a secure channel. This ensures that even if a crawler can intercept the application's state data, it cannot understand its meaning. Event stream obfuscation injects some false events and state updates into the events and state updates triggered by user interactions. These operations will not affect the actual functionality of the application, but will make it extremely difficult to understand the application logic by listening to the event stream.

[0088] For PWAs, DynamicShieldX also provides additional protection measures. It modifies the behavior of Service Worker to dynamically encrypt and decrypt cached resources to ensure that critical application resources are not directly accessed even when offline. For PWAs that use push notifications, DynamicShieldX encrypts the content of push notifications and dynamically decrypts them on the client to prevent sensitive information from being obtained by intercepting push notifications. For PWAs that use background synchronization APIs, DynamicShieldX changes the behavior of synchronization operations, introducing random delays and false synchronization operations, making it difficult to infer application behavior by analyzing synchronization patterns.

[0089] DynamicShieldX also includes a sophisticated machine learning model for real-time analysis of user behavior. This model not only considers traditional indicators such as click frequency and mouse movement patterns, but also analyzes behavioral characteristics unique to the SPA environment, such as page switching frequency, component interaction patterns, etc. The model uses an improved long short-term memory network (LSTM) that can capture the time series characteristics of user behavior, thereby more accurately identifying abnormal behavior.

[0090] In practical applications, DynamicShieldX significantly improves the security of SPA and PWA. It not only effectively prevents automated data crawling, but also resists more complex reverse engineering attempts. At the same time, due to its intelligent resource management and lazy loading strategy, in most cases, the impact of DynamicShieldX on application performance is negligible.

[0091] It is important to note that although DynamicShieldX provides strong protection, it is not foolproof. During implementation, security and user experience need to be carefully balanced to ensure that legitimate users' access is not affected. In addition, strategies need to be regularly updated and optimized to cope with evolving crawler techniques.

[0092] In general, DynamicShieldX represents an important advancement in anti-crawler technology in the field of modern web applications. It not only adapts to the special needs of SPA and PWA, but also provides new possibilities for building safer and more dynamic web applications. Future research directions may further explore how to combine these technologies with emerging web standards and frameworks to cope with the ever-changing web ecosystem.

[0093] Example 6: Distributed anti-crawler system based on edge computing. This example discusses how to use edge computing technology to build a distributed anti-crawler system to deal with large-scale, distributed crawler attacks. This method not only improves the efficiency and scalability of the anti-crawler system, but also better adapts to complex network environments and access requirements in different geographical locations.

[0094] In traditional centralized anti-crawler systems, all detection and defense logic are concentrated on the core server. This architecture may experience performance bottlenecks and response delays when facing large-scale, distributed crawler attacks. The edge computing anti-crawler system achieves faster response speed and stronger defense capabilities by delegating part of the computing and decision-making logic to the edge of the network.

[0095] The core component of the system is the "EdgeGuard" module deployed on each edge node. EdgeGuard is a lightweight but powerful anti-crawler unit that is able to independently perform basic anti-crawler tasks while keeping in sync with the central control system. The main functions of EdgeGuard include request filtering, behavior analysis, dynamic response generation, and local cache management.

[0096] Request filtering is EdgeGuard's first line of defense. It uses a modified Bloom filter algorithm that can quickly identify known malicious IP addresses and abnormal request patterns. This filter regularly updates the rule set from the central system and also updates itself based on patterns observed locally.

[0097] The behavior analysis module uses a lightweight machine learning model that is optimized for edge computing environments. This model can analyze multiple dimensions such as the time interval, sequential pattern, content characteristics, etc. of the request, and evaluate the credibility of each request in real time. The parameters of the model are synchronized from the central system regularly, but they are also fine-tuned based on local data to adapt to the characteristics of a specific geographic location or network environment.

[0098] Dynamic response generation is an innovative feature of EdgeGuard. For requests identified as suspicious, EdgeGuard does not simply reject them, but generates dynamic, obfuscated response content. These contents appear legitimate on the surface, but actually contain invalid or misleading information. This approach not only confuses crawlers, but also helps the system collect more information about crawler behavior.

[0099] The local cache management module uses an intelligent caching strategy. It not only caches static content, but also dynamically adjusts the caching strategy based on user behavior and content sensitivity. For highly dynamic or sensitive content, the system uses short-term caching or real-time generation to prevent data mining through cached content.

[0100] EdgeGuard nodes form a distributed network that enables limited information exchange and collaborative defense. For example, when a node detects a new attack pattern, it can quickly notify neighboring nodes, forming a distributed defense mechanism similar to the immune system.

[0101] The central control system plays the role of coordinator and decision maker. It aggregates data from all EdgeGuard nodes, runs more complex analysis algorithms, and formulates global defense strategies. The central system is also responsible for managing and distributing encryption keys, updating machine learning models, coordinating large-scale countermeasures, etc.

[0102] In addition, the system introduces a novel "honeypot" strategy. When suspicious activity is detected, the system can dynamically create fake service nodes that provide fake data that looks real but is actually carefully constructed. By inducing crawlers to visit these honeypots, the system can collect more information about the attacker's intentions and techniques, while also effectively distracting the crawler's attention and resources.

[0103] However, this distributed system also brings some new challenges, such as how to ensure the consistency of policies distributed on different nodes, how to achieve effective data sharing while protecting user privacy, etc. These issues need to be carefully considered and weighed during system design and implementation.

[0104] In general, the distributed anti-crawler system based on edge computing represents a new direction in anti-crawler technology. It not only improves the performance and scalability of the system, but also provides new possibilities for dealing with more complex and dynamic crawler threats. With the continuous development of edge computing technology, this distributed anti-crawler system is expected to play a greater role in the future and provide stronger support for network content protection.

[0105] Example 7: Intelligent content dynamic generation and anti-crawler integrated system. This example describes an innovative anti-crawler method that deeply integrates intelligent content dynamic generation technology with advanced anti-crawler strategies. This method can not only effectively prevent conventional data crawling, but also build a line of defense at the content level, so that even if the crawler breaks through the technical barrier, the acquired data is difficult to be effectively used.

[0106] The core of the system is a module called "ContentShield", which combines natural language processing (NLP), computer vision and dynamic content generation technology. The main goal of ContentShield is to generate unique page content for each user request, which is difficult to extract effective information in batches, while maintaining the semantic and visual consistency of the content.

[0107] ContentShield works as follows: First, the system performs deep semantic analysis on the original content to extract key concepts, entities, and relationships. This step uses the latest natural language understanding model to accurately capture the core semantic structure of the content.

[0108] Next, based on these semantic structures, the system uses a fine-tuned large-scale language model (such as an improved version of the GPT series) to dynamically generate new text content. While maintaining the original semantics, these generated contents will change in expression, wording, and structure. For example, for a product description, the system may reorganize the order of features, replace certain words with synonyms, or change the sentence structure.

[0109] For image content, the system uses a technology based on generative adversarial networks (GANs). This technology can make subtle changes to images while maintaining the main features and semantics of the image. These changes may include color adjustments, texture changes, and position adjustments of non-critical elements. These changes are almost imperceptible to the human eye, but are enough to interfere with crawler technologies based on image hashing or feature matching.

[0110] During the text and image generation process, the system will insert some "digital watermarks". These watermarks are special patterns or marks that are difficult to detect with the naked eye but can be recognized by the system. They can be used to track the source and use of content, providing a basis for subsequent copyright protection and abuse detection.

[0111] After the content is generated, the system will verify the generated content to ensure its semantic accuracy and consistency. This step uses a specially trained semantic comparison model to evaluate the semantic similarity between the generated content and the original content. If the similarity is lower than the preset threshold, the system will regenerate the content or fall back to using the original content.

[0112] To further increase complexity, ContentShield also implements a "mixed reality" content strategy. In some cases, the system generates completely fictional but plausible content elements and mixes them with real content. These fictional elements are carefully designed so that they appear credible when isolated, but reveal obvious inconsistencies if someone attempts to extract and use this information at scale.

[0113] ContentShield is tightly integrated with other anti-crawler technologies described in the previous embodiments. For example, it adjusts content generation strategies based on real-time risk assessments of DynamicShieldX (the technology described in Example 5). For requests identified as possible crawlers, the system increases the degree of content variation and the proportion of fictitious elements.

[0114] However, the use of ContentShield also brings some challenges. First, dynamic content generation increases the computing burden of the server. To solve this problem, the system implements an intelligent caching strategy. For similar requests, it reuses previously generated content and only makes minor modifications. Secondly, it also brings challenges to search engine optimization (SEO) and content consistency. To this end, the system provides special processing logic for search engine crawlers to ensure that they can index a relatively stable version.

[0115] In addition, ContentShield also includes a self-learning and optimization module. This module continuously analyzes the performance and effectiveness of the system, including the quality of content generation, the effectiveness of anti-crawler, the impact on user experience, etc. Based on these analyses, the system automatically adjusts various parameters, such as the degree of content variation, the proportion of fictional elements, the configuration of the generation model, etc., to achieve the best balance.

[0116] In general, ContentShield represents a new anti-scraping approach that uses the content itself as part of the defense. Through intelligent dynamic generation technology, the system can not only effectively prevent large-scale data crawling, but also protect the originality and value of the content. This approach opens up new possibilities for digital content protection, especially for websites and platforms that are highly dependent on the value of original content.

[0117] Example 8: Distributed trust and anti-crawler system based on blockchain. This example introduces an innovative anti-crawler method that combines blockchain technology, distributed identity authentication, and smart contracts to create a decentralized trust and access control system. This method can not only effectively prevent traditional crawler attacks, but also provide a safer and more transparent access mechanism for legitimate users.

[0118] The core of the system is a framework called "ChainGuard", which uses blockchain technology to build a distributed trust network. The main components of ChainGuard include a distributed identity authentication system, an access control mechanism based on smart contracts, and a decentralized reputation scoring system.

[0119] The distributed identity verification system is the foundation of ChainGuard. It uses a model based on Self-Sovereign Identity (SSI). In this model, each user has a unique decentralized identifier (DID) that is stored on the blockchain. Users can use this DID to prove their identity without relying on a centralized identity provider.

[0120] The identity verification process is as follows: When a user first accesses the system, they are asked to create a DID. This process involves generating a pair of public and private keys, and the public key is registered on the blockchain. Each subsequent access by the user requires a signature using the private key, and the system verifies the validity of the signature through the blockchain. This approach not only improves security, but also protects the user's privacy, because the system only needs to verify the signature and does not need to know the user's true identity.

[0121] The smart contract-based access control mechanism is the second key component of ChainGuard. The system deploys a series of smart contracts on the blockchain, which define the access rules for different resources. For example, some content may require users to have a specific reputation score, or the number of visits in the past 24 hours does not exceed a certain threshold. These rules are encoded into smart contracts, ensuring that their execution is transparent and tamper-proof.

[0122] When a user requests access to a resource, the system calls the corresponding smart contract. The contract checks the user's DID, reputation score, historical behavior and other information, and then decides whether to grant access. This process is automatic, decentralized, and does not rely on any central server.

[0123] The decentralized reputation scoring system is the third important component of ChainGuard. Each user's behavior will be recorded and scored, and this score is stored on the blockchain. The calculation of the score takes into account multiple factors, including access patterns, content interactions, feedback from other users, etc. In order to protect privacy, the actual behavior data will not be stored directly on the chain, but will be encrypted and aggregated before being uploaded to the chain.

[0124] The update of reputation score is achieved through a consensus mechanism. Participating nodes in the network (which may include content providers, high-reputation users, etc.) will regularly evaluate user behavior and submit suggestions for score updates. These suggestions are reached through a modified Proof of Stake algorithm and then updated to the blockchain.

[0125] ChainGuard also implements a "challenge-response" mechanism to dynamically verify the authenticity of users. The system randomly sends challenges to users, requiring them to complete certain tasks, such as solving a simple Turing test. The difficulty and frequency of these challenges are dynamically adjusted based on the user's reputation score. The results of the challenges affect the user's reputation score, which in turn affects their future access rights.

[0126] To improve performance and scalability, ChainGuard adopts a two-layer architecture. Critical trust and access control logic is executed on the main chain (such as Ethereum), while some high-frequency, low-value operations are performed on the side chain. This design greatly reduces the burden on the main chain while maintaining the security and decentralization of the system.

[0127] In practical applications, ChainGuard has shown significant advantages. First, it effectively prevents large-scale automated crawler attacks because each access request requires valid identity authentication and sufficient reputation score. Second, it provides a better experience for legitimate users, and high-reputation users can enjoy higher access rights and lower verification requirements. Finally, the transparency and decentralization of the entire system enhance user trust.

[0128] However, the implementation of ChainGuard also faces some challenges. First, the introduction of blockchain technology increases the complexity of the system, which may affect the initial user adoption rate. Second, the performance and latency issues of the decentralized system need to be further optimized. To solve these problems, the system introduces some middle-layer optimizations, such as using cache to store recently verified user information and reduce direct queries to the blockchain.

[0129] In general, ChainGuard represents a new direction for anti-crawler technology and access control systems. By combining blockchain technology with traditional network security methods, it not only provides a safer and more transparent access control mechanism, but also provides the possibility of building a more open and decentralized Internet ecosystem. As blockchain technology continues to mature and optimize, this anti-crawler system based on distributed trust is expected to play a greater role in the future.< / section>

Claims

1. A dynamic webpage anti-crawler method based on artificial intelligence, characterized in that: The following steps are involved: (1) Receive a request to access a web page; (2) Using a pre-trained large language model to generate obfuscated HTML elements in real time that are similar to the original web page content but do not contain key information; (3) Reconstruct the DOM structure of the web page while maintaining the consistency of visual presentation and functionality, including randomly adjusting the nesting relationship and order of elements; (4) dynamically generate and assign identifiers to web page elements so that the identifiers are different in each request; (5) Dynamically inserting interfering elements with similar structures around key elements; (6) Generate dynamic CSS styles to maintain visual consistency; (7) Return the reconstructed web page to the requester.

2. The method according to claim 1, characterized in that The large language model is a GPT series model.

3. The method according to claim 1, characterized in that The process of reconstructing the DOM structure in step (3) is completed on the server side using server-side rendering technology.

4. The method according to claim 1, characterized in that: It also includes the use of JavaScript to make secondary adjustments to the web page structure on the client side to further increase the unpredictability of the page structure.

5. The method according to claim 1, characterized in that The number and positions of the interference elements inserted in step (5) are different in each request.

6. The method according to claim 1, characterized in that It also includes the use of multi-layer nesting and redundant representation of key content to increase the difficulty of data extraction.

7. The method according to claim 1, characterized in that The dynamic CSS style uses Flexbox or Grid layout technology to maintain the consistency of the page layout.

8. A dynamic webpage anti-crawler system based on artificial intelligence, characterized in that: include: A request receiving module, used for receiving a request for accessing a web page; AI generation module, which contains a pre-trained large language model for generating obfuscated HTML elements in real time; DOM reconstruction module, used to rebuild the DOM structure of the web page while maintaining the consistency of visual presentation and functionality; An identifier generation module, used to dynamically generate and assign identifiers to web page elements; Interference element insertion module, used to dynamically insert interference elements with similar structures around key elements; CSS generation module, used to generate dynamic CSS styles that maintain visual consistency; The response sending module is used to return the reconstructed web page to the requester.

9. The system according to claim 8, characterized in that It also includes a client adjustment module for performing secondary adjustment of the web page structure using JavaScript on the client side.

10. The system according to claim 8, characterized in that The DOM reconstruction module adopts server-side rendering technology.

11. The system according to claim 8, characterized in that Also included is a content protection module for using multi-layer nesting and redundant representation for critical content.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images