Web anti-crawler methods, devices, equipment, storage media and products

Abstract

This application provides a method, apparatus, device, storage medium, and product for preventing web scraping, relating to the field of information security. The method includes: sending a first page request to a front-end server to obtain a first-screen HTML fragment and rendering it to obtain the first-screen page content; then sending a second page request to a back-end server to obtain a non-first-screen HTML fragment based on interaction with the back-end server; rendering the non-first-screen HTML fragment to obtain non-first-screen page content; and concatenating the first-screen page content and the non-first-screen page content. In the isomorphic rendering process between the front-end and back-end, this application ensures that the front-end server only returns the first-screen HTML fragment, not the complete HTML, while the browser requests other HTML fragments from the back-end server. This means the front-end server only retains the first-screen data, effectively preventing web crawlers from scraping data from the front-end server and fundamentally protecting website data security.

Description

Technical Field

[0001] This application relates to the field of information security, and in particular to a method, apparatus, device, storage medium, and product for preventing web scraping. Background Technology

[0002] In the digital information age, data security is of paramount importance. Inadequate data protection on websites can lead to data theft by web crawlers, resulting in data leaks and indirectly causing asset losses. To improve first-screen performance, some high-traffic pages often employ isomorphic rendering (a combination of server-side and client-side rendering) for their initial display. For these isomorphically rendered pages, the front-end server lacks parameter validation. Once a web crawler obtains the URL of the isomorphically rendered page, it directly accesses the page. The server returns the complete Hypertext Markup Language (HTML) content to the crawler, which can then analyze and extract the necessary information. Furthermore, if a website can be easily crawled for valuable content, competitors will become more interested, attracting more crawlers and drastically increasing server load, indirectly leading to a significant increase in production costs and even exhausting server resources, rendering the page inaccessible.

[0003] Currently, web anti-scraping methods include high-frequency anti-scraping, where the server identifies whether an IP is a crawler by identifying whether it is frequently accessing the page or carrying a fixed cookie. If the access frequency reaches a certain level, the IP will be blocked from accessing the page. However, this cannot prevent the crawler from obtaining the page content data on its first visit to the page.

[0004] Another common anti-scraping technique involves inserting code into web pages to identify user behavior. This code detects whether a user is a web crawler by observing whether they scroll or click on the page after accessing it from a specific IP address. If no user interacts with the page after accessing it, the user is considered a potential crawler and may be required to enter a verification code before being allowed to remove restrictions. However, this method still cannot prevent crawlers from obtaining page content data on their first visit, and it is difficult to apply to web pages that do not require interaction. Summary of the Invention

[0005] This application provides a webpage anti-crawler method, apparatus, device, storage medium, and product to solve the technical problem that for pages using a front-end and back-end isomorphic rendering method, crawlers can easily obtain the complete HTML content directly from the server after obtaining the URL, thus leading to data leakage.

[0006] Firstly, this application provides a webpage anti-crawler method, applied to a browser, including:

[0007] In response to a user command, a first page request is sent to the front-end server. The first page request includes the webpage address and the browser viewport size.

[0008] The first screen HTML fragment is obtained from the front-end server, and the first screen HTML fragment is rendered to obtain the first screen page content. The first screen HTML fragment is a page HTML fragment suitable for the browser viewport size obtained from the web address.

[0009] A second page request is sent to the backend server. The second page request includes the webpage address and the first screen HTML fragment.

[0010] Based on the interaction with the backend server, obtain the non-first screen HTML fragment, render the non-first screen HTML fragment to obtain the non-first screen page content, and then concatenate the first screen page content and the non-first screen page content.

[0011] Secondly, this application provides a webpage anti-crawler method applied to a front-end server, including:

[0012] In response to the browser's first page request, which includes the webpage address and the browser viewport size, the first page request is sent to the backend server.

[0013] First-screen data is obtained from the backend server. The first-screen data is page data suitable for the browser viewport size obtained from the web address. First-screen HTML fragments are generated by splicing the page data.

[0014] The HTML fragment on the first screen is then sent to the browser.

[0015] Thirdly, this application provides a webpage anti-crawler method applied to a backend server, including:

[0016] In response to a first page request from the front-end server, the first page request includes a webpage address and a browser viewport size. Page data suitable for the browser viewport size is obtained from the webpage address as first-screen data, and the first-screen data is returned to the front-end server.

[0017] In response to a second page request from the browser, the second page request including the webpage address and the first screen HTML fragment, the browser obtains a page HTML fragment other than the first screen HTML fragment from the webpage address as a non-first screen HTML fragment, and returns the non-first screen HTML fragment to the browser.

[0018] Fourthly, this application provides a webpage anti-crawler device applied to a browser, comprising:

[0019] The front-end request module is used to respond to user instructions and send a first page request to the front-end server. The first page request includes the webpage address and the browser viewport size.

[0020] The first-screen rendering module is used to obtain the first-screen HTML fragment from the front-end server, render the first-screen HTML fragment, and display the first-screen page content. The first-screen HTML fragment is a page HTML fragment suitable for the browser viewport size obtained from the web address.

[0021] The backend request module is used to send a second page request to the backend server. The second page request includes the web page address and the first screen HTML fragment.

[0022] The non-first-screen rendering module obtains non-first-screen HTML fragments based on interaction with the backend server, renders the non-first-screen HTML fragments to obtain non-first-screen page content, and concatenates the first-screen page content and the non-first-screen page content.

[0023] Fifthly, this application provides a webpage anti-crawler device applied to a front-end server, comprising:

[0024] The first page request forwarding module is used to respond to the browser's first page request, which includes the web page address and the browser viewport size, and send the first page request to the backend server.

[0025] The HTML splicing module is used to obtain first-screen data from the backend server. The first-screen data is page data suitable for the browser viewport size obtained from the web address, and the first-screen HTML fragment is generated by splicing the page data.

[0026] The feedback module is used to send the HTML fragment from the first screen to the browser.

[0027] Sixthly, this application provides a webpage anti-crawler device applied to a backend server, comprising:

[0028] The first page request processing module is used to respond to the first page request of the front-end server. The first page request includes a web page address and a browser viewport size. The module obtains page data suitable for the browser viewport size from the web page address as the first screen data and returns the first screen data to the front-end server.

[0029] The second page request processing module is used to respond to a second page request from a browser. The second page request includes the web page address and the first screen HTML fragment. The module obtains page HTML fragments other than the first screen HTML fragment from the web page address as non-first screen HTML fragments and returns the non-first screen HTML fragments to the browser.

[0030] In a seventh aspect, this application provides an electronic device including a memory, a processor, and computer-executable instructions stored in the memory and executable on the processor, wherein the processor, when executing the computer-executable instructions, implements the webpage anti-crawler method described in any one of the first aspects above.

[0031] Eighthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the webpage anti-crawler method described in any one of the first aspects.

[0032] Ninthly, this application provides a computer program product that, when run on a terminal device, causes the terminal device to execute the webpage anti-crawler method described in any of the first aspects above.

[0033] The webpage anti-crawler method, apparatus, device, storage medium, and product provided in this application obtain the first-screen HTML fragment by initiating a first page request to a front-end server, rendering it to obtain the first-screen page content, and then initiating a second page request to a back-end server to obtain non-first-screen HTML fragments based on the interaction with the back-end server, rendering the non-first-screen HTML fragments to obtain non-first-screen page content, and concatenating the first-screen page content and the non-first-screen page content. This ensures that during the isomorphic rendering process between the front-end and back-end, the front-end server only returns the first-screen HTML fragment and not the complete HTML. After the browser renders the first-screen HTML fragment, it then requests other HTML fragments from the back-end server. Thus, the front-end server only retains the first-screen data, effectively preventing crawlers from scraping data from the front-end server. Compared with existing methods for identifying and blocking crawlers, this invention fundamentally protects the security of website data. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0035] Figure 1 This is a schematic diagram illustrating an application scenario of the webpage anti-crawler method provided in the embodiments of this application;

[0036] Figure 2A flowchart of a browser-side anti-crawling method for web pages provided in an embodiment of this application;

[0037] Figure 3 A flowchart of a front-end server-side webpage anti-crawling method provided in an embodiment of this application;

[0038] Figure 4 A flowchart of a backend server-side webpage anti-crawling method provided in an embodiment of this application;

[0039] Figure 5 A diagram illustrating the interaction between a browser and front-end / back-end servers in the webpage anti-crawling method provided in this application embodiment;

[0040] Figure 6 A schematic diagram of a browser-side anti-crawler device provided in an embodiment of this application;

[0041] Figure 7 A schematic diagram of a webpage anti-crawler device on the front-end server side provided in an embodiment of this application;

[0042] Figure 8 A schematic diagram of a webpage anti-crawler device on the backend server side provided in an embodiment of this application;

[0043] Figure 9 A schematic diagram of the structure of an electronic device for the web anti-crawler device provided in this application embodiment.

[0044] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0045] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0046] It should be noted that the webpage anti-crawler method and apparatus of this application can be used in the field of information security, or in any field other than information security. The application field of the webpage anti-crawler method and apparatus of this application is not limited.

[0047] Currently, there are two main types of webpage anti-scraping techniques. One is high-frequency anti-scraping, where the server identifies whether an IP address is a web crawler by checking its frequency of access, such as by identifying whether it carries a fixed cookie. If the access frequency reaches a certain level, the accessing IP address is blocked. The other is behavior-based anti-scraping, which involves inserting code into the webpage to identify user behavior. By detecting whether a user performs actions such as scrolling or clicking after accessing the webpage from a specific IP address, the server identifies whether the user is a web crawler. If the user does not perform any actions on the webpage after accessing it, the server considers the user to be a web crawler and requires the user to enter a verification code before allowing the server to remove the restrictions.

[0048] For high-frequency anti-scraping methods, some data has already been lost in the early stages before identification can be performed. Furthermore, if the crawler's access frequency is low, the server has difficulty identifying it. For genuine users who frequently access the website, this strategy may mistakenly block them, causing the website to lose those users. Additionally, for distributed crawlers, since their access IPs are not fixed, high-frequency anti-scraping methods cannot effectively identify them. Behavioral identification anti-scraping methods can overcome some of the shortcomings of high-frequency anti-scraping, but they also expose all data to the crawler on its first visit. They can only identify the user as a crawler after the first crawl and then block the crawler's subsequent crawls. In other words, neither of these remedial anti-scraping methods can fundamentally prevent web page information from being crawled; they can only limit the frequency of web page crawling and raise the barrier to entry for crawling.

[0049] To address the aforementioned technical problems, this application aims to provide a webpage anti-crawler method, apparatus, device, storage medium, and product. The core concept of this method is that during the isomorphic rendering process between the front-end and back-end, the front-end server only returns the initial HTML fragment and not the complete HTML. After rendering the initial HTML fragment, the browser requests other HTML fragments from the back-end server. Thus, the front-end server only retains the initial data, effectively preventing crawlers from scraping all page data from the front-end server and protecting page data from the source.

[0050] To better understand the solutions of the embodiments of this application, an application scenario involved in the embodiments of this application will be introduced below.

[0051] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the webpage anti-crawler method provided in the embodiments of this application, such as... Figure 1As shown, the system includes a browser 100 and a server 200. The server 200 includes a front-end server 201 and a back-end server 202. The browser 100 can respond to user commands and send page requests to the server 200. Specifically, in the server 200, the front-end server 201 first responds to the page request from the browser 100, retrieves the initial HTML fragment from the back-end server 202, and returns the initial HTML fragment to the browser 100. While rendering the initial HTML fragment, the browser 100 continues to send page requests to the server 200. At this time, the back-end server 202 responds by directly returning the non-initial HTML fragments to the browser 100. The browser then renders the non-initial HTML fragments, thus assembling the entire page.

[0052] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0053] Figure 2 A flowchart illustrating a browser-side anti-crawler method provided in this application embodiment. Figure 2 As shown, the method in this embodiment includes:

[0054] S201: In response to a user instruction, a first page request is sent to the front-end server. The first page request includes the webpage address and the browser viewport size.

[0055] The execution entity in this application embodiment can be a browser or a client that includes a browser.

[0056] Understandably, when a user issues a command to access a page in their browser, the browser requests the page's HTML from the front-end server.

[0057] In this embodiment, only a first page request is sent to the front-end server. The first page request includes the webpage address and the browser viewport size. In other words, based on the first page request, the front-end server will only request a first-screen HTML fragment that is suitable for the browser viewport size from the back-end server, rather than the complete page HTML.

[0058] S202: Obtain the first-screen HTML fragment from the front-end server, render the first-screen HTML fragment to obtain the first-screen page content. The first-screen HTML fragment is a page HTML fragment suitable for the browser viewport size obtained from the web address.

[0059] In this embodiment, the browser only obtains the first-screen HTML fragment from the front-end server. Rendering the first-screen HTML fragment yields the first-screen page content suitable for the browser viewport, ensuring that users can obtain the first-screen information quickly and promptly without affecting the user experience.

[0060] S203: Initiate a second page request to the backend server. The second page request includes the web address and the first screen HTML fragment.

[0061] In this embodiment, while rendering the first screen HTML fragment, or after rendering the first screen HTML fragment, the browser sends a second page request to the backend. In other words, based on the second page request, the backend server will send data from other parts of the page back to the browser.

[0062] S204: Obtain non-first-screen HTML fragments based on interaction with the backend server, render the non-first-screen HTML fragments to obtain non-first-screen page content, and concatenate the first-screen page content and non-first-screen page content.

[0063] Understandably, after the backend API responds, the browser retrieves non-first-screen HTML fragments outside the page viewport and renders the non-first-screen page content. In this way, the non-first-screen page data remains on the backend server, not the frontend server. For web crawlers that attempt to obtain HTML by attacking the frontend server, the crawler can only obtain the HTML of the first screen and cannot crawl the complete webpage, thus preventing the website from having its data stolen.

[0064] The webpage anti-crawler method provided in this embodiment obtains the first-screen HTML fragment by sending a first page request to the front-end server and rendering it to obtain the first-screen page content. Then, it sends a second page request to the back-end server to obtain non-first-screen HTML fragments based on the interaction with the back-end server, renders the non-first-screen HTML fragments to obtain non-first-screen page content, and concatenates the first-screen page content and non-first-screen page content. This ensures that during the isomorphic rendering process between the front-end and back-end, the front-end server only returns the first-screen HTML fragment and not the complete HTML. After the browser renders the first-screen HTML fragment, it then requests other HTML fragments of the page from the back-end server. Thus, the front-end server only retains the first-screen data, which effectively prevents crawlers from scraping data from the front-end server. Compared with existing methods for identifying and blocking crawlers, this invention fundamentally protects the security of website data.

[0065] The technical solutions for the above-mentioned anti-crawler webpage methods are described in detail below.

[0066] In one possible implementation, the webpage anti-crawler method provided in this embodiment can send an interface request to the backend server, and after verification, obtain the target container component from the backend server and extract non-first-screen HTML fragments from the target container component.

[0067] Specifically, obtaining non-first-screen HTML fragments based on interaction with the backend server includes: sending an interface request, which is used by the backend server to verify the interface and, after successful verification, allows data to be transferred between the backend server and the browser; obtaining a target container component, which includes non-first-screen HTML fragments, which are page HTML fragments other than the first-screen HTML fragments obtained from the webpage address; and extracting the non-first-screen HTML fragments from the target container component.

[0068] Understandably, non-first-screen HTML fragments can be executed by the browser via JavaScript to request APIs, retrieve data, and then be rendered in the browser. On the backend server, during page development, the first-screen HTML fragments and non-first-screen HTML fragments can be distinguished, and the non-first-screen HTML fragments can be encapsulated as child components of the target container component.

[0069] It should be noted that non-first-screen HTML fragments can also be tagged, especially for content involving non-public information, which can be privately marked in advance. Non-first-screen HTML fragments with private tags are not obtained by the front-end server, so the browser can only obtain non-first-screen HTML fragments with private tags from the interaction with the back-end server, ensuring that non-public content cannot be easily crawled by web crawlers.

[0070] Furthermore, non-first-screen HTML fragments are set with boolean values. The true and false values ​​of the boolean values ​​are used to indicate whether the non-first-screen HTML fragments are rendered. The default value of the boolean value is false. The target container component also includes lifecycle functions, which set the boolean value to true during the overlapping period of the lifecycle functions. The lifecycle functions are only triggered on the browser side.

[0071] In other words, rendering non-first-screen HTML fragments to obtain non-first-screen page content can include: triggering the overlapping cycle of lifecycle functions with a boolean value of true; rendering non-first-screen HTML fragments to obtain non-first-screen page content.

[0072] It is understandable that the target container component can be a general React container component or a Vue container component, as long as it uses the same front-end framework as the front-end and back-end isomorphic rendering. This application does not impose any restrictions on this.

[0073] In this embodiment, by sending an interface request to the backend server and waiting for the verification to pass, the target container component is obtained from the backend server, and non-first-screen HTML fragments are extracted from the target container component, which effectively limits the amount of information obtained by the crawler.

[0074] In one possible implementation, the webpage anti-crawler method provided in this embodiment, after rendering the first screen HTML fragment, further includes: calling the activation method of the front-end framework to render the first screen HTML fragment; associating each HTML in the first screen HTML fragment with the client component; and binding each HTML with page events.

[0075] Specifically, after the browser obtains the initial HTML fragment, it can call the React framework's hydrate method to render the initial screen. At this point, the initial HTML is not recreated on the browser side; instead, the browser directly uses the initial HTML fragment returned by the front-end server, associates the initial HTML fragment with the client-side components, and binds events. In this way, the user can see the initial page and interact with it (such as clicking, typing, etc.).

[0076] In this embodiment, by activating the HTML fragment on the first screen and binding it to page events, users can interact with the content of the first screen without affecting their experience.

[0077] Figure 3 A flowchart illustrating a front-end server-side anti-crawler method provided in this application embodiment. Figure 3 As shown, the method in this embodiment includes:

[0078] S301: In response to the browser's first page request, which includes the webpage address and browser viewport size, the first page request is sent to the backend server.

[0079] The execution entity of this application embodiment can be a front-end server or a web anti-crawler system in the front-end server, wherein the web anti-crawler system can be implemented by software.

[0080] S302: Obtain the first screen data from the backend server. The first screen data is the page data suitable for the browser viewport size obtained from the web address. Generate the first screen HTML fragment based on the page data.

[0081] S303: Sends the first screen HTML fragment to the browser.

[0082] Furthermore, generating the first-screen HTML fragment based on page data splicing can include: utilizing the isomorphic mechanism of the front-end and back-end of the front-end framework, calling the front-end framework package, and splicing the page data to obtain the first-screen HTML fragment.

[0083] Understandably, in this embodiment, the main task of the front-end server is to transmit the first page request and the initial HTML data between the browser and the back-end server. After obtaining the initial data from the back-end server's interface response, the front-end server can assemble the initial HTML fragment according to business logic. Since this embodiment can use the React front-end and back-end isomorphic mechanism, the RenderToString method of the React package can be called to render the initial components into an HTML string on the front-end server and return it to the browser.

[0084] In this embodiment, the first page request and the first screen HTML data are transmitted between the browser and the backend server through the frontend server. The frontend server needs less data to render the page than the ordinary isomorphic rendering of the frontend and backend, and the rendered HTML fragments are also shorter. Therefore, the frontend server response time is faster, and the first screen performance is better than that of the ordinary isomorphic rendering of the frontend and backend.

[0085] Figure 4 A flowchart illustrating a backend server-side anti-crawler method provided in an embodiment of this application. Figure 4 As shown, the method in this embodiment includes:

[0086] S401: Responds to the first page request from the front-end server. The first page request includes the web address and the browser viewport size. It retrieves the page data suitable for the browser viewport size from the web address as the first screen data and returns the first screen data to the front-end server.

[0087] The execution entity of this application embodiment can be a backend server or a web anti-crawler system in the backend server, wherein the web anti-crawler system can be implemented by software.

[0088] S402: In response to a browser's second page request, which includes a web page address and a first-screen HTML fragment, retrieve the page HTML fragment other than the first-screen HTML fragment from the web page address as a non-first-screen HTML fragment, and return the non-first-screen HTML fragment to the browser.

[0089] Furthermore, before retrieving the page HTML fragments other than the first-screen HTML fragment from the webpage address as the first-screen HTML fragment, the method further includes: wrapping the non-first-screen HTML fragments into a target container component, setting a boolean value in the target container component, and using the boolean value to indicate the rendering state of the content in the target container component; wherein, on the server side, the boolean value is false, and the content in the target container component is not rendered; on the client side, the boolean value is true, and the content in the target container component is rendered.

[0090] In this embodiment, a boolean value can be set in the target container component to indicate whether child components (e.g., non-first-screen HTML fragments) are rendered. The default value is false, so the child components will not be rendered during server-side rendering. This boolean value is set to true during the ComponentDidMount overlap period because ComponentDidMount only executes on the browser side; that is, the child components are only rendered during client-side rendering. Components wrapped by the target container component (e.g., non-first-screen HTML fragments) will be rendered on the browser side. Furthermore, the target container component can wrap all components that are purely front-end rendered on subsequent screens, eliminating the need to write logic restricting server-side rendering in each purely front-end rendered component. Moreover, wrapping purely front-end components with a container component makes the code easier for coders to identify and maintain.

[0091] In this embodiment, during the isomorphic rendering process between the front-end and back-end, the front-end server only returns partial HTML fragments, not the complete HTML. This effectively limits the amount of information obtained by the crawler and prevents the crawler from scraping data from the front-end server. Compared with existing methods for identifying and blocking crawlers, this invention fundamentally protects the security of website data. Furthermore, since the crawler cannot obtain data, it loses interest and stops crawling the website, thereby indirectly reducing the load on the front-end server.

[0092] Figure 5 This diagram illustrates the interaction between the browser and the front-end and back-end servers in the webpage anti-crawling method provided in this application embodiment. Figure 5 As shown, the method in this embodiment includes:

[0093] In response to a user's command, the browser sends a first page request to the front-end server. The first page request includes the webpage address and the browser viewport size.

[0094] The front-end server responds to the browser's first page request and sends the first page request to the back-end server;

[0095] The backend server retrieves page data suitable for the browser viewport size from the web address as the first screen data and returns the first screen data to the frontend server.

[0096] The front-end server obtains the initial screen data from the back-end server, generates the initial screen HTML fragment, and sends the initial screen HTML fragment back to the browser;

[0097] The browser renders the first HTML fragment to obtain the content of the first screen page;

[0098] The browser sends a second page request to the backend server. The second page request includes the web page address and the first screen HTML fragment.

[0099] The backend server retrieves the page HTML fragments other than the first-screen HTML fragment from the web address as non-first-screen HTML fragments, and returns the non-first-screen HTML fragments to the browser;

[0100] The browser obtains non-first-screen HTML fragments based on its interaction with the backend server, and renders these non-first-screen HTML fragments to obtain the non-first-screen page content.

[0101] The browser combines the content of the first screen of the page with the content of other screens.

[0102] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0103] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0104] Figure 6 This is a schematic diagram of a browser-side anti-crawler device provided in an embodiment of this application. Figure 6 As shown, the webpage anti-crawler device includes:

[0105] The front-end request module 61 is used to respond to user instructions and send a first page request to the front-end server. The first page request includes the web page address and the browser viewport size.

[0106] The first screen rendering module 62 is used to obtain the first screen HTML fragment from the front-end server, render the first screen HTML fragment, and display the first screen page content. The first screen HTML fragment is a page HTML fragment suitable for the browser viewport size obtained from the web address.

[0107] The backend request module 63 is used to send a second page request to the backend server. The second page request includes the web address and the first screen HTML fragment.

[0108] The non-first-screen rendering module 64 obtains non-first-screen HTML fragments based on interaction with the backend server, renders the non-first-screen HTML fragments to obtain non-first-screen page content, and then concatenates the first-screen page content and non-first-screen page content.

[0109] In one possible design, the non-first-screen rendering module 64 is specifically used for:

[0110] Send an API request. The API request is used by the backend server to verify the API and, after successful verification, allows data to be transferred between the backend server and the browser.

[0111] Obtain the target container component, which includes non-first-screen HTML fragments. Non-first-screen HTML fragments are page HTML fragments other than the first-screen HTML fragments obtained from the web page address.

[0112] Extract non-first-screen HTML fragments from the target container component.

[0113] In one possible design, the non-first-screen HTML fragment is set with a Boolean value. The true and false values ​​of the Boolean value are used to indicate whether the non-first-screen HTML fragment is rendered. The default Boolean value is false.

[0114] The target container component also includes lifecycle functions that set boolean values ​​to true during the overlapping lifecycle functions. These lifecycle functions are only triggered on the browser side.

[0115] In one possible design, the non-first-screen rendering module 64 is specifically used for:

[0116] Triggers the overlapping cycles of lifecycle functions; the boolean value is true.

[0117] Render the HTML fragments that are not on the first screen to obtain the content of the page that is not on the first screen.

[0118] In one possible design, the first-screen rendering module 62 is also specifically used for:

[0119] The activation method of the front-end framework is called to render the first HTML fragment;

[0120] Associate each HTML element in the initial HTML fragment with the client-side components, and bind each HTML element to page events.

[0121] Figure 7 This is a schematic diagram of a webpage anti-crawler device on the front-end server side provided in an embodiment of this application. Figure 7 As shown, the webpage anti-crawler device includes:

[0122] The first page request forwarding module 71 is used to respond to the browser's first page request, which includes the web address and browser viewport size, and send the first page request to the backend server.

[0123] HTML splicing module 72 is used to obtain first-screen data from the backend server. The first-screen data is page data suitable for the browser viewport size obtained from the web address. The first-screen HTML fragment is generated by splicing the page data.

[0124] Feedback module 73 is used to send the first screen HTML fragment back to the browser.

[0125] In one possible design, the HTML splicing module 72 is specifically used for:

[0126] By utilizing the isomorphic mechanism of the front-end and back-end frameworks, the front-end framework packages are called to concatenate the page data and obtain the first screen HTML fragment.

[0127] Figure 8 This is a schematic diagram of a webpage anti-crawler device on the backend server side provided in an embodiment of this application. Figure 8 As shown, the webpage anti-crawler device includes:

[0128] The first page request processing module 81 is used to respond to the first page request from the front-end server. The first page request includes the web address and the browser viewport size. It obtains the page data suitable for the browser viewport size from the web address as the first screen data and returns the first screen data to the front-end server.

[0129] The second page request processing module 82 is used to respond to the browser's second page request, which includes a web page address and a first-screen HTML fragment. It obtains the page HTML fragment other than the first-screen HTML fragment from the web page address as a non-first-screen HTML fragment and returns the non-first-screen HTML fragment to the browser.

[0130] In one possible design, the second page request processing module 82 is also specifically used for:

[0131] Wrap the non-first-screen HTML fragment in the target container component and set a boolean value in the target container component. The boolean value is used to indicate the rendering status of the content in the target container component. Specifically, on the server side, the boolean value is false, and the content in the target container component is not rendered. On the client side, the boolean value is true, and the content in the target container component is rendered.

[0132] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0133] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0134] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0135] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0136] Figure 9 This is a schematic diagram of the electronic device used in an embodiment of this application to provide a web anti-crawler device. Figure 9 As shown, the electronic device of this embodiment includes: at least one processor 90 ( Figure 9 (Only one is shown) a processor, a memory 91, and a computer program stored in the memory 91 and executable on at least one processor 90, which, when executing the computer program, implements the steps in any of the above method embodiments.

[0137] The electronic device may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will understand that... Figure 9This is merely an example of an electronic device and does not constitute a limitation on electronic devices. It may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0138] The processor 90 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0139] The specific implementation process of processor 901 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0140] In some embodiments, memory 91 may be an internal storage unit of an electronic device, such as the memory of the electronic device. In other embodiments, memory 91 may be an external storage device of the electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, memory 91 may include both internal and external storage units of the electronic device. Memory 91 is used to store operating systems, applications, bootloaders, data, and other programs, such as program code for computer programs. Memory 91 can also be used to temporarily store data that has been output or will be output.

[0141] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0142] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0143] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an application-specific integrated circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the aforementioned electronic device.

[0144] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0145] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0146] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0147] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for preventing web scraping, characterized in that, This method is applied to browsers and includes: In response to a user command, a first page request is sent to the front-end server. The first page request includes the webpage address and the browser viewport size. The first screen HTML fragment is obtained from the front-end server, and the first screen HTML fragment is rendered to obtain the first screen page content. The first screen HTML fragment is a page HTML fragment suitable for the browser viewport size obtained from the web address. A second page request is sent to the backend server. The second page request includes the webpage address and the first screen HTML fragment. Based on the interaction with the backend server, obtain the non-first screen HTML fragment, render the non-first screen HTML fragment to obtain the non-first screen page content, and then concatenate the first screen page content and the non-first screen page content.

2. The method according to claim 1, characterized in that, The step of obtaining non-first-screen HTML fragments based on interaction with the backend server includes: Send an interface request, which is used by the backend server to verify the interface and, after successful verification, allows data to be transmitted between the backend server and the browser. Obtain a target container component, wherein the target container component includes a non-first-screen HTML fragment, wherein the non-first-screen HTML fragment is a page HTML fragment other than the first-screen HTML fragment obtained from the web page address; Extract the non-first-screen HTML fragment from the target container component.

3. The method according to claim 2, characterized in that, The non-first-screen HTML fragment is set with a boolean value. The true and false values ​​of the boolean value are used to indicate whether the non-first-screen HTML fragment is rendered. The default value of the boolean value is false. The target container component also includes a lifecycle function that sets the boolean value to true during the overlapping period of the lifecycle function. The lifecycle function is only triggered on the browser side.

4. The method according to claim 3, characterized in that, The process of rendering the non-first-screen HTML fragment to obtain non-first-screen page content includes: The overlapping period of the lifecycle function is triggered, and the boolean value is true. The non-first-screen HTML fragment is rendered to obtain the non-first-screen page content.

5. The method according to claim 1, characterized in that, After rendering the first screen HTML fragment, the process further includes: The activation method of the front-end framework is invoked to render the first screen HTML fragment; Associate each HTML element in the first-screen HTML fragment with the client-side components, and bind each HTML element with page events.

6. A webpage anti-crawler method, characterized in that, This method is applied to the front-end server, including: In response to the browser's first page request, which includes the webpage address and the browser viewport size, the first page request is sent to the backend server. First-screen data is obtained from the backend server. The first-screen data is page data suitable for the browser viewport size obtained from the web address. The first-screen HTML fragment is generated by splicing the page data. The HTML fragment on the first screen is then sent to the browser.

7. The method according to claim 6, characterized in that, The process of generating the first-screen HTML fragment based on the page data includes: By utilizing the isomorphic mechanism of the front-end and back-end frameworks, the front-end framework package is called to concatenate the page data to obtain the first screen HTML fragment.

8. A method for preventing web scraping, characterized in that, This method is applied to backend servers and includes: In response to a first page request from the front-end server, the first page request includes a webpage address and a browser viewport size. Page data suitable for the browser viewport size is obtained from the webpage address as first-screen data, and the first-screen data is returned to the front-end server. In response to a second page request from the browser, the second page request including the webpage address and the first screen HTML fragment, the browser obtains a page HTML fragment other than the first screen HTML fragment from the webpage address as a non-first screen HTML fragment, and returns the non-first screen HTML fragment to the browser.

9. The method according to claim 8, characterized in that, Before obtaining the page HTML fragment other than the first-screen HTML fragment from the webpage address as the first-screen HTML fragment, the method further includes: Wrap the non-first-screen HTML fragment in a target container component, and set a boolean value in the target container component to indicate the rendering status of the content in the target container component; wherein, on the server side, the boolean value is false, and the content in the target container component is not rendered; on the client side, the boolean value is true, and the content in the target container component is rendered.

10. A webpage anti-crawler device, the device being applied to a browser, comprising: The front-end request module is used to respond to user instructions and send a first page request to the front-end server. The first page request includes the webpage address and the browser viewport size. The first-screen rendering module is used to obtain the first-screen HTML fragment from the front-end server, render the first-screen HTML fragment, and display the first-screen page content. The first-screen HTML fragment is a page HTML fragment suitable for the browser viewport size obtained from the web address. The backend request module is used to send a second page request to the backend server. The second page request includes the web page address and the first screen HTML fragment. The non-first-screen rendering module obtains non-first-screen HTML fragments based on interaction with the backend server, renders the non-first-screen HTML fragments to obtain non-first-screen page content, and concatenates the first-screen page content and the non-first-screen page content.

11. A webpage anti-crawler device, applied to a front-end server, comprising: The first page request forwarding module is used to respond to the browser's first page request, which includes the web page address and the browser viewport size, and send the first page request to the backend server. The HTML splicing module is used to obtain first-screen data from the backend server. The first-screen data is page data suitable for the browser viewport size obtained from the web address, and the first-screen HTML fragment is generated by splicing the page data. The feedback module is used to send the HTML fragment from the first screen to the browser.

12. A webpage anti-crawler device, applied to a backend server, comprising: The first page request processing module is used to respond to the first page request of the front-end server. The first page request includes a web page address and a browser viewport size. The module obtains page data suitable for the browser viewport size from the web page address as the first screen data and returns the first screen data to the front-end server. The second page request processing module is used to respond to a second page request from a browser. The second page request includes the web page address and the first screen HTML fragment. The module obtains page HTML fragments other than the first screen HTML fragment from the web page address as non-first screen HTML fragments and returns the non-first screen HTML fragments to the browser.

13. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 9.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 9.

15. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-9.

Images