Detection method, device, electronic device and readable storage medium for automatic capture of page information

Abstract

The present disclosure discloses a detection method, device, electronic device and readable storage medium for automatic crawling of page information, which relates to the field of data processing technology, and specifically to the field of artificial intelligence technology such as big data and information flow. The specific implementation scheme is as follows: inserting a detection code into the acquired response page requested by the application, and sending the response page with the inserted detection code to the application, so that the application executes the detection code to register the DOM tree change event to be monitored and the response operation of the DOM tree change event; in response to the DOM tree structure information of the response page provided by the application, the DOM tree structure information of the response page is detected and processed to determine whether there is an abnormal behavior of automatic crawling of page information; wherein the DOM tree structure information of the response page is provided by the application through the response operation of executing the DOM tree change event; in response to determining that there is an abnormal behavior of automatic crawling of page information, abnormal response processing is performed.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and specifically to the field of artificial intelligence technology such as big data and information flow. Background Art

[0002] With the continuous development of the internet, applications (APPs) for terminals are proliferating in an endless stream. While using an application, users may encounter situations where their page information is maliciously scraped by automated crawlers. This not only leads to the theft of the application's core content but can also paralyze the application's service servers.

[0003] Therefore, it is of great significance to effectively protect the application's page information and prevent it from being maliciously captured by automatic crawlers. Summary of the Invention

[0004] The present disclosure provides a detection method, device, electronic device and readable storage medium for automatically capturing page information.

[0005] According to one aspect of the present disclosure, a detection method for automatically crawling page information is provided, comprising:

[0006] Inserting detection code into the obtained response page requested by the application, and sending the response page with the detection code inserted to the application, so that the application executes the detection code to register the DOM tree change event to be monitored and the response operation of the DOM tree change event;

[0007] In response to the DOM tree structure information of the response page provided by the application, detecting and processing the DOM tree structure information of the response page to determine whether there is abnormal behavior in the automatic crawling of page information; wherein the DOM tree structure information of the response page is provided by the application through the response operation of executing the DOM tree change event;

[0008] In response to determining that there is abnormal behavior in automatic crawling of page information, abnormal response processing is performed.

[0009] According to another aspect of the present disclosure, a detection device for automatically crawling page information is provided, comprising:

[0010] a code insertion unit, configured to insert a detection code into the obtained response page requested by the application, and send the response page with the detection code inserted therein to the application, so that the application executes the detection code to register the DOM tree change event to be monitored and the response operation to the DOM tree change event;

[0011] a structure detection unit, configured to detect and process the DOM tree structure information of the response page provided by the application, so as to determine whether there is any abnormal behavior in the automatic crawling of page information; wherein the DOM tree structure information of the response page is provided by the application through a response operation executed in response to the DOM tree change event;

[0012] The exception handling unit is used to perform exception response processing in response to determining that there is an abnormal behavior of automatic crawling of page information.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0017] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the above-mentioned aspect and any possible implementation manner.

[0018] According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method of the aspect and any possible implementation manner as described above.

[0019] It can be seen from the above technical solution that the embodiment of the present disclosure inserts a detection code into the response page requested by the obtained application, and sends the response page with the inserted detection code to the application, so that the application executes the detection code to register the DOM tree change event to be monitored and the response operation of the DOM tree change event. Then, in response to the DOM tree structure information of the response page provided by the application, the DOM tree structure information of the response page is detected and processed to determine whether there is an abnormal behavior of automatic crawling of page information, wherein the DOM tree structure information of the response page is provided by the application by executing the response operation of the DOM tree change event, so that in response to the determination of the abnormal behavior of automatic crawling of page information, abnormal response processing can be performed. Since the detection code capable of monitoring DOM tree change events is inserted in the response page, it is possible to determine whether there is an abnormal behavior of automatic crawling of page information based on the DOM tree changes of the response page, and it is possible to effectively detect some abnormal behaviors of automatic crawling of page information that cannot be detected by conventional detection based on the page content, thereby improving the reliability of automatic crawling detection of page information.

[0020] In addition, the technical solution provided by the present disclosure can effectively improve the security of application page information access.

[0021] In addition, the technical solution provided by the present disclosure can effectively improve the user experience.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] To more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without inventive effort. The drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Among them:

[0024] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0027] Figure 4 It is a block diagram of an electronic device used to implement the detection method of automatically crawling page information according to the embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] It should be noted that the terminal devices involved in the embodiments of the present disclosure may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0031] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0032] With the continuous development of the internet, applications (APPs) for terminals are proliferating in an endless stream. While using an application, users may encounter situations where their page information is maliciously scraped by automated crawlers. This not only leads to the theft of the application's core content but can also paralyze the application's service servers.

[0033] Applications can execute code written in scripting languages, page code can execute in the page space, and plug-in code executes in a special process space, which can control the Document Object Model (DOM) tree structure of the page space. However, the code in the page space is unaware of the plug-in code in the process space.

[0034] Taking crawlers as an example, crawlers can access computer system information through abnormal channels or radical technical means, causing great interference to the normal operation of the system and normal users.

[0035] Currently, one method of implementing a crawler is to use a browser plug-in mechanism and a scripting language to write an automated program to automatically capture page information.

[0036] These browser plugin-based crawlers manipulate the page's DOM tree to fulfill various functions during the crawling process. All page elements on a page can be organized into a tree structure, forming the page's DOM tree. The page's DOM tree consists of node objects and the relationships between them.

[0037] For example, each tag in a Hypertext Markup Language (HTML) page will be loaded as an element node object on the DOM tree of the HTML page; each tag's attribute will be loaded as an attribute node object on the DOM tree of the HTML page; and each tag's content will be loaded as a text node object on the DOM tree of the HTML page.

[0038] For example, consider the Button element node object in a page's DOM tree. This represents a button. During normal user access, the application simply displays the Button element, or the code reads its attributes. However, plugin crawlers modify the Button element's attributes, such as adding a flag to log the crawling process.

[0039] Because it utilizes the special permissions of the plug-in, it can bypass conventional detection of abnormal behavior of automatic crawling of page information based on the page content.

[0040] Therefore, there is an urgent need to provide a detection method that can detect abnormal behaviors of automatic crawling of page information that cannot be detected by conventional detection means.

[0041] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure, as shown in Figure 1 shown.

[0042] 101. Insert detection code into the obtained response page requested by the application, and send the response page with the inserted detection code to the application, so that the application executes the detection code to register the DOM tree change event to be monitored and the response operation of the DOM tree change event.

[0043] The detection code may be a code written in a scripting language, such as JavaScript code.

[0044] 102. In response to the DOM tree structure information of the response page provided by the application, detect and process the DOM tree structure information of the response page to determine whether there is any abnormal behavior in the automatic crawling of page information.

[0045] The DOM tree structure information of the response page is provided by the application through executing a response operation of the DOM tree change event.

[0046] 103. In response to determining that there is abnormal behavior of automatic crawling of page information, perform abnormal response processing.

[0047] At this point, by utilizing the application's standard interface to listen to the DOM tree change events of the response page, and then further combining the DOM tree structure information after the change, it is possible to detect whether there is any abnormal behavior in the automatic crawling of page information, thereby detecting abnormal behavior in the automatic crawling of page information that cannot be detected by conventional detection methods.

[0048] It should be noted that part or all of the execution entities of 101 to 103 may be applications located in the local terminal, or may also be functional units such as plug-ins or software development kits (SDKs) set in the applications located in the local terminal, or may also be processing engines located in the network-side server, or may also be distributed systems located on the network side, for example, a processing engine or distributed system in a detection device for automatic capture of page information on the network side, etc. This embodiment does not specifically limit this.

[0049] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0050] In this way, by inserting a detection code into the response page requested by the obtained application, and sending the response page with the inserted detection code to the application, so that the application can execute the detection code to register the DOM tree change event to be monitored and the response operation of the DOM tree change event, and then, in response to the DOM tree structure information of the response page provided by the application, the DOM tree structure information of the response page is detected and processed to determine whether there is an abnormal behavior of automatic crawling of page information, wherein the DOM tree structure information of the response page is provided by the application by executing the response operation of the DOM tree change event, so that in response to the determination of the existence of abnormal behavior of automatic crawling of page information, abnormal response processing can be performed. Since the detection code capable of monitoring DOM tree change events is inserted in the response page, it is possible to determine whether there is an abnormal behavior of automatic crawling of page information based on the DOM tree changes of the response page, and it is possible to effectively detect some abnormal behaviors of automatic crawling of page information that cannot be detected by conventional detection based on the page content, thereby improving the reliability of automatic crawling detection of page information.

[0051] In the present disclosure, a more preferred implementation method is that 101 to 103 can be executed by an independent device on the network side, for example, a detection device for automatically capturing page information on the network side, etc. At this time, the application side and the business server side do not need to undergo any modification and can execute the business logic normally.

[0052] Optionally, in a possible implementation of this embodiment, in 101, the response page returned by the service server to the application request can be intercepted, and then the detection code can be inserted into the intercepted response page. Then, the response page with the detection code inserted can be sent to the application.

[0053] The inserted detection code may include behavioral features of the DOM tree structure and behavioral analysis logic based on the DOM tree structure, and may be obfuscated and encrypted.

[0054] Optionally, in one possible implementation of this embodiment, after step 101, after receiving the acquisition response page provided by the business server, the application may further execute detection code inserted in the response page while outputting or reading the response page. After executing the detection code, the application may register DOM tree change events to be monitored based on the behavioral characteristics of the DOM tree structure, and register response operations for the monitored DOM tree change events based on the behavioral analysis logic based on the DOM tree structure.

[0055] The DOM tree change event may include, but is not limited to, at least one of the following events:

[0056] Element attribute change events of elements in the DOM tree;

[0057] New events added to elements of the DOM tree; and

[0058] DOM tree element removal event.

[0059] The response operation of the DOM tree change event may be providing the DOM tree structure information of the corresponding response page to the service server when the DOM tree change event occurs.

[0060] In a specific implementation process, the response operation of the DOM tree change event may be to provide the DOM tree structure information of the corresponding response page to the service server separately when the DOM tree change event occurs.

[0061] For example, the message sent by the application to the business server may be simulated, and the DOM tree structure information of the corresponding response page may be provided to the business server.

[0062] In this example, the response operation of the DOM tree change event may specifically be inserting the DOM tree structure information of the corresponding response page into a new message sent by the application to the business server when the DOM tree change event occurs, and providing the message to the business server.

[0063] The so-called new message is a new message used to provide the DOM tree structure information of the corresponding response page to the business server separately as indicated by the response operation.

[0064] Accordingly, in step 102, in response to the new message triggered by the application containing the DOM tree structure information of the response page, the DOM tree structure information of the response page may be inspected and processed to determine whether there is abnormal behavior in the automatic crawling of page information. The DOM tree structure information of the response page is inserted into the new message by the application in response to the DOM tree change event.

[0065] In another specific implementation process, the response operation of the DOM tree change event may be that when the DOM tree change event occurs, the DOM tree structure information of the corresponding response page is provided to the business server in other information sent by the application to the business server.

[0066] For example, in the page request sent by the application to the business server, the DOM tree structure information of the corresponding response page can be provided to the business server.

[0067] In this example, the response operation of the DOM tree change event may specifically be inserting the DOM tree structure information of the corresponding response page into the page request sent by the application to the business server when the DOM tree change event occurs, and providing the information to the business server.

[0068] Accordingly, in step 102, in response to a page request triggered by the application that includes the DOM tree structure information of the response page, the DOM tree structure information of the response page may be inspected and processed to determine whether there is abnormal behavior in the automatic crawling of page information. The DOM tree structure information of the response page is inserted into the page request by the application in response to the DOM tree change event.

[0069] Or, for another example, the DOM tree structure information of the corresponding response page can be provided to the business server in the heartbeat request sent by the application to the business server.

[0070] In this example, the response operation of the DOM tree change event may specifically be inserting the DOM tree structure information of the corresponding response page into the heartbeat request sent by the application to the business server when the DOM tree change event occurs, and providing the information to the business server.

[0071] Accordingly, in step 102, in response to the heartbeat request triggered by the application and containing the DOM tree structure information of the response page, the DOM tree structure information of the response page may be inspected and processed to determine whether there is abnormal behavior in the automatic crawling of page information. The DOM tree structure information of the response page is inserted into the heartbeat request by the application in response to the DOM tree change event.

[0072] Optionally, in a possible implementation of this embodiment, in 102, the DOM tree structure information of the response page sent by the application to the business server may be intercepted, and then the DOM tree structure information of the response page may be detected and processed.

[0073] Optionally, in a possible implementation of this embodiment, the DOM tree structure information of the response page may include, but is not limited to, the following information:

[0074] Structural change information of the DOM tree of the response page; or

[0075] The complete structural information of the DOM tree of the response page.

[0076] Optionally, in a possible implementation of this embodiment, in 102, the changed elements of the DOM tree can be determined based on the DOM tree structure information of the response page, and then, based on the element attributes of the changed elements of the DOM tree, it can be determined whether there is abnormal behavior in the automatic crawling of page information.

[0077] Specifically, after determining the elements of the DOM tree changes, we can further combine the element attributes of the elements of the DOM tree changes with the abnormal crawling feature database to determine whether there is abnormal behavior in the automatic crawling of page information, and then, we can further determine the crawler type and name corresponding to the abnormal behavior.

[0078] For example, based on the DOM tree structure information of the response page and further combined with the abnormal crawling feature database, it can be determined that the element that changes in the DOM tree is the body (Body) element. According to the element attribute Sideexplayingflag (custom character) of the body (Body) element, it is determined that the attribute value of the modified element attribute sideexplayingflag is 1. Then, it can be combined with the abnormal crawling feature database to determine that there is an abnormal behavior of automatic crawling of page information, and the name of the plug-in crawler is "Katalon Recorder".

[0079] Or, for another example, based on the DOM tree structure information of the response page and further combined with the abnormal crawling feature database, it can be determined that the newly added element of the DOM tree is a tag (Div) element, and based on the element attribute Style of the tag (Div) element, it can be determined that the attribute value of the added element attribute style contains text such as "!important; visibility:visible!important; width:100%!important; z-index:2147483646!important;". Then, it can be determined that there is an abnormal behavior of automatic crawling of page information in combination with the abnormal crawling feature database, and the name of the plug-in crawler is "Web Scraper".

[0080] Typically, existing crawler detection methods rely on special objects in scripting languages. However, when a crawler runs in a browser plugin's plugin space, the crawler's code signatures cannot be detected, rendering existing detection methods ineffective. Therefore, by leveraging the structural changes in the DOM tree, we implement detection based on the page's DOM behavior. This allows detection even when the crawler is running in a browser plugin's plugin space.

[0081] In the present disclosure, in 103, in response to determining that there is abnormal behavior of automatic crawling of page information, the abnormal response processing performed can be to directly reject the page request of the application, or to allow the business server to normally process the page request of the application, and normally obtain another response page requested by the application, and perform abnormal behavior alarm processing, or to block the current user of the application, or to block the current application. This embodiment does not specifically limit this.

[0082] In the present disclosure, in response to determining that there is no abnormal behavior of automatically crawling page information, normal response processing can be performed. For example, the service request of the application can be allowed to be processed by the service server, or another response page requested by the application can be obtained normally and operations 101 to 103 can be re-executed. This embodiment does not specifically limit this.

[0083] The following will take a separately deployed detection device and the operation of page request and response through the interaction between the browser and the business server as an example to explain the technical solution of the present disclosure in detail. Figure 2 shown.

[0084] 201. The browser sends a first page request to the business server.

[0085] 202. The detection device intercepts the first page request and does not perform any processing.

[0086] 203. The detection device directly forwards the intercepted first page request to the service server.

[0087] 204. The business server returns a first response page to the browser according to the first page request.

[0088] 205. The detection device intercepts the first response page and inserts a detection code into the first response page.

[0089] 206. The detection device sends a response page with the detection code inserted therein to the browser.

[0090] 207. The browser executes the detection code to register the DOM tree change event to be monitored and the response operation of the DOM tree change event.

[0091] The response operation of the DOM tree change event may specifically be that when the DOM tree change event occurs, the DOM tree structure information of the corresponding first response page is inserted into the page request sent by the browser to the business server and provided to the business server.

[0092] 208. The browser monitors the DOM tree change event, inserts the DOM tree structure information of the first response page into the second page request, and sends it to the service server.

[0093] 209. The detection device intercepts the second page request inserted into the DOM tree structure information of the first response page, and determines whether there is abnormal behavior of automatic crawling of page information based on the DOM tree structure information of the first response page.

[0094] Specifically, the detection device can intercept the second page request inserted into the DOM tree structure information of the first response page, determine the elements of the DOM tree that have changed based on the DOM tree structure information of the first response page, and further combine the abnormal capture feature database based on the element attributes of the elements of the DOM tree that have changed to determine whether there is abnormal behavior in the automatic capture of page information.

[0095] In one case, in response to determining that there is abnormal behavior in automatic crawling of page information, abnormal response processing can be performed.

[0096] For example, the second page request for inserting the DOM tree structure information of the first response page is directly rejected.

[0097] Or, for another example, the second page request inserted into the DOM tree structure information of the first response page is forwarded to the business server, and abnormal behavior alarm processing is performed. The business server processes the second page request normally, normally obtains another response page requested by the application, and re-executes operations 205 to 209.

[0098] Or, for another example, all page requests of the current user of the application are blocked.

[0099] Or, for another example, all page requests of the current application are blocked.

[0100] In another case, in response to determining that there is no abnormal behavior of automatic crawling of page information, normal response processing can be performed.

[0101] For example, the second page request inserted into the DOM tree structure information of the first response page is forwarded to the business server, and the business server processes the second page request normally.

[0102] Or, for another example, another response page requested by the application is obtained normally, and operations 205 to 209 are executed again.

[0103] In this embodiment, a detection code is inserted into the response page requested by the obtained application, and the response page with the inserted detection code is sent to the application for the application to execute the detection code to register the DOM tree change event to be monitored and the response operation of the DOM tree change event. Then, in response to the DOM tree structure information of the response page provided by the application, the DOM tree structure information of the response page is detected and processed to determine whether there is an abnormal behavior of automatic crawling of page information. The DOM tree structure information of the response page is provided by the application by executing the response operation of the DOM tree change event, so that abnormal response processing can be performed in response to the determination of the existence of abnormal behavior of automatic crawling of page information. Since the detection code capable of monitoring DOM tree change events is inserted in the response page, it is possible to determine whether there is an abnormal behavior of automatic crawling of page information based on the DOM tree change of the response page. Some abnormal behaviors of automatic crawling of page information that cannot be detected by conventional detection based on the page content can be effectively detected, thereby improving the reliability of automatic crawling detection of page information.

[0104] In addition, the technical solution provided by the present disclosure can effectively improve the security of application page information access.

[0105] In addition, the technical solution provided by the present disclosure does not require manual operation, is simple to operate, and is not prone to errors, which can further improve the security, efficiency, and reliability of access to application page information.

[0106] In addition, the technical solution provided by the present disclosure can effectively improve the user experience.

[0107] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0108] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] Figure 3 is a schematic diagram according to the third embodiment of the present disclosure, as shown in Figure 3As shown. The detection device 300 for automatic capture of page information of this embodiment may include a code insertion unit 301, a structure detection unit 302 and an exception handling unit 303. The code insertion unit 301 is used to insert a detection code into the acquired response page requested by the application, and send the response page with the detection code inserted to the application, so that the application can execute the detection code to register the DOM tree change event to be monitored and the response operation of the DOM tree change event; the structure detection unit 302 is used to detect and process the DOM tree structure information of the response page in response to the DOM tree structure information of the response page provided by the application, so as to determine whether there is an abnormal behavior of automatic capture of page information; the DOM tree structure information of the response page is provided by the application through the response operation of executing the DOM tree change event; the exception handling unit 303 is used to perform abnormal response processing in response to determining that there is an abnormal behavior of automatic capture of page information.

[0110] It should be noted that part or all of the detection device for automatic capture of page information in this embodiment can be an application located in the local terminal, or it can also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal, or it can also be a processing engine located in the network side server, or it can also be a distributed system located on the network side, for example, a processing engine or distributed system in the detection platform for automatic capture of page information on the network side, etc. This embodiment does not specifically limit this.

[0111] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0112] Optionally, in a possible implementation of this embodiment, the structure detection unit 302 can be specifically used to respond to a page request triggered by the application and containing the DOM tree structure information of the response page, and detect and process the DOM tree structure information of the response page to determine whether there is abnormal behavior in the automatic crawling of page information; wherein, the DOM tree structure information of the response page is inserted into the page request by the application by executing a response operation of the DOM tree change event.

[0113] Optionally, in a possible implementation of this embodiment, the code insertion unit 301 may be further configured to, in response to determining that there is no abnormal behavior of automatic crawling of page information, normally obtain another response page requested by the application.

[0114] Optionally, in a possible implementation of this embodiment, the DOM tree structure information of the response page may include, but is not limited to, the following information:

[0115] Structural change information of the DOM tree of the response page; or

[0116] The complete structural information of the DOM tree of the response page.

[0117] Optionally, in a possible implementation of this embodiment, the structure detection unit 302 can be specifically used to determine the changed elements of the DOM tree based on the DOM tree structure information of the response page; and determine whether there is abnormal behavior in the automatic crawling of page information based on the element attributes of the changed elements of the DOM tree.

[0118] It should be noted that Figure 1 The corresponding embodiment method, and Figure 2 The method performed by the test device in the corresponding embodiment can be implemented by the detection device for automatically capturing page information provided in this embodiment. Figure 1 and Figure 2 The relevant contents in the corresponding embodiments will not be repeated here.

[0119] In this embodiment, a detection code is inserted into the response page requested by the obtained application, and the response page with the inserted detection code is sent to the application for the application to execute the detection code to register the DOM tree change event to be monitored and the response operation of the DOM tree change event. Then, in response to the DOM tree structure information of the response page provided by the application, the DOM tree structure information of the response page is detected and processed to determine whether there is an abnormal behavior of automatic crawling of page information. The DOM tree structure information of the response page is provided by the application by executing the response operation of the DOM tree change event, so that abnormal response processing can be performed in response to the determination of the existence of abnormal behavior of automatic crawling of page information. Since the detection code capable of monitoring DOM tree change events is inserted in the response page, it is possible to determine whether there is an abnormal behavior of automatic crawling of page information based on the DOM tree change of the response page. Some abnormal behaviors of automatic crawling of page information that cannot be detected by conventional detection based on the page content can be effectively detected, thereby improving the reliability of automatic crawling detection of page information.

[0120] In addition, the technical solution provided by the present disclosure can effectively improve the security of application page information access.

[0121] In addition, the technical solution provided by the present disclosure does not require manual operation, is simple to operate, and is not prone to errors, which can further improve the security, efficiency, and reliability of access to application page information.

[0122] In addition, the technical solution provided by the present disclosure can effectively improve the user experience.

[0123] In the technical solution disclosed herein, the acquisition, storage and application of the response pages involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0124] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0125] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0126] like Figure 4 As shown, electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. Various programs and data required for the operation of electronic device 400 can also be stored in RAM 403. Computing unit 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0127] Multiple components in the electronic device 400 are connected to the I / O interface 405, including an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a magnetic disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0128] The computing unit 401 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above, such as the detection method for automatic capture of page information. For example, in some embodiments, the detection method for automatic capture of page information can be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the detection method for automatic capture of page information described above can be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the detection method for automatically capturing page information in any other appropriate manner (eg, by means of firmware).

[0129] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0130] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0131] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0133] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0134] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0135] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0136] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A detection method for automatically crawling page information, characterized in that: include: Inserting detection code into the obtained response page requested by the application, the detection code including: behavioral characteristics of a Document Object Model (DOM) tree structure and behavior analysis logic based on the DOM tree structure; and sending the response page with the detection code inserted to the application, so that the application executes the detection code to register DOM tree change events to be monitored based on the behavioral characteristics of the DOM tree structure and to register response operations to the monitored DOM tree change events based on the behavior analysis logic based on the DOM tree structure; In response to the DOM tree structure information of the response page provided by the application, detecting and processing the DOM tree structure information of the response page to determine whether there is abnormal behavior in the automatic crawling of page information; wherein the DOM tree structure information of the response page is provided by the application through the response operation of executing the DOM tree change event; In response to determining that there is abnormal behavior in automatic crawling of page information, abnormal response processing is performed.

2. The method according to claim 1, characterized in that The step of detecting and processing the DOM tree structure information of the response page provided by the application in response to the DOM tree structure information of the response page to determine whether there is abnormal behavior in the automatic crawling of page information includes: In response to a page request triggered by the application and containing DOM tree structure information of the response page, the DOM tree structure information of the response page is detected and processed to determine whether there is abnormal behavior in the automatic crawling of page information; wherein the DOM tree structure information of the response page is inserted into the page request by the application by executing a response operation of the DOM tree change event.

3. The method according to claim 1, characterized in that The method further comprises: In response to determining that there is no abnormal behavior of automatically crawling page information, another response page requested by the application is obtained normally.

4. The method according to claim 1, wherein The DOM tree structure information of the response page includes: Structural change information of the DOM tree of the response page; or The complete structural information of the DOM tree of the response page.

5. The method according to any one of claims 1 to 4, characterized in that The detecting and processing of the DOM tree structure information of the response page includes: Determining the changed elements of the DOM tree according to the DOM tree structure information of the response page; According to the element attributes of the elements whose DOM tree has changed, it is determined whether there is abnormal behavior in the automatic crawling of page information.

6. A detection device for automatically capturing page information, characterized in that: include: a code insertion unit, configured to insert detection code into the obtained response page requested by the application, the detection code including: behavioral characteristics of a Document Object Model (DOM) tree structure and behavior analysis logic based on the DOM tree structure, and send the response page with the detection code inserted to the application so that the application executes the detection code, thereby registering DOM tree change events to be monitored based on the behavioral characteristics of the DOM tree structure and registering response operations to the monitored DOM tree change events based on the behavior analysis logic based on the DOM tree structure; a structure detection unit, configured to detect and process the DOM tree structure information of the response page provided by the application, so as to determine whether there is any abnormal behavior in the automatic crawling of page information; wherein the DOM tree structure information of the response page is provided by the application through a response operation executed in response to the DOM tree change event; The exception handling unit is used to perform exception response processing in response to determining that there is an abnormal behavior of automatic crawling of page information.

7. The device according to claim 6, characterized in that The structure detection unit is specifically used to In response to a page request triggered by the application and containing DOM tree structure information of the response page, the DOM tree structure information of the response page is detected and processed to determine whether there is abnormal behavior in the automatic crawling of page information; wherein the DOM tree structure information of the response page is inserted into the page request by the application by executing a response operation of the DOM tree change event.

8. The device according to claim 6, characterized in that The code insertion unit is also used to In response to determining that there is no abnormal behavior of automatically crawling page information, another response page requested by the application is obtained normally.

9. The device according to claim 6, characterized in that The DOM tree structure information of the response page includes: Structural change information of the DOM tree of the response page; or The complete structural information of the DOM tree of the response page.

10. The device according to any one of claims 6 to 9, characterized in that The structure detection unit is specifically used to Determining the changed elements of the DOM tree according to the DOM tree structure information of the response page; and According to the element attributes of the elements whose DOM tree has changed, it is determined whether there is abnormal behavior in the automatic crawling of page information.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

Images