Data crawling method and device and server

Abstract

The invention provides a data crawling method and device and a server, and relates to the technical field of website data crawling, and the method comprises the following steps: determining an encryption state of a target website by using a current crawler algorithm and website information of the target website; when the encryption state is encrypted, obtaining a target JS file of the target website; performing algorithm restoration processing on the target JS file, determining encryption parameters of the target website, adding the encryption parameters to a current crawler algorithm, and determining a target crawler algorithm; and performing data crawling processing on the target website by utilizing a target crawler algorithm, and determining target crawling data. According to the method, the encryption parameter can be reversely determined through the JS, so that the confirmation difficulty of determining the target crawler algorithm and the target crawling data is reduced, and the data crawling efficiency is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of website data crawling, and in particular to a data crawling method, device and server. Background Art

[0002] When crawling data from a website, the crawler needs to carry login information or carry encryption parameters when accessing the details page. At present, relevant technologies have proposed that Selenium can be used to disguise itself as a browser to make requests, obtain the website's communication authentication by simulating human click actions, and crack the website's encryption parameters, thereby achieving automatic crawling of target data. However, Selenium's data crawling efficiency is low, and the requirements for version configuration are high. The corresponding driver needs to be updated, and Selenium's protection mechanism does not allow the storage of cross-domain cookies. When logging in, you need to open the web page first and then load the cookie. This refresh method is very unfriendly and will seriously affect the user experience. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a data crawling method, device and server, which can reversely determine encryption parameters through JS, thereby reducing the difficulty of determining the target crawler algorithm and the target crawling data, thereby significantly improving the data crawling efficiency.

[0004] In the first aspect, an embodiment of the present invention provides a data crawling method, the method comprising: using the current crawler algorithm and the website information of the target website to determine the encryption status of the target website, wherein the website information includes the original data of the website; when the encryption status is encrypted, obtaining the target JS file of the target website; performing algorithm restoration processing on the target JS file to determine the encryption parameters of the target website, and adding the encryption parameters to the current crawler algorithm to determine the target crawler algorithm; using the target crawler algorithm to perform data crawling processing on the target website to determine the target crawled data.

[0005] In one embodiment, the current crawler algorithm includes a user agent parameter, and the step of determining the encryption status of the target website using the current crawler algorithm and the website information of the target website includes: using the user agent parameter to obtain the communication authentication of the target website so that the current crawler algorithm performs data crawling processing on the target website to determine the first crawled data; using the first crawled data and the original data of the website to perform encryption analysis processing to determine the encryption status of the target website.

[0006] In one embodiment, the step of using crawled data and website original data to perform encryption analysis and processing to determine the encryption status of a target website includes: using first crawled data and website original data to determine target parameters in a preset parameter set; adding the target parameters to the current crawler algorithm, determining an updated crawler algorithm, and using the updated crawler algorithm to crawl data on the target website to determine second crawled data; comparing the second crawled data with the website original data; if the comparison result is inconsistent, determining that the target website is encrypted; if the comparison result is inconsistent, determining that the target website is not encrypted.

[0007] In one embodiment, the method includes: when the target website is in an unencrypted state, determining an updated crawler algorithm as a target crawler algorithm.

[0008] In one embodiment, the step of obtaining a target JS file of a target website includes: obtaining text data of the target website; determining a target value of the text data based on the text data through a preset analysis test model; and performing numerical processing on the target value to determine the target JS file.

[0009] In one embodiment, after the step of acquiring the text data of the target website, it includes: performing data diagnosis processing on the text data to determine the data diagnosis result; if the data diagnosis result is normal, sending the text data to a preset analysis test model to determine the target value of the text data; if the data diagnosis result is expired, reacquiring the text data.

[0010] In one embodiment, the step of numerically processing the target value and determining the target JS file includes: establishing an empty new stack; placing the target value of the text data at the target position of the new stack; and performing stack processing on the target value to determine the target JS file.

[0011] In the second aspect, an embodiment of the present invention also provides a data crawling device, which includes: a status confirmation module, which uses the current crawler algorithm and the website information of the target website to determine the encryption status of the target website, wherein the website information includes the original data of the website; a file acquisition module, which obtains the JS file of the target website when the encryption status is encrypted; a password restoration module, which performs algorithm restoration processing on the JS file, determines the encryption parameters of the target website, and adds the encryption parameters to the current crawler algorithm to determine the target crawler algorithm; a data crawling module, which uses the target crawler algorithm to perform data crawling processing on the target website to determine the target crawled data.

[0012] In a third aspect, an embodiment of the present invention further provides a server, comprising a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement any one of the methods provided in the first aspect.

[0013] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement any one of the methods provided in the first aspect.

[0014] The embodiments of the present invention bring the following beneficial effects:

[0015] The embodiment of the present invention provides a data crawling method, device and server. The method uses the current crawler algorithm and the website information of the target website to determine the encryption state of the target website, wherein the website information includes the original data of the website. When the encryption state is encrypted, the target JS file of the target website is obtained, the target JS file is subjected to algorithm restoration processing, the encryption parameters of the target website are determined, and the encryption parameters are added to the current crawler algorithm to determine the target crawler algorithm, thereby using the target crawler algorithm to perform data crawling processing on the target website and determine the target crawled data. The embodiment of the present invention can determine the encryption parameters through JS reverse engineering, thereby reducing the difficulty of determining the target crawler algorithm and the target crawled data, thereby significantly improving the data crawling efficiency.

[0016] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0017] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of a data crawling method provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the structure of a target crawler algorithm provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the structure of a data crawling device provided by an embodiment of the present invention;

[0022] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] At present, when crawlers crawl website data, they may encounter situations where they need to carry login information to access website data, or they need to carry some encryption parameters when accessing the details page. By disguising as a browser and making requests through selenium, human click actions can be simulated to obtain the website's communication authentication and crack the website's encryption parameters, thereby realizing automatic crawling of target data. However, the data crawling speed of this scheme is slow and it occupies more memory. Based on this, the data crawling method provided by the present invention can save the step of selenium automation window by learning JS reverse engineering, and the encryption parameters can be determined by JS reverse engineering, which can reduce the difficulty of determining the target crawler algorithm and confirming the target crawled data, thereby significantly improving the data crawling efficiency.

[0025] See also Figure 1 The flowchart of a data crawling method shown in FIG. 1 mainly includes the following steps S102 to S108:

[0026] Step S102, using the current crawler algorithm and the website information of the target website, determine the encryption status of the target website, wherein the website information includes the original data of the website, and the current crawler algorithm includes: user agent parameters (User-Agent), the user agent parameters are used to disguise as a browser to make requests and simulate human click actions. In one embodiment, the crawler algorithm needs to carry encryption parameters (Token, Sign value) when crawling data. If the encryption parameters can be obtained on the list page of the website, it can be determined that the encryption status of the target website is unencrypted. If the encryption parameters cannot be directly obtained on the list page of the website, it is determined that the encryption status of the target website is encrypted.

[0027] Step S104, when the encryption status is encrypted, obtain the target JS file of the target website. In one embodiment, the encrypted target website will encrypt the encryption parameters in the JS file, which is equivalent to a script. Therefore, it is necessary to determine the target JS file and restore the encryption parameters of the website.

[0028] Step S106, performing algorithm restoration processing on the target JS file, determining the encryption parameters of the target website, and adding the encryption parameters to the current crawler algorithm to determine the target crawler algorithm. In one embodiment, see Figure 2 The schematic diagram of the structure of a target crawler algorithm is shown. When the target website is encrypted, the target crawler algorithm includes: encryption parameters, user agent parameters and text data (Cookie) of the target website.

[0029] Step S108, use the target crawler algorithm to crawl data on the target website and determine the target crawled data. In one embodiment, the crawler algorithm is a program or script that automatically crawls World Wide Web information according to certain rules. After determining the target crawler algorithm, the target crawler algorithm can be used to automatically crawl the data of the target website, thereby obtaining the target crawled data.

[0030] The above-mentioned data crawling method provided by the embodiment of the present invention can determine the encryption parameters through JS reverse engineering, thereby reducing the difficulty of determining the target crawler algorithm and the target crawling data, thereby significantly improving the data crawling efficiency.

[0031] The embodiment of the present invention also provides an implementation method for crawling data from a website based on JS reverse engineering, for details, see (1) to (3) below:

[0032] (1) Using user agent parameters to obtain communication authentication of the target website, so that the current crawler algorithm performs data crawling processing on the target website, determines first crawled data, and uses the first crawled data and the original data of the website to perform encryption analysis processing to determine the encryption status of the target website. In one embodiment, using the first crawled data and the original data of the website, in a preset parameter set, a target parameter is determined, the target parameter is added to the current crawler algorithm, an updated crawler algorithm is determined, and the updated crawler algorithm is used to perform data crawling processing on the target website, and second crawled data is determined. The second crawled data is compared with the original data of the website: if the comparison results are inconsistent, it is determined that the target website is encrypted; if the comparison results are inconsistent, it is determined that the target website is not encrypted, wherein the preset parameters The set includes: Accept, Content-Type, Content-Language, Host, Referer and other parameters. In one embodiment, when obtaining the first crawled data, the crawler algorithm only carries the user agent parameter. If the data returned by the request (i.e., the first crawled data) and the data opened by the browser (the original data of the website) do not match, it means that other parameters need to be added to the request header (headers) of the crawler algorithm. According to the type of the target website, parameters are selected from the preset parameter set and added to the request header of the crawler algorithm, so as to perform a second test and obtain the second crawled data. If the second crawled data still does not match the original data of the website, it means that the encryption parameters of the website are not stored in the list page and the website has been encrypted.

[0033] (2) See Figure 2 A structural diagram of a target crawler algorithm is shown. When the target website is in an unencrypted state, the updated crawler algorithm is determined as the target crawler algorithm. At this time, the target crawler algorithm includes: encryption parameters (Token, Sign), user agent parameters (User-Agent) and one or more target parameters (Accept, Content-Type, Content-Language, Host, Referer).

[0034] (3) When the target website is in an encrypted state, the text data of the target website is obtained, and the target value of the text data is determined based on the text data through a preset analysis and testing model, and the target value is numerically processed to determine the target JS file. In one embodiment, after the text data of the target website is obtained, the text data needs to be subjected to data diagnosis processing to determine the data diagnosis result: if the data diagnosis result is normal, the text data is sent to the preset analysis and testing model to determine the target value of the text data; if the data diagnosis result is expired, the text data is re-acquired. In one embodiment, after the target value is determined through data diagnosis, an empty new stack is established, and the target value of the text data is placed at the target position of the new stack, and the target value is subjected to stack processing to determine the target JS file. By cracking the JS stack, the encryption algorithm can be restored and embedded into the program, so that the encryption parameters can be restored using the target JS file, so that the crawler can use the encryption parameters to perform data crawling processing on the target website and determine the target crawled data. The encrypted position can be quickly located through the stack processing through the context, so that the required target JS file and target value can be accurately found in the massive data, thereby effectively improving the data crawling rate.

[0035] To sum up, the present invention can crack the JS root stack, restore the encryption algorithm, embed it into the program, and reversely crack the encryption parameters of JS to increase the rate at which the program acquires target data, thereby reducing the difficulty of determining the target crawler algorithm and the target crawled data, thereby significantly improving the data crawling efficiency.

[0036] For the data crawling method provided in the above embodiment, the embodiment of the present invention provides a data crawling device, see Figure 3 The structure diagram of a data crawling device shown in FIG. 1 includes the following parts:

[0037] The status confirmation module 302 determines the encryption status of the target website using the current crawler algorithm and the website information of the target website, wherein the website information includes the original data of the website;

[0038] The file acquisition module 304 acquires the JS file of the target website when the encryption state is encrypted;

[0039] The password restoration module 306 performs algorithm restoration processing on the JS file, determines the encryption parameters of the target website, and adds the encryption parameters to the current crawler algorithm to determine the target crawler algorithm;

[0040] The data crawling module 308 uses a target crawler algorithm to perform data crawling processing on the target website to determine the target crawling data.

[0041] The above-mentioned data processing device provided in the embodiment of the present application can determine the encryption parameters through JS reverse engineering, thereby reducing the difficulty of determining the target crawler algorithm and the target crawling data, thereby significantly improving the data crawling efficiency.

[0042] In one embodiment, the current crawler algorithm includes a user agent parameter. When performing the step of determining the encryption status of the target website using the current crawler algorithm and the website information of the target website, the above-mentioned status confirmation module 302 is also used to: use the user agent parameter to obtain the communication authentication of the target website so that the current crawler algorithm performs data crawling processing on the target website to determine the first crawled data; use the first crawled data and the original data of the website to perform encryption analysis processing to determine the encryption status of the target website.

[0043] In one embodiment, when performing encryption analysis processing using crawled data and website original data to determine the encryption status of the target website, the above-mentioned status confirmation module 302 is also used to: use the first crawled data and the website original data to determine the target parameters in a preset parameter set; add the target parameters to the current crawler algorithm, determine the updated crawler algorithm, and use the updated crawler algorithm to crawl data on the target website to determine the second crawled data; compare the second crawled data with the website original data; if the comparison result is inconsistent, it is determined that the target website is encrypted; if the comparison result is inconsistent, it is determined that the target website is not encrypted.

[0044] In one implementation, the state confirmation module 302 is further used to: when the target website is in an unencrypted state, determine the updated crawler algorithm as the target crawler algorithm.

[0045] In one embodiment, when performing the step of obtaining the target JS file of the target website, the above-mentioned file acquisition module 304 is also used to: obtain the text data of the target website; determine the target value of the text data based on the text data through a preset analysis test model; perform numerical processing on the target value to determine the target JS file.

[0046] In one embodiment, after the step of acquiring the text data of the target website, the above-mentioned file acquisition module 304 is also used to: perform data diagnosis processing on the text data to determine the data diagnosis result; if the data diagnosis result is normal, send the text data to a preset analysis test model to determine the target value of the text data; if the data diagnosis result is expired, re-acquire the text data.

[0047] In one embodiment, when performing numerical processing on the target value and determining the target JS file, the above-mentioned file acquisition module 304 is also used to: establish an empty new stack; place the target value of the text data at the target position of the new stack; perform stack processing on the target value to determine the target JS file.

[0048] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0049] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned implementation methods.

[0050] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43, wherein the processor 40, the communication interface 43 and the memory 41 are connected via the bus 42; the processor 40 is used to execute an executable module stored in the memory 41, such as a computer program.

[0051] The memory 41 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0052] The bus 42 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0053] Among them, the memory 41 is used to store programs, and the processor 40 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0054] The processor 40 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 40. The above processor 40 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 41, and the processor 40 reads the information in the memory 41 and completes the steps of the above method in combination with its hardware.

[0055] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0056] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0057] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A data crawling method, characterized in that: The method comprises: Determine the encryption status of the target website by using the current crawler algorithm and website information of the target website, wherein the website information includes original website data; When the encryption state is encrypted, obtaining the target JS file of the target website; Performing algorithm restoration processing on the target JS file, determining encryption parameters of the target website, and adding the encryption parameters to the current crawler algorithm to determine the target crawler algorithm; The target crawler algorithm is used to perform data crawling processing on the target website to determine target crawling data.

2. The data crawling method according to claim 1, characterized in that: The current crawler algorithm includes a user agent parameter, and the step of determining the encryption status of the target website using the current crawler algorithm and the website information of the target website includes: Using the user agent parameter to obtain the communication authentication of the target website, so that the current crawler algorithm performs data crawling processing on the target website to determine the first crawled data; The first crawled data and the original website data are used to perform encryption analysis and processing to determine the encryption status of the target website.

3. The data crawling method according to claim 2, characterized in that: The step of performing encryption analysis processing using the crawled data and the original website data to determine the encryption status of the target website includes: Determine a target parameter in a preset parameter set by using the first crawled data and the original website data; Adding the target parameter to the current crawler algorithm, determining an updated crawler algorithm, and using the updated crawler algorithm to crawl data on the target website to determine second crawled data; Comparing the second crawled data with the original website data; If the comparison result is inconsistent, it is determined that the target website has been encrypted; If the comparison results are inconsistent, it is determined that the target website is not encrypted.

4. The data crawling method according to claim 3, characterized in that: The method comprises: When the target website is in an unencrypted state, the updated crawler algorithm is determined as the target crawler algorithm.

5. The data crawling method according to claim 1, characterized in that: The step of obtaining the target JS file of the target website includes: Acquire text data of the target website; Determining a target value of the text data based on the text data by using a preset analysis test model; The target value is numerically processed to determine the target JS file.

6. The data crawling method according to claim 5, characterized in that: After the step of acquiring the text data of the target website, the method further comprises: Performing data diagnosis processing on the text data to determine a data diagnosis result; If the data diagnosis result is normal, the text data is sent to a preset analysis test model to determine a target value of the text data; If the data diagnosis result is expired, the text data is acquired again.

7. The data crawling method according to claim 5, characterized in that: The step of performing numerical processing on the target value to determine the target JS file includes: Create an empty new stack; placing the target value of the text data at a target position of the new stack; The target value is subjected to stack processing to determine the target JS file.

8. A data crawling device, characterized in that: The device comprises: A status confirmation module, which uses the current crawler algorithm and website information of the target website to determine the encryption status of the target website, wherein the website information includes original website data; A file acquisition module, when the encryption state is encrypted, acquires the JS file of the target website; A password restoration module performs algorithm restoration processing on the JS file, determines the encryption parameters of the target website, and adds the encryption parameters to the current crawler algorithm to determine the target crawler algorithm; The data crawling module utilizes the target crawler algorithm to perform data crawling processing on the target website to determine the target crawling data.

9. A server, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the method according to any one of claims 1 to 7.

Images