A data acquisition method, device, apparatus and storage medium

Abstract

The application provides a data acquisition method and device, equipment and a storage medium, and relates to the technical field of data processing. The method comprises the following steps: constructing a decryption function according to an acquired browser instance, the decryption function being generated by calling a decryption process of a website where the browser instance is located; acquiring a key according to the decryption function and a target request website; and performing a data request on a server while carrying the key, so as to acquire target data. It can be seen that, by calling the built-in decryption process of the browser, the decryption function is generated, and then the key is acquired, so that the cracking difficulty of the key is reduced, and the data acquisition efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a data acquisition method and device, equipment and a storage medium. BACKGROUND

[0002] When data is parsed, data can be acquired by parsing a browser page during dynamic page crawling, or a predefined script is executed using a native function to extract page data. Specifically, a crawler usually initiates a request using data or a key stored on a user's local terminal generated by a browser, or cracks encrypted logic through reverse engineering to acquire data according to a decryption process. However, a predefined script cannot acquire an effective key in real time, resulting in a failure to request data, and reverse engineering decryption requires a large amount of time, which leads to low data acquisition efficiency. SUMMARY

[0003] The present application provides a data acquisition method, device, equipment and storage medium, which can solve the problem of low data acquisition efficiency.

[0004] To solve the above technical problems, the present application is implemented as follows:

[0005] In a first aspect, the present application provides a data acquisition method, comprising:

[0006] A decryption function is constructed according to an acquired browser instance, wherein the decryption function is generated by calling a decryption process of a website where the browser instance is located;

[0007] A key is acquired according to the decryption function and a target request website;

[0008] Data is requested from a server by carrying the key, and target data is acquired.

[0009] In a second aspect, the present application provides a data acquisition device, comprising:

[0010] A construction module is configured to construct a decryption function according to an acquired browser instance, wherein the decryption function is generated by calling a decryption process of a website where the browser instance is located;

[0011] A first acquisition module is configured to acquire a key according to the decryption function and a target request website;

[0012] A second acquisition module is configured to request data from a server by carrying the key, and acquire target data.

[0013] In a third aspect, an electronic device is provided, which comprises a processor and a memory. The memory stores programs or instructions executable by the processor. When the programs or instructions are executed by the processor, the steps of the data acquisition method according to the first aspect are implemented.

[0014] In a fourth aspect, a readable storage medium is provided, which stores programs or instructions. When the programs or instructions are executed by a processor, the steps of the data acquisition method according to the first aspect are implemented.

[0015] In the embodiments of the present application, according to the acquired browser instance, a decryption function is constructed, which is generated by calling a decryption process of a website where the browser instance is located. According to the decryption function and a target request website, a key is acquired. The key is carried to a server for data request, and target data is acquired. It can be seen that, by calling the built-in decryption process of the browser, the decryption function is generated, and then the key is acquired, which can reduce the cracking difficulty of the key, and further improve the data acquisition efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of the data acquisition method provided by the embodiments of the present application is shown;

[0017] Figure 2 A running diagram of the data acquisition device provided by the embodiments of the present application is shown;

[0018] Figure 3 A structure diagram of the data acquisition device provided by the embodiments of the present application is shown;

[0019] Figure 4 A structure diagram of the electronic device provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0020] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application. Unless otherwise defined, the technical terms or scientific terms used in the present application should be understood as the common meanings of the technical terms or scientific terms by those skilled in the art. The terms "first", "second" and similar terms used in the present application do not represent any order, number or importance, but are only used to distinguish different components. Similarly, the terms "one" or "a" and similar terms do not represent a number limitation, but represent the existence of at least one. The terms "connected" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right" and the like are only used to represent relative positional relationships, when the absolute positions of the described objects are changed, the relative positional relationships are also changed accordingly.

[0021] The data acquisition method, device, equipment and storage medium provided by the embodiments of the application are further described below with reference to the drawings.

[0022] Please refer to Figure 1 , Figure 1 The flowchart of a data acquisition method provided by the embodiments of the present application is shown in the figure. As shown in the figure, the method comprises the following steps.

[0023] In step 110, a decryption function is constructed according to the obtained browser instance.

[0024] In step 120, a key is acquired according to the decryption function and the target request website.

[0025] In step 130, data is requested from the server carrying the key to acquire target data.

[0026] Through the above steps 110, 120 and 130 of the embodiments of the present application, a decryption function is constructed according to the obtained browser instance, and the decryption function is generated by calling the decryption process of the website where the browser instance is located. Then, after acquiring the key according to the decryption function and the target request website, the key is carried to request data from the server to acquire target data. As can be seen, by calling the decryption process built-in the browser to generate the decryption function, and then acquiring the key, the difficulty of cracking the key can be reduced, and the data acquisition efficiency can be improved.

[0027] In step 110, the decryption function is generated by calling the decryption process of the website where the browser instance is located, and the decryption process can be implemented based on the existing encryption function of the website where the browser instance is located.

[0028] The starting of the browser instance can be implemented by applying a test framework (for example: Playwright), wherein the browser instance can support browsers such as Chromium, WebKit or Firefox. It should be understood that the starting of the browser instance is implemented by using Playwright, that is, it is equivalent to being applied to a real browser, and simulating manual clicking, scrolling, waiting for loading and the like to obtain dynamic content.

[0029] In this way, by using Playwright to obtain dynamic content, compared with an ordinary crawler, only static HTML (such as a web page source code) can be obtained, and the web page data is dynamically loaded by JS (needs to click a button to display), and the crawling efficiency is high.

[0030] In step 120, after initializing the browser, a target request website is input into the browser to obtain a key corresponding to the target request website.

[0031] It should be noted that the key can be stored in a memory cache for use in the next decryption function call.

[0032] In step 130, after obtaining the key, a data request is made to the browser carrying the key to obtain target data.

[0033] For example, in a certain e-commerce platform, by simulating a user accessing a product detail page through a browser, after obtaining the browser instance, a decryption function of the product detail page is constructed, wherein the function call is generated by the decryption process of the e-commerce platform. Then, the decryption function and the target request uniform resource locator (Uniform Resource Locator, URL) (that is, the target request website) are used to obtain the key of the target request URL, and then the data in the target request URL is obtained by carrying the key.

[0034] In step 120, when the target request website includes M pages, in the embodiment of the present application, the method of obtaining the key according to the decryption function and the target request website in step 120 can further include:

[0035] Step 120-01, according to the crawling request, N target pages in the M pages are crawled;

[0036] Step 120-02, according to the decryption function and the N target pages, N keys are obtained;

[0037] Wherein, M and N are integers greater than or equal to 1, and M is greater than or equal to N.

[0038] According to the above steps 120-01 to 120-02, under the premise of the crawling request, N target pages in M pages are crawled, and N keys corresponding to the target pages are obtained according to the decryption function, which can improve the efficiency of cracking and obtaining the keys.

[0039] For example, in a certain news website, the target request URL contains 10 pages (M=10), and in the case of a crawling request setting of 5, 5 target pages (N=5) are crawled, and 5 keys are obtained based on the decryption function and the 5 target pages, which are used to request different news content respectively.

[0040] Optionally, the N target pages include a first target page.

[0041] In the embodiment of step 120-02, further, the obtaining of the N keys according to the decryption function and the N target pages includes:

[0042] According to the decryption function and the initial decryption parameter of the first target page obtained by the browser, the key of the first target page is obtained, and the initial decryption parameter is stored in the server.

[0043] It should be understood that the browser is connected with the web server, and the browser can obtain the decryption parameter stored in the server, and then according to the decryption function and the decryption parameter, the key of the first target page can be obtained.

[0044] In this way, after obtaining the key of the first target page, in the embodiment of step 130, further includes:

[0045] Step 130-01, data request is carried out to the server with the key of the first target page, and first data is obtained, and the target data includes the first data.

[0046] After obtaining the key of the first target page, the key of the first target page is carried to request the web server to obtain the data of the first target page, which can improve the data acquisition efficiency.

[0047] For example, in a social media platform, the N target pages include a "home page" (first target page). The system obtains the "home page" key by the decryption function and the initial decryption parameter of the "home page" obtained by the browser (stored in the server), so that the data of the "home page" can be requested and obtained by carrying the key.

[0048] For another example, in an online education platform, the N target pages include a "course list page" (first target page). The system obtains the key by the decryption function and the initial decryption parameter of the "course list page" obtained by the browser, and requests and obtains the data of the "course list page" by carrying the key.

[0049] Optionally, the N target pages further comprise a second target page;

[0050] In an embodiment of step 120-02, further, the obtaining the N keys according to the decryption function and the N target pages further comprises:

[0051] extracting data parameters in the first data;

[0052] obtaining the key of the second target page according to the decryption function and the extracted data parameters;

[0053] Wherein, the second target page is after the first target page in the order of the target pages.

[0054] Wherein, the data parameters in the first data can include the key in the first target page, can include the initial decryption parameter in the first target page, or can include both, which is not limited by the embodiments of the present application.

[0055] It should be noted that the extracted data parameters should be stored in the memory cache for use in the next decryption function call.

[0056] In this way, after saving the first data and extracting the data parameters in the first data, the decryption function is repeatedly called to obtain the key of the second target page, and then the key of the second target page is carried to request data from the web server to obtain the target data of the second target page.

[0057] For example, in a certain e-commerce website, the N target pages include a "product list page" (first target page) and a "product detail page" (second target page). The "product list page" data is obtained first and the data parameters are extracted, then the "product detail page" key is obtained based on the decryption function and the extracted data parameters, and finally the product detail data is requested.

[0058] For another example, in a certain news platform, the N target pages include a "news classification page" (first target page) and a "specific news page" (second target page). The system first obtains the "news classification page" data and extracts the data parameters, then obtains the "specific news page" key based on the decryption function and the extracted data parameters, and finally requests the specific news data.

[0059] In the embodiments of the present application, after the decryption function is repeatedly called to obtain the key after the target request website is input, and the step of saving the target data corresponding to the target page is performed, the target data of all target pages, i.e. the crawling data, is completed.

[0060] It should be understood that the termination condition of the repeated steps until all pages (i.e., target pages) are captured is based on the page capture quantity reaching a preset threshold or the traversal depth exceeding a specified level.

[0061] Optionally, the initial decryption parameter is obtained by parsing initial hypertext markup language response content or JavaScript code of the target request website.

[0062] In an optional embodiment, in a social platform, the initial decryption parameter in the JavaScript code is extracted by simulating manual operation and parsing the initial HTML response content of the target request URL, and the initial decryption function is used for subsequent key generation.

[0063] In order to better understand the flow of the data acquisition method provided by the embodiments of the present application, the following refers to Figure 2 The data acquisition method provided by the embodiments of the present application is described.

[0064] It should be understood that the data acquisition flow can be implemented by a crawler device. As shown in Figure 2 , Figure 2 is a flowchart of the execution of the data acquisition device.

[0065] After the user initiates a data capture request, the device receives the capture request, starts a browser to obtain a browser instance, and after initializing the browser environment, the device constructs a decryption function according to the browser instance, wherein the decryption function is generated by calling the decryption process of the website. After the decryption function is constructed, the request URL is input into the browser, and the decryption function is implanted, at which time the browser requests the initial decryption parameter from the web server, encrypts the initial decryption parameter after obtaining the initial decryption parameter, and returns it to the device, and then the decryption function is called to obtain the decryption Token (i.e., the key). In this way, after obtaining the key, the device can directly request data from the web server with the key, and after receiving the target data returned by the web server (the user obtains the target data), the data is saved.

[0066] Please refer to Figure 3 , Figure 3 is a structural schematic diagram of a data acquisition device provided by the embodiments of the present application, as shown in the figure, the data acquisition device 300 comprises:

[0067] The construction module 310 is configured to construct a decryption function according to the obtained browser instance, wherein the decryption function is generated by calling the decryption process of the website where the browser instance is located;

[0068] The first acquisition module 320 is configured to acquire a key according to the decryption function and the target request website.

[0069] The second obtaining module 330 is configured to carry the key to a server to make a data request and obtain target data.

[0070] The target request website includes M pages.

[0071] The first obtaining module 320 includes:

[0072] The grabbing submodule is configured to grab N target pages in the M pages according to a grabbing request.

[0073] The obtaining submodule is configured to obtain N keys according to the decryption function and the N target pages.

[0074] Wherein, M and N are integers greater than or equal to 1, and M is greater than or equal to N.

[0075] Optionally, the N target pages include a first target page.

[0076] The obtaining submodule includes:

[0077] The first obtaining unit is configured to obtain a key of the first target page according to the decryption function and an initial decryption parameter of the first target page obtained by the browser, wherein the initial decryption parameter is stored in the server.

[0078] The second obtaining module 330 includes:

[0079] The second obtaining unit is configured to carry the key of the first target page to the server to make a data request and obtain first data, wherein the target data includes the first data.

[0080] Optionally, the N target pages further include a second target page.

[0081] The obtaining submodule further includes:

[0082] The extracting unit is configured to extract a data parameter in the first data.

[0083] The third obtaining unit is configured to obtain a key of the second target page according to the decryption function and the extracted data parameter.

[0084] Wherein, the grabbing order of the second target page is after the first target page.

[0085] Optionally, the initial decryption parameter is obtained by analyzing initial hypertext markup language response content or JavaScript code of the target request website.

[0086] The chart generation apparatus in the embodiments of the present application can be an electronic device, for example, an electronic device with an operating system, or a component in an electronic device, for example, an integrated circuit or a chip.

[0087] The data acquisition apparatus provided by the embodiments of the present application can achieve Figure 1 The method embodiments shown achieve various processes and achieve the same technical effects, and to avoid repetition, details are not repeated here.

[0088] Optionally, as Figure 4 The embodiments of the present application also provide an electronic device 400, including a processor 401 and a memory 402, the memory 402 stores programs or instructions that can run on the processor 401, when the programs or instructions are executed by the processor 401, the various steps of the above-mentioned data acquisition method embodiments are achieved, and the same technical effects can be achieved, to avoid repetition, details are not repeated here.

[0089] The embodiments of the present application provide a readable storage medium, the readable storage medium stores programs or instructions, when the programs or instructions are executed by the processor, the various processes of the above-mentioned data acquisition method embodiments are achieved, and the same technical effects can be achieved, to avoid repetition, details are not repeated here.

[0090] The processor is the processor in the terminal in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, etc. In some examples, the readable storage medium can be a non-transient readable storage medium.

[0091] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, the above-mentioned specific embodiments are only illustrative, not restrictive, and those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims, all of which belong to the protection of the present application.

Claims

1. A data acquisition method, characterized by, The method comprises the following steps: constructing a decryption function according to the obtained browser instance, the decryption function being generated by calling a decryption process of a website where the browser instance is located; obtaining a key according to the decryption function and a target request website; carrying out a data request to a server by carrying the key to obtain target data; the target request website comprises M pages; the step of obtaining the key according to the decryption function and the target request website comprises: grabbing N target pages in the M pages according to a grabbing request; obtaining N keys according to the decryption function and the N target pages; wherein M and N are both integers greater than or equal to 1, and M is greater than or equal to N; the N target pages comprise a first target page and a second target page; the step of obtaining the N keys according to the decryption function and the N target pages comprises: extracting a data parameter in first data; obtaining a key of the second target page according to the decryption function and the extracted data parameter; wherein the data parameter in the first data comprises a key in the first target page and an initial decryption parameter in the first target page, and the grabbing order of the second target page is after the first target page.

2. The method according to claim 1, wherein: the step of obtaining the N keys according to the decryption function and the N target pages comprises: obtaining a key of the first target page according to the decryption function and an initial decryption parameter of the first target page obtained through the browser, the initial decryption parameter being stored in the server; the step of carrying out the data request to the server by carrying the key to obtain the target data comprises: carrying out a data request to the server by carrying the key of the first target page to obtain first data, the target data comprising the first data.

3. The method of claim 2, wherein, the initial decryption parameter is obtained by analyzing an initial hypertext markup language response content or JavaScript code of the target request website.

4. A data acquisition device, characterized by The method comprises the following steps: a construction module is configured to construct a decryption function according to an obtained browser instance, the decryption function being generated by calling a decryption process of a website where the browser instance is located; a first obtaining module is configured to obtain a key according to the decryption function and a target request website; a second obtaining module is configured to carry out a data request to a server by carrying the key to obtain target data; the target request website comprises M pages; the first obtaining module comprises: a grabbing submodule is configured to grab N target pages in the M pages according to a grabbing request; an obtaining submodule is configured to obtain N keys according to the decryption function and the N target pages; wherein M and N are both integers greater than or equal to 1, and M is greater than or equal to N; the N target pages comprise a first target page and a second target page; the obtaining submodule comprises: an extracting unit is configured to extract a data parameter in first data; a third obtaining unit is configured to obtain a key of the second target page according to the decryption function and the extracted data parameter; The data parameter in the first data comprises a key in the first target page and an initial decryption parameter in the first target page, and the crawling sequence of the second target page is after the first target page.

5. The apparatus of claim 4, wherein, The obtaining sub-module comprises: a first obtaining unit configured to obtain the key of the first target page according to the decryption function and the initial decryption parameter of the first target page obtained by the browser, the initial decryption parameter being stored in the server; The second obtaining module comprises: a second obtaining unit configured to make a data request to the server by carrying the key of the first target page, and obtain first data, the target data comprising the first data.

6. An electronic device, comprising: A processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 3.

7. A readable storage medium characterized by, The readable storage medium stores programs or instructions, and the programs or instructions are executed by the processor to implement the steps of the method of any one of claims 1 to 3.

Images