Data acquisition method and system for adaptive webpage structure change based on reinforcement learning

Abstract

The present application relates to the technical field of data crawling, more particularly, to a data collection method and system for adaptive webpage structure change based on reinforcement learning. Two Docker containers are constructed in advance, one of which is installed with a browser program, a crawler program and a daemon process, and the other is used to store and update an Actor-Critic model; the Actor-Critic model is trained based on the reinforcement learning mode, so that the Actor-Critic model can adapt to the structure change of the target webpage, automatically learn the new crawling process, and reduce the labor cost of modifying the crawler program code. The present application solves the problem that the existing data collection cannot adapt to the webpage structure change.

Description

Technical Field

[0001] This invention relates to the field of data crawling technology, and more specifically, to: a data acquisition method based on reinforcement learning that adapts to changes in webpage structure, and a data acquisition system using this method. Background Technology

[0002] Currently, web crawlers are one of the main methods for acquiring information and data on the Internet. By automatically accessing web pages and extracting information from them, web crawlers can help users quickly and efficiently collect large amounts of data. For example, businesses can use web crawlers to monitor competitors' prices, product information, and marketing activities in order to adjust their strategies in a timely manner; academic researchers can use web crawlers to obtain large-scale text data for research such as language analysis and social network studies.

[0003] If a developer's web crawler only crawls data for a short period, and the webpage structure changes very little, then the crawler can work well. However, in some scenarios, it's necessary to monitor the content of certain webpages over a long period, such as e-commerce platforms needing to monitor product price fluctuations over months or even years. During such a long period, websites often undergo iterative updates, such as changes to the position or style of clickable buttons. This can cause pre-written crawlers to malfunction. Therefore, targeted iterative updates to the crawler are required, which consumes a significant amount of manpower.

[0004] Therefore, the inventors considered designing a data acquisition method that could adapt to changes in webpage structure. Summary of the Invention

[0005] Therefore, it is necessary to provide a data acquisition method and system based on reinforcement learning that can adapt to changes in webpage structure, addressing the problem that existing data acquisition methods cannot adapt to changes in webpage structure.

[0006] This invention is achieved using the following technical solution:

[0007] In a first aspect, the present invention discloses a data acquisition method based on reinforcement learning for adaptive webpage structure changes, used to complete a set crawling task on a target webpage.

[0008] The data acquisition method for adaptive webpage structure changes based on reinforcement learning includes the following steps:

[0009] Step 1: Build Docker container G and Docker container R.

[0010] In this system, Docker container G is used to install a browser program, a web crawler, and a daemon process; the browser program is used to load the target webpage; the web crawler program is used to control the browser program to execute actions sent by Docker container R; and the daemon process is used to obtain the status of the target webpage and calculate the immediate reward after executing the action.

[0011] The Docker container R is used to store and update the Actor-Critic model AC; AC includes: Policy Network A and Value Network C; the model in Docker container R calculates actions and updates model parameters based on the status and real-time rewards sent by the daemon process.

[0012] Step 2: Perform N rounds of formal training on the model in the Docker container R, and update the parameters of A and C through reinforcement learning until the crawler program correctly completes the set crawling task.

[0013] The maximum number of steps in the nth round of formal training is T;

[0014] The training methods for step t in the nth round of formal training include:

[0015] Step 3.1, According to state S t Calculate the action A to be performed in step t. t ;

[0016] in, include This represents the Actor-Critic model obtained in the (t-1)th step of the nth round of formal training. express The strategic network department in the middle, express The value network part in the middle; n∈[1,N], t∈[1,T-1];

[0017] Step 3.2, the crawler program first controls the browser program to execute A t Then, control the browser program to take a screenshot of the target webpage and obtain the status S. t+1 ;

[0018] Step 3.3, the daemon process obtains S t+1 And calculate A t Instant Rewards R t ;

[0019] Step 3.4, According to R t renew The parameters form According to S t+1Calculate the action A to be performed in step t+1. t+1 .

[0020] This data acquisition method based on reinforcement learning and adaptive webpage structure changes implements the method or process according to embodiments of this disclosure.

[0021] Secondly, this invention discloses a data acquisition system for adaptive webpage structure changes based on reinforcement learning, which uses the data acquisition system for adaptive webpage structure changes based on reinforcement learning disclosed in the first aspect.

[0022] The data acquisition system for adaptive webpage structure changes based on reinforcement learning includes: Docker container G, Docker container R, and a model training module.

[0023] Docker container G is used to install the browser program, web crawler, and daemon. Docker container R is used to store and update the Actor-Critic model AC. The model training module is used to perform N rounds of formal training on the model in Docker container R, updating the parameters of A and C through reinforcement learning until the web crawler correctly completes the set crawling task.

[0024] This data acquisition system based on reinforcement learning and adaptive webpage structure changes implements the methods or processes according to embodiments of this disclosure.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] 1. This invention pre-builds two Docker containers, one of which installs a browser program, a crawler program, and a daemon process, while the other is used to store and update the Actor-Critic model. This invention trains the Actor-Critic model based on reinforcement learning, enabling the Actor-Critic model to adapt to changes in the structure of the target webpage, thereby guiding the crawler program to automatically learn new crawling processes and reducing the human cost of modifying the crawler program code.

[0027] 2. During the training process, this invention converts the webpage state into a vector that matches the Actor-Critic model to realize the input, which solves the problem that the action space of existing methods is too large and difficult to design.

[0028] 3. In the training process, this invention employs cueing engineering, which converts task descriptions into text vectors and inputs them into the model, thereby letting the model know the task objective and accelerating the model training speed.

[0029] 4. In addition, the present invention also uses supervised learning for pre-training before formal training, so that the Actor-Critic model can learn some correct crawling processes in advance, thereby initializing some network parameters to accelerate the convergence speed of the model during subsequent reinforcement learning. Attached Figure Description

[0030] Figure 1 This is a flowchart of the first data acquisition method for adaptive webpage structure changes based on reinforcement learning in Embodiment 1 of the present invention;

[0031] Figure 2 for Figure 1 Data flow diagram of Docker container G and Docker container R during model training;

[0032] Figure 3 This is a flowchart of the second data acquisition method for adaptive webpage structure changes based on reinforcement learning in Embodiment 2 of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] It should be noted that when a component is said to be "installed on" another component, it can be directly on the other component or it may be in a component that is centered on it. When a component is said to be "set on" another component, it can be directly set on the other component or it may also be in a component that is centered on it. When a component is said to be "fixed to" another component, it can be directly fixed to the other component or it may also be in a component that is centered on it.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0036] Example 1

[0037] Please see Figure 1 , Figure 1 This is a flowchart of the first data acquisition method for adaptive webpage structure changes based on reinforcement learning provided in Embodiment 1. Figure 1As shown, the first data acquisition method based on reinforcement learning for adaptive webpage structure changes includes the following steps:

[0038] Step 1: Build Docker container G and Docker container R.

[0039] Docker is a mainstream tool for creating containers, allowing developers to package various applications and their dependencies into Docker image files. These images can then be installed and run on any physical device (Linux or Windows, etc.) to achieve virtualization, completely decoupling applications from the underlying device. This enables flexible migration and deployment between physical machines, freeing operations engineers from tedious environment deployment tasks, greatly improving work efficiency, and reducing potential risks during deployment. This invention utilizes Docker containers, leveraging these advantages.

[0040] See Figure 2 Docker container G is used to install browser programs, web crawlers, and daemons. Docker container R is used to store and update the Actor-Critic model AC.

[0041] The browser program is a GUI program used to load the target webpage. The web crawler program is written using the web automation tool Selenium and is used to control the browser program to execute actions sent by the Docker container R; generally, actions can be "click," "grab," "take screenshot," etc. The daemon process is a program developed based on OpenCV, used to obtain the state of the target webpage and calculate the immediate reward after executing the action.

[0042] AC consists of: Policy Network Department A and Value Network Department C. The model within the Docker container R calculates actions and updates model parameters based on the status and real-time rewards sent by the daemon process.

[0043] Step 2: Perform N rounds of formal training on the model (i.e., AC) inside the Docker container R, and update the parameters of A and C through reinforcement learning until the crawler program correctly completes the set crawling task.

[0044] The maximum number of steps in each round of formal training is T. Since the training method for each step is similar, we will take the t-th step in the nth round of formal training as an example, which includes 4 main steps:

[0045] Step 3.1, According to state S t Calculate the action A to be performed in step t. t .

[0046] in, include This represents the Actor-Critic model obtained in the (t-1)th step of the nth round of formal training. express The strategic network department in the middle, express The value network part in; t∈[1,T-1].

[0047] Step 3.2, the crawler program first controls the browser program to execute A t Then, control the browser program to take a screenshot of the target webpage and obtain the status S. t+1 .

[0048] Step 3.3, the daemon process obtains S t+1 And calculate A t Instant Rewards R t .

[0049] Step 3.4, According to R t renew The parameters form According to S t+1 Calculate the action A to be performed in step t+1. t+1 .

[0050] The following is a detailed explanation of the four main steps mentioned above:

[0051] 1. Step 3.1 includes 4 subordinate steps:

[0052] Step 3.1.1, S t Convert to match A vector S of dimension t , and input Calculate the action probability π(a|S) t ;θ t );

[0053] Wherein, π(a|S t ;θ t )express In S t The probability of outputting action a; θ t express The parameters updated at step t-1 in the nth round of formal training.

[0054] Specifically, due to S t The size is too large and may not be a good match. The dimension, therefore, needs to be S t Processed into The identifiable dimensions are then used to create a mesh and convert it into a vector S. t For example, St The resolution is 1024*768, while If the recognizable size is 1000*700, then the S needs to be removed. t The top and bottom borders are each 24 pixels wide, and the left and right borders are each 68 pixels wide, resulting in a 1000*700 pixel screenshot. This 1000*70 pixel screenshot is then divided into 100*70 grid cells, each with dimensions of 10*10 pixels. These 100*70 grid cells form a grid set, which serves as the... The action space; That is, there are 100*70 position options, and it is determined in which grid cell the operation is performed.

[0055] It's important to note that existing web crawlers use packet sniffing, obtaining the HTML DOM tree. A single DOM tree often contains tens of thousands of DOM nodes, and the structure of the DOM tree varies significantly from webpage to webpage. Directly inputting the DOM tree into a reinforcement learning model presents challenges in designing the action space, and the excessively large action space poses a significant challenge to model training. This invention, however, obtains and processes webpage screenshots to generate multiple finite grid cells, thereby reducing the action space and resolving the issues of inconsistent webpage structures and excessively large action space design.

[0056] Step 3.1.2, Based on the action probability π(a|S) t ;θ t Perform random sampling to obtain the action A to be performed in step t. t And transmit it to the crawler program;

[0057] Step 3.1.3, In S t Execute action a and output the score of action a as q(a,S) t ;w t );

[0058] Among them, w t express The parameters updated at step t-1 in the nth round of formal training;

[0059] Step 3.1.4, calculate the expected value of the action score V(S) t ;θ t ,w t ):

[0060]

[0061] 2. Step 3.2 includes 3 subordinate steps:

[0062] Step 3.2.1: The crawler program receives A from the Docker container R. t and control the browser program to execute A t ;

[0063] Step 3.2.2, the target webpage is executed A t After reaching a new state, the crawler takes a screenshot of the browser window, obtains a screenshot of the target webpage, and uses it as a snapshot. t+1 ;

[0064] Step 3.2.3: Generate the log corresponding to the crawler program. t+1 ; where log t+1 Includes: the current step number is step t, S t+1 .

[0065] 3. Step 3.3 includes two subordinate steps:

[0066] Step 3.3.1, the daemon process reads the log. t+1 and load S t+1 ;

[0067] Step 3.3.2, if S t+1 Once the set crawling task is completed, then R t Positive reward r t Otherwise, R t Set to 0;

[0068] Where S t+1 If the crawling task is not completed within T steps, then R t Negative reward -r t .

[0069] It should be noted that the examination of S t+1 Whether the set crawling task has been completed depends on the type of task. For example, if the set crawling task is to collect data, the data collection results can be verified: taking step 3.3 as an example, after the crawler program controls the browser program to execute A... t Then, check whether the collected data has been obtained; if the collected data has been obtained, store the collected data in the database and compare it with the preset verification data: if they are the same, set TRUE in the corresponding log; otherwise, set FALSE.

[0070] 4. Step 3.4 includes 4 subordinate steps:

[0071] Step 3.4.1, fix θ t Update w t For w t+1 and will Updated to

[0072] Among them, w t+1 The update methods include:

[0073] Step 3.4.1.1, calculate A t Target reward y t :

[0074] y t =R t +γ·q(A t+1 ,S t+1 ;w t );

[0075] In the formula, Rt represents the instant reward of At; γ represents the reward discount coefficient; q(A t+1 ,S t+1 ;w t )express In S t+1 Execute A t+1 and output A t+1 The score;

[0076] Step 3.4.1.2, calculate q(A) t ,S t ;w t ) and y t The loss function L(w):

[0077]

[0078] Where q(A) t ,S t ;w t ) represents A t The expected reward; w represents the value network parameter variable;

[0079] Step 3.4.1.3, based on L(w), use the gradient descent algorithm to... t Updated to w t+1 ;

[0080] in, α represents the learning rate.

[0081] Step 3.4.2, fix w t+1 Update θ t For θ t+1 and will Updated to

[0082] Where, θ t+1 The update methods include:

[0083] Step 3.4.2.1: Calculate the approximate derivative of the expected value of the action score.

[0084]

[0085] in, express After entering S t Post-output action The probability of; Indicates based on action probability Actions obtained through random sampling; Represents the action value function; θ represents the policy network parameter variable; Indicates a fixed w t+1 The expected score for the subsequent action.

[0086] The derivation process of step 3.4.2.1 is as follows:

[0087] First, derive the equivalent derivative formula for the expected value of the action score:

[0088]

[0089] In the formula, A t S represents t The action space below. Q π (S t A t ) represents the action value function, which can be represented by q(A) t ,S t ;w t () can be used as an approximation. It indicates a desire for the expected value.

[0090] From A t Random sampling action And calculate the gradient at this point, and obtain:

[0091]

[0092] It is obvious that and From A t It was obtained through random sampling. Therefore, yes The unbiased estimates can be considered to be approximately equal.

[0093] Step 3.4.2.2, based on According to the gradient ascent algorithm, θ t Updated to θ t+1 ;

[0094] in, β represents the learning rate.

[0095] Step 3.4.3, Will Updated to

[0096] Step 3.4.4, S t+1 Process into a match A vector S of dimension t+1 , and input Calculate A t+1 .

[0097] Among them, S t+1 The processing procedure and S t They are the same, so I won't go into details here.

[0098] Based on the steps described above, the model within the Docker container R can guide the crawler to automatically learn new crawling processes, thus eliminating the need for manual modification of the crawler code.

[0099] In addition, to accelerate model convergence during formal training, a task description can be introduced:

[0100] In step 3.2.3, log t+1 Also includes: the task description for step t (Str) t Str t The text contains task prompts: for example, clicking a button on a webpage.

[0101] In step 3.4.4, Str t Convert to text vector Str t and with S t+1 Enter together

[0102] so, Based on dual-vector input Str t S t+1 , It can be based on Str t Knowing the task objective can effectively improve training speed.

[0103] Similarly, in step 3.1.1, Str will also be... t-1 Convert to text vector Str t-1 and with S t Enter together Among them, Str t-1 The task description for step t-1 is included in the log. t middle.

[0104] This embodiment 1 also discloses a first data acquisition system based on reinforcement learning for adaptive webpage structure changes, which uses the data acquisition method based on reinforcement learning for adaptive webpage structure changes disclosed in embodiment 1.

[0105] The first data acquisition system based on reinforcement learning for adaptive webpage structure changes includes: a Docker container G, a Docker container R, and a model training module. Docker container G is used to install the browser program, the web crawler, and the daemon process. Docker container R is used to store and update the Actor-Critic model AC. The model training module is used to perform N rounds of formal training on the model (AC) within Docker container R, updating the parameters of A and C through reinforcement learning until the web crawler correctly completes the set crawling task.

[0106] Example 2

[0107] See Figure 3 This embodiment 2 discloses a second data acquisition method for adaptive webpage structure changes based on reinforcement learning. The difference from the method in embodiment 1 is that in step two, AC is first pre-trained for M rounds, and the parameters of A are obtained through supervised learning. Forming a pre-trained Actor-Critic model Again Perform N rounds of formal training, updating the parameters of A and C through reinforcement learning, until the crawler program correctly completes the set crawling task.

[0108] This second embodiment adds a pre-training process: This is because the network parameters in the first embodiment were initially randomly initialized, which could lead to the model needing multiple training rounds to find the correct crawling process. Adding pre-training is equivalent to setting relatively reliable parameters for the model, which can significantly accelerate the convergence speed during formal training.

[0109] Specifically, the pre-training method in this embodiment 2 includes:

[0110] Step 2.1: Perform sample operations on the sample webpage until the set crawling task is completed, and record any action x of the sample operation. l and execution x l The state y formed later l ;l∈[1,L], where L represents the total number of steps in the example operation;

[0111] Step 2.2, construct the action sample dataset X = {x1, x2, ..., x} L}, State sample dataset Y = {y1, y2, ..., y L};

[0112] Step 2.3: Perform M rounds of pre-training on A based on X and Y to obtain...

[0113] The training methods for the m-th round of pre-training include:

[0114] Step 2.3.1, y l enter In the middle, calculate the probability of the action.

[0115] in, This represents the policy network part obtained in the (m-1)th round of pre-training; express The parameters are: m∈[1,M]; l∈[1,L];

[0116] Step 2.3.2, calculate the loss function. And through the gradient descent algorithm Updated to

[0117] in, k represents the learning rate.

[0118] This embodiment 2 also discloses a second data acquisition system based on reinforcement learning for adaptive webpage structure changes, which uses the data acquisition method based on reinforcement learning for adaptive webpage structure changes disclosed in embodiment 2.

[0119] The second type of data acquisition system based on reinforcement learning for adaptive webpage structure changes includes: a Docker container G, a Docker container R, a pre-training module, and a model training module. Docker container G is used to install the browser program, web crawler, and daemon process. Docker container R is used to store and update the Actor-Critic model AC. The pre-training module is used to perform M rounds of pre-training on AC, obtaining the parameters of A through supervised learning. Forming a pre-trained Actor-Critic model The model training module is used for training... Perform N rounds of formal training, updating the parameters of A and C through reinforcement learning, until the crawler program correctly completes the set crawling task.

[0120] Example 3

[0121] This embodiment 3 discloses a readable storage medium storing computer program instructions. When the computer program instructions are read and executed by a processor, the data acquisition method based on reinforcement learning for adaptive webpage structure changes according to embodiment 1 or 2 is performed.

[0122] When applying the method of Example 1, it can be applied in the form of software, such as by designing it as a program that can run independently on a computer-readable storage medium. The computer-readable storage medium can be a USB flash drive, designed as a USB security token, and the program can be designed to start the entire method through an external trigger.

[0123] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A data collection method for completing a set crawling task on a target webpage based on reinforcement learning and adaptive webpage structure changes, characterized in that, It includes the following steps: Step 1: Build Docker container G and Docker container R; In this system, Docker container G is used to install a browser program, a web crawler, and a daemon process; the browser program is used to load the target webpage; the web crawler program is used to control the browser program to execute actions sent by Docker container R; and the daemon process is used to obtain the status of the target webpage and calculate the immediate reward after executing the action. Docker container R is used to store and update Actor-Critic models. A - C ; A - C Including: Strategic Network Department A Value Network Department C The model inside the Docker container R calculates actions and updates model parameters based on the status and real-time rewards sent by the daemon process. Step 2: Process the model within the Docker container R. N In the next round of formal training, updates are made through reinforcement learning. A , C The parameters are set until the crawler program correctly completes the set crawling task; wherein the first n maximum number of steps for the formal training of the wheel is T ; n ∈ [1, N]; The first n wheel training method includes: t The training method of the first step includes: Step 3.1, According to the state S t Calculate the first t The action to be performed A t ; in, include , ; Indicates the first n In the first round of formal training t -1 step training to obtain the Actor-Critic model; express The strategic network department in the middle, express Value Network Department in China; t ∈[1, T -1]; Step 3.1 includes: Step 3.1.1, will S t Convert to match A vector S of dimension t , and input Calculate the action probability π( a | S t ; θ t ); Where, π( a | S t ; θ t )express exist S t Down Output Action a The probability of; θ t express In the n In the first round of formal training t -1 step updated parameters; Step 3.1.2, Based on the action probability π ( a | S t ; θ t Random sampling is performed to obtain the first... t The action to be performed A t And transmit it to the crawler program; Step 3.1.3, exist S t Next action a and output the action. a The score q ( a , S t ;w t ); in, w t express In the n In the first round of formal training t -1 step updated parameters; Step 3.1.4, calculate the expected value of the action score. V ( S t ; θ t , w t ): ： Step 3.2: The crawler program first controls the browser program to execute. A t Then, control the browser program to take a screenshot of the target webpage and obtain its status. S t+1 ; Step 3.2 includes: Step 3.2.1: The crawler program receives data from the Docker container R. A t and control the execution of browser programs. A t ; Step 3.2.2, the target webpage is executed A t After reaching a new state, the crawler takes a screenshot of the browser window to obtain a screenshot of the target webpage, and uses it as... S t+1 ; Step 3.2.3: Generate logs corresponding to the crawler program. log t+1 ;in, log t+1 Including: The current step number is the th t step, S t+1 ; Step 3.3, Daemon Process Acquisition S t+1 and calculate A t Instant rewards R t ; Step 3.3 includes: Step 3.3.1, Daemon process reads log t+1 and load S t+1 ; Step 3.3.2, if S t+1 Once the set crawling task is completed, then R t Positive Reward r t ;otherwise, R t Set to 0; Among them, if S t+1 Exceed T If the crawling task is not completed by step [step], then R t Negative Rewards - r t ; Step 3.4, in accordance with R t renew , The parameters form ; in accordance with S t+1 Calculate the first t The action to be performed in step +1 A t+1 .

2. The data acquisition method for adaptive webpage structure changes based on reinforcement learning according to claim 1, characterized in that, In step two, for A - C proceed directly N In the next round of formal training, updates are made through reinforcement learning. A , C The parameters are set until the crawler program correctly completes the set crawling task; Or, first... A - C conduct M Pre-training, obtained through supervised learning A parameters This forms the pre-trained Actor-Critic model. ; then conduct N In the next round of formal training, updates are made through reinforcement learning. A , C The parameters are set until the crawler program correctly completes the set crawling task.

3. The data acquisition method for adaptive webpage structure changes based on reinforcement learning according to claim 1, characterized in that, Step 3.4 includes: Step 3.4.1, Fix θ t ,renew w t for w t+1 and will Updated to ; Step 3.4.2, Fix w t+1 ,renew θ t for θ t+1 and will Updated to ; Step 3.4.3, , Will Updated to ; Step 3.4.4, will S t+1 Process into a match A vector S of dimension t+1 , and input Calculate A t+1 .

4. The data acquisition method for adaptive webpage structure changes based on reinforcement learning according to claim 3, characterized in that, In step 3.4.1, w t+1 The update methods include: Step 3.4.1.1, Calculate A t Target reward y t : ； In the formula, R t express A t Instant rewards; Indicates the reward discount factor; express exist S t+1 Execute A t+1 and output A t+1 The score; Step 3.4.1.2, Calculation q ( A t , S t ;w t )and y t loss function L ( w ): in, q ( A t , S t ;w t )express A t Expected rewards; w Indicates the parameter variables of the value network department; Step 3.4.1.3, based on L ( w According to the gradient descent algorithm, w t Updated to w t+1 ; in, ; α This represents the learning rate.

5. The data acquisition method for adaptive webpage structure changes based on reinforcement learning according to claim 4, characterized in that, In step 3.4.2, θ t+1 The update methods include: Step 3.4.2.1: Calculate the approximate derivative of the expected value of the action score. : ； in, express In the input S t Post-output action The probability of; Indicates based on action probability Actions obtained through random sampling; Represents the action value function; ; θ Indicates the policy network parameter variables; Indicates fixed w t+1 Expected score for the subsequent action; Step 3.4.2.2, based on According to the gradient ascent algorithm θ t Updated to θ t+1 ; in, ; β This represents the learning rate.

6. The data acquisition method for adaptive webpage structure changes based on reinforcement learning according to claim 5, characterized in that, In step 3.1.1, the following will be done: Str t-1 Convert to text vector Str t-1 and with S t Enter together ;in, Str t-1 For the first t The task description for step -1 is included in the log. log t middle; In step 3.2.3, log t+1 Also includes: the t Step-by-step task description Str t ; In step 3.4.4, the following will be performed: Str t Convert to text vector Str t and with S t+1 Enter together .

7. A data acquisition system for adaptive webpage structure changes based on reinforcement learning, characterized in that, It uses the data acquisition method based on reinforcement learning for adaptive webpage structure changes as described in any one of claims 1-6; The data acquisition system for adaptive webpage structure changes based on reinforcement learning includes: Docker container G is used to install browser programs, web crawlers, and daemons; Docker container R is used to store and update the Actor-Critic model. A - C ; as well as The model training module is used to train models within the Docker container R. N In the next round of formal training, updates are made through reinforcement learning. A , C The parameters are set until the crawler program correctly completes the set crawling task.

Images