Simulated user behavior-based crawler method, device and related equipment

Abstract

This invention discloses a web crawler method, apparatus, and related equipment based on simulated user behavior. The method collects web browsing behavior data from several users, extracts features from the behavior data to obtain user browsing behavior features, and constructs a behavior feature library based on these features. Based on feature parameters of trajectory jitter characteristics and preset curve construction logic, a mouse movement trajectory conforming to user operating habits is constructed. According to preset time-series calculation logic and time-series distribution characteristics, fluctuation intervals and page dwell times consistent with user mouse operations are generated. A complete user operation behavior chain is constructed based on the fluctuation intervals, page dwell times, and mouse movement trajectory, resulting in a behavior chain. Anti-crawling feedback from the target website is monitored in real time, and the behavior chain is adjusted according to the anti-crawling feedback. Data is then crawled according to the adjusted behavior chain. This method significantly improves the anti-crawling capability of crawled data.

Description

Technical Field

[0001] This invention relates to the field of data crawling technology, and in particular to a crawling method, apparatus and related equipment based on simulated user behavior. Background Technology

[0002] With the development of network security technology, various websites (such as financial platforms and e-commerce platforms) have generally deployed advanced anti-crawler protection systems to protect data security. The core of these systems is to intercept crawlers by identifying their mechanical behavior characteristics (such as fixed click intervals, uniform scrolling trajectories, and single mouse movement paths).

[0003] Existing anti-anti-crawling solutions for web crawlers mainly focus on simple behavior randomization, such as setting a fixed range of click intervals or generating mouse trajectories of straight lines or simple curves. These solutions are easily identified by anti-crawling systems through time-series feature modeling, resulting in high anti-crawling identification risks. They can only break through basic anti-crawling strategies, and there is an urgent need for an intelligent web crawler technology solution that can accurately simulate user behavior characteristics and break through advanced anti-crawling protection. Summary of the Invention

[0004] The embodiments of the present invention provide a web crawler method, apparatus and related equipment based on simulated user behavior, which aims to solve the technical problem that existing web crawler methods are difficult to achieve effective data crawling when dealing with the anti-crawling strategies of target websites.

[0005] In a first aspect, embodiments of the present invention provide a web crawler method based on simulated user behavior, comprising: Collect behavioral data of several users browsing web pages, extract features based on the behavioral data to obtain user browsing behavior features, and build a behavior feature library based on the behavior features. The behavioral data includes trajectory jitter features, temporal distribution features, and link behavior features. Based on the feature parameters of the trajectory jitter characteristics and the preset curve construction logic, a mouse movement trajectory that conforms to the user's operating habits is constructed. Generate fluctuation intervals and webpage dwell times that match user mouse operations based on preset timing calculation logic and timing distribution characteristics; Based on the fluctuation interval time, page dwell time and mouse movement trajectory, a complete user operation behavior chain is constructed to obtain the behavior chain; The anti-crawling feedback of the target website is monitored in real time, the behavior chain is adjusted according to the anti-crawling feedback, and data is crawled according to the adjusted behavior chain.

[0006] Secondly, embodiments of the present invention provide a web crawler device based on simulated user behavior, comprising: The data acquisition module is used to collect behavioral data of several users browsing web pages, extract features based on the behavioral data to obtain behavioral features of users browsing web pages, and build a behavioral feature library based on the behavioral features. The behavioral data includes trajectory jitter features, temporal distribution features, and link behavior features. The trajectory construction module is used to construct a mouse movement trajectory that conforms to user operating habits based on the feature parameters of the trajectory jitter characteristics and the preset curve construction logic. The timing parameter calculation module is used to generate fluctuation intervals and webpage dwell times that match user mouse operations based on preset timing calculation logic and timing distribution characteristics. The link construction module is used to construct a complete behavioral link of user operation based on the fluctuation interval time, page dwell time and mouse movement trajectory, and obtain the behavioral link; The link adjustment module is used to monitor the anti-crawling feedback of the target website in real time, adjust the behavior link according to the anti-crawling feedback, and crawl data according to the adjusted behavior link.

[0007] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the web crawler method based on simulated user behavior described in the first aspect.

[0008] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the web crawler method based on simulated user behavior described in the first aspect.

[0009] This invention provides a web crawler method, apparatus, and related equipment based on simulated user behavior. The method collects web browsing behavior data from several users, extracts features from the behavior data to obtain user browsing behavior features, and constructs a behavior feature library based on these features. Based on the feature parameters of trajectory jitter characteristics and a preset curve construction logic, a mouse movement trajectory conforming to user operating habits is constructed. According to a preset time-series calculation logic and time-series distribution characteristics, fluctuation intervals and webpage dwell times consistent with user mouse operations are generated. A complete user operation behavior chain is constructed based on the fluctuation intervals, page dwell times, and mouse movement trajectory, resulting in a behavior chain. Real-time monitoring of anti-crawling feedback from the target website is performed, and the behavior chain is adjusted based on the anti-crawling feedback. Data is then crawled according to the adjusted behavior chain. This method generates mouse movement trajectories using Bézier curves, and the dynamic time-series algorithm simulates the user's "thinking-operation" time-series characteristics. Compared to traditional mechanical web crawling methods, which are more easily identified by anti-crawling mechanisms, this method significantly improves the anti-crawling capability of crawled data. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating an embodiment of the web crawler method based on simulated user behavior provided by the present invention. Figure 2 This is a schematic block diagram of a web crawler device based on simulated user behavior, provided in an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] Please see Figure 1 This is a flowchart illustrating a web crawler method based on simulated user behavior provided in an embodiment of the present invention. The method includes steps S110 to S150.

[0017] Step S110: Collect behavioral data of several users browsing web pages, extract features based on the behavioral data to obtain behavioral features of users browsing web pages, and build a behavioral feature library based on the behavioral features. The behavioral data includes trajectory jitter features, temporal distribution features and link behavior features. In this embodiment, web browsing behavior samples from at least 1000 different groups (different ages and operating habits) were collected, including data such as mouse movement trajectory coordinate sequences, click timestamps, dwell time, and scrolling speed changes; the sample data was preprocessed to remove outliers, and key features were extracted: Track jitter characteristics: During mouse movement, the offset range (0.1-2 pixels) and jitter frequency (1-3 times / second) every 10 milliseconds. Temporal distribution characteristics: probability distribution model of click interval (conforming to normal distribution), correlation between dwell time and content complexity; Link behavior characteristics: conversion probability of each stage of "browsing-thinking-clicking-staying", and the time proportion of the thinking stage (30%-60%). A behavioral feature library is constructed based on the above features to provide data support for subsequent simulations.

[0018] Step S120: Based on the feature parameters of the trajectory jitter feature and the preset curve construction logic, construct a mouse movement trajectory that conforms to the user's operating habits; In one embodiment, the curve construction logic uses a third-order Bézier curve to construct the mouse movement trajectory. The starting point P0 (current mouse position) and ending point P3 (target operation position) of the mouse trajectory are set. Two intermediate control points P1 and P2 are randomly generated, and the coordinate offset of the control points is randomly generated based on the trajectory jitter range (offset range 0.5-3 pixels). Within a preset time window (e.g., 5 milliseconds), the trajectory coordinates are randomly jittered by ±(0.1-0.8) pixels. During the process from the starting point to the ending point, the mouse movement speed changes according to a preset speed pattern (e.g., "acceleration-uniform speed-deceleration," where the speed change rate is trained based on sample data in a behavioral feature library to avoid mechanical features of uniform movement), thus obtaining the basic mouse trajectory.

[0019] In one embodiment, the curve construction logic employs a large language model (LLM) or a temporal generation model trained based on a behavioral feature library to construct the mouse trajectory. Specifically, the contextual information of the current page (including page layout complexity, target element coordinates, and previous operation history) and user behavior fingerprints extracted from the behavioral feature library (such as acceleration preferences, turning habits, and micro-jitter frequency distribution of a specific group) are encoded into prompt word vectors or conditions and input into a pre-trained trajectory generation large model. The trajectory generation large model is based on a deep neural network architecture and directly outputs a complete trajectory sequence containing ten high-precision coordinate points (x, y) and corresponding timestamps t by learning the probability distribution of massive amounts of real user mouse movement sequences. This trajectory sequence naturally possesses non-linear speed change characteristics (such as hesitation and pause at startup, deceleration correction when approaching the target, and random micro-jitter during the process), and the control points are not randomly generated by humans but are implicitly inferred by the model based on contextual semantics, thereby generating an anthropomorphic mouse movement trajectory that is highly consistent with real human operations in terms of statistical features and is difficult to be identified by traditional Bézier curve fitting detection algorithms.

[0020] Step S130: Generate the fluctuation interval time and webpage dwell time that match the user's mouse operation based on the preset timing calculation logic and timing distribution characteristics; In one embodiment, the timing calculation logic employs an improved Poisson process algorithm, generating fluctuation intervals and webpage dwell times that correspond to user mouse operations based on a dynamic timing algorithm. Specifically, based on the content complexity of the target website, an initial click interval (0.3-2.5 seconds) is matched from a behavioral feature database; a "thinking factor" is introduced to simulate the user's thinking process after browsing content, with the thinking factor randomly ranging from 0.1 to 0.8, and the click interval = base interval × (1 + thinking factor) to achieve dynamic fluctuations in timing; the first n (e.g., 3) click intervals are recorded, and the next interval generation references historical data to avoid consecutive identical or similar intervals, ensuring the naturalness of the timing distribution; based on the length and complexity of the webpage content, the page dwell time is dynamically adjusted, with the dwell time increasing by 0.2-0.5 seconds for every certain number of characters (e.g., 1000) in content length, while simultaneously adding random fluctuations of ±0.1-0.3 seconds.

[0021] In one embodiment, the time-series calculation logic employs a dynamic intelligent thinking mechanism based on a multimodal large model (LLM) to calculate time-series parameters. Specifically, the LLM is used to perform real-time semantic parsing of the current webpage, extracting key information density, text reading difficulty, interaction element hierarchy, and the user's current task intent to obtain semantic features. These semantic features are then combined with user cognitive models in a behavioral feature library and input into a pre-trained time-series decision-making large model. The time-series decision-making large model simulates the human cognitive process of "reading-understanding-decision-making," dynamically deducing the required "thinking time" in the current scenario: for areas with high complexity or high-value information, the model automatically extends the dwell time and increases the hesitation interval before clicking (simulating deep reading and weighing); for simple navigation or low-information areas, the time interval is shortened to simulate fast browsing behavior. Simultaneously, the model introduces random noise based on historical interaction links to generate fluctuating interval times and page dwell times with long-tail distribution characteristics, ensuring that the generated time-series data not only conforms to statistical laws but also possesses logical rationality based on content awareness, effectively overcoming anti-crawling time-series detection based on fixed thresholds or simple statistical laws.

[0022] Step S140: Construct a complete user operation behavior chain based on the fluctuation interval time, page dwell time, and mouse movement trajectory to obtain the behavior chain; In this embodiment, the behavioral chain includes a webpage browsing process, a webpage thinking process, a mouse click process, and a webpage feedback process. Specifically, the webpage browsing process simulates the user's eye movement, with the mouse slowly sliding along a random path (at a speed of 50-150 pixels / second), traversing the core content area of ​​the webpage, interspersed with several brief pauses (0.2-0.5 seconds in duration). The webpage thinking process, after the webpage browsing ends, simulates the user's understanding and decision-making process based on the duration of the pauses (e.g., 0.5-1.8 seconds), during which the mouse slightly jitters (without significant displacement). The mouse click process, based on the mouse movement trajectory, executes a click operation, with the mouse coordinates showing ± at the moment of the click. A random offset of 1-2 pixels simulates the inaccuracy of user clicks; the webpage feedback process simulates the user's waiting for the page response based on the fluctuation interval time (e.g., 0.3-1.2 seconds). If the page does not load in time, the dwell time is automatically extended (by 0.5-1 second) to avoid the mechanical behavior of frequent clicks; the loop of "webpage browsing process - webpage thinking process - mouse click process - webpage feedback process" is cyclical. In each loop, the behavioral characteristics (trajectory jitter, time interval, dwell time) are dynamically adjusted based on the behavioral characteristic library to avoid the solidification of behavioral patterns.

[0023] Step S150: Monitor the anti-crawling feedback of the target website in real time, adjust the behavior link according to the anti-crawling feedback, and crawl data according to the adjusted behavior link.

[0024] In this embodiment, the anti-scraping feedback of the target website is monitored in real time (such as response status codes, CAPTCHA triggers, and IP restriction prompts), and behavioral parameters are dynamically adjusted, including: 1. If a minor anti-scraping warning is detected (such as increased response delay), the randomness of the behavior is increased: the mouse jitter amplitude is increased (0.2-1 pixel), and the fluctuation range of the time interval is increased (0.2-2.8 seconds); 2. If severe anti-scraping measures are detected (such as verification commands and IP blocking), the backup behavioral model in the behavioral feature library is called, the mouse movement trajectory and timing algorithm parameters are switched, and the access is re-initiated. In addition, the successful behavioral data of each access (trajectories, timing, and link parameters not identified by anti-scraping) can be stored in the feature library, and the model can be optimized through incremental learning to improve the adaptability to different anti-scraping systems. According to the adjusted behavioral link, the target webpage data is crawled according to preset rules. During the collection process, the crawler characteristics are hidden (such as disguising real browser fingerprints and disabling automation identifiers) to ensure the legality and stability of data collection; after the collection is completed, the data is preliminarily cleaned and structured data is output.

[0025] This method collects behavioral data from several users browsing web pages, extracts features based on the behavioral data to obtain user browsing behavior characteristics, and constructs a behavior feature library based on these characteristics. Based on the feature parameters of trajectory jitter characteristics and a preset third-order Bézier curve, a mouse movement trajectory conforming to user operating habits is constructed. According to a preset dynamic temporal algorithm and temporal distribution characteristics, fluctuation intervals and page dwell times consistent with user mouse operations are generated. A complete user operation behavior chain is constructed based on the fluctuation intervals, page dwell times, and mouse movement trajectory, resulting in the behavior chain. Real-time monitoring of anti-crawling feedback from the target website is performed, and the behavior chain is adjusted based on the anti-crawling feedback, with data crawling proceeding according to the adjusted behavior chain. This method generates mouse movement trajectories using Bézier curves, and the dynamic temporal algorithm simulates the temporal characteristics of user "thinking-operation," which is more easily identified by anti-crawling mechanisms compared to traditional mechanical crawling methods, greatly improving the anti-crawling capability of user-crawled data.

[0026] This invention also provides a web crawler device based on simulated user behavior, which is used to execute any of the aforementioned web crawler methods based on simulated user behavior. Specifically, please refer to... Figure 2 , Figure 2 This is a schematic block diagram of a web crawler device based on simulated user behavior provided in an embodiment of the present invention. The web crawler device 100 based on simulated user behavior can be configured in a server.

[0027] like Figure 2 As shown, the crawler device 100 based on simulated user behavior includes a data acquisition module 110, a trajectory construction module 120, a time series parameter calculation module 130, a link construction module 140, and a link adjustment module 150.

[0028] The acquisition module 110 is used to collect behavioral data of several users browsing web pages, extract features based on the behavioral data to obtain behavioral features of users browsing web pages, and build a behavioral feature library based on the behavioral features. The behavioral data includes trajectory jitter features, temporal distribution features and link behavior features. The trajectory construction module 120 is used to construct a mouse movement trajectory that conforms to user operating habits based on the feature parameters of the trajectory jitter characteristics and the preset curve construction logic. The timing parameter calculation module 130 is used to generate fluctuation interval time and webpage dwell time that conform to user mouse operation based on preset timing calculation logic and timing distribution characteristics. Link construction module 140 is used to construct a complete behavior link of user operation based on the fluctuation interval time, page dwell time and mouse movement trajectory, and obtain the behavior link; The link adjustment module 150 is used to monitor the anti-crawling feedback of the target website in real time, adjust the behavior link according to the anti-crawling feedback, and crawl data according to the adjusted behavior link.

[0029] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the web crawler method based on simulated user behavior as described above.

[0030] In another embodiment of the invention, a computer-readable storage medium is provided. This computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the web crawling method based on simulated user behavior as described above.

[0031] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0032] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Units with the same function may be grouped into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, or it may be an electrical, mechanical, or other form of connection.

[0033] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0034] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0035] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks.

[0036] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A web crawler method based on simulating user behavior, characterized in that, include: Collect behavioral data of several users browsing web pages, extract features based on the behavioral data to obtain user browsing behavior features, and build a behavior feature library based on the behavior features. The behavioral data includes trajectory jitter features, temporal distribution features, and link behavior features. Based on the feature parameters of the trajectory jitter characteristics and the preset curve construction logic, a mouse movement trajectory that conforms to the user's operating habits is constructed. Generate fluctuation intervals and webpage dwell times that match user mouse operations based on preset timing calculation logic and timing distribution characteristics; Based on the fluctuation interval time, page dwell time and mouse movement trajectory, a complete user operation behavior chain is constructed to obtain the behavior chain; The anti-crawling feedback of the target website is monitored in real time, the behavior chain is adjusted according to the anti-crawling feedback, and data is crawled according to the adjusted behavior chain.

2. The web crawler method based on simulated user behavior as described in claim 1, characterized in that, The curve construction logic involves constructing a trajectory curve based on a third-order Bézier curve. The basic mouse trajectory for a user browsing a webpage is constructed based on the trajectory jitter range and the preset curve construction logic, including: Set the mouse trajectory start point P0 and mouse trajectory end point P3, and randomly generate two intermediate control points P1 and P2. The control point coordinate offset is randomly generated based on the trajectory jitter range. Within a preset time window, the trajectory coordinates are randomly jittered. During the process from the starting point to the ending point, the mouse movement speed changes according to a preset speed pattern to obtain the basic mouse trajectory.

3. The web crawler method based on simulated user behavior as described in claim 1, characterized in that, The process of generating fluctuation intervals and webpage dwell times that correspond to user mouse operations based on a preset dynamic timing algorithm and timing distribution characteristics includes: The initial click interval is determined based on the temporal distribution features in the aforementioned behavioral feature library; The fluctuation interval time is obtained by calculating the fluctuation interval based on the initial click interval and the preset thinking factor; The page dwell time is obtained by adjusting the mouse hover time on the webpage based on the length and complexity of the webpage content.

4. The web crawler method based on simulated user behavior as described in claim 1, characterized in that, The behavioral chain includes a web browsing process, a web page thinking process, a mouse click process, and a web page feedback process. The complete user operation behavioral chain, constructed based on the fluctuation interval time, page dwell time, and mouse movement trajectory, is obtained, including: Based on the features of the behavioral feature library, simulate the user's eye movement, adjust the mouse to slowly traverse the core content area of ​​the webpage along a random path, and construct the webpage browsing process. After browsing, the user's understanding and decision-making process regarding the webpage content is simulated based on the duration of page dwell, thus constructing the webpage thinking process; Based on the mouse movement path, a mouse click operation is performed. When the click occurs, the mouse click coordinates of the user are simulated to shift randomly, thus constructing the mouse click process. Based on the fluctuation interval time, the process of simulating the user waiting for the page to respond is simulated. If the page does not record in time, the dwell time is automatically extended to construct the webpage feedback process. The behavior chain is constructed based on the webpage browsing process, webpage thinking process, mouse click process, and webpage feedback process.

5. The web crawler method based on simulated user behavior as described in claim 1, characterized in that, The real-time monitoring of the target website's anti-scraping feedback, and the adjustment of the behavioral chain based on the anti-scraping feedback, includes: If the anti-scraping warning of the target website is triggered, the fluctuation range of mouse jitter amplitude and click interval will be increased. If the target website's verification command or IP blocking is triggered, the mouse movement trajectory and corresponding timing parameters are switched, and the access is re-initiated. The timing parameters include the fluctuation interval time and the duration of the webpage stay.

6. The web crawler method based on simulated user behavior as described in claim 1, characterized in that, The step of extracting features based on the behavioral data to obtain user browsing webpage behavioral features includes: The behavioral data is preprocessed to remove outliers.

7. A web crawler device based on simulated user behavior, characterized in that, include: The data acquisition module is used to collect behavioral data of several users browsing web pages, extract features based on the behavioral data to obtain behavioral features of users browsing web pages, and build a behavioral feature library based on the behavioral features. The behavioral data includes trajectory jitter features, temporal distribution features, and link behavior features. The trajectory construction module is used to construct a mouse movement trajectory that conforms to user operating habits based on the feature parameters of the trajectory jitter characteristics and the preset curve construction logic. The timing parameter calculation module is used to generate fluctuation intervals and webpage dwell times that match user mouse operations based on preset timing calculation logic and timing distribution characteristics. The link construction module is used to construct a complete behavioral link of user operation based on the fluctuation interval time, page dwell time and mouse movement trajectory, and obtain the behavioral link; The link adjustment module is used to monitor the anti-crawling feedback of the target website in real time, adjust the behavior link according to the anti-crawling feedback, and crawl data according to the adjusted behavior link.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the web crawling method based on simulated user behavior as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the web crawling method based on simulated user behavior as described in any one of claims 1 to 6.

Images