A data collection method and system based on page browsing behavior modeling

Abstract

The present application proposes a data collection method based on page browsing behavior modeling, comprising the following steps: a) designing and constructing a js plug-in, and embedding the js plug-in into the browser; b) obtaining a general behavior template file based on page browsing behavior; c) dynamically replacing task parameters to obtain a new behavior template file; d) parsing the new behavior template file into an ordered behavior list; e) controlling the browser to execute the behaviors in the behavior list in sequence to extract data; f) saving and exporting the data extracted on the web page. The data collection of the text data collection method of the present application is based on the simulation of real interactive behaviors and markup behaviors, and is immune to most anti-crawler mechanisms. It achieves accurate data extraction through element positioning, eliminating the secondary cleaning process, and can greatly reduce the difficulty of data collection. When the target website updates its style, it only needs to re-model the new style, reducing maintenance costs and having wide applicability.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and mainly to a data collection method and system based on page browsing behavior modeling. Background Art

[0002] Open source data collection primarily consists of traditional crawler technology and existing template-based automated collection. Traditional crawler technology collects raw web page data, which contains a large amount of HTML tags and irrelevant advertising information. Therefore, the web page data needs to be cleaned up to obtain the real data. Template-based automated collection, on the other hand, is based on a crawler framework and accurately extracts web page data by configuring web page templates, eliminating the need for secondary data cleaning. However, the data collection process still faces some of the same challenges as traditional crawlers, such as parameter encryption, IP restrictions, and image verification. These anti-crawler technologies require developers to invest more time, energy, and resources to combat them, which inevitably increases development difficulty and cost, burdening developers. Furthermore, both traditional crawler and template-based automated collection technologies rely to a certain extent on the relatively fixed style of the web page. Once the style of the web page changes, the crawler program and template collection need to be modified and redeployed accordingly, which undoubtedly increases maintenance costs.

[0003] Traditional crawler collection and template collection technologies used in the open source data collection process have the characteristics of high technical barriers and high maintenance costs. Therefore, designing a data collection method or system with low maintenance cost, immunity to most anti-crawler mechanisms and the ability to accurately extract data has become an urgent problem to be solved. Summary of the Invention

[0004] In response to the above technical problems in the prior art, the inventors proposed a data collection method and system based on page browsing behavior modeling.

[0005] According to one aspect of the present invention, a data collection method based on page browsing behavior modeling is proposed, comprising the following steps:

[0006] S1: Design and build a js plug-in and embed it into the browser;

[0007] S2: Open the starting address of the target website, enter the task parameters, browse the web page, obtain a common behavior template file based on the web browsing behavior, and save the task parameters and the common behavior template file as a template task;

[0008] S3: Import the template task, dynamically replace the task parameters in the template task, and obtain a new behavior template file;

[0009] S4: Start the target website, and the js plug-in parses the new behavior template file data into an ordered behavior list;

[0010] S5: The js plug-in controls the browser to execute the behaviors in the behavior list in order, implements the corresponding behaviors and extracts data;

[0011] S6: Repeat step S5 until all the behaviors in the behavior list are executed, and save and export the data extracted from the web page in the form of a data list. JS plug-ins are JavaScript code files with specific functions, suitable for mainstream browsers such as Chrome, Firefox, and Safari. They can be made into browser plug-ins or embedded into browsers through other tools, such as Tampermonkey and FeHelper. The plug-in monitors and records web browsing actions, models these actions, and generates a general behavior template file. Task parameters are dynamically replaced based on actual needs to generate a new behavior template file. When accessing a specific website, the plug-in parses the new behavior template file into an ordered list of actions and executes the actions in the behavior list on the web page in sequence, performing the operations and extracting data. Because the number of web page styles on a website is limited and relatively fixed, and the actions for browsing different web pages are also relatively fixed, a fixed set of behavior templates can be used to obtain data on the same type of web pages on the website, greatly reducing the difficulty of data collection. Furthermore, the data collection process is conducted in a real browser, simulating real human interaction, making it naturally immune to most anti-crawler mechanisms and reducing the time and resource consumption associated with anti-crawler attacks.

[0012] Preferably, in S1, the js plug-in includes the following four functional modules:

[0013] Web page analyzer: used to analyze the data structure of the HTML page, analyze the extraction path of the web page elements, locate the web page elements according to the extraction path, and perform component analysis on the web page elements;

[0014] Behavior recorder: used to monitor and record users' page browsing behavior on the website and save it in a list format;

[0015] Template Editor: used to display, edit and modify the content recorded by the behavior recorder, and finally obtain the behavior template file;

[0016] Template parser: used to parse the behavior template file to obtain an ordered list of behaviors, and control the browser to execute the behaviors in the ordered list of behaviors.

[0017] There are many ways to locate and extract web page elements, such as XPath positioning, cssPath positioning, etc. With the cooperation of web page analyzer, behavior recorder and template editor, a general behavior template file is obtained by performing page browsing behavior in a real browser. The template parser is used to parse the behavior template file to obtain a behavior list, and control the browser to execute the behavior in the behavior list, implement corresponding operation behavior or data extraction behavior, and output the extracted data.

[0018] Preferably, the page browsing behavior includes normal interactive behavior and marking behavior.

[0019] Normal interactive behaviors include clicking, dragging, sliding, mouse scrolling, text box input, etc., which are used to complete web page browsing. Marking behaviors are used to mark elements of data to be extracted in the web page.

[0020] Preferably, in S2, web browsing is performed, and the specific steps of obtaining a universal behavior template file based on the page browsing behavior are:

[0021] S21: The behavior recorder intercepts the normal interaction between the operator and the web page element, the web page analyzer extracts the path of the operated web page element, counts the path of the operated web page element and the corresponding operation type as an operation unit, and the behavior recorder records the operation unit;

[0022] S22: The behavior recorder intercepts the operator's marking behavior of the web page element, the web page analyzer determines the path of the marked element and the data type of the marked element, counts the obtained path of the marked element and the data type of the marked element as an extraction unit, and the behavior recorder records the extraction unit;

[0023] S23: The operation unit and the extraction unit are combined into a behavior unit. After the page browsing behavior is completed, the behavior recorder obtains an ordered list of behavior units. This ordered list of behavior units is a behavior template.

[0024] S24: Display the behavior template in the form of a flow chart in the template editor, perform actions such as editing, modifying, and deleting the behavior template, and export the behavior template to obtain a general behavior template file.

[0025] Preferably, the marking behavior is to hover the mouse over the element to be marked and confirm the marking using a custom shortcut key.

[0026] By hovering over the element to be marked and marking it with a custom shortcut key, it is possible to prevent the marking behavior from conflicting with the normal interaction behavior, so that the normal interaction behavior and the marking behavior can proceed in an orderly manner without interfering with each other.

[0027] Preferably, in S5, the js plug-in executes the behaviors in the behavior list in order on the web page, implements the corresponding operation behaviors and extracts data, and the specific steps are:

[0028] S51: Obtaining the path of the element to be operated and the specific operation behavior from the operation unit, the web page analyzer first locates the specific element according to the element path, and then the template parser performs the specific operation behavior on the specific element;

[0029] S52: Obtaining the path and data type of the element to be extracted from the extraction unit, the web page analyzer locates the element to be extracted, and the template parser extracts data of the corresponding type from the corresponding element and outputs the extracted data;

[0030] S53: Save the data output by the extraction unit and repeat step S52 until the behavior list is completely executed.

[0031] Preferably, the specific operation behaviors include simulating clicks, simulating text input, page sliding, completing parameter encryption verification, completing cookie verification, and completing dynamic requests.

[0032] The path of the element to be operated and the specific operation behavior are obtained from the operation unit. The web page analyzer first locates the specific element according to the element path, and then the template parser performs specific operation behaviors on the element, such as simulated clicks, simulated text input, page sliding, etc. This process may trigger the browser to dynamically load page data. At this time, the browser will automatically help us complete anti-crawling measures such as parameter encryption, cookie verification, and dynamic requests to ensure that all the data we want can be loaded into the web page, reducing the time and resource consumption caused by anti-crawling.

[0033] Further preferably, the data types of the markup elements include text, attributes and original HTML data.

[0034] In a second aspect, the present application proposes a computer system of an electronic device, comprising one or more components, which, when operating, implement the method described in the first aspect of the present application.

[0035] In a third aspect, the present application further proposes a computer-readable storage medium on which one or more computer programs are stored, characterized in that when the one or more computer programs are executed by a computer processor, the method described in the first aspect of the present application is implemented.

[0036] Fourthly, this application also proposes a data collection system based on page browsing behavior modeling, comprising:

[0037] js plug-in module: configured to model website page browsing behavior, obtain behavior template files, parse the behavior template files into an ordered behavior list, and execute the behaviors in the behavior list in order on the web page;

[0038] Browser module: configuration used to implement page browsing behavior and execute behaviors in the behavior list after the behavior module file is parsed;

[0039] Data processing module: configured to extract required data from web pages, save and export the data extracted on the web pages.

[0040] The present application discloses a data collection method based on page browsing behavior modeling. A js plug-in is constructed and embedded into a web page. Based on the browsing behavior of the website page, normal interactive behavior and marking behavior are monitored and recorded, a general behavior template file is obtained, and a new behavior template is obtained after dynamically replacing the task parameters. When the web page is opened, the js plug-in parses the new behavior template file into an ordered behavior list, controls the browser to execute the behaviors in the behavior list until all the behaviors in the behavior list are executed, and saves and exports the data extracted on the web page, thereby realizing the collection of data on a specific web page. The data collection method of the present application is automated by the js plug-in in the modeling and data collection processes, which greatly reduces the difficulty of data collection. Data collection is carried out in a real browser, which can simulate real human interactive behavior and is naturally immune to most anti-crawler mechanisms, reducing the time consumption caused by anti-crawler resistance. The required data can be accurately extracted through element positioning, eliminating the secondary cleaning process. For websites of different styles, only the browsing of the website needs to be remodeled to realize the collection of new styles of the target website, reducing maintenance costs and having a wider range of applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and, together with the description, serve to explain the principles of the present invention. Other embodiments and many of the expected advantages of the embodiments will be readily apparent as they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with respect to each other. Like reference numerals designate corresponding similar parts.

[0042] Figure 1 A flow chart of a data collection method based on page browsing behavior modeling is shown in the present application;

[0043] Figure 2 A schematic diagram of the js plug-in structure of a data collection method based on page browsing behavior modeling in this application is shown;

[0044] Figure 3A flowchart of obtaining a general behavior template file in a data collection method based on page browsing behavior modeling of the present application is shown;

[0045] Figure 4 A flowchart showing the behaviors in the execution behavior list of a data collection method based on page browsing behavior modeling of the present application is shown;

[0046] Figure 5a A schematic structural diagram of a computer system 600 of an electronic device according to an embodiment of the present invention is shown;

[0047] Figure 5b A structural diagram of a data collection system 700 based on page browsing behavior modeling according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] Figure 1 A flow chart of a data collection method based on page browsing behavior modeling is shown in the present application. Figure 1 As shown, a data collection method based on page browsing behavior modeling includes the following steps:

[0051] S1: Design and build a js plug-in and embed it into the browser;

[0052] S2: Open the starting address of the target website, enter the task parameters, browse the web page, obtain a common behavior template file based on the web browsing behavior, and save the task parameters and the common behavior template file as a template task;

[0053] S3: Import the template task, dynamically replace the task parameters in the template task, and obtain a new behavior template file;

[0054] S4: Start the target website, and the js plug-in parses the new behavior template file data into an ordered behavior list;

[0055] S5: The js plug-in executes the behaviors in the behavior list on the web page in order, implements the corresponding operation behaviors and extracts data;

[0056] S6: Repeat step S5 until all the behaviors in the behavior list are executed, and save and export the data extracted from the web page in the form of a data list.

[0057] A website may have thousands of web pages, but the number of web page styles is limited and relatively fixed, and the actions of browsing different web pages are also relatively fixed. Therefore, a set of fixed behavior templates can be used to obtain all data of the same type of web pages in the website by replacing the dynamic information in the behavior templates.

[0058] JS plug-ins are JavaScript code files with specific functions, suitable for mainstream browsers such as Chrome, Firefox, and Safari. They can be implemented as browser plug-ins or embedded into browsers through other tools, such as Tampermonkey and FeHelper. The plug-in monitors and records web browsing actions, models these actions, and generates a general behavior template file. Task parameters are dynamically replaced based on actual needs to generate a new behavior template file. When accessing a specific web page, the plug-in parses the new behavior template file into an ordered list of actions and executes the actions in the list sequentially on the web page, performing the operations and extracting data. This allows data from the same type of web pages on a website to be collected using a fixed set of behavior templates, greatly reducing the difficulty of data collection. Furthermore, data collection is conducted in a real browser, simulating real human interactions and inherently resistant to most anti-scraping mechanisms, reducing the time and resource consumption associated with anti-scraping efforts.

[0059] Specifically, taking Tampermonkey as an example, the method of embedding the js plug-in into the browser through other tools is as follows:

[0060] GM_addElement('script',{

[0061] 'src':'plugin.js',

[0062] 'type':'text / javascript'

[0063] });

[0064] Among them, plugin.js is the written js plug-in code.

[0065] Figure 2 The following is a schematic diagram of the js plug-in structure of a data collection method based on page browsing behavior modeling in this application. Figure 2 As shown, the js plug-in 100 includes the following four functional modules:

[0066] Web page analyzer 101: used to analyze the data structure of the web page HTML, analyze the extraction path of the web page elements, locate the web page elements according to the extraction path and perform component analysis on the web page elements;

[0067] Behavior recorder 102: used to monitor and record the user's page browsing behavior on the website and save it in a list format;

[0068] Template editor 103: used to display, edit and modify the content recorded by the behavior recorder, and finally obtain a behavior template file;

[0069] Template parser 104: used to parse the behavior template file to obtain an ordered behavior list, and control the browser to execute the behaviors in the ordered behavior list.

[0070] There are many ways to locate and extract web page elements, such as XPath positioning, cssPath positioning, etc. With the cooperation of web page analyzer, behavior recorder and template editor, a general behavior template file is obtained by performing page browsing behavior in a real browser. The template parser is used to parse the behavior template file to obtain a behavior list, and control the browser to execute the behavior in the behavior list, implement corresponding operation behavior or data extraction behavior, and output the extracted data.

[0071] Preferably, the page browsing behavior includes normal interactive behavior and marking behavior.

[0072] Normal interactive behaviors include clicking, dragging, sliding, mouse scrolling, text box input, etc., which are used to complete web page browsing. Marking behaviors are used to mark elements of data to be extracted in the web page.

[0073] Preferably, the marking behavior is to hover the mouse over the element to be marked and confirm the marking using a custom shortcut key.

[0074] By hovering over the element to be marked and marking it with a custom shortcut key, it is possible to prevent the marking behavior from conflicting with the normal interaction behavior, so that the normal interaction behavior and the marking behavior can proceed in an orderly manner without interfering with each other.

[0075] Further preferably, the data types of the markup elements include text, attributes and original HTML data.

[0076] like Figure 3 As shown, the specific steps for browsing a web page and obtaining a general behavior template file based on the page browsing behavior are as follows:

[0077] S21: The behavior recorder intercepts the normal interaction between the operator and the web page element, the web page analyzer extracts the path of the operated web page element, counts the path of the operated web page element and the corresponding operation type as an operation unit, and the behavior recorder records the operation unit;

[0078] S22: The behavior recorder intercepts the operator's marking behavior of the web page element, the web page analyzer determines the path of the marked element and the data type of the marked element, counts the obtained path of the marked element and the data type of the marked element as an extraction unit, and the behavior recorder records the extraction unit;

[0079] S23: The operation unit and the extraction unit are combined into a behavior unit. After the page browsing behavior is completed, the behavior recorder obtains an ordered list of behavior units. This ordered list of behavior units is a behavior template.

[0080] S24: Display the behavior template in the form of a flow chart in the template editor, perform actions such as editing, modifying, and deleting the behavior template, and export the behavior template to obtain a general behavior template file.

[0081] like Figure 4 As shown, the js plug-in executes the behaviors in the behavior list on the web page in order, implements the corresponding operation behaviors and extracts data. The specific steps are as follows:

[0082] S51: Obtaining the path of the element to be operated and the specific operation behavior from the operation unit, the web page analyzer first locates the specific element according to the element path, and then the template parser performs the specific operation behavior on the specific element;

[0083] S52: Obtaining the path and data type of the element to be extracted from the extraction unit, the web page analyzer locates the element to be extracted, and the template parser extracts data of the corresponding type from the corresponding element and outputs the extracted data;

[0084] S53: Save the data output by the extraction unit and repeat the above steps until the behavior list is fully executed.

[0085] Preferably, the specific operation behaviors include simulating clicks, simulating text input, page sliding, completing parameter encryption verification, completing cookie verification, and completing dynamic requests.

[0086] The path of the element to be operated and the specific operation behavior are obtained from the operation unit. The web page analyzer first locates the specific element according to the element path, and then the template parser performs specific operation behaviors on the element, such as simulated clicks, simulated text input, page sliding, etc. This process may trigger the browser to dynamically load page data. At this time, the browser will automatically help us complete anti-crawling measures such as parameter encryption, cookie verification, and dynamic requests to ensure that all the data we want can be loaded into the web page, reducing the time and resource consumption caused by anti-crawling.

[0087] Reference below Figure 5a , which shows a structural diagram of a computer system 600 suitable for implementing an electronic device of an embodiment of the present application. Figure 5a The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0088] like Figure 5a As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the system 600 are also stored in the RAM 603. The CPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0089] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 606 including a liquid crystal display (LCD) and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 610 as needed, so that a computer program read therefrom can be installed in the storage section 608 as needed.

[0090] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable storage medium described in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0091] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0092] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0093] The modules involved in the embodiments described in this application may be implemented in software or hardware. The units described may also be provided in a processor, and the names of these units do not, in certain circumstances, constitute limitations on the units themselves.

[0094] Embodiments of the present invention also relate to a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer processor, implements the method described above. The computer program includes program code for executing the method shown in the flowchart. It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable medium, or any combination thereof.

[0095] Figure 5b FIG. 7 shows a structural diagram of a data collection system 700 based on page browsing behavior modeling according to an embodiment of the present application. Figure 5b As shown, it includes a js plug-in module 701, a browser module 702 and a data processing module 703;

[0096] js plug-in module 701: configured to model page browsing behavior, obtain a behavior template file, parse the behavior template file into an ordered behavior list, and execute the behaviors in the behavior list in order on the web page;

[0097] Browser module 702: configured to implement page browsing behavior and execute behaviors on the behavior list after the behavior module file is parsed;

[0098] Data processing module 703: configured to extract required data from the web page, save and export the data extracted from the web page.

[0099] The present application discloses a data collection method based on page browsing behavior modeling, which embeds a js plug-in into a browser, opens a target website, and models the page browsing behavior. When collecting web page data, the behavior template generated by the modeling is parsed, a behavior list is generated, and the behaviors on the behavior list are executed in sequence to complete automatic browsing of web pages and data collection. Based on the simulation of real interactive behaviors and marking behaviors, the method is immune to most anti-crawler mechanisms, and accurate data extraction is achieved through element positioning, eliminating the secondary cleaning process, which can greatly reduce the difficulty of data collection. When the target website updates its style, only the new style needs to be re-modeled, which reduces maintenance costs and has wide applicability.

[0100] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A data collection method based on page browsing behavior modeling, characterized in that: The following steps are involved: S1: Design and build a js plug-in, and embed the js plug-in into the browser; the js plug-in includes a web page analyzer: used to analyze the data structure of the web page HTML, analyze the extraction path of the web page elements, locate the web page elements according to the extraction path, and perform component analysis on the web page elements; a behavior recorder: used to monitor and record the user's page browsing behavior on the website and save it in the form of a list; a template editor: used to display, edit and modify the content recorded by the behavior recorder, and finally obtain a behavior template file; wherein, the page browsing behavior includes normal interaction behavior and markup behavior; S2: Open the target website's starting address, enter task parameters, browse the webpage, obtain a common behavior template file based on the page browsing behavior, and save the task parameters and the common behavior template file as a template task. The specific steps are as follows: S21: The behavior recorder intercepts the normal interaction between the operator and the web page element, the web page analyzer extracts the path of the operated web page element, counts the path of the operated web page element and the corresponding operation type as an operation unit, and the behavior recorder records the operation unit; S22: The behavior recorder intercepts the operator's marking behavior of the web page element, the web page analyzer determines the path of the marked element and the data type of the marked element, counts the obtained path of the marked element and the data type of the marked element as an extraction unit, and the behavior recorder records the extraction unit; S23: The operation unit and the extraction unit are combined into a behavior unit. After the page browsing behavior is completed, the behavior recorder obtains an ordered list of behavior units. This ordered list of behavior units is a behavior template. S24: Displaying the behavior template in the form of a flow chart in the template editor, editing, modifying, and deleting behaviors in the behavior template, and exporting the behavior template to obtain a general behavior template file; S3: Import the template task, dynamically replace the task parameters in the template task, and obtain a new behavior template file; S4: Start the target website, and the js plug-in parses the new behavior template file data into an ordered behavior list; S5: The js plug-in controls the browser to execute the behaviors in the behavior list in order, implements the corresponding behaviors and extracts data; S6: Repeat step S5 until all the behaviors in the behavior list are executed, and save and export the data extracted from the web page in the form of a data list.

2. The data collection method based on page browsing behavior modeling according to claim 1 is characterized in that: In S1, the js plug-in includes the following functional modules: Template parser: used to parse the behavior template file to obtain an ordered list of behaviors, and control the browser to execute the behaviors in the ordered list of behaviors.

3. The data collection method based on page browsing behavior modeling according to claim 1 is characterized in that: The marking behavior is to hover the mouse over the element to be marked and confirm the marking using the custom shortcut key.

4. The data collection method based on page browsing behavior modeling according to claim 1 is characterized in that: In S5, the js plug-in executes the behaviors in the behavior list in order on the web page, implements the corresponding behaviors and extracts data. The specific steps are as follows: S51: Obtaining the path of the element to be operated and the specific operation behavior from the operation unit, the web page analyzer first locates the specific element according to the element path, and then the template parser performs the specific operation behavior on the specific element; S52: Obtaining the path and data type of the element to be extracted from the extraction unit, the web page analyzer locates the element to be extracted, and the template parser extracts data of the corresponding type from the corresponding element and outputs the extracted data; S53: Save the data output by the extraction unit and repeat the above steps until the behavior list is fully executed.

5. The data collection method based on page browsing behavior modeling according to claim 4 is characterized in that: The specific operation behaviors include simulating clicks, simulating text input, page sliding, completing parameter encryption verification, completing cookie verification, and completing dynamic requests.

6. A computer system for an electronic device, the electronic device comprising: a processor, and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the data collection method based on page browsing behavior modeling as described in any one of claims 1 to 5.

7. A computer-readable storage medium having one or more computer programs stored thereon, characterized in that: When the one or more computer programs are executed by a computer processor, the method according to any one of claims 1 to 5 is implemented.

8. A data collection system based on page browsing behavior modeling, characterized in that: include: js plug-in module: the configuration includes a web page analyzer: used to analyze the data structure of the web page html, analyze the extraction path of the web page elements, locate the web page elements according to the extraction path and perform component analysis on the web page elements; a behavior recorder: used to monitor and record the user's page browsing behavior on the website, and save it in the form of a list; a template editor: used to display, edit and modify the content recorded by the behavior recorder, and finally obtain a behavior template file; wherein, the page browsing behavior includes normal interactive behavior and marking behavior; used to obtain a behavior template file based on page browsing behavior modeling, parse the behavior template file into an ordered behavior list, and execute the behavior in the behavior list in sequence on the web page; wherein, the construction of the behavior template file is based on the following steps: the behavior recorder intercepts the normal interactive behavior of the operator with the web page elements, and the web The page analyzer extracts the path of the operated web page element, and counts the path of the operated web page element and the corresponding operation type as an operation unit, which is recorded by the behavior recorder; the behavior recorder intercepts the operator's marking behavior of the web page element, and the web page analyzer determines the path of the marked element and the data type of the marked element, and counts the obtained path of the marked element and the data type of the marked element as an extraction unit, which is recorded by the behavior recorder; the operation unit and the extraction unit are merged into a behavior unit, and after the page browsing behavior is completed, the behavior recorder obtains an ordered list of behavior units, which is a behavior template; the behavior template is displayed in the form of a flow chart in the template editor, and the behavior template is edited, modified, and deleted, and the behavior template is exported to obtain a general behavior template file; Browser module: configuration used to implement page browsing behavior and execute behaviors in the behavior list after the behavior module file is parsed; Data processing module: configured to extract required data from web pages, save and export the data extracted on the web pages.

Images