Internet clue identification system and method based on dynamic interaction

Abstract

The invention discloses an internet clue identification system and method based on dynamic interaction, and belongs to the technical field of internet big data supervision. The system comprises a browser module used for simulating browser internet surfing behaviors of a user in a preset browser mode so as to carry out dynamic interaction of browser webpages; the webpage element identification module is used for extracting interactive elements of the browser webpage and carrying out dynamic interaction with a server of the browser webpage through the interactive elements so as to identify a website map of a website corresponding to the browser webpage; and the behavior identification module is used for identifying the website map to obtain internet clues of the website. The application of the invention can identify illegal clues in the website and supervise the deep network field which cannot be touched by the existing hyperlink analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet big data supervision, and more specifically, to an Internet clue identification system and method based on dynamic interaction. Background Art

[0002] While online commodity transactions bring us an efficient and convenient life, more and more online illegal activities have greatly infringed upon the legitimate rights and interests of consumers, seriously affected the healthy development of online commodity transactions, and brought severe challenges to regulatory work.

[0003] Challenge 1: The entire process of online commodity transactions takes place in virtual space, which has a large span, fast changes, strong concealment, and difficult to identify.

[0004] Challenge 2: It is difficult to verify, define, collect evidence and enforce illegal activities in online commodity transactions.

[0005] Challenge 3: Internet supervision places high demands on business personnel, but the supervision concept is not yet mature and the supporting regulations are not yet complete.

[0006] Therefore, technologies for collecting illegal clues on the Internet have emerged. For example, the hyperlink analysis technology used by major search engines extracts web page relationships by analyzing the internal and external links in static pages, and then collects relevant pages through specific crawler scheduling methods. Finally, certain illegal behavior identification rules are used to make judgments and collect evidence, etc.

[0007] However, the internet is a multi-layered structure. The surface web lies at the surface level, accessible through standard search engines. Beneath this layer lies the deep web. Content within the deep web is inaccessible through conventional search engines. Therefore, the vast majority of illegal activity lies hidden within the deep web, beyond the reach of traditional search engines. This poses a significant challenge to traditional internet regulation techniques. Summary of the Invention

[0008] To address the above problems, the present invention proposes an Internet clue identification system based on dynamic interaction, comprising:

[0009] The browser module is used to simulate the user's browser surfing behavior in a preset browser mode to perform dynamic interaction with the browser web page;

[0010] A web page element recognition module is used to extract the interactive elements of the browser web page and dynamically interact with the server of the browser web page through the interactive elements to identify the website map of the website corresponding to the browser web page;

[0011] The behavior recognition module is used to identify the website map and obtain Internet clues of the website.

[0012] Optional, browser mode, including: no-header browser mode and interface browser mode;

[0013] Among them, the interfaceless browser mode is deployed in multiple processes on the Linux server;

[0014] Among them, the interface browser mode is used in the testing phase of the Internet clue identification system, and the interfaceless browser mode is used in the application phase of the Internet clue identification system.

[0015] Optional, interactive elements, including:

[0016] Web page input box, submit button, page turn button and internal / external links.

[0017] Optionally, the webpage element recognition module extracts interactive elements of the browser webpage, and dynamically interacts with the server of the browser webpage through the interactive elements, including:

[0018] Extracting interactive elements of the browser web page through regular expressions and / or natural language processing technology, and traversing the browser web page through the interactive elements, and scanning the server site of the website where the browser web page is located;

[0019] After extracting the interactive elements of the browser web page, determining whether the interactive elements are valid, and if valid, dynamically interacting with the server of the browser web page through the interactive elements;

[0020] Determining whether the interactive element is valid includes:

[0021] Determine whether the interactive element of the browser web page is visible. If visible, determine whether the interactive element has an integrated button and determine whether the integrated button is valid. If there is no integrated button, enter a request and determine whether to respond. If so, it is valid.

[0022] Optional, Internet leads, including:

[0023] Clues to prohibited items, sensitive words, illegal advertising and target products.

[0024] In another aspect, the present invention further proposes an Internet clue identification method based on dynamic interaction, comprising:

[0025] Use the preset browser mode to simulate the user's browser surfing behavior to achieve dynamic interaction of browser web pages;

[0026] Extracting interactive elements from the browser webpage, and dynamically interacting with the server of the browser webpage through the interactive elements to identify a website map of the website corresponding to the browser webpage;

[0027] The website map is identified to obtain Internet clues of the website.

[0028] Optional, browser mode, including: no-header browser mode and interface browser mode;

[0029] Among them, the interfaceless browser mode is deployed in multiple processes on the Linux server;

[0030] Among them, the interface browser mode is used in the testing phase of the Internet clue identification system, and the interfaceless browser mode is used in the application phase of the Internet clue identification system.

[0031] Optional, interactive elements, including:

[0032] Web page input box, submit button, page turn button and internal / external links.

[0033] Optionally, extracting interactive elements of the browser webpage and dynamically interacting with the server of the browser webpage through the interactive elements includes:

[0034] Extracting interactive elements of the browser web page through regular expressions and / or natural language processing technology, and traversing the browser web page through the interactive elements, and scanning the server site of the website where the browser web page is located;

[0035] After extracting the interactive elements of the browser web page, determining whether the interactive elements are valid, and if valid, dynamically interacting with the server of the browser web page through the interactive elements;

[0036] Determining whether the interactive element is valid includes:

[0037] Determine whether the interactive element of the browser web page is visible. If visible, determine whether the interactive element has an integrated button and determine whether the integrated button is valid. If there is no integrated button, enter a request and determine whether to respond. If so, it is valid.

[0038] Optional, Internet leads, including:

[0039] Clues to prohibited items, sensitive words, illegal advertising and target products.

[0040] In yet another aspect, the present invention further provides a computing device comprising: one or more processors;

[0041] a processor for executing one or more programs;

[0042] When the one or more programs are executed by the one or more processors, the above-described method is implemented.

[0043] In another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, the method described above is implemented.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] The present invention provides an internet clue identification system based on dynamic interaction, comprising: a browser module for simulating a user's browser surfing behavior in a preset browser mode to enable dynamic interaction with the browser webpage; a webpage element identification module for extracting interactive elements from the browser webpage and dynamically interacting with the browser webpage server through the interactive elements to identify the sitemap of the website corresponding to the browser webpage; and a behavior identification module for identifying the sitemap to obtain internet clues from the website. The application of the present invention can identify illegal clues within websites and monitor deep web areas that are inaccessible to existing hyperlink analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features, and advantages of the present invention will become more apparent through a more detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. The accompanying drawings are provided to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and are not intended to limit the present invention. In the drawings, the same reference numerals generally represent the same components or steps.

[0047] Figure 1 This is an abstract schematic diagram of module interaction in a system embodiment of the present invention;

[0048] Figure 2 A schematic diagram of module interaction in a system embodiment of the present invention;

[0049] Figure 3 This is a diagram showing the operating principle of an embodiment of the system of the present invention;

[0050] Figure 4 This is an operational flow chart of an embodiment of the system of the present invention. DETAILED DESCRIPTION

[0051] Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. These embodiments are provided to provide a thorough and complete disclosure of the present invention and to fully convey the scope of the present invention to those skilled in the art. The terminology used in the exemplary embodiments shown in the accompanying drawings is not intended to limit the present invention. In the accompanying drawings, identical elements are denoted by the same reference numerals.

[0052] Unless otherwise specified, the terms used herein (including technical terms) have the meanings commonly understood by those skilled in the art. In addition, it is understood that terms defined in commonly used dictionaries should be understood to have the same meanings as those in the context of the relevant fields, and should not be understood as idealized or overly formal meanings.

[0053] Example 1:

[0054] The present invention proposes an Internet clue identification system based on dynamic interaction, such as Figure 1 Shown, including:

[0055] The browser module is used to simulate the user's browser surfing behavior in a preset browser mode to perform dynamic interaction with the browser web page;

[0056] A web page element recognition module is used to extract the interactive elements of the browser web page and dynamically interact with the server of the browser web page through the interactive elements to identify the website map of the website corresponding to the browser web page;

[0057] The behavior recognition module is used to identify the website map and obtain Internet clues of the website.

[0058] Among them, the browser mode includes: no-interface browser mode and interface browser mode;

[0059] Among them, the interfaceless browser mode is deployed in multiple processes on the Linux server;

[0060] Among them, the interface browser mode is used in the testing phase of the Internet clue identification system, and the interfaceless browser mode is used in the application phase of the Internet clue identification system.

[0061] Among them, interactive elements include:

[0062] Web page input box, submit button, page turn button and internal / external links.

[0063] The webpage element recognition module extracts the interactive elements of the browser webpage and dynamically interacts with the server of the browser webpage through the interactive elements, including:

[0064] Extracting interactive elements of the browser web page through regular expressions and / or natural language processing technology, and traversing the browser web page through the interactive elements, and scanning the server site of the website where the browser web page is located;

[0065] After extracting the interactive elements of the browser web page, determining whether the interactive elements are valid, and if valid, dynamically interacting with the server of the browser web page through the interactive elements;

[0066] Determining whether the interactive element is valid includes:

[0067] Determine whether the interactive element of the browser web page is visible. If visible, determine whether the interactive element has an integrated button and determine whether the integrated button is valid. If there is no integrated button, enter a request and determine whether to respond. If so, it is valid.

[0068] Among them, Internet clues include:

[0069] Clues to prohibited items, sensitive words, illegal advertising and target products.

[0070] The following combination Figures 1 to 4 The present invention will be further described:

[0071] like Figure 1 The following modules are included:

[0072] The browser module simulates user behavior to interact with web pages, including both interface-less browsers and interface-based browsers.

[0073] The traditional get / post request mode has high crawling efficiency, but it cannot load front-end JS and other codes, and cannot interact in real time;

[0074] To simulate real user behavior, we chose two simulated browser modes: a browser with an interface and a browser without an interface. The browser without an interface does not require a graphical user interface and can be deployed in multiple processes on a Linux server, balancing operational convenience and crawling efficiency.

[0075] Web page element recognition module, which interacts with the website server by extracting interactive elements on the page, including input box discovery, submit button discovery, page turn button discovery, and internal / external link discovery;

[0076] Here we use technologies such as regular expressions and natural language processing to automatically identify input boxes, submit buttons, page turn buttons, and internal / external links on web pages.

[0077] By analyzing the input box type, visibility, and integrated buttons, the input box is judged to be valid and associated with a corresponding submit button. If the submit button does not exist, try to press Enter to submit the request;

[0078] By analyzing the page turning buttons, we can find "Previous Page", "Next Page", "First Page", "Last Page", etc., and conduct a comprehensive traversal of the result records;

[0079] By analyzing the internal and external links, we can get the site map of the website, which helps us get as many input boxes as possible and conduct a full-scale scan of the site.

[0080] The illegal behavior identification module helps regulators collect evidence and enforce the law by identifying illegal clues on web pages, including identification of prohibited items, sensitive words, illegal advertisements, and targeted products.

[0081] For each result page, we use machine learning and natural language processing technology to identify illegal activities;

[0082] Contraband identification: monitoring whether contraband is contained;

[0083] Sensitive word identification: monitor whether sensitive words are contained, and then further determine whether it is illegal;

[0084] Illegal advertising identification: monitors whether businesses use extreme language, engage in false advertising, and identify illegal advertising;

[0085] Target commodity identification: Retrieve target commodities of concern to regulatory agencies, with self-configuration, convenience and efficiency;

[0086] Among them, the browser module simulates user behavior to interact with web pages, including interface-free browsers and interface browsers, such as Figure 2 As shown;

[0087] The web page element recognition module interacts with the website server by extracting interactive elements on the page, including input box discovery, submit button discovery, page turn button discovery and internal / external link discovery, such as Figure 2 As shown;

[0088] The illegal behavior identification module helps regulatory agencies collect evidence and enforce the law by identifying illegal clues on the page, including identification of contraband, sensitive words, illegal advertisements, and target products. Figure 2 As shown;

[0089] like Figure 3 、 Figure 4 As shown, the specific operation process of the present invention is as follows:

[0090] We need to collect deep clues from websites whose homepage is "http: / / www.xxx.com" (xxx.com does not represent the real domain name).

[0091] Accessing "http: / / www.xxx.com" through the browser module simulates browser behavior and is less likely to trigger the website's anti-crawling mechanism. During the testing phase, we choose a browser with an interface for easier debugging. During the production phase, we switch to a browser without an interface to improve efficiency.

[0092] Through the web page element recognition module, we can identify input boxes, submit buttons, page turn buttons, internal links / external links, etc. Here, we choose to monitor the target product: "safety helmet". We automatically enter "safety helmet" in the input box, and then click the submit button to send it to the website backend and get the returned result page. If the submit button is not found, we try to use the enter button instead.

[0093] Through the illegal behavior identification module, the result page is monitored to see if it contains "safety helmet" product information, and then structured data is extracted through natural language processing technology;

[0094] Monitor whether there is a page-turning button. If so, automatically turn the page. Then follow the steps in step 3 to identify illegal activities and strive to obtain all result pages.

[0095] Monitor the internal and external links of the "http: / / www.xxx.com" page, obtain the website map, traverse the website map, and then iterate according to step 1 until the collection requirements are met.

[0096] Example 2:

[0097] In another aspect, the present invention further proposes an Internet clue identification method based on dynamic interaction, comprising:

[0098] Step 1: Using a preset browser mode, simulate the user's browser surfing behavior to perform dynamic interaction of the browser web page;

[0099] Step 2: extracting interactive elements from the browser webpage, and dynamically interacting with the server of the browser webpage through the interactive elements to identify a website map of the website corresponding to the browser webpage;

[0100] Step 3: Identify the website map to obtain Internet clues of the website.

[0101] Among them, the browser mode includes: no-interface browser mode and interface browser mode;

[0102] Among them, the interfaceless browser mode is deployed in multiple processes on the Linux server;

[0103] Among them, the interface browser mode is used in the testing phase of the Internet clue identification system, and the interfaceless browser mode is used in the application phase of the Internet clue identification system.

[0104] Among them, interactive elements include:

[0105] Web page input box, submit button, page turn button and internal / external links.

[0106] Extracting the interactive elements of the browser webpage and dynamically interacting with the server of the browser webpage through the interactive elements includes:

[0107] Extracting interactive elements of the browser web page through regular expressions and / or natural language processing technology, and traversing the browser web page through the interactive elements, and scanning the server site of the website where the browser web page is located;

[0108] After extracting the interactive elements of the browser web page, determining whether the interactive elements are valid, and if valid, dynamically interacting with the server of the browser web page through the interactive elements;

[0109] Determining whether the interactive element is valid includes:

[0110] Determine whether the interactive element of the browser web page is visible. If visible, determine whether the interactive element has an integrated button and determine whether the integrated button is valid. If there is no integrated button, enter a request and determine whether to respond. If so, it is valid.

[0111] Among them, Internet clues include:

[0112] Clues to prohibited items, sensitive words, illegal advertising and target products.

[0113] The application of the present invention can identify illegal clues within the website and supervise the deep web areas that cannot be reached by existing hyperlink analysis.

[0114] Example 3:

[0115] Based on the same inventive concept, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of the method in the above embodiment.

[0116] Example 4:

[0117] Based on the same inventive concept, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of the method in the above embodiment.

[0118] It will be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0119] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0120] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0122] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0123] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An Internet clue identification system based on dynamic interaction, characterized in that: include: The browser module is used to simulate the user's browser surfing behavior in a preset browser mode to perform dynamic interaction with the browser web page; A web page element recognition module is used to extract the interactive elements of the browser web page and dynamically interact with the server of the browser web page through the interactive elements to identify the website map of the website corresponding to the browser web page; The behavior recognition module is used to identify the website map and obtain Internet clues of the website.

2. The Internet clue identification system according to claim 1, characterized in that: The browser mode includes: a browser mode without an interface and a browser mode with an interface; Among them, the interfaceless browser mode is deployed in multiple processes on the Linux server; Among them, the interface browser mode is used in the testing phase of the Internet clue identification system, and the interfaceless browser mode is used in the application phase of the Internet clue identification system.

3. The Internet clue identification system according to claim 1, characterized in that: The webpage element recognition module extracts the interactive elements of the browser webpage, and dynamically interacts with the server of the browser webpage through the interactive elements, including: Extracting interactive elements of the browser web page through regular expressions and / or natural language processing technology, and traversing the browser web page through the interactive elements, and scanning the server site of the website where the browser web page is located; After extracting the interactive elements of the browser web page, determining whether the interactive elements are valid, and if valid, dynamically interacting with the server of the browser web page through the interactive elements; Determining whether the interactive element is valid includes: Determine whether the interactive element of the browser web page is visible. If visible, determine whether the interactive element has an integrated button and determine whether the integrated button is valid. If there is no integrated button, enter a request and determine whether to respond. If so, it is valid.

4. A method for identifying Internet clues based on dynamic interaction, characterized in that: include: Use the preset browser mode to simulate the user's browser surfing behavior to achieve dynamic interaction of browser web pages; Extracting interactive elements from the browser webpage, and dynamically interacting with the server of the browser webpage through the interactive elements to identify a website map of the website corresponding to the browser webpage; The website map is identified to obtain Internet clues of the website.

5. The Internet clue identification method according to claim 4, characterized in that: The browser mode includes: a browser mode without an interface and a browser mode with an interface; Among them, the interfaceless browser mode is deployed in multiple processes on the Linux server; Among them, the interface browser mode is used in the testing phase of the Internet clue identification system, and the interfaceless browser mode is used in the application phase of the Internet clue identification system.

6. The Internet clue identification method according to claim 4, characterized in that: The interactive elements include: Web page input box, submit button, page turn button and internal / external links.

7. The Internet clue identification method according to claim 4, characterized in that: The extracting the interactive elements of the browser web page and dynamically interacting with the server of the browser web page through the interactive elements includes: Extracting interactive elements of the browser web page through regular expressions and / or natural language processing technology, and traversing the browser web page through the interactive elements, and scanning the server site of the website where the browser web page is located; After extracting the interactive elements of the browser web page, determining whether the interactive elements are valid, and if valid, dynamically interacting with the server of the browser web page through the interactive elements; Determining whether the interactive element is valid includes: Determine whether the interactive element of the browser web page is visible. If visible, determine whether the interactive element has an integrated button and determine whether the integrated button is valid. If there is no integrated button, enter a request and determine whether to respond. If so, it is valid.

8. The Internet clue identification method according to claim 4, characterized in that: The Internet clues include: Clues to prohibited items, sensitive words, illegal advertising and target products.

9. A computer device, characterized in that: include: one or more processors; a processor for executing one or more programs; When the one or more programs are executed by the one or more processors, the method according to any one of claims 4 to 8 is implemented.

10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed, the method according to any one of claims 4 to 8 is implemented.

Images