Method for realizing web crawler based on large front-end technology

Abstract

The invention discloses a method for realizing a web crawler based on a large front-end technology, which comprises the following steps of: firstly, starting a Puppeteer by using a NodeJS to carry out a web crawling service which can simulate the operation of real human beings, click any position of a page, roll the page, drag elements, read page contents, intercept a webpage, intercept an HTTP (Hyper Text Transport Protocol) request and the like, namely, what you see is what you get; on the basis of the characteristic, the development speed which is incomparable to other crawlers can be realized; secondly, user-oriented visual interaction service is carried out, so that the user is helped to control a Puppeteer through the application, and webpage content capturing, webpage screenshot, network request interception and the like are carried out; the data storage service is used for storing related information of a user, so that the user can conveniently check own operation history, quickly initialize an application, quickly switch a crawler context environment and the like, and the optimal user experience is achieved; and finally, the pictures can be analyzed and processed through a TensorFlow deep learning framework, such as character recognition (used for verification code verification), image classification (providing personalized capture strategies for different users) and the like.

Description

Technical Field

[0001] The present invention relates to a design scheme and belongs to the technical field of large front-end development. A method for realizing a network crawler is implemented based on NodeJS and front-end technology. Background Art

[0002] CN201711016509.5 A method for realizing the capture and downloading of pictures based on Nodejs crawler, Guoyun Technology Co., Ltd. - Application date: 2017-10-25 - Main classification number: G06F17 / 30

[0003] Abstract: The present invention relates to the technical field of web page development, and in particular to a method for implementing the crawling and downloading of images based on the nodejs crawler technology. The method is to install Node.js; then implement the request module responsible for responding to network requests, the cheerio module responsible for processing the crawled data, the fs module responsible for downloading images, the path module responsible for formatting the image name, and the async module for processing asynchronous concurrency; encapsulate all method modules into a JS file and execute it, and wait for the retrieval to be completed before viewing the downloaded images. The present invention provides a convenient method for obtaining web page information.

[0004] 201810108285.9 Method and device for identifying web crawlers and extracting web crawler features - Disclosed are a method and device for identifying web crawlers and extracting web crawler features, the method comprising: determining a subset of a user access feature set, the access parameters of the access features in this subset being the same as the access parameters of at least one access feature in the overall business access feature set; determining a comprehensive access feature set based on the subset and the overall business access feature set; obtaining the values ​​of each access feature in a collection of the user access feature set and the comprehensive access feature set of a sample user from historical logs serving as training data, and selecting important access feature sets from the collection based on the values ​​of each access feature in the collection.

[0005] NodeJS is a JavaScript toolkit for writing high-performance network servers. It can quickly build a platform for network services and applications. The platform is built on Chrome's JavaScript runtime, which means that it is actually a package of the Google V8 engine (used in Google Chrome browser). The V8 engine executes Javascript very quickly and has very good performance.

[0006] NodeJS also optimizes some special use cases and provides alternative APIs to make V8 run better in non-browser environments. For example, in a server environment, processing binary data is often essential, but Javascript does not support this enough. Therefore, V8 in NodeJS adds the Buffer class to process binary data conveniently and efficiently. Therefore, NodeJS not only simply uses V8, but also optimizes it to make it better in various environments. Its main features are as follows:

[0007] 1. It is a Javascript runtime environment: it inherits the simplicity of the Javascript language, has a low learning threshold, and enables front-end engineers to get started more quickly.

[0008] 2. Rely on Chrome V8 engine for code interpretation: NodeJS is based on V8 engine, which is an open source JavaScript engine released by Google. V8 engine executes Javascript very fast and has outstanding performance.

[0009] 3. Asynchronous event-driven: The design concept of NodeJS is based on event-driven, which is asynchronous callback. Most of the APIs it provides are event-based and asynchronous. The advantage of event-driven is that it makes full use of system resources. Code execution does not need to be blocked to wait for a certain operation to complete, and limited resources are used for other tasks. The event-driven mechanism is achieved by efficiently maintaining the event loop queue in an internal single thread, without multi-threaded resource occupation and context switching.

[0010] 4. Non-blocking I / O: The advantage of non-blocking I / O is its strong performance. Combined with the above-mentioned asynchronous event-driven, one thread can handle a large number of concurrent connections and can make great use of limited hardware resources.

[0011] 5. Single thread: NodeJS is based on a single thread (main thread). All blockages will be put into a thread pool, and then the main thread will cooperate with the thread pool through a queue, making NodeJS work very efficiently.

[0012] Puppeteer is a Node.js library that provides a high-level API to control Chrome / Chromium through the development tool protocol. In theory, you can use it to do anything a browser can do, such as generating screenshots and PDFs of pages, crawling web page content, network request interception, etc. Its main features are as follows:

[0013] 1. Use NodeJS as the technology stack: front-end developers can quickly get started with development without any extra learning costs.

[0014] 2. Based on DevTools Protocol: DevTools Protocol is a communication protocol that allows Puppeteer to interact with Google Chrome or Chromium browsers. This protocol enables Puppeteer to control the browser by sending specific commands, thereby achieving functions such as automated testing, page rendering, and monitoring network requests.

[0015] 3. Easy to get started: Puppeteer can be easily started with just a few lines of js code, and the official provides many reference cases so you can get started right away.

[0016] PM2 is a daemon management tool for NodeJS applications. It helps users manage and guard applications. It can help us better manage crawler services and make applications run more securely. Its main features are as follows:

[0017] 1. Automatic restart: When the application is terminated unexpectedly, the response application will be automatically restarted.

[0018] 2. Multi-core optimization: When the service runs on a multi-core CPU, multiple instances can be started based on the number of cores to maximize resource utilization.

[0019] 3. Load balancing: When the system is running, users will be evenly distributed according to the load of each core.

[0020] 4.0-second restart: Seamless switching of applications can be achieved when updating applications. Summary of the invention

[0021] The purpose of the present invention is to realize web crawler technology by using big front-end technology, start Puppeteer service through NodeJS, control Chrome / Chromium to execute operation commands such as browsing web pages, intercepting requests, and capturing pictures, and can also customize the execution of certain program steps according to the user's JS script, and can store user information. According to different users, different users can browse their respective access history in the service, and the context environment can be quickly switched when the user uses it.

[0022] The technical solution of the present invention is a method for realizing a web crawler based on big front-end technology, which is mainly divided into three parts: Puppeteer crawler service, user-oriented visual interaction service and data storage service;

[0023] 1) User-oriented visual interactive service: When a user visits the web page started by the target service, the web page first displays the user's historical processing data, such as: intercepted web page images, captured web page content, intercepted HTTP request content, etc.; the second and most important function is to provide users with options for crawling, such as: web page content crawling, web page screenshots, intercepting network requests, and executing user-defined NodeJS code;

[0024] 2) Puppeteer crawler service: When the user operates and sends a crawling command in the visual interface, the crawler service identifies the command content and identifies and executes different crawling logics based on the command content;

[0025] 3) Data storage service: This service is used to store user information, including user information, user historical crawled data, the address of the Puppeteer instance run by the user, etc.

[0026] The main implementation steps are as follows: puppeteer is a powerful web automation tool that can automatically perform tasks and obtain web page data, so it can be used as a crawler.

[0027] Step 1: Initialize the NodeJS application for user visual interaction, develop the main interfaces such as the login interface and the operation interface, and the application is divided into view module, routing module, core service module, HTTP request module, etc.; user access is processed by the routing module to the core service module, and then to the page showing the view module. Finally, the user sends an HTTP request to the Puppeteer crawler service through page operations and waits for a response; this application mainly plays the role of interface display and interaction with the crawler service. The main technologies used are: NodeJS, Express library, ejs template engine, HTML, Javascript, CSS, HTTP, etc.;

[0028] Step 2: Initialize the data storage NodeJS application, which is mainly used for data storage and data query / write. When storing, it stores relevant information according to the user dimension, including: user ID, user historical crawled data and the response returned by the crawler. It also provides interfaces for data query and data writing. The technologies involved are: NodeJS, Express, and mongoDB.

[0029] Step 3: Initialize the Puppeteer crawler NodeJS application, install the puppeteer dependency package, use constbrowser=await puppeteer.launch() to initialize the browser object, use const page=awaitthis.browser.newPage() to create a tab page, at this time the tabId will correspond to this user ID, that is, it will be stored in the database, and then turn on CDP, const cdpSession=await page.target().createCDPSession(), then you can monitor the cdpSession to monitor network requests; if the user sends a command to intercept the web page, then run await at this time page.screenshot({ / / parameters}), pass in the corresponding parameters to take a screenshot and return it to the user; if the user intercepts a request, the cdpSession object needs to be used, cdpSession.on('xxxx',(data)=>{ / / data operation}), where the listener type needs to be passed in. After obtaining the data, the data is processed in the callback and returned to the user; if the user provides a custom JS script, then you only need to put the custom script in the sandbox to execute it, const contextObject={ / / data that the custom script depends on}; vm.createContext(contextObject); / / Pre-context isolation

[0030] const result = vm.runInContext(' / / custom script code', contextObject), at this time the code passed in by the user is executed.

[0031] Step 4: Release and deploy. Release the three services in conjunction with Jenkins and deploy them using PM2, making application operation and maintenance more efficient and stable.

[0032] Step 5: User use: the user visits the page provided by the user-oriented visual interactive service to view his / her own crawling history; or selects the crawling function he / she wants to perform, enters the URL or enters the custom NodeJS code to perform network crawling, and finally displays the response content;

[0033] Through the above steps, a method of implementing crawlers using only big front-end technology without using technologies such as Python was solved, achieving the goal of web crawling without professional crawler developers and the Python technology system.

[0034] Node.js is a JavaScript runtime environment based on the Chrome V8 engine. It uses an event-driven, non-blocking I / O model to allow JavaScript to run on the server-side development platform.

[0035] Beneficial effects: Front-end developers have a natural advantage in web-related technologies. In the past, they were limited by the weakness of front-end technologies and had little exposure to crawler-related technologies. Now there is a method to implement web crawlers based on big front-end technology, making it possible for front-end developers to develop crawler applications, reducing the employment costs of companies with insufficient talent and expanding the boundaries of front-end development.

[0036] On the basis of existing crawler technology, the present invention can also integrate advanced image recognition technology, and use deep learning frameworks such as TensorFlow.js or similar image processing libraries to realize automatic recognition and processing of image content in web pages. This function not only enhances the crawler's ability to understand static images, but also can handle dynamically loaded images or complex image processing tasks with ease. By combining machine learning models, the crawler of the present invention can intelligently parse image information and provide more dimensional support for subsequent data analysis.

[0037] In order to meet the needs of real-time data analysis, the function of real-time data stream processing can also be added to the data storage service. By using the stream module of Node.js, the present invention realizes real-time cleaning and conversion of data, ensuring data quality while improving the efficiency of data processing. This enables the crawler of the present invention to not only efficiently capture data, but also to perform quality control and preliminary analysis on the captured data in real time.

[0038] Considering the development trend of modern Web applications, the crawler can also strengthen its support for single-page applications (SPA). By supporting technologies such as AJAX and WebSocket, the crawler of the present invention can effectively capture and process dynamic content in such applications, thereby broadening the application scope of the crawler, improving the versatility of applications, and improving the development efficiency of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The overall architecture diagram for crawler services.

[0040] Figure 2 A multi-instance NodeJS application started by the PM2 daemon, each with a Chrome / Chromium instance controlled by Puppeteer. DETAILED DESCRIPTION

[0041] For example, on an encyclopedia question-and-answer website Z, a user logs in to an account on the Z website to ask a question. After the question is sent, the Z website answers the question and displays it. At this time, the user wants to use the front-end crawler service to use the website Z. The steps are as follows:

[0042] Step 1: Develop a visual interactive web application A that meets the requirements, with a question input box and an answer display area. Use HTTP requests to send questions, and the crawler service will fetch the answers and return them to A for display. Of course, you can also choose WebSocket for network interaction to achieve the purpose of interaction.

[0043] Step 2: Develop storage application C for persistent service of user information;

[0044] Step 3: Develop Puppeteer crawler application B according to step 3 in the invention content. At this time, there is no need to retain multiple commands, and you only need to concentrate on developing programs suitable for website Z; the solution is as follows: 1. When a user accesses application B, the user will be assigned to a specific NodeJS service by PM2, such as Figure 2 , at this time, the id of the current NodeJS service instance is stored in the database, and the Z website is opened for user login; second, after login, the instance tab ID of Chrome / Chromium started by Puppeteer at this time and the page object under the tab are stored to enter the question and answer page; third, after the user knows that the login is successful, he uses application A to send a question, and application B obtains the question, focuses the mouse on the input box, enters the user's question, and clicks send. At this time, CDP is used to monitor the request and send it, and the request ID_1 is remembered, waiting for the Z application to respond; 4. When the Z application responds, CDP is used to monitor the received request, obtain the request ID_2, and compare the request ID_1 and ID_2. If they are equal, the data in the request is directly returned to application A for display to the user, and the data is saved to service C for displaying historical visits the next time the user visits; the overall steps are as follows Figure 1 As shown. Steps to start and use services A, B, and C:

[0045] Step 1: Service A starts the storage service to provide other applications with data acquisition and writing persistence capabilities;

[0046] Step 2: Service B uses PM2 to start the Puppeteer crawler service cluster, initializes the service based on the data of the storage service, and responds to user operation commands;

[0047] Step 3: Service C starts a visual interactive service for users, which includes optional crawling function options and provides different operations for different options. For example, when the crawling web content option is selected, a web address input box will be provided; when the custom script option is selected, a code input box will be displayed, and users can enter NodeJS code to customize the operation;

[0048] Step 4: The user visits the website provided by service C to check his / her usage record, or selects the function he / she needs to use to crawl the network.

[0049] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for implementing a web crawler based on big front-end technology, characterized in that: It has Puppeteer crawler service module, user-oriented visual interaction service and data storage service; 1) Puppeteer crawler service: In the Node environment, the daemon PM2 is used to start the corresponding number of Puppeteer instances according to the load requirements and machine performance, and different browsers are started according to the environment. The production environment enables headless browsers to save server resources; the development environment starts the head mode to facilitate development; CDPSession is started at startup to obtain network request data; this service crawls different resources and captures pictures according to different usage requirements; 2) User-oriented visual interactive service: Users can select corresponding crawling modes according to different usage scenarios, such as obtaining web content and capturing web images. This service also allows users to upload custom nodejs scripts to control Puppeteer to crawl. 3) Data storage service: used to store various user data information, including the user's access history, the context of the captured web pages, the Puppeteer instance used by the user, and the tab information in the instance; The steps to use are as follows: Step 1: Initialize the NodeJS application for user visual interaction, develop the main interfaces such as the login interface and the operation interface, and the application is divided into view module, routing module, core service module, HTTP request module, etc.; user access is processed by the routing module to the core service module, and then to the page showing the view module. Finally, the user sends an HTTP request to the Puppeteer crawler service through page operations and waits for a response; this application mainly plays the role of interface display and interaction with the crawler service. The main technologies used are: NodeJS, Express library, ejs template engine, HTML, Javascript, CSS, HTTP, etc.; Step 2: Initialize the data storage NodeJS application, which is mainly used for data storage and data query / write. When storing, it stores relevant information according to the user dimension, including: user ID, user historical crawled data and the response returned by the crawler. It also provides interfaces for data query and data writing. The technologies involved are: NodeJS, Express, and mongoDB. Step 3: Initialize the Puppeteer crawler NodeJS application, install the puppeteer dependency package, use const browser = await puppeteer.launch() to initialize the browser object, use const page = awaitthis.browser.newPage() to create a tab page, at this time the tabId will correspond to this user ID, that is, it will be stored in the database, and then turn on CDP, const cdpSession = await page.target().createCDPSession(), then you can monitor the cdpSession to monitor network requests; if the user sends a command to intercept the web page, then run await at this time page.screenshot({ / / parameters}), pass in the corresponding parameters to take a screenshot and return it to the user; if the user intercepts a request, the cdpSession object needs to be used, cdpSession.on('xxxx',(data)=>{ / / data operation}), where the listener type needs to be passed in. After obtaining the data, the data is processed in the callback and returned to the user; if the user provides a custom JS script, then you only need to put the custom script in the sandbox to execute it, const contextObject={ / / data that the custom script depends on}; vm.createContext(contextObject); / / Pre-context isolation const result = vm.runInContext(' / / custom script code', contextObject), at this time the code passed in by the user is executed. Step 4: Release and deploy. Release the three services in conjunction with Jenkins and deploy them using PM2, making application operation and maintenance more efficient and stable. Step 5: User use: The user visits the page provided by the user-oriented visual interactive service to view his or her own crawling history; or selects the crawling function he or she wants to perform, enters the URL or enters the custom NodeJS code to perform network crawling, and finally displays the response content.

2. The method for implementing a web crawler based on big front-end technology according to claim 1 is characterized in that: After the question is sent to the Z website, use the front-end crawler service to use the website Z. The steps are as follows: Step 1: Develop a visual interactive web application service A that meets the requirements, with a question input box and an answer display area, use HTTP requests to send questions, and the crawler service will return the answers to A for display after crawling them; or choose WebSocket for network interaction to achieve the interactive purpose; Step 2: Develop a storage application service C, which is used for persistent services of various user information; Step 3: Develop a Puppeteer crawler application service B in accordance with step 3 in the invention content. At this time, there is no need to retain multiple commands, and you only need to concentrate on developing programs suitable for website Z; the solution is as follows: 1) When a user accesses application B, the user will be assigned to a specific NodeJS service by PM2. At this time, the id of the current NodeJS service instance is stored and stored in the database, and Z is opened 1) The website logs in the user; 2) After logging in, the instance tab ID of Chrome / Chromium started by Puppeteer at this time and the page object under the tab are stored to enter the question and answer page; 3) After the user knows that the login is successful, he uses the service A application to send a question. The service B application obtains the question, focuses the mouse on the input box, enters the user's question, and clicks send. At this time, the CDP is used to listen for the request to be sent, and the request ID_1 is remembered, waiting for the Z application to respond; 4) When the Z application responds, the CDP is used to listen to the received request, obtain the request ID_2, compare the request ID_1 and ID_2, if they are equal, directly return the data in the request to application A for display to the user, and save the data in service C for displaying historical visits the next time the application is visited.

3. The method for implementing a web crawler based on big front-end technology according to claim 3 is characterized in that: Step 1: Service A starts the storage service to provide other applications with data acquisition and writing persistence capabilities; Step 2: Service B uses PM2 to start the Puppeteer crawler service cluster, initializes the service based on the data of the storage service, and responds to user operation commands; Step 3: Service C starts a visual interactive service for users, which includes optional crawling function options and provides different operations for different options. For example, when the crawling web content option is selected, a web address input box will be provided; when the custom script option is selected, a code input box will be displayed, and users can enter NodeJS code to customize the operation; Step 4: The user visits the website provided by service C to check his / her usage record, or selects the function he / she needs to use to crawl the network.

Images