Deployment method, device, equipment and storage medium of whole network data monitoring system

Abstract

The present application discloses a deployment method, device, equipment and storage medium of a network-wide data monitoring system. The system uses Docker to package a Scrapyd code image, and pushes the Scrapyd code image to a Kubernetes environment for deployment to complete the Scrapyd containerized deployment; uses Docker to package a Spiderkeeper service image, and pushes the image to a Kubernetes environment for deployment to complete the Spiderkeeper containerized deployment; based on the deployed Scrapyd container and Spiderkeeper container, a new security authentication mechanism is added according to the Spiderkeeper source code; based on the security authentication mechanism, Spiderkeeper is logged in to create a crawler scheduled scheduling task for collecting network-wide data. The high efficiency, security and stability of the data collection service are achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a deployment method, device, equipment and storage medium for a full-network data monitoring system. Background Art

[0002] Social hot issues have always attracted the attention of the majority of netizens. In the era of new media and big data, the transmission path and speed of hot information have increased exponentially with the support of major Internet platforms and big data technologies. In order to ensure the timeliness and authority of information release, higher requirements are placed on the efficiency, security and stability of data collection services.

[0003] At present, data collection and monitoring of hot spots in the entire network are mainly carried out through crawlers. However, the traditional crawler framework has a large amount of code and high crawler scheduling costs, and cannot monitor data stably and efficiently, making it difficult to cope with the high-frequency big data of the entire network. Summary of the invention

[0004] The present application provides a deployment method, device, equipment and storage medium for a full-network data monitoring system to solve the technical problem that the current data monitoring system is difficult to meet the requirements of efficiency, security and stability of data collection services.

[0005] In order to solve the above technical problems, in a first aspect, the present application provides a deployment method of a network-wide data monitoring system, comprising:

[0006] Use Docker to package the Scrapyd code image, and push the Scrapyd code image to the Kubernetes environment for deployment to complete the Scrapyd containerized deployment;

[0007] Use Docker to package the Spiderkeeper service image, and push the Spiderkeeper service image to the Kubernetes environment for deployment to complete the Spiderkeeper container deployment;

[0008] Based on the deployed Scrapyd container and Spiderkeeper container, and according to the Spiderkeeper source code, a new security authentication mechanism is added to verify the identity credentials between the Spiderkeeper backend and the Scrapyd backend;

[0009] Log in to Spiderkeeper based on the security authentication mechanism to create a crawler scheduled task for collecting data from the entire network.

[0010] In some implementations, packaging the Scrapyd code image using Docker and pushing the Scrapyd code image to a Kubernetes environment for deployment includes:

[0011] Configure the first configuration file of Jenkinsfile, the first configuration file of Dockerfile, and the first configuration file of deployment.yaml;

[0012] Upload the Scrapyd code, the first configuration file of Jenkinsfile, the first configuration file of Dockerfile, and the first configuration file of deployment.yaml to the GitLab code management platform to complete the code synchronization update;

[0013] Pull the latest Scrapyd code from the GitLab code management platform through Jenkins, and generate the Scrapyd code image by combining the first configuration file of Jenkinsfile and the first configuration file of Dockerfile;

[0014] The Scrapyd code image is pushed to the Kunbernets environment through Jenkins, and the project is deployed in combination with the first configuration file deployment.yaml.

[0015] In some implementations, the Spiderkeeper service image is packaged using Docker, and the Spiderkeeper service image is pushed to a Kubernetes environment for deployment to complete the Spiderkeeper containerized deployment:

[0016] Configure the second configuration file of Jenkinsfile, the second configuration file of Dockerfile, and the second configuration file of deployment.yaml;

[0017] Upload the Spiderkeeper code, Jenkinsfile second configuration file, Dockerfile second configuration file, and deployment.yaml second configuration file to the GitLab code management platform to complete the code synchronization update;

[0018] Pull the latest Spiderkeeper code from the GitLab code management platform through Jenkins, and generate the Spiderkeeper service image in combination with the second configuration file of Jenkinsfile and the second configuration file of Dockerfile;

[0019] The Spiderkeeper service image is pushed to the Kunbernets environment through Jenkins, and the project is deployed in combination with the second configuration file deployment.yaml.

[0020] In some implementations, based on the deployed Scrapyd container and Spiderkeeper container, according to the Spiderkeeper source code, a new security authentication mechanism for verifying identity credentials in the Spiderkeeper backend and the Scrapyd backend is added, including:

[0021] Download Scrapyd dependencies through pip command;

[0022] Set the username and password for Scrapyd service security authentication in the scrapyd.conf configuration file;

[0023] Log in to GitHub and download the Spiderkeeper project source code;

[0024] Add user authentication information to the requests code in the Spiderkeeper project source code. The user authentication information is the username and password in the scrapyd.conf configuration file.

[0025] In some implementations, logging into Spiderkeeper based on the security authentication mechanism to create a crawler scheduled task for collecting data from the entire network includes:

[0026] Based on the security authentication mechanism, log in to the Spiderkeeper user interface;

[0027] Create a crawler project in the Manage interface of the Spiderkeeper user interface;

[0028] Based on the preset command, package the crawler project, and upload the packaged crawler project to the Spiderkeeper service through the Deploy interface;

[0029] Add a crawler task on the Periodic Jobs page of the Spiderkeeper service and configure the scheduling period of the crawler task on the scheduler.

[0030] In a second aspect, the present application also provides a deployment device for a network-wide data monitoring system, including:

[0031] The first deployment module is used to package the Scrapyd code image using Docker, and push the Scrapyd code image to the Kubernetes environment for deployment to complete the Scrapyd containerized deployment;

[0032] The second deployment module is used to package the Spiderkeeper service image using Docker, and push the Spiderkeeper service image to the Kubernetes environment for deployment to complete the Spiderkeeper containerized deployment;

[0033] Added a new module for deploying Scrapyd and Spiderkeeper containers. Based on the Spiderkeeper source code, a new security authentication mechanism is added to verify the identity credentials between the Spiderkeeper backend and the Scrapyd backend.

[0034] A creation module is used to log in to Spiderkeeper based on the security authentication mechanism to create a crawler scheduled task for collecting data from the entire network.

[0035] In some implementations, the newly added module is specifically used to:

[0036] Download Scrapyd dependencies through pip command;

[0037] Set the username and password for Scrapyd service security authentication in the scrapyd.conf configuration file;

[0038] Log in to GitHub and download the Spiderkeeper project source code;

[0039] Add user authentication information to the requests code in the Spiderkeeper project source code. The user authentication information is the username and password in the scrapyd.conf configuration file.

[0040] In some implementations, the creation module is specifically used to:

[0041] Based on the security authentication mechanism, log in to the Spiderkeeper user interface;

[0042] Create a crawler project in the Manage interface of the Spiderkeeper user interface;

[0043] Based on the preset command, package the crawler project, and upload the packaged crawler project to the Spiderkeeper service through the Deploy interface;

[0044] Add a crawler task on the Periodic Jobs page of the Spiderkeeper service and configure the scheduling period of the crawler task on the scheduler.

[0045] In a third aspect, the present application also provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the deployment method of the full-network data monitoring system as described in the first aspect is implemented.

[0046] In a fourth aspect, the present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the deployment method of the full-network data monitoring system as described in the first aspect.

[0047] Compared with the prior art, this application has at least the following beneficial effects:

[0048] By using Docker to package the Scrapyd code image, and pushing the Scrapyd code image to the Kubernetes environment for deployment, the Scrapyd containerized deployment is completed; the Spiderkeeper service image is packaged by Docker, and the image is pushed to the Kubernetes environment for deployment to complete the Spiderkeeper containerized deployment; based on the deployed Scrapyd container and Spiderkeeper container, according to the Spiderkeeper source code, a new security authentication mechanism for verifying identity credentials in the Spiderkeeper background and the Scrapyd background is added; based on the security authentication mechanism, Spiderkeeper is logged in to create a crawler scheduled scheduling task for collecting data from the entire network. The code volume of the Scrapy crawler framework is only one-third of the original, which realizes the agile development of data collection, which is conducive to the rapid response of product customization development and iterative updates; Scrapyd+Spiderkeeper provides interface scheduling management and security authentication mechanism for crawler tasks, reduces the maintenance cost of crawler scheduling, and strengthens the security of crawler services. In addition, the project supports containerized deployment, and can support one-click automatic deployment in the Kubernets environment to achieve integrated management of project launch, resource allocation, task scheduling, and background monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a deployment method of a network-wide data monitoring system according to an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the structure of a network-wide data monitoring system according to an embodiment of the present application;

[0051] Figure 3 A schematic diagram of the process of containerized deployment shown in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the data collection security authentication process shown in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of a deployment device for a network-wide data monitoring system according to an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the structure of a computer device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] Please refer to Figure 1 , Figure 1 The flowchart of a distance protection multi-setting zone setting method provided in the embodiment of the present application is shown in FIG. The distance protection multi-setting zone setting method in the embodiment of the present application can be applied to computer devices, including but not limited to smart phones, laptops, tablet computers, desktop computers, physical servers, cloud servers and other devices. Figure 1 As shown, the distance protection multi-setting zone setting method of this embodiment includes steps S101 to S104, which are described in detail as follows:

[0057] Step S101, using Docker to package the Scrapyd code image, and pushing the Scrapyd code image to the Kubernetes environment for deployment, so as to complete the Scrapyd containerized deployment;

[0058] Step S102, using Docker to package the Spiderkeeper service image, and pushing the image to the Kubernetes environment for deployment, so as to complete the Spiderkeeper container deployment;

[0059] Step S103, based on the deployed Scrapyd container and Spiderkeeper container, according to the Spiderkeeper source code, a new security authentication mechanism is added for verifying identity credentials in the Spiderkeeper backend and the Scrapyd backend;

[0060] Step S104: Log in to Spiderkeeper based on the security authentication mechanism to create a crawler scheduled task for collecting data from the entire network.

[0061] In this embodiment, the whole network data monitoring system is as follows: Figure 2 As shown. For the collection of hot data from the entire network, automatic monitoring of hot search topics of users across the network is realized. Among them, the data collection project uses Scrapy, Scrapyd, and Spiderkeeper as the underlying architecture: Compared with traditional crawlers, the amount of code using the Scrapy crawler framework is only one-third of the original, which realizes agile development of data collection, which is conducive to customized product development and rapid response to iterative updates; Scrapyd+Spiderkeeper provides an interface-based scheduling management and security authentication mechanism for crawler tasks, reducing the maintenance cost of crawler scheduling, while strengthening the security of crawler services. On the other hand, the project supports containerized deployment, and can support one-click automatic deployment in the Kubernets environment, realizing the integrated management of project launch, resource allocation, task scheduling, and background monitoring.

[0062] In some embodiments, the step S101 includes:

[0063] Configure the first configuration file of Jenkinsfile, the first configuration file of Dockerfile, and the first configuration file of deployment.yaml;

[0064] Upload the Scrapyd code, the first configuration file of Jenkinsfile, the first configuration file of Dockerfile, and the first configuration file of deployment.yaml to the GitLab code management platform to complete the code synchronization update;

[0065] Pull the latest Scrapyd code from the GitLab code management platform through Jenkins, and generate the Scrapyd code image by combining the first configuration file of Jenkinsfile and the first configuration file of Dockerfile;

[0066] The Scrapyd code image is pushed to the Kunbernets environment through Jenkins, and the project is deployed in combination with the first configuration file deployment.yaml.

[0067] In this embodiment, if Figure 3As shown, configure the Jenkinsfile configuration file, Dockerfile configuration file, and deployment.yaml configuration file: Jenkinsfile configuration file: divided into three stages: Stage 1 configures the Docker image information (including: warehouse address, project name, environment variables, image tag, etc.), Stage 2 configures the build command of the Docker image, and Stage 3 configures the Docker image deployment command to Kubernetes; Dockerfile configuration file: configures the internal information of the Docker image, including: Python version, project directory, download method of third-party dependencies, time zone setting, project port, startup command, etc.; deployment.yaml configuration file: configures relevant information in the Kubernetes environment, including: namespace, image name, environment variables, mount path, project port, etc. Upload Scrapyd code, Jenkinsfile, Dockerfile, and deployment.yaml to the GitLab code management platform to complete the code update synchronization operation. Pull the latest code from the GitLab code management platform through Jenkins, and combine the configuration information of Jenkinsfile with the Dockerfile image configuration file to achieve one-click image generation. After the image is built, push the image to the Kunbernets environment through Jenkins, and deploy the project in combination with the deployment.yaml configuration file.

[0068] In some embodiments, the step S102 includes:

[0069] Configure the second configuration file of Jenkinsfile, the second configuration file of Dockerfile, and the second configuration file of deployment.yaml;

[0070] Upload the Spiderkeeper code, Jenkinsfile second configuration file, Dockerfile second configuration file, and deployment.yaml second configuration file to the GitLab code management platform to complete the code synchronization update;

[0071] Pull the latest Spiderkeeper code from the GitLab code management platform through Jenkins, and generate the Spiderkeeper service image in combination with the second configuration file of Jenkinsfile and the second configuration file of Dockerfile;

[0072] The Spiderkeeper service image is pushed to the Kunbernets environment through Jenkins, and the project is deployed in combination with the second configuration file deployment.yaml.

[0073] In this embodiment, the Jenkinsfile configuration file, Dockerfile configuration file, and deployment.yaml configuration file are configured: Jenkinsfile configuration file: divided into three stages: Stage 1 configures Docker image information (including: warehouse address, project name, environment variables, image tag, etc.), Stage 2 configures the build command of the Docker image, and Stage 3 configures the Docker image deployment command to Kubernetes; Dockerfile configuration file: configures the internal information of the Docker image, including: Python version, project directory, download method of third-party dependencies, time zone setting, project port, startup command, etc.; deployment.yaml configuration file: configures relevant information under the Kubernetes environment, including: namespace, image name, environment variables, mount path, project port, etc. Upload Spiderkeeper code, Jenkinsfile, Dockerfile, and deployment.yaml to the GitLab code management platform to complete the code update synchronization operation. Pull the latest code from the GitLab code management platform through Jenkins, and combine the configuration information of Jenkinfile with the Dockerfile image configuration file to achieve one-click generation of the image. After the image is built, push the image to the Kunbernets environment through Jenkins, and deploy the project in combination with the deployment.yaml configuration file.

[0074] In some embodiments, the step S103 includes:

[0075] Download Scrapyd dependencies through pip command;

[0076] Set the username and password for Scrapyd service security authentication in the scrapyd.conf configuration file;

[0077] Log in to GitHub and download the Spiderkeeper project source code;

[0078] Add user authentication information to the requests code in the Spiderkeeper project source code. The user authentication information is the username and password in the scrapyd.conf configuration file.

[0079] In this embodiment, the Scrapyd dependency is downloaded through the pip command. The old version of Scrapyd does not support the user authentication function. The current Scrapyd-1.3.0 version supports this function. Set the username and password for the Scrapyd service security authentication in the scrapyd.conf configuration file. Log in to the GitHub website and download the Spiderkeeper project source code. Based on the Spiderkeeper source code, modify the http.py code in the util directory in the source code, and add user authentication to the code involving the requests part. The authenticated username and password are the username and password in scrapyd.conf.

[0080] In some embodiments, the step S104 includes:

[0081] Based on the security authentication mechanism, log in to the Spiderkeeper user interface;

[0082] Create a crawler project in the Manage interface of the Spiderkeeper user interface;

[0083] Based on the preset command, package the crawler project, and upload the packaged crawler project to the Spiderkeeper service through the Deploy interface;

[0084] Add a crawler task on the Periodic Jobs page of the Spiderkeeper service and configure the scheduling period of the crawler task on the scheduler.

[0085] In this embodiment, log in to the Spiderkeeper user interface and enter the user name and password; create a crawler project in the Manage interface; package the crawler project through the scrapyd-deploy-build-egg output.egg command, and upload the output.egg package to the Spiderkeeper service through the Deploy interface; add a crawler task in the Periodic Jobs page and configure the task scheduling period on the scheduler. The data collection security authentication process is as follows Figure 4 shown.

[0086] In order to implement the deployment method of the network-wide data monitoring system corresponding to the above method embodiment, to achieve the corresponding functions and technical effects. Figure 5 , Figure 5 The structural block diagram of a deployment device of a network-wide data monitoring system provided by an embodiment of the present application is shown. For ease of description, only the parts related to the present embodiment are shown. The deployment device of the network-wide data monitoring system provided by an embodiment of the present application includes:

[0087] The first deployment module 501 is used to package the Scrapyd code image using Docker, and push the Scrapyd code image to the Kubernetes environment for deployment, so as to complete the Scrapyd containerized deployment;

[0088] The second deployment module 502 is used to package the Spiderkeeper service image using Docker and push the image to the Kubernetes environment for deployment to complete the Spiderkeeper container deployment;

[0089] Added module 503, which is used to add a security authentication mechanism for verifying identity credentials between the Spiderkeeper backend and the Scrapyd backend based on the deployed Scrapyd container and Spiderkeeper container and the Spiderkeeper source code;

[0090] The creation module 504 is used to log in to Spiderkeeper based on the security authentication mechanism to create a crawler scheduled task for collecting data from the entire network.

[0091] In some embodiments, the first deployment module 501 is specifically used to:

[0092] Configure the first configuration file of Jenkinsfile, the first configuration file of Dockerfile, and the first configuration file of deployment.yaml;

[0093] Upload the Scrapyd code, the first configuration file of Jenkinsfile, the first configuration file of Dockerfile, and the first configuration file of deployment.yaml to the GitLab code management platform to complete the code synchronization update;

[0094] Pull the latest Scrapyd code from the GitLab code management platform through Jenkins, and generate the Scrapyd code image by combining the first configuration file of Jenkinsfile and the first configuration file of Dockerfile;

[0095] The Scrapyd code image is pushed to the Kunbernets environment through Jenkins, and the project is deployed in combination with the first configuration file deployment.yaml.

[0096] In some embodiments, the second deployment module 502 is specifically configured to:

[0097] Configure the second configuration file of Jenkinsfile, the second configuration file of Dockerfile, and the second configuration file of deployment.yaml;

[0098] Upload the Spiderkeeper code, Jenkinsfile second configuration file, Dockerfile second configuration file, and deployment.yaml second configuration file to the GitLab code management platform to complete the code synchronization update;

[0099] Pull the latest Spiderkeeper code from the GitLab code management platform through Jenkins, and generate the Spiderkeeper service image in combination with the second configuration file of Jenkinsfile and the second configuration file of Dockerfile;

[0100] The Spiderkeeper service image is pushed to the Kunbernets environment through Jenkins, and the project is deployed in combination with the second configuration file deployment.yaml.

[0101] In some embodiments, the newly added module 503 is specifically used to:

[0102] Download Scrapyd dependencies through pip command;

[0103] Set the username and password for Scrapyd service security authentication in the scrapyd.conf configuration file;

[0104] Log in to GitHub and download the Spiderkeeper project source code;

[0105] Add user authentication information to the requests code in the Spiderkeeper project source code. The user authentication information is the username and password in the scrapyd.conf configuration file.

[0106] In some embodiments, the creation module 504 is specifically used to:

[0107] Based on the security authentication mechanism, log in to the Spiderkeeper user interface;

[0108] Create a crawler project in the Manage interface of the Spiderkeeper user interface;

[0109] Based on the preset command, package the crawler project, and upload the packaged crawler project to the Spiderkeeper service through the Deploy interface;

[0110] Add a crawler task on the Periodic Jobs page of the Spiderkeeper service and configure the scheduling period of the crawler task on the scheduler.

[0111] The deployment device of the above-mentioned whole network data monitoring system can implement the deployment method of the whole network data monitoring system of the above-mentioned method embodiment. The optional items in the above-mentioned method embodiment are also applicable to this embodiment and will not be described in detail here. The rest of the contents of the embodiment of this application can refer to the contents of the above-mentioned method embodiment, and will not be repeated in this embodiment.

[0112] Figure 6 This is a schematic diagram of the structure of a computer device provided in one embodiment of the present application. Figure 6 As shown, the computer device 6 of this embodiment includes: at least one processor 60 ( Figure 6 Only one is shown), a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, wherein the processor 60 implements the steps of any of the above method embodiments when executing the computer program 62.

[0113] The computer device 6 may be a computing device such as a smart phone, a tablet computer, a desktop computer, a cloud server, etc. The computer device may include but is not limited to a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 6 It is only an example of the computer device 6 and does not constitute a limitation on the computer device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0114] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0115] In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the computer device 6. The memory 61 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0116] In addition, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0117] An embodiment of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device implements the steps in the above-mentioned method embodiments when executing the computer device.

[0118] In several embodiments provided in the present application, it is understood that each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved.

[0119] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0120] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A method for deploying a network-wide data monitoring system, characterized in that: include: Configure the first Jenkinsfile configuration file, the first Dockerfile configuration file, and the first deployment.yaml configuration file; the first Jenkinsfile configuration file includes: Docker image information, Docker image build commands, and Docker image deployment to Kubernetes commands; Upload the Scrapyd code, the first configuration file of Jenkinsfile, the first configuration file of Dockerfile, and the first configuration file of deployment.yaml to the GitLab code management platform to complete the code synchronization update; Pull the latest Scrapyd code from the GitLab code management platform through Jenkins, and generate the Scrapyd code image by combining the first configuration file of Jenkinsfile and the first configuration file of Dockerfile; Push the Scrapyd code image to the Kunbernets environment through Jenkins, and deploy the project in combination with the first configuration file deployment.yaml to complete the Scrapyd container deployment; Configure the second configuration file of Jenkinsfile, the second configuration file of Dockerfile, and the second configuration file of deployment.yaml; the second configuration file of Jenkinsfile includes: Docker image information, the command for building the Docker image, and the command for deploying the Docker image to Kubernetes; Upload the Spiderkeeper code, Jenkinsfile second configuration file, Dockerfile second configuration file, and deployment.yaml second configuration file to the GitLab code management platform to complete the code synchronization update; Pull the latest Spiderkeeper code from the GitLab code management platform through Jenkins, and generate a Spiderkeeper service image by combining the second configuration file of Jenkinsfile and the second configuration file of Dockerfile; Push the Spiderkeeper service image to the Kunbernets environment through Jenkins, and deploy the project in combination with the second configuration file of deployment.yaml to complete the containerized deployment of Spiderkeeper; Based on the deployed Scrapyd container and Spiderkeeper container, and according to the Spiderkeeper source code, a new security authentication mechanism is added to verify the identity credentials between the Spiderkeeper backend and the Scrapyd backend; Log in to Spiderkeeper based on the security authentication mechanism to create a crawler scheduled task for collecting data from the entire network.

2. The method for deploying a network-wide data monitoring system according to claim 1, characterized in that: Based on the deployed Scrapyd container and Spiderkeeper container, according to the Spiderkeeper source code, a new security authentication mechanism is added to verify the identity credentials in the Spiderkeeper background and the Scrapyd background, including: Download Scrapyd dependencies through pip command; Set the username and password for Scrapyd service security authentication in the scrapyd.conf configuration file; Log in to GitHub and download the Spiderkeeper project source code; Add user authentication information to the requests code in the Spiderkeeper project source code. The user authentication information is the username and password in the scrapyd.conf configuration file.

3. The deployment method of the whole network data monitoring system according to claim 1, characterized in that: The step of logging into Spiderkeeper based on the security authentication mechanism to create a crawler scheduled task for collecting data from the entire network includes: Based on the security authentication mechanism, log in to the Spiderkeeper user interface; Create a crawler project in the Manage interface of the Spiderkeeper user interface; Based on the preset command, package the crawler project, and upload the packaged crawler project to the Spiderkeeper service through the Deploy interface; Add a crawler task on the Periodic Jobs page of the Spiderkeeper service and configure the scheduling period of the crawler task on the scheduler.

4. A deployment device for a network-wide data monitoring system, characterized in that: include: The first deployment module is used to configure the first Jenkinsfile configuration file, the first Dockerfile configuration file, and the deployment.yaml configuration file; wherein the first Jenkinsfile configuration file includes: Docker image information, a command for building the Docker image, and a command for deploying the Docker image to Kubernetes; Upload the Scrapyd code, the first configuration file of Jenkinsfile, the first configuration file of Dockerfile, and the first configuration file of deployment.yaml to the GitLab code management platform to complete the code synchronization update; Pull the latest Scrapyd code from the GitLab code management platform through Jenkins, and generate the Scrapyd code image by combining the first configuration file of Jenkinsfile and the first configuration file of Dockerfile; Push the Scrapyd code image to the Kunbernets environment through Jenkins, and deploy the project in combination with the first configuration file deployment.yaml to complete the Scrapyd container deployment; The second deployment module is used to configure the second Jenkinsfile configuration file, the second Dockerfile configuration file, and the deployment.yaml configuration file; wherein the second Jenkinsfile configuration file includes: Docker image information, a command for building the Docker image, and a command for deploying the Docker image to Kubernetes; Upload the Spiderkeeper code, Jenkinsfile second configuration file, Dockerfile second configuration file, and deployment.yaml second configuration file to the GitLab code management platform to complete the code synchronization update; Pull the latest Spiderkeeper code from the GitLab code management platform through Jenkins, and generate a Spiderkeeper service image by combining the second configuration file of Jenkinsfile and the second configuration file of Dockerfile; Push the Spiderkeeper service image to the Kunbernets environment through Jenkins, and deploy the project in combination with the second configuration file of deployment.yaml to complete the containerized deployment of Spiderkeeper; Added a new module for deploying Scrapyd and Spiderkeeper containers. Based on the Spiderkeeper source code, a new security authentication mechanism is added to verify the identity credentials between the Spiderkeeper backend and the Scrapyd backend. A creation module is used to log in to Spiderkeeper based on the security authentication mechanism to create a crawler scheduled task for collecting data from the entire network.

5. The deployment device of the whole network data monitoring system according to claim 4, characterized in that: The newly added module is specifically used for: Download Scrapyd dependencies through pip command; Set the username and password for Scrapyd service security authentication in the scrapyd.conf configuration file; Log in to GitHub and download the Spiderkeeper project source code; Add user authentication information to the requests code in the Spiderkeeper project source code. The user authentication information is the username and password in the scrapyd.conf configuration file.

6. The deployment device of the whole network data monitoring system according to claim 5, characterized in that: The creation module is specifically used for: Based on the security authentication mechanism, log in to the Spiderkeeper user interface; Create a crawler project in the Manage interface of the Spiderkeeper user interface; Based on the preset command, package the crawler project, and upload the packaged crawler project to the Spiderkeeper service through the Deploy interface; Add a crawler task on the Periodic Jobs page of the Spiderkeeper service and configure the scheduling period of the crawler task on the scheduler.

7. A computer device, characterized in that: It includes a processor and a memory, the memory is used to store a computer program, and when the computer program is executed by the processor, it implements the deployment method of the whole network data monitoring system as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the deployment method of the full-network data monitoring system as described in any one of claims 1 to 3.

Images