A network detection system and method

Abstract

The present application provides a network detection system and method, which relates to the field of Internet technology. The detection system includes a user end, a central control end, an instruction distribution server end and a detection node, wherein the central control end responds to the detection task requirement instruction sent by the user end, and determines at least one target detection node for performing the detection task based on the task information of the detection task required by the user end in the detection task requirement instruction; the central control end generates a detection task execution instruction and sends it to the instruction distribution server end; the instruction distribution server end adds the detection task indicated by the detection task execution instruction to the task queue corresponding to the target detection node; the detection node sends a detection task acquisition instruction to the instruction distribution server end to obtain the detection task to be performed by the detection node from the task queue corresponding to the detection node, so as to improve the resource utilization rate of the detection system and its adaptability to complex environments.

Description

Technical Field

[0001] The present application relates to the technical field of network detection, and in particular to a network detection system and method. Background Art

[0002] The current mainstream network mapping platform is mainly a hybrid structure, that is, a structure that combines centralization and peer-to-peer. Among them, the centralized structure selects a central node among the detection nodes to control the remaining detection nodes to complete the detection tasks and receive the detection results. However, this single-level structure is easy to trace, and the review and experimental development, testing and deployment processes require the participation of platform operators. After executing a certain type of task, if there is no same task issued, the detection node will be idle, resulting in a waste of detection resources and low resource utilization. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a network detection system and method to improve the resource utilization of the detection system and the adaptability to complex environments.

[0004] In a first aspect, the present invention provides a network detection system, which includes a user end, a central control end, an instruction distribution server end and a detection node, wherein the central control end responds to a detection task requirement instruction sent by the user end, and determines at least one target detection node that executes the detection task based on the task information of the detection task required by the user end in the detection task requirement instruction; the central control end generates a detection task execution instruction and sends it to the instruction distribution server end; the instruction distribution server end adds the detection task indicated by the detection task execution instruction to a task queue corresponding to the target detection node; the detection node sends a detection task acquisition instruction to the instruction distribution server end to obtain the detection task to be executed by the detection node from the task queue corresponding to the detection node.

[0005] In an optional implementation, the central control end responds to the detection task requirement instruction sent by the user end, which includes at least the task information of the detection task required by the user end, determines the corresponding detection node list and feeds it back to the user end; the user end selects at least one target detection node from the received detection node list, and sends the identifier of the target detection node back to the central control end.

[0006] In an optional implementation, the central control end divides the detection task based on the number of target detection nodes to generate a corresponding plurality of detection sub-tasks; and the central control end generates a detection task execution instruction based on the detection sub-tasks and sends it to the instruction distribution server. The detection task execution instruction includes at least the task information of the detection sub-task and the identifier of the target detection node to execute the detection sub-task.

[0007] In an optional implementation, the system also includes a data backhaul server, to which the detection node sends corresponding detection task execution status information and detection task execution results; and the central control end feeds back the detection task execution status information or detection task execution results corresponding to the target detection node to the user end.

[0008] In an optional implementation, the central control end responds to a detection task status query instruction sent by the user end, where the detection task status query instruction is used to indicate an identifier of the detection task; the central control end forwards the detection task status query instruction to the data return server to obtain the detection task execution status information and feed it back to the user end.

[0009] In an optional implementation, the central control end responds to the detection task result query instruction sent by the user end, and the detection task status query instruction is used to indicate the identification of the detection task; the central control end forwards the detection task result query instruction to the data return server to obtain the detection task execution result information and feedback it to the user end.

[0010] In an optional implementation, the detection node encrypts the detection task result information and sends it to the data return server.

[0011] In a second aspect, the present invention provides a network detection method, which includes responding to a detection task requirement instruction, and determining at least one target detection node that executes the detection task based on the task information of the detection task required by the user end in the detection task requirement instruction; generating a detection task execution instruction and sending it to an instruction distribution server; adding the detection task indicated by the detection task execution instruction to a task queue corresponding to the target detection node; and responding to a detection task acquisition instruction to acquire the detection task to be executed by the detection node from the task queue corresponding to the detection node.

[0012] In an optional embodiment, in response to a detection task requirement instruction sent by a user, the detection task requirement instruction includes at least task information of the detection task required by the user, a corresponding detection node list is determined and fed back to the user to obtain at least one target detection node selected by the user in the detection node list.

[0013] In an optional embodiment, the detection task is divided based on the number of target detection nodes to generate a corresponding plurality of detection sub-tasks; and a detection task execution instruction is generated based on the detection sub-tasks, the detection task execution instruction including at least task information of the detection sub-task and an identifier of the target detection node to execute the detection sub-task.

[0014] The present application provides a network detection system and method, wherein the detection system includes a user end, a central control end, an instruction distribution server end, and a detection node, wherein the central control end responds to the detection task requirement instruction sent by the user end, and determines at least one target detection node for executing the detection task based on the task information of the detection task required by the user end in the detection task requirement instruction; the central control end generates a detection task execution instruction and sends it to the instruction distribution server end; the instruction distribution server end adds the detection task indicated by the detection task execution instruction to the task queue corresponding to the target detection node; the detection node sends a detection task acquisition instruction to the instruction distribution server end to obtain the detection task to be executed by the detection node from the task queue corresponding to the detection node. The resource utilization rate of the detection system and the adaptability to complex environments are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A schematic diagram of the structure of a network detection system provided in an embodiment of the present application;

[0017] Figure 2 A communication flow chart of a network detection system provided in an embodiment of the present application when issuing a network crawler task;

[0018] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] First, the applicable scenario of the present application is described. The technical solution of the present application can be applied to the construction of a network detection platform.

[0020] The current cyberspace mapping platform faces problems such as single function, high requirements for the operating environment, high costs for renting VPS (Virtual Private Server) and servers, inflexible node addition and exit that have a significant impact on task execution, insufficient task execution concurrency, and low resource utilization.

[0021] Based on this, the present application uses ad hoc network technology to build a multi-environment adaptive detection network system.

[0022] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0023] Figure 1 A schematic diagram of the structure of a network detection system provided in an embodiment of the present application. Figure 1 As shown, the present application provides a network detection system, which includes a user end, a central control end, a command distribution server end, a data return server end and a detection node.

[0024] The central control end here is used to manage the detection nodes. The instruction distribution server and the data return server are used to issue instructions and pull data between the central control end and the detection nodes. The central control end, the instruction distribution server and the data return server can specifically be servers.

[0025] In order to improve the operational stability and complex environment coping capabilities of the network detection system, the central control end here can use a cloud host or a physical host.

[0026] The network detection system provided in the embodiment of the present application has a fast and highly concurrent management mechanism. In view of the problem that different types of resources are difficult to organize dynamically, a multi-level centralized network structure is designed to realize the dynamic organization of the detection network, and multiple types of interfaces are designed based on the HTTPS protocol to realize communication between nodes, and finally the design of a heterogeneous multi-environment detection network organization and management model is completed.

[0027] The network detection system provided in the embodiment of the present application aims to solve the problem of complicated detection resource management and scheduling process under multi-tasking conditions, designs simple steps for task management and detection program distribution, and designs API (Application Programming Interface) in combination with specific functional requirements, including task management API, node management API, detection program management API, etc., to realize the issuance of detection tasks, result feedback, status feedback, operation feedback messages, etc.

[0028] Next, we can take a web crawler as an example to illustrate the workflow of the network detection system.

[0029] like Figure 2 As shown, Figure 2 A communication flow chart of a network detection system for issuing a network crawler task provided in an embodiment of the present application specifically includes:

[0030] S10. The user terminal sends a detection task requirement instruction to the central control terminal.

[0031] S11, the central control end responds to the detection task requirement instruction sent by the user end and determines the corresponding detection node list.

[0032] The user end can specifically call the API of the central control node to view functions such as node list, task information, node status information, etc.

[0033] The detection task requirement instruction here includes at least task information of the detection task required by the user terminal and user terminal information. The task information is used to indicate the detection task required by the user terminal, such as topology detection or web crawler, etc. The user terminal information may include the user terminal identification, the user terminal geographical location, etc.

[0034] The central control terminal can select a detection node that matches the current detection task according to the task type, geographical location, occupied resource size, detection node load, etc. of the detection task, form a detection node list and feed it back to the user terminal. Exemplarily, a detection node whose distance to the electronic device to be detected is less than a preset distance value can be selected to generate a detection node list.

[0035] S12. The central control end feeds back the detection node list to the user end.

[0036] S13: The user terminal selects at least one target detection node from the received detection node list.

[0037] The user end can select the number of target detection nodes to perform the current detection task according to actual conditions, and select the corresponding target detection node in the detection node list.

[0038] S14. The user end sends the identifier of the target detection node back to the central control end.

[0039] S15. The central control end divides the detection task based on the number of target detection nodes to generate a corresponding plurality of detection subtasks.

[0040] It is understandable that when the number of target detection nodes is 1, there is no need to split the detection task.

[0041] S16: The central control terminal generates a detection task execution instruction based on the detection subtask and sends it to the instruction distribution server. The detection task execution instruction at least includes the task information of the detection subtask and the identifier of the target detection node to execute the detection subtask.

[0042] S17. The instruction distribution server adds the detection task indicated by the detection task execution instruction to the task queue corresponding to the target detection node.

[0043] The instruction distribution server can construct a task queue corresponding to each detection node one by one. In each task queue, each detection task or detection subtask is sorted from high to low according to the priority.

[0044] S18. The detection node sends a detection task acquisition instruction to the instruction distribution server.

[0045] Here, the detection node may obtain the next detection task to be executed from the instruction distribution server when the previous detection task / detection subtask is about to end.

[0046] The detection node can also obtain the next detection task to be executed from the instruction distribution server at a preset frequency.

[0047] S19. The instruction distribution server determines the detection task to be executed by the detection node from the corresponding task queue and sends it to the detection node.

[0048] The instruction distribution server sends the detection task / detection subtask with the highest current priority to the corresponding detection node in the corresponding task queue.

[0049] The present application provides a network detection system, in which the central control end can dynamically classify detection nodes based on detection tasks, so that the detection network can execute multiple types of tasks concurrently, thereby improving resource utilization, optimizing the performance of the network detection system, and enabling the network detection system to operate stably in complex environments.

[0050] In one embodiment of the present application, the system further includes a data return service end, the detection node sends the corresponding detection task execution status information and detection task execution result to the data return service end, and the central control end feeds back the detection task execution status information or detection task execution result corresponding to the target detection node to the user end.

[0051] The detection node can perform detection tasks in the following ways:

[0052] The detection node is pre-installed with a resident program for controlling the operation of the detection node.

[0053] The task execution logic in the resident program is specifically to load the corresponding detection program according to the priority of the detection task to complete the detection task. For example, when the detection task is a web crawler, the detection program here is a data crawler program.

[0054] The detection program of the detection node can save the detection task execution status information of the detection task by calling the resident program status API and send it back to the data return server. The detection node can also package and encrypt the log file (detection task execution result information) corresponding to the completed detection task and send it back to the data return server.

[0055] The data return server may store the detection task execution status information and detection task execution result information fed back by each detection node.

[0056] Furthermore, the data feedback server and the instruction distribution server can also improve the security of the system.

[0057] In one embodiment of the present application, the user terminal can query the status of the detection task in the following manner:

[0058] The central control end responds to the detection task status query instruction sent by the user end, and the detection task status query instruction is used to indicate the identification of the detection task. The central control end forwards the detection task status query instruction to the data return server to obtain the detection task execution status information and feed it back to the user end.

[0059] The user can pull the results of the detection task in the following ways:

[0060] The central control end responds to the detection task result query instruction sent by the user end. The detection task status query instruction is used to indicate the identification of the detection task; the central control end forwards the detection task result query instruction to the data return server to obtain the detection task execution result information and feedback it to the user end.

[0061] Based on the same inventive concept, a network detection method is also provided in an embodiment of the present application, the method comprising responding to a detection task requirement instruction, and determining at least one target detection node that performs the detection task based on the task information of the detection task required by the user end in the detection task requirement instruction; generating a detection task execution instruction and sending it to an instruction distribution server; adding the detection task indicated by the detection task execution instruction to a task queue corresponding to the target detection node; and responding to a detection task acquisition instruction to obtain the detection task to be performed by the detection node from the task queue corresponding to the detection node.

[0062] In a preferred embodiment, it specifically includes responding to a detection task requirement instruction sent by a user, wherein the detection task requirement instruction includes at least task information of the detection task required by the user, determining a corresponding detection node list and feeding it back to the user to obtain at least one target detection node selected by the user in the detection node list.

[0063] In a preferred embodiment, the detection task is divided based on the number of target detection nodes to generate a corresponding plurality of detection subtasks; and a detection task execution instruction is generated based on the detection subtask, wherein the detection task execution instruction includes at least task information of the detection subtask and an identifier of the target detection node to execute the detection subtask.

[0064] A network detection method provided by the present application utilizes the characteristics of self-organizing networks to construct a variety of node access mechanisms, so that multiple types of nodes can be quickly added and deleted, providing a faster and more compatible structure, thereby solving the problem of single detection node type utilization in existing detection platforms. In addition, a scheduling strategy is designed to build a collaborative scheduling mechanism for the network, improve the utilization rate of detection resources, and propose a collaborative scheduling technology based on self-organizing network technology, which can unify resource management, reduce the complexity of upper-level application development, and reduce development workload and expenses.

[0065] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 3 As shown in , the electronic device 300 includes a processor 310 , a memory 320 and a bus 330 .

[0066] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 communicates with the memory 320 through the bus 330. When the machine-readable instructions are executed by the processor 310, the steps of a network detection method in the aforementioned embodiment can be executed. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0067] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a network detection method in the aforementioned embodiment can be executed. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0068] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0069] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0070] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0071] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0072] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0073] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0074] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A network detection system, characterized in that: The detection system includes a user terminal, a central control terminal, an instruction distribution server terminal and a detection node, wherein: The central control end responds to the detection task requirement instruction sent by the user end, and determines at least one target detection node for executing the detection task based on the task information of the detection task required by the user end in the detection task requirement instruction; The central control end generates the detection task execution instruction and sends it to the instruction distribution server; The instruction distribution server adds the detection task indicated by the detection task execution instruction to the task queue corresponding to the target detection node; The detection node sends a detection task acquisition instruction to the instruction distribution server to obtain the detection task to be executed by the detection node from the task queue corresponding to the detection node; The central control end responds to the detection task requirement instruction sent by the user end, wherein the detection task requirement instruction at least includes task information of the detection task required by the user end, determines a corresponding detection node list and feeds it back to the user end; The user end selects at least one target detection node from the received detection node list, and sends the identifier of the target detection node back to the central control end; The central control end divides the detection task based on the number of target detection nodes to generate a corresponding plurality of detection subtasks; and The central control end generates a detection task execution instruction based on the detection subtask and sends it to the instruction distribution server, wherein the detection task execution instruction at least includes task information of the detection subtask and an identifier of a target detection node to execute the detection subtask; The system also includes a data return service end, The detection node sends the corresponding detection task execution status information and detection task execution results to the data return server; and The central control end feeds back the detection task execution status information or detection task execution result corresponding to the target detection node to the user end.

2. The system according to claim 1, characterized in that The central control end responds to the detection task status query instruction sent by the user end, wherein the detection task status query instruction is used to indicate the identification of the detection task; The central control end forwards the detection task status query instruction to the data return service end to obtain the detection task execution status information and feed it back to the user end.

3. The system according to claim 1, characterized in that The central control end responds to the detection task result query instruction sent by the user end, wherein the detection task result query instruction is used to indicate the identification of the detection task; The central control end forwards the detection task result query instruction to the data return service end to obtain the detection task execution result information and feed it back to the user end.

4. The system according to claim 1, characterized in that The detection node encrypts the detection task result information and sends it to the data return server.

5. A network detection method, characterized in that: The method comprises: In response to the detection task requirement instruction, based on the task information of the detection task required by the user end in the detection task requirement instruction, determine at least one target detection node for executing the detection task; Generate detection task execution instructions and send them to the instruction distribution server; Add the detection task indicated by the detection task execution instruction to the task queue corresponding to the target detection node; In response to the detection task acquisition instruction, the detection task to be executed by the detection node is acquired from the task queue corresponding to the detection node; and responding to a detection task requirement instruction sent by a user, wherein the detection task requirement instruction includes at least task information of the detection task required by the user, determining a corresponding detection node list and feeding it back to the user, so as to obtain at least one target detection node selected by the user in the detection node list; Based on the number of target detection nodes, the detection task is divided to generate a corresponding plurality of detection subtasks; and Generate a detection task execution instruction based on the detection subtask, wherein the detection task execution instruction at least includes task information of the detection subtask and an identifier of a target detection node to execute the detection subtask; Feedback the detection task execution status information or detection task execution result corresponding to the target detection node to the user end.

Images