Method, device, electronic device and storage medium for massive IP detection
Through the combination of goroutine concurrent execution unit and ICMP monitoring and response analysis steps, the problems of low IP detection efficiency and high resource consumption are solved, and efficient detection and resource management of multiple IP addresses are achieved, which improves system stability.
Patent Information
- Application Number
- CN202411187250.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-08-28
AI Technical Summary
The existing IP detection method is inefficient and consumes serious system resources when probing large amounts of IP addresses, especially in the case of network congestion or slow response of the target host, which may cause system downtime.
The goroutine concurrent execution unit is used to process the IP address to be responsive, and the ICMP monitoring and response analysis steps are used to monitor the ICMP response asynchronously, and the timeout mechanism is used to avoid waiting indefinitely, control resource usage, and allocate reasonable concurrent numbers to manage background monitoring resources.
It improves the efficiency of IP detection, reduces resource waste and consumption, greatly alleviates the problems of low efficiency and serious resource consumption, realizes connectivity detection of multiple target hosts, and enhances the stability and scalability of the system.
Smart Images

Figure CN118714049B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer networks and communications, and in particular to a method, device, electronic equipment and storage medium for massive IP detection. Background Art
[0002] IP liveness detection, or IP survival detection, is a network management and troubleshooting tool used to determine whether the target IP address is reachable. It usually sends a network data packet to the target IP address and determines whether the target host is alive based on the returned response. ICMP Echo Request (Ping Command): Uses ICMP (Internet Control Message Protocol) protocol. The sender sends an ICMP Echo request packet to the target host, which contains a unique identifier (used to characterize the liveness request) and a sequence number (used to characterize the sender). If the target host is alive and receives this request, it replies with an ICMP Echo reply packet containing the same identifier and sequence number. The sender determines whether the target host is alive based on whether the reply packet is received.
[0003] In the above live detection scheme, when a large number of IP addresses need to be detected, sending and waiting for ICMP Echo replies one by one will cause time delay. Especially in the case of network congestion or slow response of the target host, this delay will be more obvious. When there is no specific tool to scan (i.e. detect) a special subnet segment (i.e. the network segment of the IP address that needs to be detected), network engineers need to manually use detection tools to detect IPs, which is extremely inefficient; in addition, when performing massive IP detection, if a single-threaded or simple multi-threaded scanning method is used, it may cause unreasonable use of system resources (such as network bandwidth, CPU, memory, etc.), affecting other normal services, and in serious cases, it may cause system downtime. This is especially obvious when scanning a large number of unreachable or slow-responding targets.
[0004] In summary, the existing IP live detection method has the technical problems of low efficiency and serious consumption of system resources when detecting a large number of IP addresses. Summary of the invention
[0005] In view of this, the purpose of the present invention is to provide a method, device, electronic device and storage medium for massive IP detection, so as to alleviate the technical problems of low efficiency and serious system resource consumption of existing IP detection methods when detecting a large number of IP addresses.
[0006] In a first aspect, an embodiment of the present invention provides a method for massive IP detection, including:
[0007] Obtain a list of IP addresses to be detected, and divide the IP addresses to be detected in the list of IP addresses to be detected to obtain at least one detection task block, wherein the detection task block contains a preset number of IP addresses to be detected;
[0008] A goroutine concurrent execution unit is created for each of the live detection task blocks, so that each of the goroutine concurrent execution units performs live detection processing on a preset number of to-be-detected IP addresses in the corresponding live detection task block in parallel;
[0009] Each goroutine concurrent execution unit performs detection processing on a preset number of target IP addresses to be detected in the corresponding target detection task block according to the following local ICMP monitoring and response analysis steps:
[0010] Initialize basic variables and create a monitoring goroutine, wherein the basic variables include: variables used to indicate whether the live detection process is terminated and variables for temporarily storing results. The monitoring goroutine is used to continuously monitor the ICMP responses returned by each target host. When the ICMP response is monitored, the source IP address in the ICMP response is added as a live IP to the variable for temporarily storing results.
[0011] Traversing each target IP address to be detected in the target detection task block, parsing each target IP address to be detected, obtaining a network object corresponding to each target IP address to be detected, and custom-generating a corresponding ICMP detection request for each target IP address to be detected;
[0012] Each of the network objects sends a corresponding ICMP live detection request to a corresponding target host, so that the target host returns an ICMP response to the listening goroutine according to the network communication protocol, and then the listening goroutine adds the source IP address in the ICMP response as a live IP to the variable of the temporary storage result, wherein the timer records the time when the first ICMP live detection request is sent as the starting live detection time;
[0013] If all target IP addresses to be detected in the target detection task block belong to the variables of the temporary storage result, or the current detection duration determined based on the start detection time recorded by the timer reaches the target detection duration, the detection processing of the target detection task block is terminated, and the variable used to characterize whether the detection processing is terminated is set to yes, the continuous monitoring of the monitoring goroutine is terminated, and the used resources are released, wherein the target detection duration is calculated according to the number of target IP addresses to be detected in the target detection task block;
[0014] The source IP addresses stored in the variable of the temporary storage result are summarized to obtain a summary result of the surviving IP addresses.
[0015] Furthermore, a list of active IP addresses to be detected is obtained, including:
[0016] Obtaining manually input IP data, wherein the IP data includes: IP data represented by CIDR, and / or IP data represented by IP address range;
[0017] The list of active IP addresses to be detected is obtained by calculation according to the representation method of the IP data.
[0018] Further, each of the target IP addresses to be detected is parsed, including:
[0019] Use net.ResolveIPAddr to resolve each of the target IP addresses to be detected, and obtain the network object corresponding to each of the target IP addresses to be detected, wherein the network object includes: *net.IPAddr object;
[0020] Each of the network objects sends a corresponding ICMP liveness request to a corresponding target host, including:
[0021] Each of the network objects calls conn.WriteTo to send the corresponding ICMP detection request to the corresponding target host.
[0022] Further, if the current goroutine concurrent execution unit fails to perform detection processing on a preset number of target IP addresses to be detected in the corresponding target detection task block according to the local ICMP listening and response analysis step, the current goroutine concurrent execution unit performs detection processing on a preset number of target IP addresses to be detected in the corresponding target detection task block according to the following non-listening ICMP IP detection processing step:
[0023] Determine the maximum concurrency number according to the number of target live IP addresses to be detected in the target live task block and a preset value, and create a buffer channel according to the maximum concurrency number, wherein the buffer channel is used to control the number of concurrent live detection goroutines to be at most the maximum concurrency number;
[0024] Traverse each target IP address to be detected in the target detection task block, and start a detection goroutine for each target IP address to be detected;
[0025] Record the number of the probing goroutines, and determine the probing goroutines to be executed in the probing goroutines according to the buffer channel, wherein the number of the probing goroutines to be executed is the number of probing goroutines currently allowed to be executed in parallel by the buffer channel;
[0026] During the execution of each of the to-be-executed detection goroutines, a ping function based on the ICMP protocol is called to detect the activity of the target to-be-detected IP address corresponding to each of the to-be-executed detection goroutines;
[0027] If the ping function based on the ICMP protocol returns a live detection result, then according to the live detection result, the active target IP address to be detected is added to the variable for temporarily storing the result;
[0028] If the pending detection goroutine is completed, the number of completed pending detection goroutines is recorded, and the parallel execution permission of the pending detection goroutine in the buffer channel is released, so that other unexecuted detection goroutines enjoy the permission of parallel execution in the buffer channel, until the number of completed pending detection goroutines is the same as the number of target pending detection IP addresses in the target detection task block, and the detection processing of the target detection task block is completed;
[0029] closing the buffer channel;
[0030] The source IP addresses stored in the variable of the temporary storage result are summarized to obtain a summary result of the surviving IP addresses.
[0031] Further, the maximum concurrency number is determined according to the number of target IP addresses to be detected in the target detection task block and a preset value, including:
[0032] The smaller value between the number of target IP addresses to be detected in the target detection task block and the preset value is used as the maximum concurrent number.
[0033] Furthermore, the method further comprises:
[0034] If the target network object sends the corresponding ICMP detection request to the corresponding target host, and the target host does not return an ICMP response to the listening goroutine, a retry is performed according to a preset retry strategy, wherein the preset retry strategy includes: a retry strategy with a preset number of retries and a retry strategy with an adjusted retry interval.
[0035] Furthermore, the preset retry strategy is obtained by predicting the retry strategy of the current network information using a retry strategy prediction model.
[0036] In a second aspect, an embodiment of the present invention further provides a device for detecting massive IP activity, including:
[0037] An acquisition unit is used to acquire a list of IP addresses to be detected, and divide the IP addresses to be detected in the list of IP addresses to be detected to obtain at least one detection task block, wherein the detection task block contains a preset number of IP addresses to be detected;
[0038] A creation unit is used to create a goroutine concurrent execution unit for each of the live detection task blocks, so that each of the goroutine concurrent execution units performs live detection processing on a preset number of to-be-detected IP addresses in the corresponding live detection task block in parallel;
[0039] The live detection processing unit is used for each goroutine concurrent execution unit to perform live detection processing on a preset number of target live detection IP addresses in the corresponding target live detection task block according to the following local ICMP monitoring and response analysis steps:
[0040] Initialize basic variables and create a monitoring goroutine, wherein the basic variables include: variables used to indicate whether the live detection process is terminated and variables for temporarily storing results. The monitoring goroutine is used to continuously monitor the ICMP responses returned by each target host. When the ICMP response is monitored, the source IP address in the ICMP response is added as a live IP to the variable for temporarily storing results.
[0041] Traversing each target IP address to be detected in the target detection task block, parsing each target IP address to be detected, obtaining a network object corresponding to each target IP address to be detected, and custom-generating a corresponding ICMP detection request for each target IP address to be detected;
[0042] Each of the network objects sends a corresponding ICMP live detection request to a corresponding target host, so that the target host returns an ICMP response to the listening goroutine according to the network communication protocol, and then the listening goroutine adds the source IP address in the ICMP response as a live IP to the variable of the temporary storage result, wherein the timer records the time when the first ICMP live detection request is sent as the starting live detection time;
[0043] If all target IP addresses to be detected in the target detection task block belong to the variables of the temporary storage result, or the current detection duration determined based on the start detection time recorded by the timer reaches the target detection duration, the detection processing of the target detection task block is terminated, and the variable used to characterize whether the detection processing is terminated is set to yes, the continuous monitoring of the monitoring goroutine is terminated, and the used resources are released, wherein the target detection duration is calculated according to the number of target IP addresses to be detected in the target detection task block;
[0044] The source IP addresses stored in the variable of the temporary storage result are summarized to obtain a summary result of the surviving IP addresses.
[0045] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.
[0046] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to execute any method described in the first aspect above.
[0047] In an embodiment of the present invention, a method for massive IP live detection is provided, including: obtaining a list of IP addresses to be detected, and dividing the IP addresses to be detected in the list of IP addresses to be detected to obtain at least one live detection task block, wherein the live detection task block contains a preset number of IP addresses to be detected; creating a goroutine concurrent execution unit for each live detection task block, so that each goroutine concurrent execution unit performs live detection processing on the preset number of IP addresses to be detected in the corresponding live detection task block in parallel; each goroutine concurrent execution unit performs live detection processing on the preset number of target IP addresses to be detected in the corresponding target live detection task block according to the following local ICMP monitoring and response analysis steps: initializing basic variables and creating a monitoring goroutine, wherein the basic variables include: variables for indicating whether the live detection processing is terminated and variables for temporarily storing results, and the monitoring goroutine is used to continuously monitor the ICMP response returned by each target host, and when the ICMP response is monitored, the source IP address in the ICMP response is added as a surviving IP to the variable for temporarily storing results; traversing each target IP address to be detected in the target live detection task block, for each target The target IP address to be detected is parsed to obtain the network object corresponding to each target IP address to be detected, and the corresponding ICMP detection request is customized for each target IP address to be detected; each network object sends the corresponding ICMP detection request to the corresponding target host, so that the target host returns an ICMP response to the listening goroutine according to the network communication protocol, and then the listening goroutine adds the source IP address in the ICMP response as the surviving IP to the variable that temporarily stores the result, wherein the timer records the time when the first ICMP detection request is sent as the starting detection time; If all the target IP addresses to be detected in the target detection task block belong to the variable of the temporary storage result, or the current detection duration determined based on the starting detection time recorded by the timer reaches the target detection duration, the detection processing of the target detection task block is terminated, and the variable used to characterize whether the detection processing is terminated is set to yes, the continuous monitoring of the monitoring goroutine is terminated, and the used resources are released, wherein the target detection duration is calculated according to the number of target IP addresses to be detected in the target detection task block; the source IP addresses stored in the variable of the temporary storage result are summarized to obtain the summary result of the surviving IP addresses.From the above description, it can be known that in the method for massive IP detection of the present invention, each goroutine concurrent execution unit performs detection processing on a preset number of target IP addresses to be detected in the corresponding detection task block in parallel, wherein each goroutine concurrent execution unit performs detection processing on a preset number of target IP addresses to be detected in the corresponding target detection task block according to the local ICMP monitoring and response analysis steps, and uses the listening goroutine to asynchronously (the sending of ICMP detection request is sent by the network object, and the ICMP response is monitored by the listening goroutine, so it is called asynchronous) monitor the ICMP response, and each network object can simultaneously send the corresponding ICMP detection request to the corresponding The corresponding target host is then continuously monitored by the monitoring goroutine for the ICMP responses returned by each target host, and the source IP address in the ICMP response is added as the live IP to the variable for temporarily storing the result, thereby achieving the purpose of simultaneously detecting connectivity for multiple target hosts and improving the efficiency of live detection. At the same time, a timeout mechanism is used to avoid indefinite waiting, thereby increasing the robustness of the algorithm. In addition, by controlling the variables for whether the live detection process is terminated and the goroutine concurrent execution unit, the background monitoring resources can be effectively managed, thereby avoiding resource waste, unreasonable use and high resource consumption, thereby alleviating the technical problems of low efficiency and serious system resource consumption of the existing IP live detection method when detecting a large number of IP addresses. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0049] Figure 1 A flowchart of a method for detecting massive IP activity provided by an embodiment of the present invention;
[0050] Figure 2 A flowchart of another method for detecting massive IP activity provided by an embodiment of the present invention;
[0051] Figure 3 A schematic diagram of a device for detecting massive IP activity provided by an embodiment of the present invention;
[0052] Figure 4 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0053] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] Traditional IP live detection methods are inefficient and consume a lot of system resources when detecting a large number of IP addresses.
[0055] Based on this, in the method for massive IP detection of the present invention, each goroutine concurrent execution unit performs detection processing on a preset number of target IP addresses to be detected in the corresponding detection task block in parallel, wherein each goroutine concurrent execution unit performs detection processing on a preset number of target IP addresses to be detected in the corresponding target detection task block according to the local ICMP monitoring and response analysis steps, and uses the listening goroutine to asynchronously (sending ICMP detection requests is sent by the network object, and the ICMP responses are monitored by the listening goroutine, so it is called asynchronous) monitor ICMP responses, and each network object The corresponding ICMP live detection request can be sent to the corresponding target host at the same time, and then the monitoring goroutine will continue to monitor the ICMP response returned by each target host, and add the source IP address in the ICMP response as the live IP to the variable for temporary storage of the result, so as to achieve the purpose of connectivity detection for multiple target hosts at the same time and improve the efficiency of live detection. At the same time, the timeout mechanism is used to avoid indefinite waiting, which increases the robustness of the algorithm. In addition, through the variables of whether the live detection process is terminated and the control of the goroutine concurrent execution unit, the background monitoring resources can be effectively managed to avoid resource waste, unreasonable use and high resource consumption.
[0056] To facilitate understanding of this embodiment, a method for detecting massive IP activity disclosed in an embodiment of the present invention is first introduced in detail.
[0057] Embodiment 1:
[0058] According to an embodiment of the present invention, an embodiment of a method for massive IP detection is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0059] Figure 1 FIG. 4 is a flow chart of a method for detecting massive IP activity according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:
[0060] Step S102, obtaining a list of IP addresses to be detected, and dividing the IP addresses to be detected in the list of IP addresses to be detected to obtain at least one detection task block, wherein the detection task block contains a preset number of IP addresses to be detected;
[0061] First, some related concepts in the present invention are introduced:
[0062] ICMP (Internet Control Message Protocol) is an Internet control message protocol. It is a sub-protocol of the TCP / IP protocol suite, used to transmit control messages between IP hosts and routers. Control messages refer to network messages such as whether the network is connected, whether the host is reachable, and whether the route is available. Although these control messages do not transmit user data, they play an important role in the transmission of user data.
[0063] IP (Internet Protocol) is a network layer protocol in the TCP / IP system. The purpose of designing IP is to improve the scalability of the network: first, to solve Internet problems and realize the interconnection of large-scale, heterogeneous networks; second, to separate the coupling relationship between top-level network applications and bottom-level network technologies to facilitate the independent development of the two. According to the end-to-end design principle, IP only provides a connectionless, unreliable, best-effort data packet transmission service for the host.
[0064] Go (also known as Golang) is a statically strongly typed, compiled language developed by Robert Griesemer, Rob Pike and Ken Thompson of Google. The syntax of Go language is similar to C, but its functions include: memory safety, GC (garbage collection), structural form and CSP-style concurrent computing.
[0065] Goroutine is a concurrent execution unit in the Go language. The underlying layer of goroutine is implemented using coroutine, which is a user thread running in user mode (kernel mode, user mode), which can be scheduled by the language and framework layer. Go implements a scheduler at the language level. At the same time, it encapsulates the network and IO libraries, shields the complex details at the operating system level, and provides unified keyword support at the language level.
[0066] The application scenarios of the present invention mainly involve the following network element devices:
[0067] Live detection server: A server responsible for performing IP live detection tasks, equipped with the technical solution of the present invention. It communicates with the network device to be detected through a network interface, sends ICMP live detection requests and receives ICMP responses. The live detection server can be an independent physical server, or a virtual machine or container instance in a cloud environment.
[0068] Network equipment: The network equipment where the IP address to be detected is located, including routers, switches, servers, terminals, etc. These devices communicate with the live detection server through the network, receive ICMP live detection requests and return ICMP responses.
[0069] The live detection server is connected to the network device through a network interface (such as an Ethernet interface) to form a star or mesh network topology. When the live detection server sends an ICMP live detection request to the network device, it selects the best communication path through the IP layer routing. After receiving the ICMP live detection request, the network device generates and sends an ICMP response based on its own status and the processing logic of the network protocol stack.
[0070] During implementation, the liveness detection server obtains a list of IP addresses to be detected, and divides the IP addresses to be detected in the list to obtain at least one liveness detection task block, each of which contains a preset number of IP addresses to be detected.
[0071] It should be noted that before obtaining the list of IP addresses to be detected, the detection server initializes and sets parameters for the system corresponding to the method for massive IP detection of the present invention, sets the number of Go goroutines (i.e., goroutine concurrent execution units), task block size, retry strategy and other parameters; after being ready, it waits to receive the detection task.
[0072] Step S104, creating a goroutine concurrent execution unit for each live detection task block, so that each goroutine concurrent execution unit performs live detection processing on a preset number of to-be-detected IP addresses in the corresponding live detection task block in parallel;
[0073] Specifically, the liveness detection server creates a goroutine concurrent execution unit for each liveness detection task block, and passes the liveness detection task block to the goroutine concurrent execution unit for liveness detection processing. Multiple goroutine concurrent execution units execute liveness detection tasks in parallel for the corresponding liveness detection task blocks (specifically a preset number of IP addresses to be detected).
[0074] Step S106, each goroutine concurrent execution unit performs detection processing on a preset number of target to-be-detected IP addresses in the corresponding target detection task block according to the following local ICMP monitoring and response analysis steps:
[0075] Initialize the basic variables and create a monitoring goroutine. The basic variables include: variables used to indicate whether the live detection process has terminated and variables for temporarily storing results. The monitoring goroutine is used to continuously monitor the ICMP responses returned by each target host. When the ICMP response is monitored, the source IP address in the ICMP response is added as the live IP to the variable for temporarily storing results.
[0076] Specifically, a Boolean variable endflag is initialized first, which is used to indicate whether the live detection process is terminated. When initialized, the Boolean variable endflag is false, indicating that the live detection process has not been terminated. The execution of the monitoring goroutine is controlled by the value of this variable.
[0077] At the same time, create a listening goroutine, which will loop infinitely and continuously listen for ICMP responses. Whenever a non-nil (empty) source IP address sourceIP is listened, the source IP address is added as a live IP to a global active host list (assuming there is a global active host list AliveHosts), and is also added to the variable chanHosts for temporary storage of results. When other applications need to obtain information about live IPs later, the string representation of the source IP address is sent through chanHosts.
[0078] Traverse each target IP address to be detected in the target detection task block, parse each target IP address to be detected, obtain the network object corresponding to each target IP address to be detected, and customize the corresponding ICMP detection request for each target IP address to be detected;
[0079] Each network object sends the corresponding ICMP live detection request to the corresponding target host, so that the target host returns an ICMP response to the listening goroutine according to the network communication protocol, and then the listening goroutine adds the source IP address in the ICMP response as the live IP to the variable that temporarily stores the result, where the timer records the time when the first ICMP live detection request is sent as the starting live detection time;
[0080] Specifically, the corresponding ICMP liveness detection request is an ICMP liveness detection request (specifically, an IcmpByte message) corresponding to the target liveness detection IP address corresponding to the network object.
[0081] Set a timer start to record the time when the first ICMP detection request is sent as the starting detection time, and then enter a waiting and listening loop.
[0082] If all the target IP addresses to be detected in the target detection task block belong to the variables for temporary storage results, or the current detection duration determined based on the start detection time recorded by the timer reaches the target detection duration, the detection processing of the target detection task block ends, and the variable used to characterize whether the detection processing is terminated is set to yes, the continuous monitoring of the monitoring goroutine is terminated, and the used resources are released, where the target detection duration is calculated based on the number of target IP addresses to be detected in the target detection task block;
[0083] Specifically, the fact that each target IP address to be detected as active belongs to a variable for temporarily storing results indicates that each target IP address to be detected as active is active, that is, detection of each target IP address to be detected as active has been completed.
[0084] The current detection time is the time between the start detection time and the current time.
[0085] The target detection time is calculated according to the number of target IP addresses to be detected in the target detection task block. The more the number of target IP addresses to be detected, the longer the target detection time. It can be calculated by presetting multiple piecewise functions. For example, when the number of target IP addresses to be detected is less than 1000, the target detection time and the number of target IP addresses to be detected are in a first functional relationship; when the number of target IP addresses to be detected is not less than 1000 and less than 5000, the target detection time and the number of target IP addresses to be detected are in a second functional relationship, and so on.
[0086] After the detection process of the target detection task block is completed, set endflag to true (that is, set the variable used to indicate whether the detection process is terminated to yes), which will terminate the listening loop of the monitoring goroutine, close the ICMP connection (the network object no longer sends ICMP detection requests), and release all resources.
[0087] The source IP addresses stored in the variable for temporarily storing the result are summarized to obtain a summary result of the surviving IP addresses.
[0088] Specifically, after all goroutine concurrent execution units complete the liveness detection task, the liveness detection server releases the used resources, summarizes the source IP addresses stored in the variable for temporary storage of results, obtains the summary result of the live IP addresses, outputs or stores the summary result in a preset format (such as a CSV file, database record, etc.), and obtains a complete list of live IP addresses for subsequent network management or security monitoring.
[0089] It should be noted that, in addition to the ICMP protocol, other network protocols and tools can also be combined to perform IP detection. For example, the SYN packet of the TCP protocol can be used to detect the activity of the IP address by trying to establish a TCP connection. In addition, existing network scanning tools or libraries can also be used to detect the activity, such as Nmap, etc. These tools and protocols can provide more abundant detection means, improve the accuracy and flexibility of detection. When implemented, the summary results of the surviving IP addresses obtained by the method provided by the present invention are compared and verified with the detection results obtained by other methods, and the source IP addresses that are detected as active by the above two methods are used as the final detection results.
[0090] In an embodiment of the present invention, a method for massive IP live detection is provided, including: obtaining a list of IP addresses to be detected, and dividing the IP addresses to be detected in the list of IP addresses to be detected to obtain at least one live detection task block, wherein the live detection task block contains a preset number of IP addresses to be detected; creating a goroutine concurrent execution unit for each live detection task block, so that each goroutine concurrent execution unit performs live detection processing on the preset number of IP addresses to be detected in the corresponding live detection task block in parallel; each goroutine concurrent execution unit performs live detection processing on the preset number of target IP addresses to be detected in the corresponding target live detection task block according to the following local ICMP monitoring and response analysis steps: initializing basic variables and creating a monitoring goroutine, wherein the basic variables include: variables for indicating whether the live detection processing is terminated and variables for temporarily storing results, and the monitoring goroutine is used to continuously monitor the ICMP response returned by each target host, and when the ICMP response is monitored, the source IP address in the ICMP response is added as a surviving IP to the variable for temporarily storing results; traversing each target IP address to be detected in the target live detection task block, for each target The target IP address to be detected is parsed to obtain the network object corresponding to each target IP address to be detected, and the corresponding ICMP detection request is customized for each target IP address to be detected; each network object sends the corresponding ICMP detection request to the corresponding target host, so that the target host returns an ICMP response to the listening goroutine according to the network communication protocol, and then the listening goroutine adds the source IP address in the ICMP response as the surviving IP to the variable that temporarily stores the result, wherein the timer records the time when the first ICMP detection request is sent as the starting detection time; If all the target IP addresses to be detected in the target detection task block belong to the variable of the temporary storage result, or the current detection duration determined based on the starting detection time recorded by the timer reaches the target detection duration, the detection processing of the target detection task block is terminated, and the variable used to characterize whether the detection processing is terminated is set to yes, the continuous monitoring of the monitoring goroutine is terminated, and the used resources are released, wherein the target detection duration is calculated according to the number of target IP addresses to be detected in the target detection task block; the source IP addresses stored in the variable of the temporary storage result are summarized to obtain the summary result of the surviving IP addresses.From the above description, it can be known that in the method for massive IP detection of the present invention, each goroutine concurrent execution unit performs detection processing on a preset number of target IP addresses to be detected in the corresponding detection task block in parallel, wherein each goroutine concurrent execution unit performs detection processing on a preset number of target IP addresses to be detected in the corresponding target detection task block according to the local ICMP monitoring and response analysis steps, and uses the listening goroutine to asynchronously (the sending of ICMP detection request is sent by the network object, and the ICMP response is monitored by the listening goroutine, so it is called asynchronous) monitor the ICMP response, and each network object can simultaneously send the corresponding ICMP detection request to the corresponding The corresponding target host is then continuously monitored by the monitoring goroutine for the ICMP responses returned by each target host, and the source IP address in the ICMP response is added as the live IP to the variable for temporarily storing the result, thereby achieving the purpose of simultaneously detecting connectivity for multiple target hosts and improving the efficiency of live detection. At the same time, a timeout mechanism is used to avoid indefinite waiting, thereby increasing the robustness of the algorithm. In addition, by controlling the variables for whether the live detection process is terminated and the goroutine concurrent execution unit, the background monitoring resources can be effectively managed, thereby avoiding resource waste, unreasonable use and high resource consumption, thereby alleviating the technical problems of low efficiency and serious system resource consumption of the existing IP live detection method when detecting a large number of IP addresses.
[0091] The above content briefly introduces the method for detecting massive IP activity of the present invention. The specific contents involved are described in detail below.
[0092] In an optional embodiment of the present invention, obtaining a list of active IP addresses to be detected specifically includes the following steps:
[0093] (1) Obtaining manually input IP data, wherein the IP data includes: IP data represented by CIDR and / or IP data represented by IP address range;
[0094] (2) Calculate the list of active IP addresses to be detected based on the representation of IP data.
[0095] In an optional embodiment of the present invention, each target IP address to be detected is parsed, specifically including the following steps:
[0096] Use net.ResolveIPAddr to resolve each target IP address to be detected, and obtain the network object corresponding to each target IP address to be detected, where the network object includes: *net.IPAddr object;
[0097] Each network object sends the corresponding ICMP liveness request to the corresponding target host, which specifically includes the following steps:
[0098] Each network object calls conn.WriteTo to send the corresponding ICMP detection request to the corresponding target host.
[0099] In an alternative embodiment of the present invention, reference Figure 2 If the current goroutine concurrent execution unit fails to perform live detection processing on the preset number of target IP addresses to be detected in the corresponding target detection task block according to the local ICMP listening and response analysis steps, the current goroutine concurrent execution unit performs live detection processing on the preset number of target IP addresses to be detected in the corresponding target detection task block according to the following non-listening ICMP IP detection processing steps:
[0100] (1) Determine the maximum concurrency based on the number of target IP addresses to be detected in the target detection task block and the preset value, and create a buffer channel based on the maximum concurrency. The buffer channel is used to control the number of concurrent detection goroutines to be the maximum concurrency.
[0101] Specifically, the smaller value between the number of target IP addresses to be detected in the target detection task block and the preset value is used as the maximum concurrent number num to control concurrent resources and reduce resource consumption. The above preset value ensures appropriate use of resources according to the host hardware configuration.
[0102] Create a buffered channel limiter based on the maximum concurrency to control the number of concurrent liveness probing goroutines. The size is num. Assuming the maximum concurrency is 100, only 100 liveness probing goroutines are allowed to execute liveness probing in parallel at the same time. The buffered channel limiter can be understood as a channel with the maximum concurrency. Only one liveness probing goroutine can be executed on a channel at a time. If another liveness probing goroutine also wants to execute on the channel, it needs to wait for the liveness probing goroutine being executed on the channel to complete before obtaining the execution permission.
[0103] (2) Traverse each target IP address to be detected in the target detection task block, and start a detection goroutine for each target IP address to be detected;
[0104] Specifically, the detection goroutine is used to detect the corresponding target IP address to be detected.
[0105] (3) Record the number of probing goroutines and determine the probing goroutines to be executed in the probing goroutines according to the buffer channel. The number of probing goroutines to be executed is the number of probing goroutines that the buffer channel currently allows to execute in parallel.
[0106] Specifically, the number of active goroutines is recorded, that is, the count of WaitGroup is increased using wg.Add(1), which means that a task is added to the coroutine, telling the program to wait for the task to be completed.
[0107] At the same time, the pending detection goroutine is determined in the detection goroutine according to the buffer channel, where the number of pending detection goroutines is the number of detection goroutines that the buffer channel currently allows to execute in parallel. For example, if the number of detection goroutines that the buffer channel currently allows to execute in parallel is 10, then the pending detection goroutines are the 10 unexecuted detection goroutines in the detection goroutine. The pending detection goroutine is used to detect the corresponding target IP address to be detected.
[0108] (4) During the execution of each pending detection goroutine, the ping function based on the ICMP protocol is called to detect the activity of the target pending detection IP address corresponding to each pending detection goroutine;
[0109] Specifically, use code to write a ping function icmpalive (customized) based on the ICMP protocol. There is only one icmpalive, which can be called during the execution of each pending detection goroutine to detect the activity of the target pending detection IP address corresponding to each pending detection goroutine. When Icmpalive detects the activity of the target pending detection IP address corresponding to each pending detection goroutine, it is also implemented by sending ICMP detection requests and receiving ICMP responses, which will not be repeated here.
[0110] (5) If the ping function based on the ICMP protocol returns a live detection result, then the active target IP address to be detected is added to the variable for temporarily storing the result according to the live detection result;
[0111] Specifically, if the ping function icmpalive based on the ICMP protocol returns true, it means that the target host is active, the active host count is updated, and the corresponding target IP address to be detected is added to the variable chanHosts for temporarily storing the result.
[0112] (6) If the pending detection goroutine is completed, the number of pending detection goroutines that have been completed is recorded, and the parallel execution permission of the pending detection goroutine in the buffer channel is released, so that other unexecuted detection goroutines have the permission to execute in parallel in the buffer channel, until the number of pending detection goroutines that have been completed is the same as the number of target pending detection IP addresses in the target detection task block, and the detection processing of the target detection task block is completed;
[0113] Specifically, the number of pending goroutines that have completed execution is recorded, that is, wg.Done() is called to notify completion; and a value is received from the buffer channel limiter to release the execution permission, that is, the parallel execution permission of the pending goroutine in the buffer channel is released.
[0114] Use wg.Wait() to wait for all active goroutines to complete.
[0115] (7) Close the buffer channel to ensure that all resources are released;
[0116] (8) Summarize the source IP addresses (i.e., the target IP addresses to be detected) stored in the variable of the temporary storage result to obtain the summary result of the surviving IP addresses.
[0117] This process is the same as the specific process of the aforementioned related content, and will not be repeated here.
[0118] The non-listening ICMP IP detection process synchronizes all ICMP detection goroutines by limiting the maximum concurrency and using sync.WaitGroup (i.e. wg.Add(1), wg.Done() and wg.Wit()), ensuring effective resource management and accurate detection of host activity status.
[0119] In an optional embodiment of the present invention, the method further comprises the following steps:
[0120] If the target network object sends the corresponding ICMP liveness request to the corresponding target host, and the target host does not return an ICMP response to the listening goroutine, a retry is performed according to the preset retry strategy, where the preset retry strategy includes: a retry strategy with a preset number of retries and a retry strategy with an adjusted retry interval.
[0121] For the process of ICMP IP detection without listening, if the ping function based on the ICMP protocol sends an ICMP live detection request and does not receive an ICMP response, a retry will be performed to increase the success rate of live detection and reduce misjudgment caused by network fluctuations.
[0122] In an optional embodiment of the present invention, the preset retry strategy is obtained by predicting the retry strategy of the current network information using a retry strategy prediction model.
[0123] Specifically, the retry strategy is optimized by introducing machine learning algorithms. Based on historical live detection data (such as the survival time, response time, and number of retries of different IP addresses) and network information, the retry strategy prediction model is trained to predict the activity probability and response time of different IP addresses in different time periods. Based on the prediction results, the number of retries, retry interval and other parameters are dynamically adjusted to make the retry strategy more intelligent and efficient.
[0124] This invention is an IP address detection technology based on the concurrency characteristics of the Go language and the ICMP (Internet Control Message Protocol) protocol. This technology is particularly suitable for scenarios where a large number of IP addresses need to be detected quickly and efficiently. The following are some possible usage scenarios:
[0125] Network monitoring and management: In the scenario of network monitoring and management, administrators need to regularly check whether the devices in the network are online to ensure the stability and availability of the network. This technology can be used to regularly perform IP detection tasks to promptly discover and resolve faults or anomalies in the network.
[0126] Network security: In the field of network security, this technology can be used to check for potential threats that may exist in the network. For example, an attacker may launch a distributed denial of service (DDoS) attack by controlling a large number of zombie hosts. Through this technology, the security team can quickly find out which IP addresses are active and identify and isolate potential attack sources accordingly.
[0127] IoT device management: In the IoT scenario, a large number of devices are connected to the network through IP addresses. Through this technology, administrators of IoT devices can regularly check the online status of devices to ensure the normal operation and timely response of devices.
[0128] Cloud service providers: For cloud service providers, this technology can be used to monitor the availability of cloud service resources they provide. For example, cloud service providers may regularly perform IP liveness detection tasks to ensure that the virtual machines, containers, or other cloud service resources they provide are active, thereby providing users with stable and reliable services.
[0129] Emergency response and disaster recovery: In emergency response and disaster recovery scenarios, this technology can be used to quickly assess whether devices in the network are alive. In the event of a natural disaster, cyber attack or other emergency, this technology can help organizations quickly understand which devices are still available and develop appropriate recovery strategies.
[0130] The network technology involved in the present invention is suitable for scenarios such as large-scale network management, security monitoring, and cloud service providers. Through the technical solution of the present invention, the status information of network devices can be quickly and accurately obtained, providing strong support for network troubleshooting, security auditing, and resource optimization.
[0131] The present invention can solve the problem of how to efficiently detect and determine the active status of a large number of IP addresses in a large-scale network. It solves:
[0132] A. Efficiency issue: Traditional IP detection methods often use serial or simple concurrent methods (manually detecting IPs one by one through ping tools), which is inefficient and cannot quickly process massive IP addresses. The technology of the present invention can process detection requests for multiple IP addresses at the same time by utilizing the concurrent characteristics of the Go language, greatly improving detection efficiency;
[0133] B. Accuracy: In the case of massive IP detection, how to ensure the accuracy of the detection results is a challenge. The technology of the present invention uses the ICMP protocol for automatic detection. The ICMP protocol is a sub-protocol of the TCP / IP protocol family, which is used to transmit control messages between IP hosts and routers. After the detection is completed, the results are automatically summarized, avoiding the errors caused by manual summary, and can accurately reflect the active status of the IP address;
[0134] C. Resource consumption problem: When performing a large number of IP address detections, how to control resource consumption and avoid excessive burden on the network and system is an important issue. The present invention reduces unnecessary resource consumption and improves the stability and scalability of the system through optimization algorithms and concurrency control.
[0135] The effects of the present invention specifically include the following points:
[0136] Efficient concurrent processing: By using the Go language's goroutine, efficient concurrent processing is achieved. The IP addresses to be detected are divided into multiple task blocks, and each goroutine concurrent execution unit is responsible for processing the IP address detection tasks in a task block. This concurrent processing method can make full use of system resources and greatly improve the detection speed, especially when processing massive IP addresses;
[0137] Accurately determine IP activity: Based on the ICMP protocol, the request message is sent and the response is received, which can accurately determine the activity status of the IP address. By parsing the content of the response message, it can eliminate misjudgments caused by network fluctuations or temporary failures, and improve the accuracy of detection;
[0138] Resource utilization optimization: This technology avoids excessive resource consumption through reasonable resource allocation and scheduling. When performing IP detection, this technology will dynamically adjust the number of concurrent detections according to the system load to ensure the reasonable utilization of system resources;
[0139] Convenient result summary and output: The detection results are summarized and output in a preset format, which is convenient for subsequent network management or security monitoring. Users can customize the output format according to their needs, such as exporting to a file or sending to a specified mailbox, which improves the convenience of use.
[0140] The key points of the two exploration processes of the present invention are summarized and explained below:
[0141] Key points of the local ICMP monitoring and response analysis algorithm:
[0142] Background continuous monitoring: Continuous monitoring of ICMP responses is achieved through the monitoring goroutine running in the background, which allows the algorithm to detect active hosts in the network in real time;
[0143] Dynamic timeout mechanism: The algorithm dynamically calculates the timeout (i.e., target detection duration) based on the length of the hostslist (i.e., the number of target IP addresses to be detected in the target detection task block), so that the detection process can adapt to network environments of different sizes;
[0144] Real-time active host update: Once an active host is detected, it is immediately added to the active host list and notified through the channel chanHosts, ensuring the real-time nature of the information;
[0145] Resource management: Use the endflag variable to control the execution and termination of the monitoring goroutine, effectively avoiding resource leakage;
[0146] Global active host list: Introduce the global active host list AliveHosts to centrally manage the active hosts discovered during the detection process to facilitate subsequent processing.
[0147] Beneficial effects:
[0148] Real-time: Ability to monitor and respond to ICMP messages in real time and quickly discover active hosts;
[0149] Dynamic timeout: dynamically calculate the timeout based on the length of hostslist to adapt to network environments of different sizes;
[0150] Resource management: Through endflag and goroutine concurrent execution unit control, background monitoring resources are effectively managed to avoid resource waste, unreasonable use and high resource consumption.
[0151] Key points of the non-listening ICMP IP detection processing algorithm:
[0152] Maximum concurrency control: By setting the maximum concurrency num, the algorithm can control the use of concurrent resources according to the length of the host list (that is, the number of target IP addresses to be detected in the target detection task block) and the preset maximum value (for example, 1000), thereby reducing resource consumption;
[0153] Buffer channel control concurrency: Use the limiter channel to control the number of goroutines that perform ICMP detection to ensure that the maximum concurrency is not exceeded;
[0154] Concurrent detection Goroutine execution: Utilize the concurrent characteristics of goroutine to detect multiple hosts at the same time, thus improving the detection efficiency;
[0155] Active host count update: When an active host is detected, the active host count is updated through livewg.Add(1) and the host name is sent through chanHosts, achieving a quick response to the active host;
[0156] Resource synchronization and release: Use WaitGroup to synchronize the completion of all goroutines and release resources after each active goroutine is executed, ensuring effective resource management;
[0157] Simple resource closing process: The algorithm has designed a clear resource closing process, which ensures that all resources are released by closing the limiter channel, thus improving the stability of the algorithm.
[0158] Beneficial effects:
[0159] Concurrency: By enabling concurrent execution of goroutines, detection efficiency is improved, especially in large-scale networks.
[0160] Resource optimization: By controlling the maximum number of concurrent connections, resource consumption is reduced and system stability and performance are improved;
[0161] Flexibility: The algorithm can dynamically adjust the maximum number of concurrent requests based on the length of the host list (i.e., the number of target IP addresses to be detected in the target detection task block) to adapt to different network sizes.
[0162] Simplicity: The algorithm has a clear structure and is easy to understand and maintain.
[0163] Embodiment 2:
[0164] The embodiment of the present invention further provides a device for massive IP liveness detection, which is mainly used to execute the method for massive IP liveness detection provided in the first embodiment of the present invention. The device for massive IP liveness detection provided in the embodiment of the present invention is specifically introduced below.
[0165] Figure 3 is a schematic diagram of a device for detecting massive IP activity according to an embodiment of the present invention. Figure 3 As shown, the device mainly includes: an acquisition unit 10, a creation unit 20 and a detection processing unit 30, wherein:
[0166] An acquisition unit is used to acquire a list of IP addresses to be detected, and divide the IP addresses to be detected in the list of IP addresses to be detected to obtain at least one detection task block, wherein the detection task block contains a preset number of IP addresses to be detected;
[0167] A creation unit is used to create a goroutine concurrent execution unit for each live detection task block, so that each goroutine concurrent execution unit performs live detection processing on a preset number of to-be-detected IP addresses in the corresponding live detection task block in parallel;
[0168] The live detection processing unit is used for each goroutine concurrent execution unit to perform live detection processing on a preset number of target live detection IP addresses in the corresponding target live detection task block according to the following local ICMP monitoring and response analysis steps:
[0169] Initialize the basic variables and create a monitoring goroutine. The basic variables include: variables used to indicate whether the live detection process has terminated and variables for temporarily storing results. The monitoring goroutine is used to continuously monitor the ICMP responses returned by each target host. When the ICMP response is monitored, the source IP address in the ICMP response is added as the live IP to the variable for temporarily storing results.
[0170] Traverse each target IP address to be detected in the target detection task block, parse each target IP address to be detected, obtain the network object corresponding to each target IP address to be detected, and customize the corresponding ICMP detection request for each target IP address to be detected;
[0171] Each network object sends the corresponding ICMP live detection request to the corresponding target host, so that the target host returns an ICMP response to the listening goroutine according to the network communication protocol, and then the listening goroutine adds the source IP address in the ICMP response as the live IP to the variable that temporarily stores the result, where the timer records the time when the first ICMP live detection request is sent as the starting live detection time;
[0172] If all the target IP addresses to be detected in the target detection task block belong to the variables for temporary storage results, or the current detection duration determined based on the start detection time recorded by the timer reaches the target detection duration, the detection processing of the target detection task block ends, and the variable used to characterize whether the detection processing is terminated is set to yes, the continuous monitoring of the monitoring goroutine is terminated, and the used resources are released, where the target detection duration is calculated based on the number of target IP addresses to be detected in the target detection task block;
[0173] The source IP addresses stored in the variable for temporarily storing the result are summarized to obtain a summary result of the surviving IP addresses.
[0174] In an embodiment of the present invention, a device for massive IP live detection is provided, comprising: obtaining a list of IP addresses to be detected, and dividing the IP addresses to be detected in the list of IP addresses to be detected to obtain at least one live detection task block, wherein the live detection task block contains a preset number of IP addresses to be detected; creating a goroutine concurrent execution unit for each live detection task block, so that each goroutine concurrent execution unit performs live detection processing on the preset number of IP addresses to be detected in the corresponding live detection task block in parallel; each goroutine concurrent execution unit performs live detection processing on the preset number of target IP addresses to be detected in the corresponding target live detection task block according to the following local ICMP monitoring and response analysis steps: initializing basic variables and creating a monitoring goroutine, wherein the basic variables include: variables for indicating whether the live detection processing is terminated and variables for temporarily storing results, and the monitoring goroutine is used to continuously monitor the ICMP response returned by each target host, and when the ICMP response is monitored, the source IP address in the ICMP response is added as a surviving IP to the variable for temporarily storing results; traversing each target IP address to be detected in the target live detection task block, for each target The target IP address to be detected is parsed to obtain the network object corresponding to each target IP address to be detected, and the corresponding ICMP detection request is customized for each target IP address to be detected; each network object sends the corresponding ICMP detection request to the corresponding target host, so that the target host returns an ICMP response to the listening goroutine according to the network communication protocol, and then the listening goroutine adds the source IP address in the ICMP response as the surviving IP to the variable that temporarily stores the result, wherein the timer records the time when the first ICMP detection request is sent as the starting detection time; If all the target IP addresses to be detected in the target detection task block belong to the variable of the temporary storage result, or the current detection duration determined based on the starting detection time recorded by the timer reaches the target detection duration, the detection processing of the target detection task block is terminated, and the variable used to characterize whether the detection processing is terminated is set to yes, the continuous monitoring of the monitoring goroutine is terminated, and the used resources are released, wherein the target detection duration is calculated according to the number of target IP addresses to be detected in the target detection task block; the source IP addresses stored in the variable of the temporary storage result are summarized to obtain the summary result of the surviving IP addresses.From the above description, it can be known that in the device for massive IP detection of the present invention, each goroutine concurrent execution unit performs detection processing on a preset number of target IP addresses to be detected in the corresponding detection task block in parallel, wherein each goroutine concurrent execution unit performs detection processing on a preset number of target IP addresses to be detected in the corresponding target detection task block according to the local ICMP monitoring and response analysis steps, and uses the listening goroutine to asynchronously (the sending of ICMP detection request is sent by the network object, and the ICMP response is monitored by the listening goroutine, so it is called asynchronous) monitor the ICMP response, and each network object can simultaneously send the corresponding ICMP detection request to the corresponding The corresponding target host is then continuously monitored by the monitoring goroutine for the ICMP responses returned by each target host, and the source IP address in the ICMP response is added as the live IP to the variable for temporarily storing the result, thereby achieving the purpose of simultaneously detecting connectivity for multiple target hosts and improving the efficiency of live detection. At the same time, a timeout mechanism is used to avoid indefinite waiting, thereby increasing the robustness of the algorithm. In addition, by controlling the variables for whether the live detection process is terminated and the goroutine concurrent execution unit, the background monitoring resources can be effectively managed, thereby avoiding resource waste, unreasonable use and high resource consumption, thereby alleviating the technical problems of low efficiency and serious system resource consumption of the existing IP live detection method when detecting a large number of IP addresses.
[0175] Optionally, the acquisition unit is further used to: acquire manually input IP data, wherein the IP data includes: IP data represented by CIDR, and / or IP data represented by IP address range; and calculate the list of active IP addresses to be detected according to the representation of the IP data.
[0176] Optionally, the live detection processing unit is also used to: use net.ResolveIPAddr to resolve each target IP address to be detected, and obtain the network object corresponding to each target IP address to be detected, wherein the network object includes: *net.IPAddr object; the live detection processing unit is also used to: each network object calls conn.WriteTo to send the corresponding ICMP live detection request to the corresponding target host.
[0177] Optionally, the detection processing unit is also used to: determine the maximum concurrency number according to the number of target IP addresses to be detected in the target detection task block and a preset value, and create a buffer channel according to the maximum concurrency number, wherein the buffer channel is used to control the number of concurrent detection goroutines to be the maximum concurrency number; traverse each target IP address to be detected in the target detection task block, and start a detection goroutine for each target IP address to be detected; record the number of detection goroutines, and determine the detection goroutine to be executed in the detection goroutine according to the buffer channel, wherein the number of detection goroutines to be executed is the number of detection goroutines currently allowed to be executed in parallel by the buffer channel; during the execution of each detection goroutine to be executed, call the ping function based on the ICMP protocol to detect each detection goroutine to be executed ine corresponding to the target to-be-detected live IP address; if the ping function based on the ICMP protocol returns a detection result, then according to the detection result, the active corresponding target to-be-detected live IP address is added to the variable for temporary storage of the result; if the to-be-executed detection live goroutine is executed, then the number of completed to-be-executed detection live goroutines is recorded, and the parallel execution permission of the to-be-executed detection live goroutine in the buffer channel is released, so that other unexecuted detection live goroutines enjoy the permission to execute in parallel in the buffer channel, until the number of completed to-be-executed detection live goroutines is the same as the number of target to-be-detected live IP addresses in the target detection live task block, and the detection processing of the target detection live task block is completed; the buffer channel is closed; the source IP addresses stored in the variable for temporary storage of the result are summarized to obtain the summary result of the surviving IP addresses.
[0178] Optionally, the liveness detection processing unit is further configured to: use the smaller value between the number of target liveness detection IP addresses in the target liveness detection task block and a preset value as the maximum concurrent number.
[0179] Optionally, the liveness detection processing unit is also used for: if the target network object sends the corresponding ICMP liveness detection request to the corresponding target host, and the target host does not return an ICMP response to the listening goroutine, then retry according to a preset retry strategy, wherein the preset retry strategy includes: a retry strategy with a preset number of retries and a retry strategy with an adjusted retry interval.
[0180] Optionally, the preset retry strategy is obtained by a retry strategy prediction model performing retry strategy prediction on current network information.
[0181] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.
[0182] like Figure 4 As shown, an electronic device 600 provided in an embodiment of the present application includes: a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 through the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the method for massive IP detection as described above.
[0183] Specifically, the memory 602 and the processor 601 can be general-purpose memories and processors, which are not specifically limited here. When the processor 601 runs the computer program stored in the memory 602, the method for detecting massive IP activity can be executed.
[0184] The processor 601 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 601. The above processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 602, and the processor 601 reads the information in the memory 602 and completes the steps of the above method in combination with its hardware.
[0185] Corresponding to the above-mentioned method for detecting massive IP activity, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to execute the steps of the above-mentioned method for detecting massive IP activity.
[0186] The device for massive IP detection provided in the embodiment of the present application can be specific hardware on the device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in the embodiment of the present application are the same as those of the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.
[0187] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.
[0188] For another example, the flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, method and computer program product according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0189] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0190] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0191] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method for massive IP detection described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.
[0192] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.
[0193] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solution of the present application, rather than to limit it. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solution recorded in the aforementioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solution deviate from the scope of the technical solution of the embodiment of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A method for detecting massive IP activity. It is characterized in that include: Obtain a list of IP addresses to be detected, and divide the IP addresses to be detected in the list of IP addresses to be detected to obtain at least one detection task block, wherein the detection task block contains a preset number of IP addresses to be detected; A goroutine concurrent execution unit is created for each of the live detection task blocks, so that each of the goroutine concurrent execution units performs live detection processing on a preset number of to-be-detected IP addresses in the corresponding live detection task block in parallel; Each goroutine concurrent execution unit performs live detection processing on a preset number of target live detection IP addresses in the corresponding target live detection task block according to the following local ICMP monitoring and response analysis steps: Initialize basic variables and create a monitoring goroutine, wherein the basic variables include: variables used to indicate whether the live detection process is terminated and variables for temporarily storing results. The monitoring goroutine is used to continuously monitor the ICMP responses returned by each target host. When the ICMP response is monitored, the source IP address in the ICMP response is added as a live IP to the variable for temporarily storing results. Traversing each target IP address to be detected in the target detection task block, parsing each target IP address to be detected, obtaining a network object corresponding to each target IP address to be detected, and custom-generating a corresponding ICMP detection request for each target IP address to be detected; Each of the network objects sends a corresponding ICMP live detection request to a corresponding target host, so that the target host returns an ICMP response to the listening goroutine according to the network communication protocol, and then the listening goroutine adds the source IP address in the ICMP response as a live IP to the variable of the temporary storage result, wherein the timer records the time when the first ICMP live detection request is sent as the starting live detection time; If all target IP addresses to be detected in the target detection task block belong to the variables of the temporary storage result, or the current detection duration determined based on the start detection time recorded by the timer reaches the target detection duration, the detection processing of the target detection task block is terminated, and the variable used to characterize whether the detection processing is terminated is set to yes, the continuous monitoring of the monitoring goroutine is terminated, and the used resources are released, wherein the target detection duration is calculated according to the number of target IP addresses to be detected in the target detection task block; The source IP addresses stored in the variable of the temporary storage result are summarized to obtain a summary result of the surviving IP addresses.
2. The method according to claim 1, It is characterized in that Get the list of active IP addresses to be detected, including: Acquire manually input IP data, wherein the IP data includes: IP data represented by CIDR, and / or IP data represented by IP address range; The list of active IP addresses to be detected is obtained by calculation according to the representation method of the IP data.
3. The method according to claim 1, It is characterized in that Each of the target IP addresses to be detected is parsed, including: Use net.ResolveIPAddr to resolve each of the target IP addresses to be detected, and obtain the network object corresponding to each of the target IP addresses to be detected, wherein the network object includes: *net.IPAddr object; Each of the network objects sends a corresponding ICMP liveness request to a corresponding target host, including: Each of the network objects calls conn.WriteTo to send the corresponding ICMP detection request to the corresponding target host.
4. The method according to claim 1, It is characterized in that If the current goroutine concurrent execution unit fails to perform live detection processing on a preset number of target live IP addresses to be detected in the corresponding target live detection task block according to the local ICMP listening and response analysis steps, the current goroutine concurrent execution unit performs live detection processing on a preset number of target live IP addresses to be detected in the corresponding target live detection task block according to the following non-listening ICMP IP detection processing steps: Determine the maximum concurrency number according to the number of target live IP addresses to be detected in the target live task block and a preset value, and create a buffer channel according to the maximum concurrency number, wherein the buffer channel is used to control the number of concurrent live detection goroutines to be at most the maximum concurrency number; Traverse each target IP address to be detected in the target detection task block, and start a detection goroutine for each target IP address to be detected; Record the number of the probing goroutines, and determine the probing goroutines to be executed in the probing goroutines according to the buffer channel, wherein the number of the probing goroutines to be executed is the number of probing goroutines currently allowed to be executed in parallel by the buffer channel; During the execution of each of the to-be-executed detection goroutines, a ping function based on the ICMP protocol is called to detect the activity of the target to-be-detected IP address corresponding to each of the to-be-executed detection goroutines; If the ping function based on the ICMP protocol returns a live detection result, then according to the live detection result, the active target IP address to be detected is added to the variable for temporarily storing the result; If the pending detection goroutine is completed, the number of completed pending detection goroutines is recorded, and the parallel execution permission of the pending detection goroutine in the buffer channel is released, so that other unexecuted detection goroutines enjoy the permission of parallel execution in the buffer channel, until the number of completed pending detection goroutines is the same as the number of target pending detection IP addresses in the target detection task block, and the detection processing of the target detection task block is completed; closing the buffer channel; The source IP addresses stored in the variable of the temporary storage result are summarized to obtain a summary result of the surviving IP addresses.
5. The method according to claim 4, It is characterized in that The maximum concurrency number is determined according to the number of target IP addresses to be detected in the target detection task block and a preset value, including: The smaller value between the number of target IP addresses to be detected in the target detection task block and the preset value is used as the maximum concurrent number.
6. The method according to claim 1, It is characterized in that The method further comprises: If the target network object sends the corresponding ICMP detection request to the corresponding target host, and the target host does not return an ICMP response to the listening goroutine, a retry is performed according to a preset retry strategy, wherein the preset retry strategy includes: a retry strategy with a preset number of retries and a retry strategy with an adjusted retry interval.
7. The method according to claim 6, It is characterized in that The preset retry strategy is obtained by the retry strategy prediction model performing retry strategy prediction on the current network information.
8. A device for detecting massive IP activity. It is characterized in that include: An acquisition unit is used to acquire a list of IP addresses to be detected, and divide the IP addresses to be detected in the list of IP addresses to be detected to obtain at least one detection task block, wherein the detection task block contains a preset number of IP addresses to be detected; A creation unit is used to create a goroutine concurrent execution unit for each of the live detection task blocks, so that each of the goroutine concurrent execution units performs live detection processing on a preset number of to-be-detected IP addresses in the corresponding live detection task block in parallel; The live detection processing unit is used for each goroutine concurrent execution unit to perform live detection processing on a preset number of target live detection IP addresses in the corresponding target live detection task block according to the following local ICMP monitoring and response analysis steps: Initialize basic variables and create a monitoring goroutine, wherein the basic variables include: variables used to indicate whether the live detection process is terminated and variables for temporarily storing results. The monitoring goroutine is used to continuously monitor the ICMP responses returned by each target host. When the ICMP response is monitored, the source IP address in the ICMP response is added as a live IP to the variable for temporarily storing results. Traversing each target IP address to be detected in the target detection task block, parsing each target IP address to be detected, obtaining a network object corresponding to each target IP address to be detected, and custom-generating a corresponding ICMP detection request for each target IP address to be detected; Each of the network objects sends a corresponding ICMP live detection request to a corresponding target host, so that the target host returns an ICMP response to the listening goroutine according to the network communication protocol, and then the listening goroutine adds the source IP address in the ICMP response as a live IP to the variable of the temporary storage result, wherein the timer records the time when the first ICMP live detection request is sent as the starting live detection time; If all target IP addresses to be detected in the target detection task block belong to the variables of the temporary storage result, or the current detection duration determined based on the start detection time recorded by the timer reaches the target detection duration, the detection processing of the target detection task block is terminated, and the variable used to characterize whether the detection processing is terminated is set to yes, the continuous monitoring of the monitoring goroutine is terminated, and the used resources are released, wherein the target detection duration is calculated according to the number of target IP addresses to be detected in the target detection task block; The source IP addresses stored in the variable of the temporary storage result are summarized to obtain a summary result of the surviving IP addresses.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to execute the method according to any one of claims 1 to 7.
