A multi-cloud collaborative distributed network security scanning method

Abstract

The application discloses a kind of multi-cloud collaborative distributed network security scanning method, it is related to network security technical field;The method comprises: receiving the network security scanning task submitted by user, quantitatively analyzes user authentication information, historical behavior information and current task information, obtains user confidence, and dynamically limits scanning authority accordingly;According to the number of online agents, the effective scanning target is dynamically fragmented, and the fragment is mapped to multiple cloud platforms according to the preset strategy;The scanning fragment is issued to the cloud host through the WSS encryption tunnel, and the cloud host is distributed to the scanning agent node using the lock-free concurrent distribution mechanism;Scan agent node executes multi-stage parallel scanning and reports the result in real time through streaming reporter;The master control center carries out aggregate analysis and unique index deduplication to multi-source result, and recycles and redistributes timeout fragment, improves the efficiency and real-time performance of large-scale network security scanning through multi-cloud collaboration, dynamic fragmentation, lock-free concurrency and streaming return.

Description

Technical Field

[0001] This invention relates to the field of network security technology, specifically to a multi-cloud collaborative distributed network security scanning method. Background Technology

[0002] With the rapid development of enterprise digital transformation and cloud computing technology, the scale of enterprise network assets is constantly expanding, and the network environment is gradually showing characteristics of cross-regional, multi-cloud, and dynamic. A large number of servers, containers, web applications, API interfaces, and IoT devices are continuously connecting to the Internet, resulting in a continuous expansion of the network attack surface. To promptly detect security vulnerabilities, weak passwords, abnormally open ports, and service configuration defects in network systems, network security scanning technology is widely used in enterprise security operations, vulnerability management, and attack and defense drills.

[0003] Most existing network security scanning solutions adopt a standalone or single-cloud centralized scanning architecture. They typically use a single server or fixed cluster to perform port probing, service identification, and vulnerability detection on target assets. However, when faced with hundreds of thousands or even millions of internet assets, the traditional standalone scanning mode is prone to bottlenecks in CPU, memory, and network bandwidth resources as the task scale increases, resulting in a significant decrease in scanning efficiency. Large-scale scanning tasks often need to last for several hours or even days, making it difficult to meet the needs of real-time security detection. Summary of the Invention

[0004] To address the problems in related technologies, this invention provides a multi-cloud collaborative distributed network security scanning method to overcome the aforementioned technical problems in existing related technologies.

[0005] To solve the aforementioned technical problem, the present invention is achieved through the following technical solution:

[0006] A multi-cloud collaborative distributed network security scanning method includes:

[0007] The system receives network security scanning tasks submitted by users, obtains a list of targets to be scanned and scanning parameters; based on the user's authentication information, historical behavior information, and dynamic constraints of the current scanning task's target list through ternary parsing, it selects valid targets from the target list; it performs dynamic sharding based on the number of valid targets and available scanning resources, and maps each shard to different cloud platforms; it distributes the shards through an encrypted communication channel; it performs lock-free concurrent allocation of the shards within the corresponding cloud platform, executes network security scanning, and reports the scanning results in real time.

[0008] Preferably, the authentication information includes at least the authentication token issuance time, authentication transition records, and the transition time corresponding to each authentication transition; the token age is calculated based on the time difference between the authentication token issuance time and the current time, and the token validity is calculated based on an exponential decay model; the transition frequency is calculated based on the number of authentication transitions, and a transition penalty is calculated based on each authentication transition time; then, the authentication stability is calculated based on the transition frequency and the transition penalty, and the authentication credibility is obtained by weighted fusion of the token validity and the authentication stability, specifically including:

[0009] The token age (Age) is calculated by taking the time difference between the token issuance time and the current time, and then calculated according to the formula. The validity period of the token is calculated. Let P be the time decay constant; calculate the transition frequency P based on the transitions present in the authentication and the time of each transition; then, perform time decay on the transitions based on the time of each transition to obtain the transition penalty term F for each transition, calculated using the following formula: Where now represents the current time. This represents the transition time of the i-th transition; Let P be the transition time decay constant; the closer the transition occurs to the current moment, the larger the corresponding term, indicating that the current authentication state is more unstable; then, the transition frequency P and the transition penalty term F are used to calculate the authentication stability, and the calculation formula is: , where α and β are the frequency penalty coefficient and the jump penalty coefficient, respectively; the token validity and authentication stability are weighted and summed, and then multiplied by 100 to obtain the authentication credibility, with the sum of the weights being one.

[0010] Preferably, the historical behavior information includes at least historical submission records, the submission result corresponding to each historical submission record, and the submission time; wherein, the submission result includes successful submission, partially successful submission, and failed submission; a result indicator variable is assigned to each historical submission record, and a time decay weight is calculated based on the submission time; the result indicator variable and the time decay weight are combined to obtain the success contribution and failure contribution respectively, and the behavior credibility is calculated based on the success contribution and failure contribution; specifically including:

[0011] For each historical commit record, a binary indicator variable is assigned to a successful commit (1), a binary indicator variable to a failed commit (0), and a binary indicator variable to a partially successful commit (0.5). This yields the binary indicator variable for each historical commit record. The decay weight for each historical commit record is obtained by applying time decay based on the commit time, calculated using the following formula: ,in This represents the time difference between the submission time and the current time. Let $\frac{ ... The failure contribution is summed for all commit records: This involves multiplying the binary indicator variable of all commit records by the corresponding decay weight, and then summing the results to obtain the failure contribution. To make the suppression of credibility by failure more significant, a failure penalty coefficient λ > 1 is set; the contribution of success is... Failure Contribution The failure penalty coefficient λ is obtained through the formula Calculate the behavioral credibility , where ε is a minimal constant to prevent the denominator from being zero.

[0012] Preferably, the current scanning task includes at least a list of targets to be scanned, target type, scanning type, scanning port range, concurrency level, timeout, target source type, whether full pipeline scanning is enabled, and task submission time; a set of keywords is parsed from the targets to be scanned in the current scanning task, and compared with a preset keyword library to obtain the hit keywords; a comprehensive effective weight is assigned to each hit keyword, and the sensitivity of the scanned target is calculated based on the coverage of the hit keywords in the set; the overlap between the current scanning task and the user's historical scans is compared and analyzed to obtain the overlap degree, and finally, the task credibility is calculated based on the sensitivity of the scanning task and the overlap degree of historical scanning tasks; specifically including:

[0013] A keyword library is pre-established, with each keyword corresponding to a basic sensitivity weight. For any target in the scanning task, its text content is identified, all keywords are parsed, and the hit keywords are extracted. A positional weight is assigned to each hit keyword based on its location within the scanning task; the positional weight indicates the sensitivity of the hit keyword's location. The basic sensitivity weight of the hit keyword is then multiplied by the positional weight to obtain the effective weight of the hit keyword. Finally, the effective weights of multiple hit keywords are non-linearly aggregated to obtain the comprehensive effective weight of the scanned target. The non-linear aggregation formula is as follows: Where k represents the index of the hit keyword, The effective weight for hitting keyword k is defined; finally, the sensitivity of the scanning task is calculated using a coverage amplification nonlinear model, which combines the effective weight E and the hit coverage R. The coverage amplification nonlinear model is as follows: , where γ is the coverage magnification factor, and γ > 0;

[0014] The system retrieves the user's historical scanning tasks, their lists of targets to be scanned, and scan times. It then compares these historical tasks with the current scanning task's target list to identify duplicate targets, extracting the number of duplicate targets. The system also counts the number of unique targets in both lists, representing the intersection and union of targets between the two scanning tasks. Dividing the intersection count by the union count yields the scan overlap rate, which in turn determines the scan overlap rate between the current scanning task and all historical scanning tasks. Finally, the scan times of historical tasks are calculated using the time decay formula. After reduction, the attenuation weight q of the historical scanning task is obtained, where This represents the time difference between the scan time and the current time. To verify the attenuation constant, the scan overlap is then weighted and summed according to the attenuation weights of its corresponding historical scan tasks, and then divided by the sum of the attenuation weights to obtain the historical overlap Q. The sensitivity of the scan task is combined with the attenuated overlap to obtain the task reliability. The calculation formula is: .

[0015] Preferably, each keyword in the keyword library corresponds to a basic sensitivity weight, thereby obtaining the basic sensitivity weight of the hit keyword. Furthermore, a position weight is assigned based on the location of the hit keyword in the scanning target. The effective weight of the hit keyword is calculated based on the basic sensitivity weight and the position weight. Finally, the effective weights of multiple hit keywords are nonlinearly aggregated to obtain the comprehensive effective weight of the scanning target.

[0016] Preferably, the dynamic constraints of the ternary parsing include: determining user confidence based on authentication credibility, behavior credibility, and task credibility, and limiting the upper limit of allowed target weights, allowed scan types, and allowed concurrency levels in the current list of targets to be scanned based on the user confidence; when the effective weight of a scanned target exceeds the upper limit of the target weight, the corresponding scanned target is removed from the list of targets to be scanned, thereby selecting effective scanned targets from the list of targets to be scanned in the scanning task.

[0017] Preferably, the number of available scanning resources is obtained through a three-level verification, which includes the following steps: querying the scanning agent records that remain active within a preset time range in the database heartbeat table to obtain the first online number; summarizing the scanning agent lists returned by each cloud platform to obtain the second online number; querying the number of instances running on the cloud platform to obtain the third online number; when the deviation between the online number of one layer and the online number of the other two layers exceeds a preset threshold, the online number of the corresponding layer is judged as an outlier and removed, and then the final number of available scanning resources, i.e., the final number of online agents, is calculated based on the remaining online number.

[0018] Preferably, the basic segmentation granularity is calculated based on the number of effective scanning targets and the final number of available scanning resources, and the basic segmentation granularity is trimmed according to the preset segmentation granularity range corresponding to the scanning type to obtain the final segmentation granularity.

[0019] Preferably, mapping each scan shard to different cloud platforms includes: selecting a corresponding sharding strategy based on the scan target type and the cloud platform resource status; the sharding strategy includes at least one of round-robin allocation, IP segment hash allocation, domain name hash allocation, weight allocation, and single cloud allocation.

[0020] Preferably, the encrypted communication channel adopts a WSS-based encrypted long connection method and uses JWT token authentication to reuse task distribution, result reporting, command control and status synchronization messages; the lock-free concurrent allocation adopts PostgreSQL's FOR UPDATE SKIP LOCKED mechanism, so that when multiple scan agents pull scan shards at the same time, they automatically skip the locked scan shards.

[0021] The present invention has the following beneficial effects:

[0022] This invention first dynamically adjusts user scanning task permissions based on a three-element dynamic constraint of authentication credibility, behavior credibility, and task credibility before the task enters scheduling. This makes scanning permissions no longer dependent on static roles or fixed whitelists, but rather change in real time according to the user's authentication status, historical operation stability, and current task risk. Second, it dynamically calculates the sharding granularity based on the online agent's three-level verification results and distributes tasks to multiple cloud platforms according to target characteristics. The cloud master control machine allocates tasks to scanning agent nodes in a lock-free concurrent manner. Combined with multi-engine parallel scanning, streaming result reporting, unified deduplication and aggregation, and timeout recovery and reassignment processes on the agent side, this reduces the CPU overload, memory accumulation, and network bandwidth congestion problems that easily occur in large-scale target scenarios when using centralized scanning on a single machine or single cloud. It improves scanning throughput efficiency, resource utilization, and fault recovery capabilities, thereby effectively solving the problems of long time consumption, poor real-time performance, insufficient cross-cloud collaboration, and low degree of automation in large-scale network security scanning.

[0023] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, the drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 The flowchart illustrates a multi-cloud collaborative distributed network security scanning method provided by this invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Application Scenario: Most existing network security scanning solutions adopt a standalone or single-cloud centralized scanning architecture. They typically use a single server or fixed cluster to perform port probing, service identification, and vulnerability detection on target assets. However, when facing hundreds of thousands or even millions of internet assets, the traditional standalone scanning mode is prone to bottlenecks in CPU, memory, and network bandwidth resources as the task scale increases, resulting in a significant decrease in scanning efficiency. Large-scale scanning tasks often require several hours or even days to complete, making it difficult to meet real-time security detection requirements. This embodiment is applied to a multi-cloud collaborative distributed scanning platform consisting of a main control center, multiple cloud master controllers, and multiple scanning agent nodes. The main control center is deployed in a local data center, each cloud master controller is deployed on a different cloud platform, and each scanning agent node is deployed in an elastic computing instance on the corresponding cloud platform. The main control center is responsible for global task management, multi-cloud resource orchestration, and result aggregation and analysis. The cloud master controllers are responsible for task consumption, node management, and result collection within their respective clouds, while the scanning agent nodes are responsible for specific scan execution and streaming reporting.

[0028] To address the problems mentioned in the background section, such as Figure 1 As shown, this embodiment of the invention provides a multi-cloud collaborative distributed network security scanning method, specifically including:

[0029] Step 1: The main control center receives the network security scanning task submitted by the user, obtains the list of targets to be scanned and the corresponding scanning parameters. The list of targets includes at least one of IP addresses, domain names, CIDR network segments, or URL addresses. The scanning parameters include scan type, scan port range, concurrency level, timeout, target source type, whether full pipeline scanning is enabled, and task submission time. Before entering the scheduling process, the task's legality is verified, specifically including:

[0030] 101. Based on the user submitting the network security scanning task, obtain user information, which includes at least identity identifier, authentication information, historical behavior information, and current scanning task information. Authentication information includes the authentication token issuance time, authentication transitions, and the time of each transition. Transitions refer to token refreshes, login session switching, secondary authentication triggers, etc. Frequent transitions usually indicate an unstable authentication chain. Calculate the token age (Age) by the time difference between the token issuance time and the current time, and then apply the formula... The validity period of the token is calculated. The validity period is a constant, which is set to 24 hours in this embodiment. The older the token is, the shorter its validity period. The frequency P is calculated based on the authentication transitions and the duration of each transition. Then, the transition penalty F for each transition is obtained by applying time decay based on the duration of each transition. The calculation formula is as follows: Where now represents the current time. This represents the transition time of the i-th transition; The transition time decay constant is set to 12 hours in this embodiment. The closer the transition occurs to the current moment, the larger the corresponding term, indicating a more unstable current authentication state. The authentication stability is then calculated using the transition frequency P and the transition penalty term F, with the following formula: Where α and β are the frequency penalty coefficient and the jump penalty coefficient, respectively. In this embodiment, the values ​​are 0.8 and 1.2, respectively. The token validity and authentication stability are weighted and summed, and then multiplied by 100 to obtain the authentication credibility. The sum of the weights is one. It should be noted that in this embodiment, the weight of token validity is less than the weight of authentication stability to highlight the impact of jump stability on authentication credibility, because multiple jumps in a short period of time usually reflect the current authentication risk better than simple token obsolescence.

[0031] 102. Historical behavior information includes each historical commit record, each commit result, and the commit time. The commit result includes successful, partially successful, and failed commits. For each historical commit record, a successful commit is assigned a binary indicator variable of 1, a failed commit is assigned a binary indicator variable of 0, and a partially successful commit is assigned a binary indicator variable of 0.5. This yields the binary indicator variable for each historical commit record. The decay weight for each historical commit record is obtained by applying time decay based on the commit time, calculated using the following formula: ,in This represents the time difference between the submission time and the current time. The behavior decay constant is set to 72 hours in this embodiment. The success contribution is obtained by summing the success portions of all commit records: multiplying the binary indicator variable of all commit records by the decay weight of the corresponding commit record, and then summing the results. The failure contribution is summed for all commit records: This involves multiplying the binary indicator variable of all commit records by the corresponding decay weight, and then summing the results to obtain the failure contribution. To make the suppression of credibility by failure more significant, this embodiment sets a failure penalty coefficient λ, where λ > 1 and takes a value of 1.5; the contribution of success is... Failure Contribution The failure penalty coefficient λ is obtained through the formula Calculate the behavioral credibility Where ε is a minimal constant to prevent the denominator from being zero, and in this embodiment, it is taken as 10. -6 It should be noted that after both successes and failures undergo time decay, the more successes accumulated and the fewer failures accumulated, the higher the credibility of the behavior; if there are many recent failures, the credibility of the behavior will drop rapidly. Because of the time decay, recent failures will have a more significant impact on current permissions than failures from a long time ago.

[0032] 103. Extract the list of targets to be scanned and the corresponding scanning parameters from the pre-scan task. The scanning parameters include at least one of IP address, domain name, CIDR network segment, or URL address. The scanning parameters must include at least the list of targets to be scanned, target type, scan type, scan port range, concurrency level, timeout, target source type, whether full pipeline scanning is enabled, and task submission time. A keyword library should be pre-established, with each keyword corresponding to a basic sensitivity weight. It should be noted that different keywords have different indicative capabilities of target risk. For example, keywords such as login entry, management directory, console interface, authentication credentials, and sensitive configurations are more likely to point to high-risk resources and should therefore be assigned higher basic sensitivity weights. Keywords that only represent general business descriptions... A lower base sensitivity weight is assigned. For any target in the scanning task, its text content is identified, all keywords are parsed, and the hit keywords are extracted. A position weight is assigned based on the position of the hit keyword in the scanning task. The position weight indicates the sensitivity of the hit keyword's position. In this embodiment, keywords appearing in the login path, management directory, and console interface have higher position weights, ranging from 0.8 to 1.0; keywords appearing only in the main text description have lower position weights, ranging from 0.2 to 0.4. Then, the base sensitivity weight of the hit keyword is multiplied by the position weight to obtain the effective weight of the hit keyword. The effective weights of multiple hit keywords are then non-linearly aggregated to obtain the comprehensive effective weight of the scanning target. The non-linear aggregation formula is: Where k represents the index of the hit keyword, The effective weight for hitting keyword k is defined; finally, the sensitivity of the scanning task is calculated using a coverage amplification nonlinear model, which combines the effective weight E and the hit coverage R. The coverage amplification nonlinear model is as follows: , where γ is the coverage magnification factor, γ>0, and in this embodiment, γ is taken as 2.0;

[0033] The system retrieves the user's historical scanning tasks, their lists of targets to be scanned, and scan times. It then compares these historical tasks with the current scanning task's target list to identify duplicate targets, extracting the number of duplicate targets. The system also counts the number of unique targets in both lists, representing the intersection and union of targets between the two scanning tasks. Dividing the intersection count by the union count yields the scan overlap rate, which in turn determines the scan overlap rate between the current scanning task and all historical scanning tasks. Finally, the scan times of historical tasks are calculated using the time decay formula. After reduction, the attenuation weight q of the historical scanning task is obtained, where This represents the time difference between the scan time and the current time. To verify the attenuation constant, in this embodiment, a value of 30 days is used. The scan overlap is then weighted and summed according to the attenuation weights of its corresponding historical scan tasks, and divided by the sum of the attenuation weights to obtain the historical overlap Q. It should be noted that since more recent historical tasks better represent the current customer's actual usage pattern, the overlap of each historical task needs to be attenuated according to the submission time. The overall task sensitivity is combined with the attenuated overlap to obtain the task credibility. The calculation formula is: It should be noted that the less sensitive the task, the better. The larger the value, the higher the credibility of the task; the closer the task is to historical normal patterns. The larger the value, the higher the credibility of the task.

[0034] 104. The authentication credibility, behavior credibility, and task credibility are weighted and summed according to preset weights to obtain the user confidence score. The sum of the weights is one. In this embodiment, the weights of behavior credibility and task credibility can be slightly higher than those of authentication credibility. The weights of authentication credibility, behavior credibility, and task credibility are set to 0.25, 0.35, and 0.40, respectively. A preset confidence interval is set. In this embodiment, the confidence interval is set to [40, 80]. If the user confidence score is greater than the maximum value of the confidence interval, the main control center allows the user to perform scanning on targets with effective weights less than or equal to one, with a concurrency level limited to 16. At the same time, the user is allowed to enter the complete multi-cloud collaborative sharding and scheduling process, and a successful submission record is accumulated. If the user confidence score is within the confidence interval, the main control center allows the user to perform scanning on targets with effective weights less than or equal to one. Scanning targets with a weight of 0.60 or less will be scanned. The scan type can be expanded to include port scanning, service identification, and vulnerability scanning within a limited scope. The range of scanned ports is limited to 1-49152, the concurrency level is limited to 4, and full pipeline scanning is not allowed. One partially successful submission record will be accumulated. If the user's confidence level is less than the minimum value of the confidence interval, the control center will determine that the user does not have the permission to execute the scan task, directly reject the task from entering the scheduling process, retain only the audit record, and accumulate one failed submission record. Within any of the above permission intervals, if the sensitivity of a scan target exceeds the upper limit of the sensitivity allowed in that interval, the target will be immediately removed from the current list of targets to be scanned. If the target list is empty after removal, the task will not enter the subsequent sharding and scheduling process.

[0035] Step two: After completing user confidence verification and filtering out invalid scan targets, the main control center no longer uses the single-machine centralized scanning method. Instead, based on the current number of online agents and the available resource status of each cloud platform, it performs dynamic sharding and multi-cloud mapping on the remaining valid scan targets. Specifically, this includes:

[0036] 201. Based on the database heartbeat table, the Agent list returned by each cloud master control machine, and the number of running instances on the cloud platform, a three-level verification is performed on the number of online Agents to reflect the current available computing power as accurately as possible; including:

[0037] The control center queries the database heartbeat table, extracts Agent records that have maintained an active heartbeat within the last 5 minutes, and uses records with heartbeat time within a threshold and an online status field as the first-layer online Agent candidate set, denoted as N1. Since the database heartbeat table may be affected by synchronization delays, short-term network jitter, or write lag, the control center further queries the Agent lists returned by each cloud master controller in parallel, summarizing and deduplicating the currently registered online Agents of each cloud master controller to obtain the second-layer online quantity N2. Finally, the control center queries the number of instances running on each cloud platform, and combines this with whether the instance has a preset Agent running identifier, whether it is running, and whether it has completed the necessary startup registration to obtain the third-layer online quantity N3. In this embodiment, the three-layer verification is not a simple layer-by-layer replacement, but rather a consistency verification performed in the order of database heartbeat priority, cloud master controller report second, and cloud instance status as a fallback: if N1, N2, and... If the deviations between N3 are all within the preset tolerance range, the weighted consistent value obtained by weighting the online numbers of the three layers according to the preset weights is used as the final online Agent number. If the relative deviation between the online number of a certain layer and the online number of the other two layers exceeds the preset threshold, the online number of that layer is judged as an outlier, and the data of that layer is removed from the final online Agent number calculation. The online Agent number is recalculated only based on the remaining normal layer data. In this embodiment, the weights of the three layers of data are set to 0.5, 0.3 and 0.2 respectively, and the final online Agent number is defined as: Nagent=round(0.5×N1+0.3×N2+0.2×N3) / (0.5+0.3+0.2). If a certain layer is temporarily unavailable, the weights are re-normalized according to the remaining layers and the calculation is performed to avoid the distortion of a single data source leading to incorrect sharding granularity judgment, thereby ensuring that subsequent sharding and multi-cloud scheduling are based on a stable resource awareness foundation.

[0038] 202. After obtaining the final number of online agents, the control center calculates the basic shard granularity based on the scanning targets selected in step one, specifically including:

[0039] The total number of remaining scan targets after legality verification and sensitivity filtering is counted as the number of valid scan targets. This number is then divided by the final number of online agents, and the result is rounded up to obtain the number of targets per agent. To ensure that the execution time of a single shard after sharding matches the scan type, the number of targets per agent is pruned according to the scan type to form the final shard granularity. The scan type is determined by the scan parameters in step one. For different scan types, a corresponding shard granularity range is pre-configured. This range is obtained from offline benchmark testing, i.e., the average single-target execution latency, CPU usage, and internal memory usage are statistically analyzed on a standard target set for that scan type. The peak data and network egress consumption are combined with the allowed task timeout limit for this scan type to deduce a reasonable sharding range. In this embodiment, taking a full pipeline scan as an example, the shard size range is set to 20-500. For other types such as port scanning, service identification, and vulnerability scanning, a corresponding range mapping table is established according to the resource consumption characteristics of each engine and the preset execution budget. Thus, the target number corresponding to a single Agent is trimmed according to the sharding granularity range corresponding to its scan type to obtain the final sharding granularity. That is, first, it is determined whether the basic sharding granularity is lower than the lower limit of this scan type. If it is lower, it is increased to the lower limit; if it is higher than the upper limit, it is compressed to the upper limit; if it is within the range, it remains unchanged.

[0040] 203. After completing the sharding granularity calculation, each shard is mapped to different cloud platforms according to a preset sharding strategy. The sharding strategy includes at least one of the following: round-robin allocation, IP segment hash allocation, domain name hash allocation, weight allocation, and single-cloud allocation, and is switched at runtime through the strategy mode. Specifically, during execution, the main control center first reads the cloud platform type, region, available quota, and current online agent scale reported by each cloud master control machine to form a cloud resource view, and selects a mapping strategy based on the characteristics of the task target. In this embodiment, for domain name targets, a primary domain name hash strategy is used to ensure that targets under the same primary domain name enter the same cloud platform as much as possible, reducing cross-cloud duplicate scanning. For targets with relatively regular IP structures, an IP segment hash strategy is used to locally concentrate targets within the same network segment. Regarding cloud resource capabilities... For cases with significant differences, a weighted allocation strategy is adopted to distribute the number of shards according to the proportion of available computing power on the cloud platform. For cases where it is necessary to avoid the characteristics of a single exit point to the greatest extent, a round-robin allocation strategy is adopted to send shards to different cloud platforms in sequence. After mapping, each shard is distributed to the corresponding cloud master machine through a WSS encrypted tunnel, and the cloud master machine performs lock-free concurrent allocation within its own cloud. Through a three-level splitting approach of global sharding, cloud-level mapping, and cloud-level concurrency, the large-scale scanning task originally concentrated on a single machine is decomposed into multiple concurrent, independently executable, and cross-cloud shards. This distributes CPU pressure, memory pressure, and network bandwidth pressure to a multi-cloud, multi-node, and multi-agent parallel system, ultimately achieving stable throughput and real-time performance improvement for large-scale tasks.

[0041] Step 3: After sharding is completed, the master control center distributes the scan shards to the corresponding cloud master controllers through the WSS encrypted tunnel. The WSS channel uses JWT tokens for authentication and multiplexes four types of messages—task distribution, result reporting, command control, and status synchronization—within the same communication link, thereby reducing the additional overhead caused by frequent connection establishment and multi-channel parallel maintenance. After receiving the shards, each cloud master controller writes the shards into its local cloud task queue and then distributes the tasks to the scanning agent nodes within its cloud through PostgreSQL's FOR UPDATESKIP LOCKED mechanism. This allows multiple agents to automatically skip locked shards when concurrently pulling tasks at the same time, avoiding mutual waiting and lock contention blocking. The cloud scheduling changes from the traditional sequential distribution mode to a lock-free concurrent allocation mode, which can significantly improve the task throughput within a single cloud and allow more agents to enter the actual scanning state in a timely manner, rather than remaining in an idle state waiting for tasks. When this mechanism is combined with the aforementioned multi-cloud sharding, the global task pressure can be first distributed to different cloud platforms, and then further distributed to multiple agents within each cloud platform, thereby forming a multi-level parallel execution structure.

[0042] Step four: After receiving the task fragments, the scanning agent node sequentially executes stages such as port probing, service identification, liveness detection, and vulnerability scanning according to the fragment content. During the vulnerability scanning stage, multiple scanning engines are supported to start in parallel, and multiple engines share a unified result callback function so that the scanning results output by different engines can be uniformly aggregated into the same processing chain. In specific execution, the Agent can call Masscan to complete large-scale port probing, call Nmap to complete service identification, call Httpx to complete web liveness detection, and combine vulnerability scanning engines such as Nuclei, Afrog, Fscan, and Rscan to perform vulnerability matching. Since these scanning actions are split and executed concurrently on multiple Agents on multiple cloud platforms, a single node no longer bears all targets and all scanning stages. Therefore, the problems of CPU overload, memory cache expansion, and network egress bandwidth congestion that are prone to occur in single-machine mode are significantly alleviated. Meanwhile, the scanning agent node has a built-in StreamReporter that uploads results in batches according to a preset rhythm, instead of waiting for all tasks to complete before uploading all results at once. In this embodiment, it uses a method of accumulating 10 results or reporting once every 5 seconds. This avoids the high-frequency small packet transmission and link jitter caused by immediately sending back every result, and also prevents results from accumulating in local memory for a long time, which would increase resource consumption. After the scan is completed, the Agent is automatically destroyed according to the task completion status to release cloud instance resources and reduce invalid running time.

[0043] Step 5: Each scanning agent node first transmits the scan results back to the corresponding cloud master control machine via a streaming reporter. Then, the cloud master control machine reports the results to the master control center through a WSS tunnel. After receiving the results from multiple cloud platforms, multiple agents, and multiple scanning engines, the master control center performs unified aggregation and analysis on the results and performs deduplication processing according to unique index rules to avoid the same target, the same port, or the same vulnerability being counted repeatedly. Since the output formats of different scanning engines are not completely consistent, the result fields are uniformly mapped during the aggregation process. Information such as hosts, ports, services, vulnerability identifiers, and risk levels from different engines are standardized before being stored in the database, thus forming a structured, searchable, and associative unified result set. For issues caused by network fluctuations or abnormal agent exits during the scanning process... For shards that are not completed due to reasons such as cloud master control machine connection interruption, the master control center reclaims them according to preset timeout thresholds and reallocates them to other still-online Agents or other available cloud platforms for continued execution. The timeout thresholds are set differently based on the scan type. In this embodiment, port scans are limited to no more than 10 minutes, service identification to no more than 20 minutes, vulnerability scans to no more than 30 minutes, and full pipeline scans to no more than 60 minutes. Each shard can be retried a maximum of 3 times. If the message middleware Kafka is available, the results are preferentially transmitted via Kafka transparent proxy. When Kafka is unavailable, it automatically degrades to database queue mode, and if necessary, further degrades to memory queue mode to ensure that the scanning task can continue without interruption in the event of partial failures or resource fluctuations. Through the above process of result aggregation, deduplication, timeout reclamation, and reallocation, a complete closed-loop processing link is ultimately formed, encompassing task sharding, multi-cloud parallel execution, streaming result feedback, aggregation analysis, and abnormal shard reclamation. This effectively supports the continuous, stable, and high-throughput operation of large-scale network security scanning tasks in a multi-cloud environment.

[0044] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0045] The preferred embodiments of the invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A multi-cloud collaborative distributed network security scanning method, characterized in that, include: The system receives network security scanning tasks submitted by users, obtains a list of targets to be scanned and scanning parameters; based on the user's authentication information, historical behavior information, and dynamic constraints of the current scanning task's ternary parsing of the target list, it filters out valid targets from the target list; the current scanning task includes at least the target list, target type, scanning type, scanning port range, concurrency level, timeout, target source type, whether full pipeline scanning is enabled, and task submission time; it parses a set of keywords from the targets to be scanned in the current scanning task and compares it with a preset keyword database to obtain the hit keywords; it then assigns each hit keyword a specific keyword to the target list. A comprehensive effective weight is assigned, and the sensitivity of the scanned target is calculated based on the coverage of the keywords hit. The overlap between the current scan task and the user's historical scan tasks is compared and analyzed to obtain the overlap. Finally, the reliability of the task is calculated based on the sensitivity of the scan task and the overlap of historical scan tasks. Each keyword in the keyword library corresponds to a basic sensitivity weight, thereby obtaining the basic sensitivity weight of the hit keyword. A position weight is assigned according to the position of the hit keyword in the scanned target. The effective weight of the hit keyword is calculated based on the basic sensitivity weight and the position weight. The effective weights of multiple hit keywords are nonlinearly aggregated to obtain the comprehensive effective weight of the scanned target. Dynamic sharding is performed based on the number of valid scanning targets and the number of available scanning resources, and each scanning shard is mapped to a different cloud platform; the scanning shards are distributed through an encrypted communication channel; the scanning shards are allocated concurrently without locks within the corresponding cloud platform, network security scanning is performed, and the scanning results are reported in real time.

2. The multi-cloud collaborative distributed network security scanning method according to claim 1, characterized in that, The authentication information includes at least the authentication token issuance time, authentication transition records, and the transition time corresponding to each authentication transition; the token age is calculated based on the time difference between the authentication token issuance time and the current time, and the token validity is calculated based on the exponential decay model; The jump frequency is calculated based on the number of authentication jumps, and the jump penalty is calculated based on the jump time of each authentication jump. The authentication stability is then calculated based on the jump frequency and jump penalty, and the authentication credibility is obtained by weighted fusion of the token validity and authentication stability.

3. The multi-cloud collaborative distributed network security scanning method according to claim 2, characterized in that, The historical behavior information includes at least historical submission records, the submission result corresponding to each historical submission record, and the submission time; wherein, the submission result includes successful submission, partially successful submission, and failed submission; each historical submission record is assigned a result indicator variable, and a time decay weight is calculated based on the submission time; the result indicator variable and the time decay weight are combined to obtain the success contribution and failure contribution respectively, and the behavior credibility is calculated based on the success contribution and failure contribution.

4. The multi-cloud collaborative distributed network security scanning method according to claim 1, characterized in that, The dynamic constraints of the ternary parsing include: determining user confidence based on authentication credibility, behavior credibility, and task credibility, and limiting the upper limit of allowed target weights, allowed scan types, and allowed concurrency levels in the current list of targets to be scanned based on the user confidence; when the effective weight of a scanned target exceeds the upper limit of target weights, the corresponding scanned target is removed from the list of targets to be scanned, thereby selecting effective scanned targets from the list of targets to be scanned in the scanning task.

5. The multi-cloud collaborative distributed network security scanning method according to claim 1, characterized in that, The number of available scanning resources is obtained through a three-level verification process. The three-level verification process includes: querying the scanning agent records that remain active within a preset time range in the database heartbeat table to obtain the first online number; summarizing the scanning agent lists returned by each cloud platform to obtain the second online number; querying the number of instances running on the cloud platform to obtain the third online number; when the deviation between the online number of one layer and the online number of the other two layers exceeds a preset threshold, the online number of the corresponding layer is judged as an outlier and removed, and then the final number of available scanning resources, i.e., the final number of online agents, is calculated based on the remaining online number.

6. The multi-cloud collaborative distributed network security scanning method according to claim 5, characterized in that, The basic granularity is calculated based on the number of effective scanning targets and the final number of available scanning resources. The basic granularity is then trimmed according to the preset granularity range corresponding to the scanning type to obtain the final granularity.

7. The multi-cloud collaborative distributed network security scanning method according to claim 1, characterized in that, Mapping each scan shard to a different cloud platform includes: selecting a corresponding sharding strategy based on the scan target type and the cloud platform resource status; the sharding strategy includes at least one of round-robin allocation, IP segment hash allocation, domain name hash allocation, weight allocation, and single cloud allocation.

8. The multi-cloud collaborative distributed network security scanning method according to claim 1, characterized in that, The encrypted communication channel adopts a WSS-based encrypted long connection method and uses JWT token authentication to reuse task distribution, result reporting, command control and status synchronization messages; the lock-free concurrent allocation adopts PostgreSQL's FOR UPDATE SKIPLOCKED mechanism, which enables multiple scan agents to automatically skip locked scan fragments when pulling scan fragments at the same time.

Images