Data acquisition method, data acquisition system and related equipment

Abstract

The invention relates to a data acquisition method, a data acquisition system and related equipment. The method comprises the following steps: in response to starting of a data acquisition task, adaptively selecting a target acquisition channel from a plurality of acquisition channels through an acquisition scheduling engine, and issuing the data acquisition task to a corresponding target terminal, so that the target terminal acquires target data through the target acquisition channel; performing credibility verification on target data returned by the target terminal through the security verification engine; and storing the target data passing the credibility verification through a data storage module. The method at least partially solves the problems of high data acquisition task failure rate, insufficient data acquisition integrity and narrow scheme application scene due to the adoption of a data acquisition system with a preset single fixed acquisition channel in a data acquisition method in related technologies.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data collection, in particular to a data collection method, a data collection system and related equipment. BACKGROUND

[0002] With the rapid development of enterprise digitization and informatization, the number of servers and terminal devices deployed in data centers, cloud platforms and various network environments is growing explosively. Under this background, asset safety inspection, configuration compliance verification, network security level protection evaluation and other operation and safety management work all rely on efficient and reliable data collection systems to obtain accurate terminal information.

[0003] The data collection system applied in the automatic data collection method in the related technology usually presets a single fixed collection channel for data collection. When the network environment where the target terminal is located appears network policy adjustment, firewall rule tightening or main collection channel failure, the data collection system cannot dynamically perceive and switch to other available channels, resulting in a significant increase in collection task failure rate, which is difficult to guarantee the completeness of collection coverage. The above phenomenon often occurs when dealing with large-scale, cross-network and high-security requirement collection scenarios, thereby limiting the applicable scenarios of the data collection method in the related scheme.

[0004] In view of the above problems in the related technology, no effective solution has been proposed so far. SUMMARY

[0005] The data collection method, the data collection system and the related equipment provided by the embodiments of the present application at least partially solve the problem that the data collection method in the related technology has a high data collection task failure rate, insufficient data collection integrity and narrow scheme applicable scenarios due to the use of a data collection system with a preset single fixed collection channel.

[0006] In order to solve the above problems, one aspect of an embodiment of the present application provides a data collection method applied to a data collection system, wherein the data collection system includes a collection scheduling engine, a security verification engine and a data storage module; the method comprises: In response to the start of the data collection task, the target collection channel is adaptively selected from the plurality of collection channels by the collection scheduling engine, and the data collection task is issued to the corresponding target terminal, so that the target terminal collects target data via the target collection channel; wherein the step of adaptively selecting the target collection channel comprises: based on the network detection module, the network state of each collection channel is detected to determine the available collection channels by the collection scheduling engine; according to the type and safety requirements of the data collection task, the candidate collection channels that meet the policy are selected from the available collection channels based on the policy matching module; according to the historical channel performance data, the channel weight of each candidate collection channel is calculated based on the weight calculation module; according to the channel weight, the target collection channel is selected from the candidate collection channels based on the channel decision module; The target data returned by the target terminal is trusted by the security verification engine; The target data that passes the trusted verification is stored by the data storage module.

[0007] In some embodiments, the step of trusted verification of the target data returned by the target terminal by the security verification engine comprises: After the target terminal receives the data collection task, the current time window reference is obtained from the time window management module via the security verification engine, and after collecting the target data, the target terminal takes the current time window reference and the collection script as input, generates a first verification code by calculating the HMAC algorithm, and sends the target data, the current time window reference and the first verification code to the security verification engine; The security verification engine sends a second verification code query request to the verification code cache module based on the received current time window reference, and compares the second verification code returned by the verification code cache module with the first verification code. If they are consistent, it is determined that the target data passes the trusted verification; wherein the second verification code is pre-calculated and stored in the verification code cache module by the time window management module based on the current time window reference and the pre-stored script by the HMAC algorithm after generating the current time window reference based on the fixed time window.

[0008] In some embodiments, the verification code cache module adopts a double window cache structure for saving the second verification code corresponding to the current time window reference and the last time window reference respectively; After the security verification engine sends a second verification code query request to the verification code cache module, the verification code cache module indexes and queries in the double window cache according to the received current time window reference, and returns the corresponding second verification code if it is hit.

[0009] In some embodiments, after the step of adaptively selecting a target collection channel from a plurality of collection channels by the collection scheduling engine and issuing a data collection task to a corresponding target terminal to enable the target terminal to collect target data via the target collection channel, the method further comprises: sending, by the collection scheduling engine, result information of the current data collection task to a feedback learning module to enable the feedback learning module to update historical channel performance data based on the result information and return the updated historical channel performance data to the collection scheduling engine for channel weight calculation in a next data collection task.

[0010] In some embodiments, when the number of target collection channels is multiple, the method further comprises: sending, by the security verification engine, channel collection results corresponding to a plurality of target collection channels that pass the trusted verification to a consistency fusion module; requesting, by the consistency fusion module, confidence weights of each target collection channel from the weight calculation module, and performing weighted fusion on the channel collection results corresponding to the plurality of target collection channels based on the confidence weights to obtain a final collection result, and sending the final collection result to the data storage module for storage.

[0011] In some embodiments, the confidence weights are dynamically calculated by the weight calculation module based on at least one of the following factors: data integrity, collection delay, and / or data consistency.

[0012] In some embodiments, the target collection channel is at least one of an SSH channel, a light probe channel, or a Web script channel.

[0013] To solve the above problems, in one aspect of an embodiment of the present application, a data collection system is provided, comprising: a collection scheduling engine configured to adaptively select a target collection channel from a plurality of collection channels in response to the start of a data collection task and issue a data collection task to a corresponding target terminal to enable the target terminal to collect target data via the target collection channel; wherein the collection scheduling engine is further configured to: determine available collection channels based on network detection of network states of each collection channel by a network detection module; filter candidate collection channels that meet a policy from the available collection channels based on a policy matching module according to the type and security requirements of the data collection task; calculate channel weights of each candidate collection channel based on a weight calculation module according to historical channel performance data; and select a target collection channel from the candidate collection channels based on a channel decision module according to the channel weights. a security verification engine configured to perform trusted verification on the target data returned by the target terminal; a data storage module configured to store the target data that passes the trusted verification.

[0014] To solve the above problems, one aspect of an embodiment of the present application provides an electronic device, comprising a processor and a memory storing a program, the program comprising instructions which, when executed by the processor, cause the processor to perform any of the above data collection methods.

[0015] To solve the above problems, one aspect of an embodiment of the present application provides a non-transitory machine-readable medium storing computer instructions for causing a computer to perform any of the above data collection methods.

[0016] The embodiment of the present application has the following beneficial effects: by responding to the start of a data collection task, the collection scheduling engine adaptively selects a target collection channel from multiple collection channels, and issues a data collection task to the corresponding target terminal, so that the target terminal collects target data via the target collection channel; wherein the step of adaptively selecting the target collection channel comprises: based on the network detection module, the network state of each collection channel is detected to determine the available collection channels by the collection scheduling engine; according to the type and security requirements of the data collection task, the candidate collection channels that meet the policy are filtered from the available collection channels based on the policy matching module; according to the historical channel performance data, the channel weight of each candidate collection channel is calculated based on the weight calculation module; according to the channel weight, the target collection channel is selected from the candidate collection channels based on the channel decision module; the target data returned by the target terminal is trusted by the security verification engine; the target data that passes the trusted verification is stored by the data storage module. The technical means overcomes the problem in the related art that the data collection method using a preset single fixed data collection system has a high data collection task failure rate, insufficient data collection integrity, and a narrow application scenario. Based on the data collection system comprising a collection scheduling engine, a security verification engine and a data storage module, the data collection method provided by the present application adaptively schedules multiple channels, so that the data collection system can automatically bypass the collection channels that are faulty, congested or prohibited by policy, dynamically select the optimal path (i.e. the target collection channel), significantly reduce the situation of collection failure due to the unavailability of a single channel, and ensure that the data collection task can be successfully performed in a variable network environment; at the same time, the returned data is forcibly and trusted by the independent security verification engine, effectively preventing malicious data from being stored due to tampering or using unauthorized scripts for collection during the transmission process, and improving the quality of the collected data.

[0017] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the application will be apparent from the description of the embodiments and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other embodiments can also be obtained from these drawings without creative labor.

[0019] Figure 1 is the main flowchart of a data acquisition method of an embodiment of the present application; Figure 2 is the main flowchart of adaptive multi-channel scheduling based on a collection scheduling engine of an embodiment of the present application; Figure 3 is the main flowchart of realizing data trusted verification of an embodiment of the present application; Figure 4 is the main flowchart of realizing data trusted verification of another embodiment of the present application; Figure 5 is the main module diagram of a data acquisition system of an embodiment of the present application; Figure 6 is the main framework diagram of a data acquisition system of another embodiment of the present application; Figure 7 is the structural diagram of an electronic device of the present application. DETAILED DESCRIPTION

[0020] Embodiments of the present application will be described in more detail by making reference to the drawings. Although some embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather these embodiments are provided to make the present application more thorough and complete. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes, and are not intended to limit the scope of protection of the present application.

[0021] In the process of developing a data collection system, it is found that the data collection scheme in the related art has at least the following technical problems in performing a data collection task, especially in the face of large-scale, cross-network, high-security requirement collection scenarios: (1) The data collection system applied in the related technical scheme usually presets a single fixed collection channel (such as relying only on SSH or only through a dedicated agent). When the network environment of the target terminal appears network policy adjustment, firewall rule tightening or the main collection channel fails, the system cannot dynamically perceive and switch to other available channels (such as lightweight probes or WebAPI), resulting in a significant increase in collection task failure rate, and it is difficult to guarantee the completeness of collection coverage. (2) In order to ensure the authenticity of the collection script and data and prevent tampering and replay attacks, the traditional data collection scheme usually adopts real-time signature verification or complex two-way authentication. Although this verification method improves the security to a certain extent, in the scenario of a large number of terminal concurrent collection, the server needs to perform high-intensity cryptographic operations (such as real-time generation or verification of digital signatures) for each request, causing huge computing pressure, resulting in a sharp increase in system response delay, forming a significant performance bottleneck. (3) The traditional data collection scheme usually lacks a chain verification mechanism that links "collection script credibility", "collection timing effectiveness" and "data source reliability", which leads to the inability to effectively identify risks introduced by malicious tampering of scripts, replay of collection requests or hijacking of data in the transmission process, and the overall credibility of the collection result is questionable.

[0022] In order to solve the above problems, the embodiment of the present application provides a data collection method applied to a data collection system, the data collection system comprising a collection scheduling engine, a security verification engine and a data storage module; as shown in the figure, the data collection method mainly comprises: Figure 1 Step S101, in response to the start of a data collection task, adaptively selecting a target collection channel from a plurality of collection channels through the collection scheduling engine, and issuing a data collection task to the corresponding target terminal, so that the target terminal collects target data through the target collection channel; wherein the step of adaptively selecting the target collection channel comprises: determining the available collection channels by detecting the network state of each collection channel based on the network detection module through the collection scheduling engine; according to the type and security requirements of the data collection task, screening the candidate collection channels that meet the policy from the available collection channels based on the policy matching module; according to the historical channel performance data, calculating the channel weight of each candidate collection channel based on the weight calculation module; according to the channel weight, selecting the target collection channel from the candidate collection channels based on the channel decision module.

[0023] Step S102, performing credible verification on the target data returned by the target terminal through the security verification engine.

[0024] ​In step S103, the target data that passes the trusted verification is stored by the data storage module.

[0025] The data collection method provided in the application is suitable for collection scenarios such as security evaluation and terminal security inspection, and has good data collection effect in large-scale, cross-network and high-security collection scenarios.

[0026] Based on the above settings, based on the data collection system including the collection scheduling engine, the security verification engine and the data storage module, the data collection method provided in the application realizes adaptive multi-channel scheduling, so that the data collection system can automatically bypass the collection channels that are faulty, congested or prohibited by policy, dynamically select the optimal path (i.e. the target collection channel), significantly reduce the situation of collection failure caused by the unavailability of a single channel, and ensure that the data collection task can be successfully executed in a variable network environment. At the same time, the independent security verification engine performs forced trusted verification on the returned data, effectively preventing malicious data from being stored due to tampering during transmission or using unauthorized scripts for collection, and improving the quality of the collected data. Further, based on the organic combination and coordination of the two key technologies of adaptive selection of the best collection channel based on multi-dimensional information and independent and forced trusted verification before data storage, an intelligent, flexible, safe and reliable data collection process is constructed. On the one hand, it responds to the uncertainty of the underlying network environment through dynamic scheduling, and on the other hand, it defends against security risks of the data itself through architectural verification, ultimately cooperatively realizes the successful execution of the data collection task in a high complexity scenario, and guarantees the accuracy and integrity of the collected data.

[0027] Specifically, the adaptive multi-channel scheduling mechanism provided in the application is realized by using the collection scheduling engine from four dimensions of dynamic perception, policy filtering, weight ordering and intelligent decision-making. According to a specific embodiment of the application, as shown in Figure 2 the main process of realizing adaptive multi-channel scheduling based on the collection scheduling engine includes: S201, based on the network perception module, performing port detection and protocol availability testing on each collection channel to determine available collection channels; S202, based on the policy matching module, performing task type matching and security domain filtering on each available collection channel according to the type and security requirements of the data collection task to determine candidate collection channels; S203, based on the weight calculation module, calculating the channel weight of each candidate collection channel according to the channel collection success rate, channel delay and channel security level indicated in the historical channel performance data of each candidate collection channel; S204, based on the channel decision module, selecting the optimal channel from the candidate collection channels as the target collection channel according to the channel weight.

[0028] That is, using the collection scheduling engine, the network state (such as port connectivity, protocol availability) of each collection channel is actively detected based on the network detection module, a real-time available collection channel set is obtained, and the latest factual basis is provided for subsequent adaptive channel decision; then, based on the strategy matching module, the available collection channel is matched with the specific requirements (such as security level, collection type) of the current data collection task, and the non-compliant channel is filtered out to obtain a candidate collection channel set, so as to ensure that the channel result selected subsequently conforms to the safety specification; then, based on the weight calculation module, historical channel performance data (such as historical response delay, success rate, channel security level, etc. of each collection channel) is introduced, and a quantitative channel weight is calculated for each candidate collection channel, so that the selection basis is upgraded from a simple on / off state to a performance weight ranking that comprehensively considers the historical channel performance; finally, based on the channel decision module, one or more optimal target collection channels are determined based on the channel weight. Based on the closed-loop decision-making process, the system can always select the target collection channel with the best historical performance from the currently available and compliant channels to execute the current data collection task, thereby directly responding to network fluctuations and policy changes and improving the collection success rate.

[0029] Further, in the data collection method provided by the application, the scheduling of the data transmission channel (responsible for the collection scheduling engine) and the authenticity verification of the data (responsible for the security verification engine) are decoupled in architecture by applying an independent security verification engine, which means that even if the collection scheduling engine successfully selects a suitable target collection channel and obtains target data, the target data will not be directly collected and verified. All target data returned by the target terminal must be verified for authenticity by the security verification engine (which can be any mechanism that ensures data authenticity and integrity, such as code verification, digital signature verification, source authentication, etc.), thereby ensuring that the target data stored in the data storage module is not only collected by the optimal collection channel, but also verified for authenticity, thereby fundamentally ensuring the authenticity and integrity of the data.

[0030] In addition, it can be understood that the above-mentioned data collection method provided by the application forms a complete automatic closed loop from adaptive multi-channel selection, task issuance to data verification and storage, and the data collection system applied can make intelligent decisions based on real-time network status, predefined policies and historical performance data, greatly reducing the need for manual intervention and improving operational efficiency.

[0031] In some examples, the above-mentioned target collection channel provided by the application is at least one of an SSH channel, a light probe channel or a Web script channel.

[0032] Among them, the SSH channel: makes use of the remote security management protocol commonly supported by standard Linux / Unix servers and network devices, which realizes high-privilege and efficient collection of management devices with command-line interface and opened SSH service. The light probe channel: usually refers to a resident memory lightweight agent program that communicates with the server through reverse connection or long connection. It realizes the penetration collection of devices behind network address translation (NAT) and firewalls, and can perform continuous monitoring and real-time response. The Web script channel: makes use of the HTTP / HTTPS protocol commonly supported by modern operating systems and applications, which realizes collection under the extremely strict network policy that only allows Web traffic (80 / 443 ports) to pass, and collection of Web-type devices / applications that cannot install SSH service or dedicated probes.

[0033] Based on the above settings, the compatibility and environmental adaptability of the data collection system are expanded, and the data collection method can still maintain high collection coverage in heterogeneous and complex network and terminal environment.

[0034] For example, using the Web script channel in an environment that only opens HTTP / HTTPS ports, using the SSH channel in an environment that allows management communication, and using the light probe reverse connection in a strictly restricted environment, thereby breaking through the restrictions of network policy. For another example, using SSH or Web script for devices that cannot install Agent, and using Web script with lower overhead for resource-constrained devices, so that the data collection system can cope with the diversity of target terminals in computing power, pre-installed software and permissions, and the data collection method can adapt to the differences in terminal resources and configurations, expanding the applicable scenarios of the scheme. In addition, when a certain type of collection channel fails on a large scale due to global policy adjustment or universal failure (for example, SSH ports are uniformly blocked), the data collection system can immediately switch to other types of channels to realize the redundancy backup of collection capability.

[0035] In some embodiments, the step of performing trusted verification on the target data returned by the target terminal by the security verification engine comprises: after the target terminal receives the data collection task, the target terminal acquires the current time window reference from the time window management module via the security verification engine; after the target data is collected, the target terminal takes the current time window reference and the collection script as inputs, generates a first verification code by calculating using the HMAC algorithm, and sends the target data, the current time window reference, and the first verification code to the security verification engine; the security verification engine sends a second verification code query request to the verification code cache module based on the received current time window reference, and compares the second verification code returned by the verification code cache module with the first verification code; if they are consistent, it is determined that the target data passes the trusted verification; wherein the second verification code is pre-calculated by the time window management module based on the current time window reference and the pre-stored script using the HMAC algorithm after the current time window reference is generated based on the fixed time window, and is stored in the verification code cache module.

[0036] Based on the above setting, the heavy cryptographic calculation (HMAC) of the server in the related technical solution is separated from real-time request processing and changed to a pre-calculation and cache query mode, so that the response delay of single data verification is extremely low (can reach milliseconds), which can easily cope with terminal concurrent reporting of more than ten thousand, and solves the service performance bottleneck problem caused by traditional real-time signature verification. At the same time, since the verification code (HMAC) is bound with the collection script content and the time window reference, any tampering of the script or data will cause verification failure, ensuring the integrity of the collection script and the data; and using the fixed time window as the dynamic change reference for determining the current time window makes each verification code valid only within a specific time window, so that even if an attacker intercepts a valid "data-verification code" combination, it cannot be reused in the next time window, effectively resisting replay attacks.

[0037] According to a specific embodiment of the present application, as Figure 3 , Figure 4As shown, the main process of trusted verification of target data can include: S301, in response to server startup, the time window management module generates a current time window reference according to a fixed time window; S302, the HMAC calculation module pre-computes a second verification code according to the current time window reference and a pre-stored script; S303, after the target terminal receives a data collection task, the target terminal acquires the current time window reference from the time window management module via the security verification engine, and after collecting the target data, the target terminal takes the current time window reference and the collection script as input, generates a first verification code through the HMAC algorithm, and sends the target data, the current time window reference and the first verification code to the security verification engine; S304, the security verification engine sends a second verification code query request to the verification code cache module based on the received current time window reference, and compares the second verification code returned by the verification code cache module with the first verification code.

[0038] That is, after the server starts, the time window management module generates a unified and periodically changing current time window reference T BASE according to a fixed time window (for example, 15 minutes). Since T BASE changes with time, the HMAC verification code calculated based on it also has timeliness. A data packet that is out of date will inevitably mismatch the current or last valid time window of the server, and thus will cause verification failure, which introduces a time dimension for the security verification mechanism of the entire data collection scheme, and fundamentally cuts off the path of replay attacks.

[0039] According to another specific embodiment of the present application, before the terminal executes the data collection task, the time window management module uses the same pre-stored script and the current T BASE as the terminal to calculate (i.e., pre-compute) the verification code (i.e., the second verification code) of all legal requests within the current time window reference, and stores it in the verification code cache module. When the target terminal reports the target data, the security verification engine no longer needs to perform time-consuming HMAC calculation, but directly uses the T BASE reported by the terminal as a key to initiate a quick query to the verification code cache module, and compares the returned pre-stored value (second verification code) with the reported value (first verification code) of the terminal to complete the trusted verification of the data. Accordingly, the response of the server to each request is converted from O(n) cryptographic calculation to O(1) cache query, thereby achieving high concurrency performance.

[0040] The HMAC algorithm can be HMAC-MD5, HMAC-SHA256, or a national secret SM3-HMAC algorithm. The terminal uses the same algorithm (HMAC) and input (T_BASE + collection script) as the server to generate the first verification code. Thus, only the terminal that holds the correct script and executes it within the legal time window can generate a verifiable HMAC code. Any tampering with the script or use of unauthorized scripts will result in a first verification code that does not match the second verification code pre-stored by the server, thereby achieving strong verification of data source and integrity.

[0041] In some embodiments, the verification code cache module adopts a double-window cache structure to store the second verification codes corresponding to the current time window reference and the last time window reference, respectively. After receiving a second verification code query request from the security verification engine, the verification code cache module performs an index query in the double-window cache according to the received current time window reference, and directly returns the corresponding second verification code if a hit is found.

[0042] Based on the above settings, the verification code cache module adopting the double-window cache structure solves the problem that, at the moment of switching between fixed time windows (e.g., every 15 minutes), due to slight time asynchronization or network delay between the server and the terminal, legal data is misjudged as verification failure, eliminates verification jitter, guarantees business continuity, and ensures that the data collection process will not be interrupted intermittently due to time boundary problems. It can be understood that even before and after the fixed time window switches, the data verification operation of the data collection system is still completed through cache query, without the need to fall back to real-time calculation or complex exception handling processes, maintaining stable and low-latency verification processes and preventing performance fluctuations caused by boundary problems. This design provides a tolerant buffer interval for the small clock deviation between the terminal and the server and network transmission delay, reduces the stringent requirements of the system on absolute time synchronization, and enhances the adaptability of the system under non-ideal network conditions.

[0043] According to the embodiment of the present application, the verification code cache module does not only save the second verification code corresponding to the current valid time window at any time, but also saves the second verification codes corresponding to two adjacent time windows, i.e., (T_BASE_current, HMAC_current) and (T_BASE_previous, HMAC_previous), in the memory at the same time. This makes the security verification engine expand the verification range from one time window to two continuous time windows when receiving a request carrying T_BASE. No matter the request logically belongs to the old time window just ended or the new time window just started, the corresponding correct second verification code has been stored in the verification code cache module, thereby avoiding the critical state of "unable to query the corresponding second verification code" caused by the switching of the time window.

[0044] According to a specific embodiment of the present application, the query logic for index query in the double-window cache can be designed to preferentially find the second verification code corresponding to the current time window, and find the second verification code corresponding to the previous time window if the first search fails. When the T_BASE reported by the terminal is consistent with the current time window of the server, the query immediately hits in the current time window cache; when the terminal is still using the T_BASE of the previous window, the query fails in the current time window, but immediately hits in the previous time window cache. The entire verification process does not introduce any time-consuming calculation or logical judgment, and still maintains the high-performance mode of cache query-comparison, smoothly processing the boundary data.

[0045] It can be understood that the cache structure only retains the second verification codes corresponding to the current time window and the previous time window, respectively, instead of all historical time windows, which means that the tolerance period for the old time window verification code is limited (usually one window length). This design balances between allowing certain clock deviation and delay and preventing expired replay attacks. It gives the terminal enough time (one time window period) to complete the transition from the old time window to the new time window, and ensures that the old verification codes of earlier periods (such as the previous day or the previous week) cannot be reused, because they have been overwritten by the new "previous time window" data in the cache. Thus, without sacrificing the security core, the clock asynchronization and boundary state problems commonly seen in distributed systems are solved.

[0046] In addition, the above-mentioned data collection method also supports traceability of the fusion result, i.e., the channel weight / confidence weight of each target collection channel and the original data participating in the calculation can be recorded, so as to facilitate subsequent security audit or algorithm optimization.

[0047] In some of the embodiments, after the step of adaptively selecting a target collection channel from the plurality of collection channels by the collection scheduling engine and issuing a data collection task to the corresponding target terminal to enable the target terminal to collect target data via the target collection channel, the method further comprises: sending, by the collection scheduling engine, result information of the current data collection task to the feedback learning module to enable the feedback learning module to update the historical channel performance data based on the result information and return the updated historical channel performance data to the collection scheduling engine for channel weight calculation in the next data collection task.

[0048] Based on the above settings, in the adaptive multi-channel scheduling process, the weight calculation for channel selection by the collection scheduling engine is not based on initial, static or outdated historical data, but on the historical channel performance data updated based on the latest and most real collection results, so that the channel weight coefficient can truly reflect the real-time performance of each collection channel in the current environment.

[0049] According to the embodiments of the present application, by introducing the feedback learning module, the channel weight of the collection channel with continuous failure and low performance can be automatically identified and reduced, or even excluded in the strategy, to avoid the data collection system repeatedly selecting the collection channel with low performance, causing the delay or failure of the collection task; the performance recovery or newly appeared high-quality collection channel can also be found in time, and the channel weight thereof is increased, to ensure that the data collection system always tends to use the current optimal collection path.

[0050] According to a specific embodiment of the embodiments of the present application, the data collection system will feed back the collection results (success rate, delay, error type of the target collection channel) to the weight technology module via the feedback learning module after the current data collection task is executed, to form an adaptive adjustment mechanism. For example, if the SSH collection channel fails for three times in succession, the weight is automatically reduced and switched to the probe channel; if the delay of the probe channel is high, the Web script channel is temporarily preferred; if the task is of a sensitive type (configuration collection), only the channel with high security level is allowed. Based on the feedback learning mechanism, the intelligent scheduling of self-learning + strategy driving + multi-channel multiplexing is realized. Further, the feedback learning mechanism at least has the following technical advantages: (1) self-learning optimization: the channel selection is self-adjusted according to the historical success rate; (2) zero manual configuration: the administrator does not need to manually select the collection method; (3) high coverage rate: the collection can still be performed in the isolated network segment, VPN or policy limited scene; (4) intelligent degradation: the backup path is automatically enabled when the main channel fails; (5) multi-channel fusion: the unified decision of SSH, probe and Web script is supported.

[0051] In some embodiments, when the number of target collection channels is multiple, the method further comprises: sending, by the security verification engine, the channel collection results corresponding to the multiple target collection channels that pass the trusted verification to the consistency fusion module; requesting, by the consistency fusion module, the confidence weight of each target collection channel from the weight calculation module, and performing weighted fusion on the channel collection results corresponding to the multiple target collection channels based on the confidence weight to obtain a final collection result, and sending the final collection result to the data storage module for storage.

[0052] Based on the above settings, by integrating the collection results of multiple independent target collection channels, random errors, transient interference or local errors that may exist in a single channel can be intelligently eliminated.

[0053] According to a specific embodiment of the present application, when the collection results returned by different target collection channels differ or even contradict each other (for example, the return value of channel A is 100 and the return value of channel B is 120), the system does not simply choose one or report an error, but performs weighted arbitration according to the "trustworthiness" of each channel to output a most reasonable value. Further, even if one or several target collection channels provide incorrect data due to temporary failure or specific attack, as long as not all high-confidence-weight collection channels fail at the same time, the data collection method provided by the present application can still correct the final result through the data of other reliable channels, ensuring the robustness of the overall output. The confidence weight is dynamically calculated by the weight calculation module based on at least one of the completeness of the data provided by each collection channel, the collection delay, and / or the data consistency.

[0054] Specifically, the consistency fusion module does not itself determine which result is more trustworthy, but requests the confidence weight (a quantitative index, the calculation basis of which (such as completeness, delay, consistency) itself reflects the reliability of the channel and the quality of the data) of each channel from the professional weight calculation module. After receiving the collection results of each target collection channel and the corresponding confidence weight, the consistency fusion module performs a weighted fusion algorithm (such as weighted average), the basic principle of which is: the higher the weight of a channel, the greater the proportion of the data provided by the channel in the final result.

[0055] The data collection method provided in the embodiments of the present application overcomes the problems in the prior art that the data collection method using a data collection system with a preset single fixed collection channel has a high failure rate of data collection tasks, insufficient data collection integrity, and a narrow application scenario, by the technical means that the target collection channel is adaptively selected from the plurality of collection channels by the collection scheduling engine in response to the start of the data collection task, and the data collection task is issued to the corresponding target terminal, so that the target terminal collects the target data via the target collection channel; wherein the step of adaptively selecting the target collection channel comprises: determining the available collection channels based on the network state of each collection channel detected by the network detection module based on the collection scheduling engine; filtering the candidate collection channels that meet the policy from the available collection channels based on the policy matching module according to the type and security requirements of the data collection task; calculating the channel weight of each candidate collection channel based on the weight calculation module according to the historical channel performance data; selecting the target collection channel from the candidate collection channels based on the channel decision module according to the channel weight; performing trusted verification on the target data returned by the target terminal by the security verification engine; and storing the target data that passes the trusted verification by the data storage module. Based on the data collection system including the collection scheduling engine, the security verification engine, and the data storage module, the data collection method adaptively schedules multiple channels, so that the data collection system can automatically bypass the collection channels that are faulty, congested, or prohibited by policy, dynamically select the optimal path (i.e., the target collection channel), significantly reduce the situation of collection failure due to the unavailability of a single channel, and ensure that the data collection task can be successfully executed in a variable network environment; at the same time, the independent security verification engine performs mandatory trusted verification on the returned data, effectively prevents malicious data from being stored due to tampering during transmission or using unauthorized scripts for collection, and improves the quality of the collected data.

[0056] Based on the above-mentioned data collection method provided in the embodiments of the present application, the present application further provides a data collection system, as shown in Figure 5 The data collection system 500 includes a collection scheduling engine 501, a security verification engine 502, and a data storage module 503 deployed on a server; wherein: The collection scheduling engine 501 is configured to adaptively select a target collection channel from a plurality of collection channels in response to the start of a data collection task, and to issue the data collection task to a corresponding target terminal, so that the target terminal collects target data via the target collection channel; wherein the collection scheduling engine is further configured to: determine available collection channels based on the network detection module detecting the network state of each collection channel; select candidate collection channels that meet the policy from the available collection channels according to the type and security requirements of the data collection task based on the policy matching module; calculate the channel weight of each candidate collection channel based on the weight calculation module according to the historical channel performance data; and select the target collection channel from the candidate collection channels based on the channel decision module according to the channel weight. The security verification engine 502 is configured to perform trusted verification on the target data returned by the target terminal. The data storage module 503 is configured to store the target data that passes the trusted verification.

[0057] Based on the above settings, based on the data collection system including the collection scheduling engine, the security verification engine and the data storage module, through adaptive multi-channel scheduling, the data collection system can automatically bypass the collection channels that are faulty, congested or prohibited by policy, dynamically select the optimal path (i.e. the target collection channel), significantly reduce the situation of collection failure due to the unavailability of a single channel, and ensure that the data collection task can be successfully executed in a variable network environment; at the same time, the independent security verification engine performs mandatory trusted verification on the returned data, effectively preventing malicious data from being stored due to tampering during transmission or using unauthorized scripts for collection, and improving the quality of the collected data. Further, based on the organic combination and coordination of the two key technologies of adaptive selection of the best collection channel based on multi-dimensional information and independent and mandatory trusted verification before data storage, an intelligent, flexible, safe and reliable data collection process is constructed, which on the one hand responds to the uncertainty of the underlying network environment through dynamic scheduling, and on the other hand defends against security risks of the data itself through architectural verification, ultimately achieving the successful execution of the data collection task in a high complexity scenario and ensuring the accuracy and integrity of the collected data.

[0058] As shown in Figure 6 The above data collection system deployed in the modules of the server further includes a consistency fusion module. When the number of target collection channels is multiple, the security verification engine sends the channel collection results corresponding to the multiple target collection channels that pass the trusted verification to the consistency fusion module; the consistency fusion module is configured to request the confidence weight of each target collection channel from the weight calculation module, and to perform weighted fusion on the channel collection results corresponding to the multiple target collection channels based on the confidence weight, to obtain the final collection result, and to send the final collection result to the data storage module for storage.

[0059] Based on the above settings, by synthesizing the collection results of multiple independent target collection channels, random errors, transient interference or local errors that may exist in a single channel can be intelligently eliminated.

[0060] As shown in Figure 6 The data collection system further includes a plurality of terminal systems, and the terminal systems correspond to a plurality of collection channels such as SSH channels, light probe channels, Web script channels or other collection channels.

[0061] Further, as shown in Figure 6 The modules deployed in the server system of the data collection system further include a plurality of sub-modules for performing specific operations, and the specific operations performed by each module and the technical effects achieved are described in the foregoing data collection method, and will not be repeated here.

[0062] Meanwhile, since the data collection system 500 of the present application is used to implement any one of the above data collection methods, the execution operations of each part in the data collection system are consistent with the foregoing scheme, and all the beneficial effects corresponding to the above scheme can be achieved, and the present application will not be repeated here.

[0063] The embodiment of the present application also provides a non-transitory machine readable medium storing a computer program, wherein the computer program is used to make the computer execute the method of the embodiment of the present application when executed by the processor of the computer.

[0064] The embodiment of the present application also provides a computer program product including a computer program, wherein the computer program is used to make the computer execute the method of the embodiment of the present application when executed by the processor of the computer. The computer program product should be understood as a software product for realizing the above method of the present application mainly through the computer program.

[0065] The embodiment of the present application also provides an electronic device including at least one processor and a memory connected in communication with the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to make the electronic device execute the method of the embodiment of the present application when executed by the at least one processor.

[0066] Reference Figure 7, a block diagram of an electronic device that can be a server or a client as an embodiment of the present application will now be described, which is an example of a hardware device that can be applied to aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer devices such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components, their connections, and their functions, as described herein, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0067] As shown in FIG. 7, Figure 7 The electronic device includes a computing unit 701 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. Various programs and data required for the operation of the electronic device can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0068] Various components in the electronic device are connected to the I / O interface 705, including an input unit 706, an output unit 707, the storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information to the electronic device, and can receive inputted digital or character information, and generate key signal inputs related to user settings and / or function controls of the electronic device. The output unit 707 can be any type of device that can present information, and can include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include, but is not limited to, a magnetic disk, an optical disk. The communication unit 709 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include, but is not limited to, a modem, a network card, an infrared communication device, and / or a wireless communication transceiver, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0069] The computing unit 701 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, CPUs, graphics processing units (GPUs), various specialized artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, or the like. The computing unit 701 performs various methods and processes described above. For example, in some embodiments, the method embodiments of the present application can be implemented as a computer program tangibly embodied in a machine-readable medium, e.g., the storage unit 708. In some embodiments, portions or all of the computer program can be loaded and / or installed onto the electronic device via the ROM 702 and / or the communication unit 709. In some embodiments, the computing unit 701 can be configured, by way of other any suitable means, e.g., by way of firmware, to perform the methods described above.

[0070] Computer programs used to implement embodiments of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program

[0071] In the context of embodiments of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared signals, or any suitable combination thereof. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0072] It should be noted that the term "comprising" and its derivations used in the embodiments of the present application are open-ended, that is, "comprising but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The terms "a" and "an" are intended to mean "one or more" unless otherwise indicated. The terms "including" and / or "containing", or variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, or contains a list of elements is not necessarily limited to those elements, but can include additional elements not expressly listed or inherent to such process, method, article, or apparatus.

[0073] The information / data (including but not limited to user device information / data, user personal information / data, information / data for analysis, stored information / data, displayed information / data, etc.) involved in the embodiments of the present application is information / data permitted by the user or agreed by all parties, and the collection, use and processing of related information / data need to comply with relevant laws, regulations and standards, and provide corresponding operation portal for the user to choose to agree or refuse.

[0074] The various steps described in the method embodiments provided by the embodiments of the present application can be executed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of protection of the present application is not limited in this respect.

[0075] The term "embodiment" in the specification refers to the specific features, structures or characteristics described in connection with the embodiments can be included in at least one embodiment of the present application. The presence of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean independence or alternative to other embodiments. Each embodiment in the specification is described in a relevant manner, and the same or similar parts between each embodiment are referred to each other. In particular, for device, equipment, system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts are referred to the part of the method embodiment.

[0076] The above-described embodiments only express several embodiments of the present application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the scope of protection. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A data acquisition method, characterized in that, The method is applied to a data acquisition system, which includes an acquisition scheduling engine, a security verification engine, and a data storage module; the method includes: In response to the initiation of a data acquisition task, the acquisition scheduling engine adaptively selects a target acquisition channel from multiple acquisition channels and issues a data acquisition task to the corresponding target terminal, so that the target terminal acquires target data via the target acquisition channel. The adaptive selection of the target acquisition channel includes: using the acquisition scheduling engine, detecting the network status of each acquisition channel based on the network detection module to determine available acquisition channels; filtering candidate acquisition channels that meet the policy requirements from the available acquisition channels based on the policy matching module according to the type and security requirements of the data acquisition task; calculating the channel weight of each candidate acquisition channel based on historical channel performance data using the weight calculation module; and selecting the target acquisition channel from the candidate acquisition channels based on the channel weights using the channel decision module. The security verification engine performs a trusted verification of the target data returned by the target terminal. The target data that has passed the trusted verification is stored through the data storage module.

2. The method according to claim 1, characterized in that, The step of performing trusted verification on the target data returned by the target terminal through the security verification engine includes: After receiving the data collection task, the target terminal obtains the current time window reference from the time window management module through the security verification engine. After collecting the target data, the target terminal uses the current time window reference and the collection script as input, calculates and generates a first verification code through the HMAC algorithm, and sends the target data, the current time window reference and the first verification code to the security verification engine. The security verification engine sends a second verification code query request to the verification code caching module based on the received current time window benchmark, and compares the second verification code returned by the verification code caching module with the first verification code. If they match, the target data is determined to have passed the trusted verification. The second verification code is generated by the time window management module based on a fixed time window benchmark, and then pre-calculated and stored in the verification code caching module using the HMAC algorithm based on the current time window benchmark and a pre-stored script.

3. The method according to claim 2, characterized in that, The verification code caching module adopts a dual-window caching structure to store the second verification code corresponding to the current time window benchmark and the previous time window benchmark respectively. After the security verification engine sends a second verification code query request to the verification code caching module, the verification code caching module performs an index query in the dual-window cache based on the received current time window benchmark. If a match is found, the corresponding second verification code is returned directly.

4. The method according to claim 1, characterized in that, After the step of adaptively selecting a target acquisition channel from multiple acquisition channels through the acquisition scheduling engine and issuing a data acquisition task to the corresponding target terminal so that the target terminal acquires target data via the target acquisition channel, the method further includes: The acquisition scheduling engine sends the result information of this data acquisition task to the feedback learning module, so that the feedback learning module updates the historical channel performance data based on the result information and returns the updated historical channel performance data to the acquisition scheduling engine for channel weight calculation in the next data acquisition task.

5. The method according to claim 1, characterized in that, When the number of target acquisition channels is multiple, the method further includes: The security verification engine sends the channel acquisition results corresponding to the multiple target acquisition channels that have passed trusted verification to the consistency fusion module. The consistency fusion module requests the confidence weight of each target acquisition channel from the weight calculation module. The consistency fusion module performs weighted fusion on the acquisition results of the multiple target acquisition channels based on the confidence weight to obtain the final acquisition result, and sends the final acquisition result to the data storage module for storage.

6. The method according to claim 5, characterized in that, The confidence weight is dynamically calculated by the weight calculation module based on at least one of the following factors: the integrity of the data provided by each acquisition channel, the acquisition delay, and / or the data consistency.

7. The method according to claim 1, characterized in that, The target acquisition channel is at least one of an SSH channel, a light probe channel, or a Web script channel.

8. A data acquisition system, characterized in that, include: A data acquisition scheduling engine is used to adaptively select a target acquisition channel from multiple acquisition channels in response to the initiation of a data acquisition task, and to issue a data acquisition task to the corresponding target terminal so that the target terminal can acquire target data through the target acquisition channel. The acquisition scheduling engine is further used to: detect the network status of each acquisition channel based on a network detection module to determine available acquisition channels; filter candidate acquisition channels that meet the policy based on a policy matching module according to the type and security requirements of the data acquisition task; calculate the channel weight of each candidate acquisition channel based on a weight calculation module according to historical channel performance data; and select the target acquisition channel from the candidate acquisition channels based on the channel weights and a channel decision module. A security verification engine is used to perform trusted verification on the target data returned by the target terminal; A data storage module is used to store the target data that has passed trusted verification.

9. An electronic device, comprising: A processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory machine-readable medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 7.

Images