A data acquisition method, a data acquisition system, and related equipment

Abstract

This application relates to a data acquisition method, a data acquisition system, and related equipment. The method includes: in response to the initiation of a data acquisition task, adaptively selecting a target acquisition channel from multiple acquisition channels using an acquisition scheduling engine, and issuing a data acquisition task to the corresponding target terminal, so that the target terminal acquires target data via the target acquisition channel; performing trusted verification on the target data returned by the target terminal using a security verification engine; and storing the trusted verified target data using a data storage module. This method at least partially solves the problems of high data acquisition task failure rate, insufficient data acquisition integrity, and narrow applicability scenarios in related technologies due to the use of a pre-set single fixed acquisition channel data acquisition system.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and in particular to a data acquisition method, a data acquisition system and related equipment. Background Technology

[0002] With the rapid development of enterprise digitalization and informatization, the number of servers and terminal devices deployed in data centers, cloud platforms, and various network environments has exploded. Against this backdrop, operational and security management tasks such as asset security checks, configuration compliance verification, and network security level protection assessments all rely on efficient and reliable data acquisition systems to obtain accurate terminal information.

[0003] The data acquisition systems used in automated data acquisition methods in related technologies typically use a single fixed acquisition channel for data acquisition. When network policies are adjusted, firewall rules are tightened, or the main acquisition channel fails in the network environment where the target terminal is located, the data acquisition system cannot dynamically detect and switch to other available channels, resulting in a significant increase in the failure rate of acquisition tasks and difficulty in ensuring the completeness of acquisition coverage. The above phenomenon often occurs, especially when dealing with large-scale, cross-network, and high-security acquisition scenarios, which in turn limits the applicable scenarios of the data acquisition methods in related solutions.

[0004] There is currently no effective solution to the aforementioned problems in the relevant technologies. Summary of the Invention

[0005] The data acquisition method, data acquisition system, and related equipment provided in this invention at least partially solve the problems of high failure rate, insufficient data acquisition integrity, and narrow applicable scenarios in related technologies due to the use of a data acquisition system with a preset single fixed acquisition channel.

[0006] To address the aforementioned problems, one aspect of this invention provides a data acquisition method applied to a data acquisition system, the data acquisition system including an acquisition scheduling engine, a security verification engine, and a data storage module; the method includes:

[0007] In response to the initiation of a data acquisition task, the acquisition scheduling engine adaptively selects a target acquisition channel from multiple acquisition channels and issues a data acquisition task to the corresponding target terminal, so that the target terminal acquires target data via the target acquisition channel. The adaptive selection of the target acquisition channel includes: using the acquisition scheduling engine, detecting the network status of each acquisition channel based on the network detection module to determine available acquisition channels; filtering candidate acquisition channels that meet the policy requirements from the available acquisition channels based on the policy matching module according to the type and security requirements of the data acquisition task; calculating the channel weight of each candidate acquisition channel based on historical channel performance data using the weight calculation module; and selecting the target acquisition channel from the candidate acquisition channels based on the channel weights using the channel decision module.

[0008] The security verification engine performs a trusted verification of the target data returned by the target terminal.

[0009] The target data that has passed the trusted verification is stored through the data storage module.

[0010] In some embodiments, the step of performing trusted verification on the target data returned by the target terminal through the security verification engine includes:

[0011] After receiving the data collection task, the target terminal obtains the current time window reference from the time window management module through the security verification engine. After collecting the target data, the target terminal uses the current time window reference and the collection script as input, calculates and generates a first verification code through the HMAC algorithm, and sends the target data, the current time window reference and the first verification code to the security verification engine.

[0012] The security verification engine sends a second verification code query request to the verification code caching module based on the received current time window benchmark, and compares the second verification code returned by the verification code caching module with the first verification code. If they match, the target data is determined to have passed the trusted verification. The second verification code is generated by the time window management module based on a fixed time window benchmark, and then pre-calculated and stored in the verification code caching module using the HMAC algorithm based on the current time window benchmark and a pre-stored script.

[0013] In some of these embodiments, the verification code caching module adopts a dual-window caching structure to store the second verification codes corresponding to the current time window reference and the previous time window reference, respectively.

[0014] After the security verification engine sends a second verification code query request to the verification code caching module, the verification code caching module performs an index query in the dual-window cache based on the received current time window benchmark. If a match is found, the corresponding second verification code is returned directly.

[0015] In some embodiments, after the step of adaptively selecting a target acquisition channel from multiple acquisition channels through the acquisition scheduling engine and issuing a data acquisition task to the corresponding target terminal so that the target terminal acquires target data via the target acquisition channel, the method further includes:

[0016] The acquisition scheduling engine sends the result information of this data acquisition task to the feedback learning module, so that the feedback learning module updates the historical channel performance data based on the result information and returns the updated historical channel performance data to the acquisition scheduling engine for channel weight calculation in the next data acquisition task.

[0017] In some embodiments, when the number of target acquisition channels is multiple, the method further includes:

[0018] The security verification engine sends the channel acquisition results corresponding to the multiple target acquisition channels that have passed trusted verification to the consistency fusion module.

[0019] The consistency fusion module requests the confidence weight of each target acquisition channel from the weight calculation module. The consistency fusion module performs weighted fusion on the acquisition results of the multiple target acquisition channels based on the confidence weight to obtain the final acquisition result, and sends the final acquisition result to the data storage module for storage.

[0020] In some embodiments, the confidence weight is dynamically calculated by the weight calculation module based on at least one of the following factors: the integrity of the data provided by each acquisition channel, acquisition latency, and / or data consistency.

[0021] In some embodiments, the target acquisition channel is at least one of an SSH channel, a light probe channel, or a Web script channel.

[0022] To address the above problems, one aspect of this invention provides a data acquisition system, comprising:

[0023] A data acquisition scheduling engine is used to adaptively select a target acquisition channel from multiple acquisition channels in response to the initiation of a data acquisition task, and to issue a data acquisition task to the corresponding target terminal so that the target terminal can acquire target data through the target acquisition channel. The acquisition scheduling engine is further used to: detect the network status of each acquisition channel based on a network detection module to determine available acquisition channels; filter candidate acquisition channels that meet the policy based on a policy matching module according to the type and security requirements of the data acquisition task; calculate the channel weight of each candidate acquisition channel based on a weight calculation module according to historical channel performance data; and select the target acquisition channel from the candidate acquisition channels based on the channel weights and a channel decision module.

[0024] A security verification engine is used to perform trusted verification on the target data returned by the target terminal;

[0025] A data storage module is used to store the target data that has passed trusted verification.

[0026] To address the aforementioned problems, one aspect of this invention provides an electronic device, including: a processor and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform any of the aforementioned data acquisition methods.

[0027] To address the aforementioned problems, one aspect of this invention provides a non-transitory machine-readable medium storing computer instructions for causing a computer to execute any of the aforementioned data acquisition methods.

[0028] The beneficial effects of this invention are as follows: By adopting a data acquisition task-responsive approach, the acquisition scheduling engine adaptively selects a target acquisition channel from multiple acquisition channels and issues a data acquisition task to the corresponding target terminal, enabling the target terminal to acquire target data via the target acquisition channel. The adaptive selection of the target acquisition channel includes: using the acquisition scheduling engine, detecting the network status of each acquisition channel based on the network detection module to determine available acquisition channels; filtering candidate acquisition channels that meet the policy requirements from available acquisition channels based on the policy matching module according to the type and security requirements of the data acquisition task; calculating the channel weight of each candidate acquisition channel based on the weight calculation module according to historical channel performance data; selecting the target acquisition channel from the candidate acquisition channels based on the channel weight and the channel decision module; performing trusted verification on the target data returned by the target terminal through a security verification engine; and storing the trusted verified target data through a data storage module. This overcomes the problems of high failure rates, insufficient data acquisition integrity, and narrow applicability scenarios in related technologies due to the use of a pre-set single fixed acquisition channel data acquisition system. Based on a data acquisition system comprising a data acquisition scheduling engine, a security verification engine, and a data storage module, the data acquisition method provided in this application utilizes adaptive multi-channel scheduling to enable the data acquisition system to automatically bypass faulty, congested, or policy-prohibited acquisition channels and dynamically select the optimal path (i.e., the target acquisition channel). This significantly reduces acquisition failures caused by the unavailability of a single channel and ensures that data acquisition tasks can be successfully executed even in variable network environments. Simultaneously, the independent security verification engine performs mandatory trusted verification on the returned data, effectively preventing malicious data from being stored due to tampering during transmission or the use of unauthorized scripts, thereby improving the quality of the acquired data.

[0029] Details of one or more embodiments of the present invention are set forth in the following drawings and description, so that other features, objects and advantages of the invention will be more readily understood. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the main flow of a data acquisition method according to one embodiment of the present invention;

[0032] Figure 2This is a schematic diagram of the main process of adaptive multi-channel scheduling based on a data acquisition scheduling engine in one embodiment of the present invention;

[0033] Figure 3 This is a schematic diagram of the main process for implementing data trust verification in one embodiment of the present invention;

[0034] Figure 4 This is a schematic diagram of the main process for implementing data trust verification in another embodiment of the present invention;

[0035] Figure 5 This is a schematic diagram of the main modules of a data acquisition system according to one embodiment of the present invention;

[0036] Figure 6 This is a schematic diagram of the main framework of a data acquisition system according to another embodiment of the present invention;

[0037] Figure 7 This is a schematic diagram of the electronic device of the present invention. Detailed Implementation

[0038] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0039] During the development of the data acquisition system, it was found that the data acquisition schemes in related technologies have at least the following technical problems when performing data acquisition tasks, especially when facing large-scale, cross-network, and high-security acquisition scenarios: (1) The data acquisition systems used in related technical solutions usually pre-set a single fixed acquisition channel (such as relying only on SSH or only through a dedicated agent). When the network policy in the network environment where the target terminal is located is adjusted, the firewall rules are tightened, or the main acquisition channel fails, the system cannot dynamically sense and switch to other available channels (such as lightweight probes or WebAPIs), which leads to a significant increase in the failure rate of acquisition tasks and makes it difficult to ensure the completeness of acquisition coverage. (2) In order to ensure the authenticity of the acquisition script and data and prevent tampering and replay attacks, traditional data acquisition schemes often use real-time signature verification or complex two-way authentication. Although this verification method improves security to a certain extent, in the scenario of a large number of terminals collecting data concurrently, the server needs to perform high-intensity cryptographic operations for each request (such as real-time generation or verification of digital signatures), which causes huge computational pressure, resulting in a sharp increase in system response delay and forming an obvious performance bottleneck. (3) Traditional data acquisition schemes usually lack a chain verification mechanism that links “trustworthy acquisition script”, “valid acquisition timing” and “reliable data source”. This makes it impossible to effectively identify risks introduced by malicious script tampering, replay of acquisition requests or hijacking of data during transmission, and the overall credibility of the acquisition results is questionable.

[0040] To address the aforementioned problems, embodiments of the present invention provide a data acquisition method applied to a data acquisition system, which includes an acquisition scheduling engine, a security verification engine, and a data storage module; such as Figure 1 As shown, this data acquisition method mainly includes:

[0041] Step S101: In response to the initiation of the data acquisition task, the acquisition scheduling engine adaptively selects a target acquisition channel from multiple acquisition channels and issues a data acquisition task to the corresponding target terminal, so that the target terminal can acquire target data through the target acquisition channel. The adaptive selection of the target acquisition channel includes: using the acquisition scheduling engine, detecting the network status of each acquisition channel based on the network detection module to determine the available acquisition channels; filtering candidate acquisition channels that meet the policy based on the policy matching module according to the type and security requirements of the data acquisition task; calculating the channel weight of each candidate acquisition channel based on the weight calculation module according to historical channel performance data; and selecting the target acquisition channel from the candidate acquisition channels based on the channel weight and the channel decision module.

[0042] Step S102: Perform trusted verification on the target data returned by the target terminal through the security verification engine.

[0043] Step S103: Store the target data that has passed the trusted verification through the data storage module.

[0044] The data collection method provided in this application is applicable to data collection scenarios such as security assessment and terminal security inspection, and it also has good data collection effect in dealing with large-scale, cross-network, and high-security collection scenarios.

[0045] Based on the above setup, and using a data acquisition system comprising a data acquisition scheduling engine, a security verification engine, and a data storage module, the data acquisition method provided in this application, through adaptive multi-channel scheduling, enables the data acquisition system to automatically bypass faulty, congested, or policy-prohibited acquisition channels, dynamically selecting the optimal path (i.e., the target acquisition channel). This significantly reduces acquisition failures caused by the unavailability of a single channel, ensuring that data acquisition tasks can be successfully executed even in variable network environments. Simultaneously, the independent security verification engine performs mandatory trusted verification on the returned data, effectively preventing malicious data from being tampered with during transmission or acquired using unauthorized scripts, thus improving the quality of the acquired data. Furthermore, the organic combination and coordination of these two key technologies—adaptive selection of the optimal acquisition channel based on multi-dimensional information and independent and mandatory trusted verification before data entry—constructs an intelligent, flexible, secure, and reliable data acquisition process. On the one hand, it addresses the uncertainties of the underlying network environment through dynamic scheduling; on the other hand, it defends against the security risks of the data itself through architectural verification. Ultimately, this collaborative approach enables the successful execution of data acquisition tasks in highly complex scenarios, ensuring the accuracy and integrity of the acquired data.

[0046] Specifically, the adaptive multi-channel scheduling mechanism provided in this application is implemented using a data acquisition scheduling engine from four dimensions: dynamic perception, policy filtering, weight ranking, and intelligent decision-making. According to a specific implementation method of this application, as follows... Figure 2 As shown, the main process of adaptive multi-channel scheduling based on the acquisition scheduling engine includes: S201, performing port detection and protocol availability testing on each acquisition channel based on the network awareness module to determine available acquisition channels; S202, performing task type matching and security domain filtering on each available acquisition channel based on the policy matching module according to the type and security requirements of the data acquisition task to determine candidate acquisition channels; S203, calculating the channel weight of each candidate acquisition channel based on the channel acquisition success rate, channel latency, and channel security level indicated in the historical channel performance data of each candidate acquisition channel by the weight calculation module; S204, selecting the optimal channel from the candidate acquisition channels as the target acquisition channel based on the channel decision module according to the channel weight.

[0047] The system employs a data acquisition scheduling engine to proactively detect the network status of each acquisition channel (e.g., port connectivity, protocol availability) using a network detection module. This obtains a set of real-time available acquisition channels, providing the latest factual basis for subsequent adaptive channel selection. Next, a policy matching module matches available acquisition channels with the specific requirements of the current data acquisition task (e.g., security level, acquisition type), filtering out non-compliant channels to obtain a candidate acquisition channel set. This ensures that the channel selection in the subsequent adaptive selection complies with security standards. Then, a weight calculation module incorporates historical channel performance data (e.g., historical response latency, success rate, channel security level, etc.) to calculate a quantified channel weight for each candidate acquisition channel. This upgrades the selection criterion from simple on / off status to a performance weight ranking that comprehensively considers historical channel performance. Finally, a channel decision module determines one or more optimal target acquisition channels based on the channel weights. This closed-loop decision-making process ensures that the system can always select the historically best-performing target acquisition channel from the currently available and compliant channels to execute the current data acquisition task. This allows the system to directly address network fluctuations and policy changes, thereby improving the acquisition success rate.

[0048] Furthermore, the data acquisition method provided in this application decouples the scheduling of data transmission channels (handled by the acquisition scheduling engine) from the verification of the data's credibility (handled by the security verification engine) in its architecture by applying an independent security verification engine. This means that even if the acquisition scheduling engine successfully selects a suitable target acquisition channel and acquires the target data, the target data will not be directly accepted. All target data returned by the target terminal must undergo credible verification by the security verification engine (which can be any mechanism that ensures the authenticity and integrity of the data, such as checksum comparison, digital signature verification, or source authentication). This ensures that the target data ultimately stored in the data storage module is not only acquired through the optimal acquisition channel but has also undergone credible verification, thereby fundamentally guaranteeing the authenticity and integrity of the data.

[0049] Furthermore, it is understood that the data acquisition method provided in this application forms a complete automated closed loop from adaptive multi-channel selection and task distribution to data verification and storage. The data acquisition system applied can make intelligent decisions based on real-time network status, predefined strategies and historical performance data, which greatly reduces the need for manual intervention and improves operation and maintenance efficiency.

[0050] In some examples, the target acquisition channel provided in this application is at least one of an SSH channel, a light probe channel, or a Web script channel.

[0051] Among them, the SSH tunnel utilizes a remote security management protocol widely supported by standard Linux / Unix servers and network devices, enabling high-privilege and efficient data collection on management devices with command-line interfaces and open SSH services. The lightweight probe tunnel typically refers to a lightweight, memory-resident agent that communicates with the server via reverse connections or long-lived connections, enabling penetrating data collection behind Network Address Translation (NAT) and firewalls, and providing continuous monitoring and real-time response. The Web script tunnel utilizes the HTTP / HTTPS protocol widely supported by modern operating systems and applications, enabling data collection under extremely strict network policies that only allow web traffic (ports 80 / 443), as well as data collection on web-based devices / applications that cannot install SSH services or dedicated probes.

[0052] Based on the above settings, the compatibility and environmental adaptability of the data acquisition system are expanded, ensuring that the data acquisition method can still maintain high acquisition coverage in heterogeneous and complex network and terminal environments.

[0053] For example, using a Web script channel in an environment with only HTTP / HTTPS ports open, an SSH channel in an environment allowing management communication, and a lightweight probe reverse connection in a strictly restricted environment can bypass network policy restrictions. Furthermore, using SSH or Web scripts on devices where agents cannot be installed, and using the lower-overhead Web scripts on resource-constrained devices, allows the data acquisition system to cope with the diversity of target terminals in terms of computing power, pre-installed software, and permissions. This enables the data acquisition method to adapt to differences in terminal resources and configurations, expanding the applicable scenarios of the solution. In addition, when a certain type of acquisition channel fails extensively due to global policy adjustments or widespread faults (e.g., SSH ports are uniformly blocked), the data acquisition system can immediately switch to other types of channels, achieving redundant backup of acquisition capabilities.

[0054] In some embodiments, the step of performing trusted verification of target data returned by the target terminal through the security verification engine includes: after receiving the data acquisition task, the target terminal obtains the current time window benchmark from the time window management module through the security verification engine; after the target data is acquired, the target terminal uses the current time window benchmark and the acquisition script as input to calculate and generate a first verification code through the HMAC algorithm, and sends the target data, the current time window benchmark, and the first verification code to the security verification engine; based on the received current time window benchmark, the security verification engine sends a second verification code query request to the verification code caching module, and compares the second verification code returned by the verification code caching module with the first verification code; if they match, the target data is determined to have passed trusted verification; wherein, the second verification code is pre-calculated by the time window management module based on the current time window benchmark generated based on a fixed time window, and then stored in the verification code caching module based on the current time window benchmark and a pre-stored script through the HMAC algorithm.

[0055] Based on the above settings, the heavy-duty cryptographic computation (HMAC) on the server side in related technical solutions is removed from real-time request processing and transformed into a pre-computation and cache query mode. This results in extremely low response latency for single data verification (down to milliseconds), easily handling concurrent reporting from tens of thousands of terminals, thus solving the server-side performance bottleneck problem caused by traditional real-time signature verification. Simultaneously, because the CAPTCHA (HMAC) is bound to the content of the collection script and a time window benchmark, any tampering with the script or data will lead to verification failure, ensuring the integrity of the collection script and data. Furthermore, using a fixed time window as the dynamic benchmark for determining the current time window ensures that each CAPTCHA is only valid within a specific time window. Even if an attacker intercepts a valid "data-CAPTCHA" combination, they cannot reuse it in the next time window, effectively resisting replay attacks.

[0056] According to a specific implementation of the embodiments of this application, such as Figure 3 , Figure 4As shown, the main process for trusted verification of target data may include: S301, in response to server startup, the time window management module generates the current time window benchmark based on a fixed time window; S302, the HMAC calculation module pre-calculates the second verification code based on the current time window benchmark and the pre-stored script; S303, after receiving the data acquisition task, the target terminal obtains the current time window benchmark from the time window management module through the security verification engine. After the target data is acquired, the target terminal uses the current time window benchmark and the acquisition script as input to calculate and generate the first verification code through the HMAC algorithm, and sends the target data, the current time window benchmark, and the first verification code to the security verification engine; S304, based on the received current time window benchmark, the security verification engine sends a second verification code query request to the verification code caching module, and compares the second verification code returned by the verification code caching module with the first verification code.

[0057] In other words, after the server starts, the time window management module generates a unified, periodically changing current time window baseline T_BASE according to a fixed time window (e.g., 15 minutes). Since T_BASE changes over time, the HMAC verification code calculated based on it also has a time limit. An outdated data packet will inevitably have a T_BASE that does not match the server's current or previous valid time window, leading to verification failure. This introduces a time dimension to the security verification mechanism of the entire data acquisition scheme, fundamentally cutting off the path for replay attacks.

[0058] According to another specific implementation of the embodiments of this application, before the terminal executes the data collection task, the time window management module uses the same pre-stored script and current T_BASE as the terminal to pre-calculate (i.e., pre-compute) the verification codes (i.e., the second verification codes) that should be present for all legitimate requests within the current time window baseline, and stores them in the verification code cache module. When the target terminal reports the target data, the security verification engine no longer needs to perform time-consuming HMAC calculations, but directly uses the T_BASE reported by the terminal as the key to initiate a fast query to the verification code cache module, and performs a simple comparison between the returned pre-stored value (the second verification code) and the value reported by the terminal (the first verification code) to complete the trusted verification of the data. Accordingly, the server's response to each request is transformed from O(n) cryptographic calculation to O(1) cache query, thereby achieving high concurrency performance.

[0059] The HMAC algorithm mentioned above can be HMAC-MD5, HMAC-SHA256, or the Chinese national cryptographic algorithm SM3-HMAC. The terminal uses the exact same algorithm (HMAC) and input (T_BASE + data collection script) as the server to generate the first verification code. This ensures that only terminals with the correct script and executing it within the legal time window can generate a verifiable HMAC code. Any modification to the script or use of an unauthorized script will result in the calculated first verification code being inconsistent with the second verification code pre-stored on the server, thus achieving strong verification of data source and integrity.

[0060] In some embodiments, the verification code caching module adopts a dual-window caching structure to store the second verification codes corresponding to the current time window benchmark and the previous time window benchmark respectively. After the security verification engine sends a second verification code query request to the verification code caching module, the verification code caching module performs an index query in the dual-window cache according to the received current time window benchmark. If a match is found, the corresponding second verification code is returned directly.

[0061] Based on the above settings, the CAPTCHA caching module, employing a dual-window caching structure, resolves the issue of legitimate data being mistakenly judged as verification failure due to minor time asynchrony or network latency between the server and terminal at the moment of switching between fixed time windows (e.g., every 15 minutes). This eliminates verification jitter, ensures business continuity, and guarantees that the data acquisition process will not experience intermittent interruptions due to time boundary issues. Understandably, even before and after switching between fixed time windows, the data acquisition system still completes data verification operations through cache queries, without needing to fall back to real-time calculations or complex exception handling processes. This maintains a stable and low-latency verification process, preventing performance fluctuations caused by boundary issues. This design provides a tolerant buffer for small clock deviations and network transmission delays between the terminal and server, reducing the system's stringent requirements for absolute time synchronization and enhancing its adaptability under non-ideal network conditions.

[0062] According to the embodiments of this application, the verification code caching module does not only store the second verification code corresponding to the current valid time window at any given time, but also simultaneously stores the second verification codes corresponding to two adjacent time windows in memory, namely (T_BASE_current, HMAC_current) and (T_BASE_previous, HMAC_previous). This allows the security verification engine to expand its verification scope from one time window to two consecutive time windows when it receives a request carrying T_BASE. Regardless of whether the request logically belongs to an old time window that has just ended or a new time window that has just begun, its corresponding correct second verification code has already been stored in the verification code caching module, thereby avoiding the critical state of "unable to find the corresponding second verification code" caused by the instantaneous event of time window switching.

[0063] According to a specific implementation of this application, the query logic for index lookup in the dual-window cache can be designed to first search for the second verification code corresponding to the current time window; if no match is found, then search for the second verification code corresponding to the previous time window. When the T_BASE reported by the terminal is consistent with the server's current time window, the query immediately hits the cache in the current time window; when the terminal is still using the T_BASE of the previous window, the query will fail in the current time window, but will then successfully hit the cache in the previous time window. The entire verification process does not introduce any time-consuming calculations or logical judgments, maintaining the high-performance cache query-comparison mode and smoothly handling boundary data.

[0064] Understandably, the cache structure only retains the second CAPTCHA corresponding to the current time window and the previous time window, rather than all historical time windows. This means that the tolerance period for CAPTCHAs from older time windows is limited (usually one window duration). This design strikes a balance between allowing a certain degree of clock skew and latency and preventing replay attacks from expired data. It gives the terminal enough time (one time window period) to complete the transition from the old time window to the new time window, while ensuring that older CAPTCHAs (such as those from the previous day or week) cannot be reused because they have already been overwritten in the cache by the new "previous time window" data. Thus, without sacrificing core security, it solves the common clock asynchrony and boundary state problems in distributed systems.

[0065] In addition, the above data acquisition method also supports traceability of fusion results, that is, the channel weight / confidence weight of each target acquisition channel and the original data involved in the calculation can be recorded to facilitate subsequent security audits or algorithm optimization.

[0066] In some embodiments, after the step of adaptively selecting a target acquisition channel from multiple acquisition channels through the acquisition scheduling engine and issuing a data acquisition task to the corresponding target terminal so that the target terminal can acquire target data through the target acquisition channel, the method further includes: sending the result information of this data acquisition task to the feedback learning module through the acquisition scheduling engine so that the feedback learning module updates the historical channel performance data based on the result information and returns the updated historical channel performance data to the acquisition scheduling engine for channel weight calculation in the next data acquisition task.

[0067] Based on the above settings, the acquisition scheduling engine uses the following as the basis for calculating the weights for channel selection during the adaptive multi-channel scheduling process: instead of relying on initial, static, or outdated historical data, it uses historical channel performance data updated based on the latest and most accurate acquisition results. This allows the channel weight coefficients to truly reflect the real-time performance of each acquisition channel in the current environment.

[0068] According to the embodiments of this application, by introducing a feedback learning module, it is possible to automatically identify and reduce the channel weight of acquisition channels that fail continuously or have low performance, and even exclude them in terms of strategy, so as to avoid the data acquisition system repeatedly selecting low-performance acquisition channels, causing acquisition task delays or failures; it is also possible to promptly discover high-quality acquisition channels that have recovered in performance or have newly emerged, and increase their channel weight, so as to ensure that the data acquisition system always tends to use the current optimal acquisition path.

[0069] According to a specific implementation of the embodiments of this application, after the current data acquisition task is executed, the data acquisition system will feed back the acquisition results (success rate, latency, and error type of the target acquisition channel) to the weighting technology module through the feedback learning module to form an adaptive adjustment mechanism. For example: if the SSH acquisition channel fails 3 times in a row, it will automatically be downgraded and switched to the probe channel; if the probe channel has high latency, the Web script channel will be temporarily prioritized; if the task is a sensitive type (configuration acquisition), only the channel with high security level will be allowed. Based on this feedback learning mechanism, intelligent scheduling of self-learning + policy-driven + multi-channel reuse is realized. Furthermore, the feedback learning mechanism has at least the following technical advantages: (1) Self-learning optimization: the channel selection adjusts itself according to the historical success rate; (2) Zero manual configuration: the administrator does not need to manually select the acquisition method; (3) High coverage: it can still collect data in isolated network segments, VPNs, or policy-restricted scenarios; (4) Intelligent degradation: the backup path is automatically enabled when the main channel fails; (5) Multi-channel fusion: it supports unified decision-making for SSH, probes, and Web scripts.

[0070] In some embodiments, when there are multiple target acquisition channels, the method further includes: sending the channel acquisition results corresponding to the multiple target acquisition channels that have passed trusted verification to the consistency fusion module through the security verification engine; requesting the confidence weight of each target acquisition channel from the weight calculation module through the consistency fusion module; performing weighted fusion of the channel acquisition results corresponding to the multiple target acquisition channels based on the confidence weight to obtain the final acquisition result; and sending the final acquisition result to the data storage module for storage.

[0071] Based on the above settings, by integrating the acquisition results from multiple independent target acquisition channels, it is possible to intelligently eliminate random errors, transient interference, or localized errors that may exist in a single channel.

[0072] According to a specific implementation of this application, when the acquisition results returned by different target acquisition channels differ or even contradict each other (for example, channel A returns a value of 100, and channel B returns a value of 120), the system does not simply randomly select one or report an error, but performs weighted arbitration based on the "confidence" of each channel to output the most reasonable value. Furthermore, even if one or more target acquisition channels provide erroneous data due to temporary failure or a specific attack, as long as not all acquisition channels with high confidence weights fail simultaneously, the data acquisition method provided by this application can still correct the final result using data from other reliable channels, ensuring the robustness of the overall output. The aforementioned confidence weights are dynamically calculated by the weight calculation module based on at least one of the following factors: the completeness of the data provided by each acquisition channel, acquisition latency, and / or data consistency.

[0073] Specifically, the consistency fusion module does not determine which result is more reliable on its own. Instead, it requests the confidence weights of each channel from the professional weight calculation module (a quantitative indicator whose calculation basis (such as integrity, latency, and consistency) itself reflects the reliability and data quality of the channel). After receiving the acquisition results and corresponding confidence weights of each target acquisition channel, the consistency fusion module executes a weighted fusion algorithm (such as weighted average). The basic principle is that the higher the weight of a channel, the greater the proportion of its data in the final result.

[0074] The data acquisition method provided in this embodiment of the invention overcomes the problems of high failure rate, insufficient data acquisition integrity, and narrow applicability of related technologies due to the use of a data acquisition system with a pre-set single fixed acquisition channel. This is achieved by adaptively selecting a target acquisition channel from multiple acquisition channels in response to the initiation of a data acquisition task, and then issuing a data acquisition task to the corresponding target terminal, enabling the target terminal to acquire target data via the target acquisition channel. The adaptive selection of the target acquisition channel includes: using the acquisition scheduling engine to detect the network status of each acquisition channel based on a network detection module to determine available acquisition channels; filtering candidate acquisition channels that meet the policy based on the type and security requirements of the data acquisition task using a policy matching module; calculating the channel weight of each candidate acquisition channel based on historical channel performance data using a weight calculation module; selecting the target acquisition channel from the candidate acquisition channels based on the channel weight and a channel decision module; performing trusted verification on the target data returned by the target terminal using a security verification engine; and storing the trusted verified target data using a data storage module. Based on a data acquisition system comprising a data acquisition scheduling engine, a security verification engine, and a data storage module, the data acquisition method provided in this application utilizes adaptive multi-channel scheduling to enable the data acquisition system to automatically bypass faulty, congested, or policy-prohibited acquisition channels and dynamically select the optimal path (i.e., the target acquisition channel). This significantly reduces acquisition failures caused by the unavailability of a single channel, ensuring that data acquisition tasks can be successfully executed even in variable network environments. Simultaneously, the independent security verification engine performs mandatory trusted verification on the returned data, effectively preventing malicious data from being stored due to tampering during transmission or the use of unauthorized scripts, thus improving the quality of the acquired data.

[0075] Based on the data acquisition method provided in the embodiments of the present invention, the embodiments of the present invention also provide a data acquisition system, such as... Figure 5 As shown, the data acquisition system 500 includes an acquisition scheduling engine 501, a security verification engine 502, and a data storage module 503 deployed on the server side; wherein:

[0076] The data acquisition scheduling engine 501 is used to respond to the initiation of a data acquisition task by adaptively selecting a target acquisition channel from multiple acquisition channels and issuing a data acquisition task to the corresponding target terminal, so that the target terminal can acquire target data through the target acquisition channel. The acquisition scheduling engine is also used to: detect the network status of each acquisition channel based on the network detection module to determine available acquisition channels; filter candidate acquisition channels that meet the policy based on the policy matching module according to the type and security requirements of the data acquisition task; calculate the channel weight of each candidate acquisition channel based on the weight calculation module according to historical channel performance data; and select the target acquisition channel from the candidate acquisition channels based on the channel weight and the channel decision module.

[0077] Security verification engine 502 is used to perform trusted verification on the target data returned by the target terminal.

[0078] The data storage module 503 is used to store target data that has passed trusted verification.

[0079] Based on the above setup, the data acquisition system, comprising a data acquisition scheduling engine, a security verification engine, and a data storage module, utilizes adaptive multi-channel scheduling to automatically bypass faulty, congested, or policy-prohibited acquisition channels, dynamically selecting the optimal path (i.e., the target acquisition channel). This significantly reduces acquisition failures due to the unavailability of a single channel, ensuring successful execution of data acquisition tasks even in dynamic network environments. Simultaneously, an independent security verification engine performs mandatory trusted verification on the returned data, effectively preventing malicious data from being tampered with during transmission or acquired using unauthorized scripts, thus improving the quality of the acquired data. Furthermore, the organic combination and coordination of these two key technologies—adaptive selection of the optimal acquisition channel based on multi-dimensional information and independent and mandatory trusted verification before data entry—builds an intelligent, flexible, secure, and reliable data acquisition process. On one hand, dynamic scheduling addresses the uncertainties of the underlying network environment; on the other hand, architectural verification defends against security risks inherent in the data itself. Ultimately, this collaborative approach enables successful execution of data acquisition tasks in highly complex scenarios, ensuring the accuracy and integrity of the acquired data.

[0080] like Figure 6As shown, the data acquisition system deployed on the server side also includes a consistency fusion module. When there are multiple target acquisition channels, the security verification engine sends the channel acquisition results corresponding to the multiple target acquisition channels that have passed trusted verification to the consistency fusion module. The consistency fusion module is used to: request the confidence weight of each target acquisition channel from the weight calculation module, and perform weighted fusion on the channel acquisition results corresponding to the multiple target acquisition channels based on the confidence weight to obtain the final acquisition result, and send the final acquisition result to the data storage module for storage.

[0081] Based on the above settings, by integrating the acquisition results from multiple independent target acquisition channels, it is possible to intelligently eliminate random errors, transient interference, or localized errors that may exist in a single channel.

[0082] like Figure 6 As shown, the data acquisition system also includes multiple terminal systems, each with multiple acquisition channels, such as SSH channels, light probe channels, Web script channels, or other acquisition channels.

[0083] Furthermore, such as Figure 6 As shown, the modules deployed in the data acquisition system, especially the server system, also include multiple sub-modules for performing specific operations. The specific operations performed by each module and the technical effects achieved are described in the aforementioned data acquisition method, and will not be repeated here.

[0084] Meanwhile, since this application is based on the data acquisition system 500 to implement any of the data acquisition methods described above, the execution operations of each part of the data acquisition system are consistent with those in the aforementioned scheme, and can achieve all the beneficial effects corresponding to the aforementioned scheme. This application will not elaborate further here.

[0085] This invention also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of this invention.

[0086] This invention also provides a computer program product, including a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform the methods of embodiments of this invention. The computer program product should be understood as a software product that primarily implements the methods described above through a computer program.

[0087] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, which, when executed by the at least one processor, causes the electronic device to perform the method of this invention.

[0088] refer to Figure 7 The present invention will now be described in the form of a structural block diagram of an electronic device that can serve as an embodiment of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0089] like Figure 7 As shown, the electronic device includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. The RAM 703 may also store various programs and data required for the operation of the electronic device. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0090] Multiple components in the electronic device are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information into the electronic device. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 707 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disks and optical discs. Communication unit 709 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, and / or wireless communication transceivers, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0091] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, CPUs, graphics processing units (GPUs), various special-purpose artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the method embodiments of the present invention can be implemented as a computer program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 702 and / or communication unit 709. In some embodiments, the computing unit 701 can be configured to perform the methods described above by any other suitable means (e.g., by means of firmware).

[0092] Computer programs for implementing the methods of embodiments of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0093] In the context of embodiments of the present invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0094] It should be noted that the term "comprising" and its variations used in the embodiments of the present invention are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that, unless explicitly indicated otherwise in the context, they should be understood as "one or more".

[0095] The information / data involved in the embodiments of this invention (including but not limited to user device information / data, user personal information / data, information / data used for analysis, stored information / data, displayed information / data, etc.) are all information / data that has been permitted by the user or fully agreed upon by all parties. Furthermore, the collection, use and processing of such information / data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to agree or refuse.

[0096] The steps described in the method embodiments provided by the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0097] The term "embodiment" in this specification refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily imply the same embodiment, nor does it imply independence or alternativeity from other embodiments. The various embodiments in this specification are described in a related manner, with reference made to the similar or identical parts between the embodiments. In particular, for apparatus, device, and system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, and relevant details are referred to in the description of the method embodiments.

[0098] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A data acquisition method, characterized in that, The method is applied to a data acquisition system, which includes an acquisition scheduling engine, a security verification engine, and a data storage module; the method includes: In response to the initiation of a data acquisition task, the acquisition scheduling engine adaptively selects a target acquisition channel from multiple acquisition channels and issues a data acquisition task to the corresponding target terminal, so that the target terminal acquires target data via the target acquisition channel. The adaptive selection of the target acquisition channel includes: using the acquisition scheduling engine, detecting the network status of each acquisition channel based on the network detection module to determine available acquisition channels; filtering candidate acquisition channels that meet the policy requirements from the available acquisition channels based on the policy matching module according to the type and security requirements of the data acquisition task; calculating the channel weight of each candidate acquisition channel based on historical channel performance data using the weight calculation module; and selecting the target acquisition channel from the candidate acquisition channels based on the channel weights using the channel decision module. The security verification engine performs trusted verification on the target data returned by the target terminal. The steps of performing trusted verification on the target data returned by the target terminal through the security verification engine include: after receiving the data acquisition task, the target terminal obtains the current time window benchmark from the time window management module via the security verification engine; after acquiring the target data, the target terminal uses the current time window benchmark and the acquisition script as input to calculate and generate a first verification code using the HMAC algorithm, and sends the target data, the current time window benchmark, and the first verification code to the security verification engine; based on the received current time window benchmark, the security verification engine sends a second verification code query request to the verification code caching module, and compares the second verification code returned by the verification code caching module with the first verification code; if they match, the target data is determined to have passed trusted verification; wherein the second verification code is pre-calculated by the time window management module based on the current time window benchmark generated based on a fixed time window, and then stored in the verification code caching module using the HMAC algorithm based on the current time window benchmark and a pre-stored script. The target data that has passed the trusted verification is stored through the data storage module.

2. The method according to claim 1, characterized in that, The verification code caching module adopts a dual-window caching structure to store the second verification code corresponding to the current time window benchmark and the previous time window benchmark respectively. After the security verification engine sends a second verification code query request to the verification code caching module, the verification code caching module performs an index query in the dual-window cache based on the received current time window benchmark. If a match is found, the corresponding second verification code is returned directly.

3. The method according to claim 1, characterized in that, After the step of adaptively selecting a target acquisition channel from multiple acquisition channels through the acquisition scheduling engine and issuing a data acquisition task to the corresponding target terminal so that the target terminal acquires target data via the target acquisition channel, the method further includes: The acquisition scheduling engine sends the result information of this data acquisition task to the feedback learning module, so that the feedback learning module updates the historical channel performance data based on the result information and returns the updated historical channel performance data to the acquisition scheduling engine for channel weight calculation in the next data acquisition task.

4. The method according to claim 1, characterized in that, When the number of target acquisition channels is multiple, the method further includes: The security verification engine sends the channel acquisition results corresponding to the multiple target acquisition channels that have passed trusted verification to the consistency fusion module. The consistency fusion module requests the confidence weight of each target acquisition channel from the weight calculation module. The consistency fusion module performs weighted fusion on the acquisition results of the multiple target acquisition channels based on the confidence weight to obtain the final acquisition result, and sends the final acquisition result to the data storage module for storage.

5. The method according to claim 4, characterized in that, The confidence weight is dynamically calculated by the weight calculation module based on at least one of the following factors: the integrity of the data provided by each acquisition channel, the acquisition delay, and / or the data consistency.

6. The method according to claim 1, characterized in that, The target acquisition channel is at least one of an SSH channel, a light probe channel, or a Web script channel.

7. A data acquisition system, characterized in that, include: A data acquisition scheduling engine is used to adaptively select a target acquisition channel from multiple acquisition channels in response to the initiation of a data acquisition task, and to issue a data acquisition task to the corresponding target terminal so that the target terminal can acquire target data through the target acquisition channel. The acquisition scheduling engine is further used to: detect the network status of each acquisition channel based on a network detection module to determine available acquisition channels; filter candidate acquisition channels that meet the policy based on a policy matching module according to the type and security requirements of the data acquisition task; calculate the channel weight of each candidate acquisition channel based on a weight calculation module according to historical channel performance data; and select the target acquisition channel from the candidate acquisition channels based on the channel weights and a channel decision module. A security verification engine is used to perform trusted verification on the target data returned by the target terminal. The security verification engine is further configured to: after receiving the data acquisition task, the target terminal obtains the current time window benchmark from the time window management module via the security verification engine; after completing the acquisition of the target data, the target terminal uses the current time window benchmark and the acquisition script as input to calculate and generate a first verification code using the HMAC algorithm, and sends the target data, the current time window benchmark, and the first verification code to the security verification engine; based on the received current time window benchmark, the security verification engine sends a second verification code query request to the verification code caching module, and compares the second verification code returned by the verification code caching module with the first verification code; if they match, the target data is determined to have passed trusted verification; wherein the second verification code is pre-calculated by the time window management module based on the current time window benchmark generated by the time window management module based on a fixed time window, and then stored in the verification code caching module using the HMAC algorithm based on the current time window benchmark and a pre-stored script. A data storage module is used to store the target data that has passed trusted verification.

8. An electronic device, comprising: A processor and a memory storing a program, characterized in that the program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory machine-readable medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 6.

Images