A data collection method, device and medium based on a one-way network environment

Abstract

The embodiments of this specification disclose a data collection method, device and medium based on a unidirectional network environment, which relate to the field of data collection technology. The method includes: generating corresponding collection task configuration information through an external network server deployed in the external network, each data collection task corresponds to at least one data source, and the internal network client is deployed in the internal network node; writing the collection task configuration information corresponding to each data collection task into a pre-built external network business library; based on a specified trigger of at least one specified internal network client, pulling the specified collection task configuration information of the specified internal network client from the external network business library; executing the specified data collection task according to the specified collection task configuration information through the specified internal network client to generate specified collection data; backfilling the specified collection data into the external network business library, so that the external network server can read the specified collection data in the external network business library, thereby realizing unidirectional data collection from the external network to the internal network.

Description

Technical Field

[0001] This specification relates to the field of data collection technology, and in particular to a data collection method, device, and medium based on a unidirectional network environment. Background Art

[0002] With the development of internet technology, data plays a vital role in various enterprises and institutions. Collecting and analyzing various types of data can play a role in prediction, analysis, or supervision. Industrial data and business data, for example, all need to go through the process of collection and processing. In this process, data collection is the source of data and is related to the entire process. Therefore, the role of data collection cannot be ignored.

[0003] With the development of various businesses, a special type of data collection scenario has emerged, such as data collection scenarios that require high security and confidentiality in enterprises, governments, or the military. Most existing data collection methods use two-way transmission between two parties in different organizations to achieve data collection, but they cannot meet the scenario requirements of data security and data confidentiality. The development of information technology has led to the widespread application of one-way network environments. In a one-way network environment, due to the one-way nature of the network, traditional data collection methods cannot achieve data collection, such as directly accessing the target server or intercepting network data packets. Therefore, existing data collection methods cannot perform data collection in scenarios that meet the requirements of data security and data confidentiality. Summary of the Invention

[0004] One or more embodiments of this specification provide a data collection method, device, and medium based on a unidirectional network environment, which are used to solve the following technical problem: the existing data collection method cannot collect data in scenarios that meet the requirements of data security and data confidentiality.

[0005] One or more embodiments of this specification adopt the following technical solutions:

[0006] One or more embodiments of the present specification provide a data collection method based on a unidirectional network environment, the method comprising: determining at least one data collection task through an external network server deployed in an external network to generate collection task configuration information corresponding to each data collection task, wherein each data collection task corresponds to at least one data source, each data source is stored in a corresponding intranet client, and the intranet client is deployed in an intranet node; writing the collection task configuration information corresponding to each data collection task into a pre-built external network service library, wherein the external network service library allows access by multiple intranet clients; based on a specified trigger of at least one specified intranet client, pulling the specified collection task configuration information of the specified data collection task corresponding to the specified intranet client from the external network service library; executing the specified data collection task through the specified intranet client in accordance with the specified collection task configuration information to generate specified collection data; and backfilling the specified collection data into the external network service library so that the external network server can read the specified collection data in the external network service library, thereby realizing unidirectional data collection from the external network to the intranet.

[0007] Furthermore, the collection task configuration information corresponding to each of the data collection tasks is generated, specifically including: deploying the external network server in the external network, and determining, through the external network server, multiple data sources to be collected and rules to be collected corresponding to the data collection task, wherein the rules to be collected are used to represent the table information to be collected or the field information to be collected in the data source; determining the intranet client to which each of the data sources to be collected belongs according to the data source to be collected and the pre-acquired intranet client information table, wherein the intranet client information table includes multiple intranet client addresses and the data source in each intranet client; establishing a correspondence between the data collection task and the intranet client address based on the intranet client address to which each of the data sources to be collected belongs and the data collection task corresponding to the data to be collected; configuring the collection task configuration information for the data collection task, wherein the collection task configuration information includes the data source to be collected, the rules to be collected, and the correspondence between the data collection task and the intranet client address.

[0008] Furthermore, based on the specified trigger of at least one specified intranet client, the specified collection task configuration information of the specified data collection task corresponding to the specified intranet client is pulled from the extranet service library, specifically including: pre-initializing the specified intranet client; under the trigger of the completion of the initialization setting, obtaining the current client information of the specified intranet client; based on the current client information, pulling the specified data collection task corresponding to the current client information in the extranet service library; and determining the specified collection task configuration information through the specified data collection task.

[0009] Furthermore, the designated intranet client is initialized in advance, specifically including: obtaining the client information of the designated intranet client under the startup trigger of the designated intranet client, wherein the client information includes the intranet node where the client is located; writing the client information into the client information set of the external network service library, registering the designated intranet client, and initializing the designated intranet client.

[0010] Furthermore, based on the designated trigger of at least one designated intranet client, before pulling the designated collection task configuration information of the designated data collection task corresponding to the designated intranet client in the external network business library, the method also includes: obtaining at least one data source and at least one rule to be collected corresponding to each of the data collection tasks; dividing each of the data collection tasks into multiple data collection subtasks according to the at least one data source and at least one rule to be collected corresponding to each of the data collection tasks, wherein each data collection subtask includes a data source to be collected and a rule to be collected; determining the intranet client address of the data source to be collected according to the data source to be collected corresponding to each of the data collection subtasks, so as to determine the designated intranet client corresponding to each of the data collection subtasks.

[0011] Furthermore, after writing the collection task configuration information corresponding to each of the data collection tasks into a pre-built external network business library, the method also includes: obtaining a client information set in the external network business library through the external network server, wherein the client information set includes heartbeat data uploaded by multiple internal network clients; based on the client information set, performing status monitoring on multiple internal network clients to determine the operating status of each of the internal network clients, wherein the operating status includes normal operation and abnormal operation; when there is a first internal network client with abnormal operation, in the external network business library, setting the status of the specified data collection subtask corresponding to the first internal network client to a task suspension status to wait for the restart of the first internal network client, wherein the specified data collection subtask is an unfinished data collection subtask of the first internal network client.

[0012] Furthermore, before writing the collection task configuration information corresponding to each of the data collection tasks into the pre-built external network business library, the method further includes: building an external network business library, wherein the external network business library is located in the external network; interacting between the external network client service end and the plurality of the internal network clients through the external network business library, specifically including: obtaining the client status information and collection result information of each of the internal network clients; writing the client status information and the collection result information of the internal network client into the external network business library through the internal network client; writing the pre-acquired collection status information and the collection result information of the internal network client into the external network business library through the external network server end; The system controls the external network service terminal to obtain the client status information, the collection result information and the collection status information in the external network service library; monitors the operation status of the internal network client according to the client status information; obtains the collection data of the internal network client according to the collection result information, and displays the data on the external network service terminal; performs task status detection on the collection task of the internal network client according to the collection status information, wherein the task status includes task completion status, task suspension status, task progress status and task waiting status.

[0013] Furthermore, the intranet client has the authority to actively communicate with the external network, and the external network server does not have the authority to actively communicate with the intranet.

[0014] One or more embodiments of this specification provide a data collection device based on a unidirectional network environment, including:

[0015] at least one processor; and,

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0018] At least one data collection task is determined by an external network server deployed in the external network to generate collection task configuration information corresponding to each of the data collection tasks, wherein each of the data collection tasks corresponds to at least one data source, each of the data source is stored in a corresponding intranet client, and the intranet client is deployed in an intranet node; the collection task configuration information corresponding to each of the data collection tasks is written to a pre-built external network business library, wherein the external network business library allows multiple intranet clients to access; based on a specified trigger of at least one specified intranet client, the specified collection task configuration information of the specified data collection task corresponding to the specified intranet client is pulled from the external network business library; through the specified intranet client, the specified data collection task is executed in accordance with the specified collection task configuration information to generate specified collection data; the specified collection data is backfilled into the external network business library, so that the external network server can read the specified collection data in the external network business library to realize one-way data collection from the external network to the intranet.

[0019] One or more embodiments of this specification provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to:

[0020] At least one data collection task is determined by an external network server deployed in the external network to generate collection task configuration information corresponding to each of the data collection tasks, wherein each of the data collection tasks corresponds to at least one data source, each of the data source is stored in a corresponding intranet client, and the intranet client is deployed in an intranet node; the collection task configuration information corresponding to each of the data collection tasks is written to a pre-built external network business library, wherein the external network business library allows multiple intranet clients to access; based on a specified trigger of at least one specified intranet client, the specified collection task configuration information of the specified data collection task corresponding to the specified intranet client is pulled from the external network business library; through the specified intranet client, the specified data collection task is executed in accordance with the specified collection task configuration information to generate specified collection data; the specified collection data is backfilled into the external network business library, so that the external network server can read the specified collection data in the external network business library to realize one-way data collection from the external network to the intranet.

[0021] At least one of the above-mentioned technical solutions adopted in the embodiments of this specification can achieve the following beneficial effects: through the above-mentioned technical solution, the task configuration information corresponding to the data collection task is written into the external network business library through the external network server, and the internal network client pulls the information from the external network business library to obtain the collection task configuration information, executes the data collection task locally according to the configuration information, and writes the collection result back to the external network business library, and the external network server pulls the collection result in the external network business library. By constructing the external network business library, the server and the client indirectly complete data interaction through the external network business library, realize data collection from the internal network, and forward the data to the external network of the unidirectional network, and share the information of the external network server and the internal network client to realize data collection in the unidirectional network environment, realize data collection in the unidirectional network environment, ensure the security and confidentiality of data collection, realize data collection and result display, and improve the efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some of the embodiments described in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0023] Figure 1 A flow chart of a data collection method based on a one-way network environment provided in an embodiment of this specification;

[0024] Figure 2 This is a schematic diagram of an application of a data collection method based on a one-way network environment provided in an embodiment of this specification;

[0025] Figure 3 This is a structural diagram of a data acquisition device based on a unidirectional network environment provided in an embodiment of this specification. DETAILED DESCRIPTION

[0026] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this specification without creative work should fall within the scope of protection of this specification.

[0027] With the development of internet technology, data plays a vital role in various enterprises and institutions. Collecting and analyzing various types of data can play a role in prediction, analysis, or supervision. Industrial data and business data, for example, all need to go through the process of collection and processing. In this process, data collection is the source of data and is related to the entire process. Therefore, the role of data collection cannot be ignored.

[0028] With the development of various businesses, a special type of data collection scenario has emerged, such as data collection scenarios that require high security and confidentiality in enterprises, governments or the military. Most existing data collection methods are two-way transmission between two parties in different organizations to achieve data collection, but they cannot meet the scenario requirements of data security and data confidentiality. The development of information technology has led to the widespread application of one-way network environments. In a unidirectional network environment, due to the unidirectional nature of the network, traditional data collection methods cannot achieve data collection, such as directly accessing the target server or intercepting network data packets. Therefore, existing data collection methods cannot perform data collection under the scenario requirements of data security and data confidentiality. The embodiment of this specification provides a data collection method based on a unidirectional network environment. It should be noted that the execution subject in the embodiment of this specification can be a server or any device with data processing capabilities. Figure 1 A flow chart of a data collection method based on a one-way network environment provided in an embodiment of this specification is shown as follows: Figure 1 As shown, it mainly includes the following steps:

[0029] Step S101: determining at least one data collection task through an external network server deployed in the external network to generate collection task configuration information corresponding to each data collection task.

[0030] Among them, each data collection task corresponds to at least one data source, each data source is stored in the corresponding intranet client, and the intranet client is deployed in the intranet node; the intranet client has the authority to actively communicate with the external network, and the external network server does not have the authority to actively communicate with the intranet.

[0031] In one embodiment of the present specification, the extranet and the intranet are one-way network environments, the intranet has the authority to access the extranet, but the extranet cannot actively communicate with the intranet. That is to say, the intranet client can actively communicate with the extranet, and the extranet server cannot communicate with the intranet. The extranet server is a service device deployed in the extranet, and the intranet client is deployed on each intranet node. It should be noted that the data source for data to be collected is stored in the intranet client, or it can be called an information library to be collected. Here, the data source for data to be collected stored in the intranet client means that the intranet client has the collection authority with the data source for the data to be collected, and it can be that the intranet client has a connection relationship or a communication relationship with the data source for storing the data to be collected. The data collection task corresponding to the data collection is determined by the external server deployed in the extranet, and the data collection task here can be multiple data collection tasks.

[0032] Generate collection task configuration information corresponding to each data collection task, specifically including: deploying the external network server in the external network, and determining, through the external network server, multiple data sources to be collected and rules to be collected corresponding to the data collection task, wherein the rules to be collected are used to represent the table information to be collected or the field information to be collected in the data source of the data to be collected; determining the intranet client to which each data source to be collected belongs based on the data source to be collected and the pre-acquired intranet client information table, wherein the intranet client information table includes multiple intranet client addresses and the data source in each intranet client; establishing a correspondence between the data collection task and the intranet client address based on the intranet client address to which each data source to be collected belongs and the data collection task corresponding to the data to be collected; configuring the collection task configuration information for the data collection task, wherein the collection task configuration information includes the data source to be collected, the rules to be collected and the correspondence between the data collection task and the intranet client address.

[0033] In one embodiment of the present specification, an external network server is deployed in an external network. The external network server determines multiple data sources and rules for data collection tasks. The data sources include the data source connection Uniform Resource Locator (URL), username, password, and the client address of the intranet client where the data source is located, which can be in the form of an IP address. The rules represent the table information or field information of the data source, including mappings between SQL statements or scripts and table information, and between SQL statements or scripts and field information. These rules specify the specific data tables and fields in the data source to be executed. The data collection task and the data sources have a one-to-many relationship. A data collection task may need to collect data from multiple data sources. The data sources also have a one-to-many relationship with the rules. Each data source may collect data from different tables and fields. Therefore, a data collection task corresponds to at least one data processing subtask, and each data collection subtask includes one data source and one rule.

[0034] In one embodiment of the present specification, an intranet client information table is pre-constructed, and the intranet client information includes the client addresses of the intranet clients deployed in each intranet node and the data source of each intranet client. Based on the data source of each data to be collected and the pre-acquired intranet client information table, the intranet client to which each data source to be collected belongs is determined, and a correspondence between the data collection task and the intranet client address is established. According to the above content, collection task configuration information is configured for the data collection task, and the collection task configuration information includes the data source to be collected, the rules to be collected, and the correspondence between the data collection task and the intranet client address.

[0035] Before writing the collection task configuration information corresponding to each data collection task into a pre-built external network business library, the method also includes: building an external network business library, wherein the external network business library is located in the external network; interacting between the external network client server and multiple internal network clients through the external network business library, specifically including: obtaining client status information and collection result information of each internal network client; writing the client status information and the collection result information of the internal network client into the external network business library through the internal network client; writing the pre-acquired collection status information into the external network business library through the external network server; controlling the external network server to obtain the client status information, the collection result information and the collection status information in the external network business library; monitoring the operation status of the internal network client based on the client status information; obtaining the collection data of the internal network client based on the collection result information, and displaying the data on the external network server; and detecting the task status of the collection task of the internal network client based on the collection status information, wherein the task status includes task completion status, task suspension status, task progress status and task waiting status.

[0036] In one embodiment of the present specification, an extranet service library is constructed in the extranet, and interaction is carried out between the extranet client server and multiple intranet clients through the extranet service library. The extranet service library deployed in the extranet is shared by the extranet server and the intranet clients to realize data collection work in a one-way network, and the extranet server can also realize status monitoring of the intranet client, display of collection result data, and task detection, etc.

[0037] In one embodiment of the present specification, client status information, collection result information, and collection status information are obtained for each intranet client. The client status information and collection result information are obtained by uploading them from the intranet client, and the intranet client periodically reports the operating status and writes back the collection result data. The collection status information is obtained by the extranet client based on analysis of the collection result information and client status information. The initial status of all collection subtasks is set to the task waiting state in the extranet service library. When the intranet client pulls a subtask, the status of the subtask is set to the task progressing state. When the intranet client writes back the collection result data, the status of the subtask is set to the task completed state. When the intranet client runs abnormally, the unfinished subtask is set to the task suspended state. The client status information and the collection result information of the intranet client are written to the extranet service library by the intranet client, and the collection status information is written to the extranet service library by the extranet client.

[0038] Control the external network server and obtain the client status information, collection result information and collection status information from the external network business library. Based on the client status information, monitor the operation status of the internal network client. When the internal network client reports the operation status at regular intervals, the operation status of the internal network client can be judged according to the client heartbeat time. For clients with abnormal operation, the server changes the client status to the client stop status, changes the status of the subtask that has not been completed by the client to the task suspension status, and is responsible for cleaning up the partial collection result information that has been output by the subtask. After the client is restarted next time, the task is re-read to collect data. Based on the collection result information, the collection data of the internal network client is obtained and the data is displayed on the external network server. Based on the collection status information, the task status of the collection task of the internal network client is detected. The task status includes task completion status, task suspension status, task progress status and task waiting status.

[0039] Step S102: writing the collection task configuration information corresponding to each data collection task into a pre-built external network service library.

[0040] Among them, the extranet service library allows multiple intranet clients to access;

[0041] In one embodiment of the present specification, the collection task configuration information corresponding to each data collection task is written into the external network service library through the external network server.

[0042] Step S103: based on the designated trigger of at least one designated intranet client, pull the designated collection task configuration information of the designated data collection task corresponding to the designated intranet client from the extranet service library.

[0043] Based on the specified trigger of at least one specified intranet client, before pulling the specified collection task configuration information of the specified data collection task corresponding to the specified intranet client in the external network business library, the method also includes: obtaining at least one data source and at least one rule to be collected corresponding to each data collection task; dividing each data collection task into multiple data collection subtasks according to the at least one data source and at least one rule to be collected corresponding to each data collection task, wherein each data collection subtask includes a data source to be collected and a rule to be collected; according to the data source to be collected corresponding to each data collection subtask, determining the intranet client address of the data source to be collected, so as to determine the specified intranet client corresponding to each data collection subtask.

[0044] In one embodiment of the present specification, each data collection task corresponds to at least one data source and at least one rule to be collected. The data collection task is divided into multiple data collection subtasks based on the data source and the rules to be collected. Each data collection subtask includes a data source to be collected and a rule to be collected. Based on the data source to be collected corresponding to each data collection subtask, the address of the intranet client to which the data source belongs is determined. A designated intranet client corresponding to each subtask is determined from multiple intranet clients at multiple nodes in the intranet.

[0045] Based on the specified trigger of at least one specified intranet client, the specified collection task configuration information of the specified data collection task corresponding to the specified intranet client is pulled from the extranet business library, specifically including: pre-initializing the specified intranet client; under the trigger of the completion of the initialization setting, obtaining the current client information of the specified intranet client; based on the current client information, pulling the specified data collection task corresponding to the current client information in the extranet business library; and determining the specified collection task configuration information through the specified data collection task.

[0046] In one embodiment of the present specification, prior to data collection, each intranet client is initialized, thereby initializing the configuration of a designated intranet client. After the initialization of the designated intranet client is completed, the current client information of the designated intranet client is obtained. The client information here can be the client IP address. Based on the client IP address, the designated data collection task corresponding to the client IP address is retrieved from the external network service library, and the designated collection task configuration information corresponding to the designated data collection task is obtained.

[0047] Initialize the designated intranet client in advance, specifically including: obtaining the client information of the designated intranet client under the startup trigger of the designated intranet client, wherein the client information includes the intranet node where the client is located; writing the client information into the client information set of the external network business library, registering the designated intranet client, and initializing the designated intranet client.

[0048] In one embodiment of the present disclosure, upon startup of a designated intranet client, client information of the designated intranet client is obtained. The client information may include the intranet node where the client resides. The client information is written to a client information collection in an extranet service library, and the designated intranet client is registered to implement initialization of the designated intranet client.

[0049] Step S104 : executing the designated data collection task and generating designated collection data according to the designated collection task configuration information through the designated intranet client.

[0050] In one embodiment of the present specification, after the designated intranet client pulls the task information corresponding to its own IP address, it locally parses and executes the collection task according to the pulled designated collection task configuration information to generate designated collection data.

[0051] Step S105 , backfilling the designated collected data into the external network service library, so that the external network server can read the designated collected data in the external network service library, thereby realizing one-way data collection from the external network to the internal network.

[0052] In one embodiment of the present specification, the intranet client is designated to write the task execution result, that is, the designated collection data, back to the external network business library. After the current collection subtask is completed, the next collection subtask is pulled. It can be written back to the external network business library after each subtask is completed, or it can be written back at the same time after all subtasks in the current collection task are completed. The external network server can query the external network business library to obtain the operating status of multiple intranet clients and monitor each intranet client; it can also read the collection result data of each task from the external network business library for subsequent business processing.

[0053] Figure 2 This is a schematic diagram of an application of a data collection method based on a one-way network environment provided in an embodiment of this specification. Figure 2 As shown, Figure 2 For data quality inspection applications, the VPN zone is the extranet area, where the extranet server and extranet service library, namely the data quality inspection server and quality inspection service library, are deployed. In the intranet node zone, there are multiple intranet nodes, each of which is deployed with a node client.

[0054] The data quality inspection server manages data sources, inspection rules, inspection tasks, client monitoring, node information maintenance, and inspection result display. The quality inspection service library includes client information sets, rule information sets, task information sets, physical node information sets, data source information sets, and result information sets. The data quality inspection server maintains the corresponding information sets in the quality inspection service library by maintaining data source information, maintaining rule information, and adding data collection tasks. It also reads inspection results, client status, and logs from the quality inspection service library.

[0055] The node client registers its node using the physical node information set, reads the collection task by pulling the task information set, collects data from the to-be-collected information repository, generates inspection results, and writes the inspection results back to the result information set. Furthermore, the node client can write heartbeat information to the client information set to implement heartbeat maintenance.

[0056] Through the above technical solution, the task configuration information corresponding to the data collection task is written into the external network business library through the external network server, and the internal network client pulls the information from the external network business library to obtain the collection task configuration information, executes the data collection task locally according to the configuration information, and writes the collection result back to the external network business library. The external network server pulls the collection result in the external network business library, and by constructing the external network business library, the server and the client indirectly complete data interaction through the external network business library, realizes data collection from the internal network, and forwards the data to the external network of the one-way network, and shares the information of the external network server and the internal network client to realize data collection in the one-way network environment, realizes data collection in the one-way network environment, ensures the security and confidentiality of data collection, can realize data collection and result display, and improves the efficiency of data processing.

[0057] The embodiment of this specification also provides a data acquisition device based on a one-way network environment, such as Figure 3 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0058] Through an external network server deployed in the external network, at least one data collection task is determined to generate collection task configuration information corresponding to each data collection task, wherein each data collection task corresponds to at least one data source, and each data source is stored in a corresponding intranet client, and the intranet client is deployed in an intranet node; the collection task configuration information corresponding to each data collection task is written to a pre-built external network business library, wherein the external network business library allows multiple intranet clients to access; based on a specified trigger of at least one specified intranet client, the specified collection task configuration information of the specified data collection task corresponding to the specified intranet client is pulled from the external network business library; through the specified intranet client, the specified data collection task is executed according to the specified collection task configuration information to generate specified collection data; the specified collection data is backfilled into the external network business library, so that the external network server can read the specified collection data in the external network business library to realize one-way data collection from the external network to the intranet.

[0059] The embodiments of this specification also provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured as follows:

[0060] Through an external network server deployed in the external network, at least one data collection task is determined to generate collection task configuration information corresponding to each data collection task, wherein each data collection task corresponds to at least one data source, and each data source is stored in a corresponding intranet client, and the intranet client is deployed in an intranet node; the collection task configuration information corresponding to each data collection task is written to a pre-built external network business library, wherein the external network business library allows multiple intranet clients to access; based on a specified trigger of at least one specified intranet client, the specified collection task configuration information of the specified data collection task corresponding to the specified intranet client is pulled from the external network business library; through the specified intranet client, the specified data collection task is executed according to the specified collection task configuration information to generate specified collection data; the specified collection data is backfilled into the external network business library, so that the external network server can read the specified collection data in the external network business library to realize one-way data collection from the external network to the intranet.

[0061] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant details, refer to the descriptions of the method embodiments.

[0062] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0063] The devices and media provided in the embodiments of this specification correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0064] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0066] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0067] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0068] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0069] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0070] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0071] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0072] The foregoing description is merely one or more embodiments of this specification and is not intended to limit this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of one or more embodiments of this specification are intended to be within the scope of the claims of this specification.

Claims

1. A data collection method based on a one-way network environment, characterized in that: The method comprises: Determine at least one data collection task by an external network server deployed in the external network to generate collection task configuration information corresponding to each data collection task, wherein each data collection task corresponds to at least one data source, each data source is stored in a corresponding intranet client, and the intranet client is deployed in an intranet node; Writing the collection task configuration information corresponding to each data collection task into a pre-built external network service library, wherein the external network service library allows multiple internal network clients to access it; Based on a specified trigger of at least one specified intranet client, pulling specified collection task configuration information of a specified data collection task corresponding to the specified intranet client from the extranet service library; Execute the specified data collection task and generate the specified collection data according to the specified collection task configuration information through the specified intranet client; Backfilling the specified collected data into the external network service library so that the external network service end can read the specified collected data in the external network service library to realize one-way data collection from the external network to the internal network; Generate collection task configuration information corresponding to each data collection task, specifically including: Deploying the external network server in the external network, and determining, through the external network server, multiple data sources and collection rules for the data collection task, wherein the collection rules are used to indicate table information or field information of the data in the data source; Determining the intranet client to which each data source to be collected belongs based on the data source to be collected and a pre-acquired intranet client information table, wherein the intranet client information table includes multiple intranet client addresses and the data source in each intranet client; Based on the address of the intranet client of each data source to be collected and the data collection task corresponding to the data to be collected, a corresponding relationship between the data collection task and the intranet client address is established; Configuring the data collection task with the collection task configuration information, wherein the collection task configuration information includes the data source to be collected, the rules to be collected, and the correspondence between the data collection task and the intranet client address; Pulling, based on a designated trigger of at least one designated intranet client, designated collection task configuration information of a designated data collection task corresponding to the designated intranet client from the extranet service library specifically includes: Initialize the designated intranet client in advance; Upon triggering completion of the initialization setting, obtaining current client information of the designated intranet client; Based on the current client information, pulling the specified data collection task corresponding to the current client information from the external network service library; Determining the configuration information of the designated data collection task through the designated data collection task; The intranet client has the authority to actively communicate with the external network, and the external network server does not have the authority to actively communicate with the intranet.

2. The data collection method based on a one-way network environment according to claim 1, characterized in that: Initialize the specified intranet client in advance, specifically including: When the designated intranet client is started and triggered, client information of the designated intranet client is obtained, wherein the client information includes the intranet node where the client is located; The client information is written into the client information set of the external network service library, and the designated intranet client is registered to initialize the designated intranet client.

3. The data collection method based on a one-way network environment according to claim 1, characterized in that: Based on a designated trigger of at least one designated intranet client, before pulling designated collection task configuration information of a designated data collection task corresponding to the designated intranet client from the extranet service library, the method further includes: Obtaining at least one data source and at least one to-be-collected rule corresponding to each of the data collection tasks; According to at least one data source and at least one rule to be collected corresponding to each data collection task, each data collection task is divided into a plurality of data collection subtasks, wherein each data collection subtask includes a data source to be collected and a rule to be collected; According to the data source to be collected corresponding to each data collection subtask, the address of the intranet client to which the data source to be collected belongs is determined, so as to determine the designated intranet client corresponding to each data collection subtask.

4. The data collection method based on a one-way network environment according to claim 1, characterized in that: After writing the collection task configuration information corresponding to each data collection task into the pre-built external network service library, the method further includes: Obtaining, through the external network service end, a client information set in the external network service library, wherein the client information set includes heartbeat data uploaded by multiple internal network clients; Based on the client information set, status monitoring is performed on the plurality of the intranet clients to determine the operating status of each of the intranet clients, wherein the operating status includes normal operation and abnormal operation; When there is a first intranet client that operates abnormally, in the extranet service library, the status of the specified data collection subtask corresponding to the first intranet client is set to a task suspension state to wait for the restart of the first intranet client, wherein the specified data collection subtask is an unfinished data collection subtask of the first intranet client.

5. The data collection method based on a one-way network environment according to claim 1, characterized in that: Before writing the collection task configuration information corresponding to each data collection task into the pre-built external network service library, the method further includes: Building an external network service library, wherein the external network service library is located on the external network; The interaction between the external network server and the plurality of the internal network clients is carried out through the external network service library, specifically including: Obtaining client status information and collection result information of each of the intranet clients; Writing the client status information and the collection result information of the intranet client into the extranet service library through the intranet client; Writing the pre-acquired collection status information into the external network service database through the external network server; Controlling the external network server to obtain the client status information, the collection result information, and the collection status information from the external network service library; Monitoring the operation status of the intranet client according to the client status information; According to the collection result information, the collected data of the intranet client is obtained, and the data is displayed on the extranet server; According to the collection status information, a task status detection is performed on the collection task of the intranet client, wherein the task status includes a task completion status, a task suspension status, a task progress status and a task waiting status.

6. A data acquisition device based on a one-way network environment, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute the method according to any one of claims 1 to 5.

7. A non-volatile computer storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are configured to execute the method according to any one of claims 1 to 5.

Images