Data acquisition system and method, computer storage medium and electronic equipment

Abstract

The invention relates to the field of data processing, and provides a data acquisition system, a data acquisition method, a computer storage medium and an electronic device.The data acquisition system comprises a task scheduling unit, a data acquisition unit and a data writing-in unit, the data acquisition unit comprises a plurality of acquisition nodes, and the task scheduling unit is connected with the data writing-in unit. The task allocation module is used for allocating acquisition tasks to the plurality of acquisition nodes and adjusting the acquisition tasks according to the states of the plurality of acquisition nodes; the plurality of acquisition nodes are used for acquiring service data from equipment deployed in an isolated network so as to complete the acquisition task; and the data writing unit is used for writing the business data into a specified database and recording the writing progress of the business data in real time in the process of writing the business data into the specified database so as to perform next data writing according to the writing progress. According to the data acquisition system, data loss can be avoided, and robustness is improved.

Description

Background Art

[0002] Network isolation technology refers to the exchange of information and sharing of resources between two or more computers or networks on the basis of disconnection. In other words, network isolation technology can not only physically isolate two networks, but also exchange data in a secure network environment.

[0003] Currently, if you need to collect data generated by devices deployed in a local isolated network, you usually send the data to a designated storage device through the device's own SysLog. However, this solution is prone to data loss.

[0004] In view of this, there is an urgent need in this field to develop a new data acquisition system and method.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure. Summary of the invention

[0006] The purpose of the present disclosure is to provide a data acquisition system, a data acquisition method, a computer storage medium and an electronic device, thereby overcoming the technical problem of easily causing data loss due to the limitations of related technologies at least to a certain extent.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a data acquisition system is provided, the data acquisition system comprising a task scheduling unit, a data acquisition unit and a data writing unit, the data acquisition unit comprising a plurality of acquisition nodes, wherein: the task scheduling unit is used to assign acquisition tasks to the plurality of acquisition nodes, and adjust the acquisition tasks according to the status of the plurality of acquisition nodes; the plurality of acquisition nodes are used to collect business data from devices deployed in an isolated network to complete the acquisition tasks; the data writing unit is used to write the business data into a designated database, and, in the process of writing the business data into the designated database, record the writing progress of the business data in real time, so as to perform the next data writing according to the writing progress.

[0009] In an exemplary embodiment of the present disclosure, the task scheduling unit assigns acquisition tasks to the multiple acquisition nodes, including: determining a probability value for assigning the current acquisition task to each of the acquisition nodes based on a current state characterization value of each of the acquisition nodes, a state parameter of each of the acquisition nodes, and a comprehensive state parameter of the multiple acquisition nodes; determining a maximum probability value from the multiple probability values; and assigning the current acquisition task to the acquisition node corresponding to the maximum probability value.

[0010] In an exemplary embodiment of the present disclosure, the task scheduling unit assigns acquisition tasks to the multiple acquisition nodes, including: determining the probability value of assigning the current acquisition task to each of the acquisition nodes based on the current state characterization value of each of the acquisition nodes, the state parameters of each of the acquisition nodes and the comprehensive state parameters of the multiple acquisition nodes, the historical task execution data of each of the acquisition nodes and the comprehensive historical task execution data of the multiple acquisition nodes; determining the maximum probability value from the multiple probability values; and assigning the current acquisition task to the acquisition node corresponding to the maximum probability value.

[0011] In an exemplary embodiment of the present disclosure, the task scheduling unit assigns acquisition tasks to the multiple acquisition nodes, including: determining the probability value of assigning the current acquisition task to each of the acquisition nodes based on the current state characterization value of each of the acquisition nodes, the state parameters of each of the acquisition nodes and the comprehensive state parameters of the multiple acquisition nodes, the weight coefficients corresponding to the state parameters, the historical task execution data of each of the acquisition nodes and the comprehensive historical task execution data of the multiple acquisition nodes, and the weight coefficients corresponding to the historical task execution data; determining the maximum probability value from the multiple probability values; and assigning the current acquisition task to the acquisition node corresponding to the maximum probability value.

[0012] In an exemplary embodiment of the present disclosure, the state parameter includes at least one resource indicator value; the resource indicator value includes: the amount of free memory, the amount of free disk and the number of free slots.

[0013] In an exemplary embodiment of the present disclosure, the task scheduling unit is further configured to: receive heartbeat data packets sent by the multiple collection nodes at a preset sending frequency; and evaluate the status of the multiple collection nodes according to the number of timeout sending times of the heartbeat data packets.

[0014] In an exemplary embodiment of the present disclosure, the task scheduling unit adjusts the acquisition task according to the status of the multiple acquisition nodes, including: when any acquisition node is evaluated to be in an abnormal state, obtaining the execution progress of the target acquisition task assigned to the acquisition node; scheduling the target acquisition task to other target acquisition nodes in a normal state, so that the target acquisition node continues to execute the target acquisition task based on the execution progress.

[0015] In an exemplary embodiment of the present disclosure, the data acquisition system also includes a data processing unit, which is used to filter and process the business data according to preset business conditions; wherein the preset business conditions include any one or more combinations of the following forms: a preset structured query language, a preset regular expression, and business conditions configured by the user.

[0016] In an exemplary embodiment of the present disclosure, the data processing unit is further used to: obtain metadata change information corresponding to the business data; and synchronize the metadata change information to the business data.

[0017] In an exemplary embodiment of the present disclosure, the data writing unit is also used to: adjust the amount of data written to the designated database at a single time according to the writing performance of the designated database; the writing performance includes the data writing rate of the designated database; and write the business data or the business data after filtering processing into the designated database in batches according to the data amount.

[0018] According to a second aspect of the present disclosure, a data collection method is provided, comprising: a task scheduling unit assigns collection tasks to multiple collection nodes, and adjusts the collection tasks according to the status of the multiple collection nodes; the multiple collection nodes collect business data from devices deployed in an isolated network to complete the collection tasks; a data writing unit writes the business data to a designated database, and, in the process of writing the business data to the designated database, records the writing progress of the business data in real time, so as to perform the next data writing according to the writing progress.

[0019] According to a third aspect of the present disclosure, there is provided a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the data collection method described in the second aspect is implemented.

[0020] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the data collection method described in the second aspect above by executing the executable instructions.

[0021] It can be seen from the above technical solutions that the data acquisition system, data acquisition method, computer storage medium and electronic device in the exemplary embodiments of the present disclosure have at least the following advantages and positive effects:

[0022] In the technical solutions provided by some embodiments of the present disclosure, on the one hand, the task scheduling unit allocates collection tasks to multiple collection nodes, and adjusts the collection tasks according to the status of multiple collection nodes, and can timely switch the collection node that executes the collection task when the node status is abnormal, so as to avoid the collection task being shelved, thereby causing the problem of data loss. On the other hand, the data writing unit writes the business data collected by multiple collection nodes into the designated database, and in the process of writing the business data into the designated database, records the writing progress of the business data in real time, so as to write the next data according to the writing progress, which can avoid data loss caused by weak network environment or system failure, thereby ensuring the reliability of data synchronization.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0025] Figure 1 A schematic diagram showing a process of sending business data through the SysLog service in the related art;

[0026] Figure 2 A schematic diagram showing a process of collecting business data by developing an agent in the related art;

[0027] Figure 3 A schematic diagram showing the structure of a data acquisition system in an exemplary embodiment of the present disclosure is shown;

[0028] Figure 4 A schematic diagram showing a process of allocating collection tasks to multiple collection nodes by a task scheduling unit in an embodiment of the present disclosure is shown;

[0029] Figure 5 A schematic diagram showing a process of allocating collection tasks to multiple collection nodes by another task scheduling unit in an embodiment of the present disclosure;

[0030] Figure 6 A schematic diagram showing a process flow of allocating collection tasks to multiple collection nodes by a task scheduling unit in another embodiment of the present disclosure is shown;

[0031] Figure 7 The overall architecture diagram of the data acquisition system in the embodiment of the present disclosure is shown;

[0032] Figure 8 A schematic diagram showing a flow chart of a data collection method in an embodiment of the present disclosure;

[0033] Fig. 9 A schematic structural diagram of an electronic device in an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0035] The terms "a", "an", "the" and "said" are used in this specification to indicate the presence of one or more elements / components / etc.; the terms "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first" and "second" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0036] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings represent the same or similar parts, and their repeated descriptions will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0037] Currently, for devices deployed on a local isolated network, if you need to collect data generated by the device for use by the business system, there are two mainstream implementation methods: one is to send data to a designated storage end through the SysLog service, and the other is to develop an agent to send the data collected by the agent deployed on the device end to the designated storage end.

[0038] Among them, SysLog is often called system log or system record, which is a standard for transmitting record messages on the Internet Protocol (TCP / IP) network.

[0039] refer to Figure 1 , Figure 1 A schematic diagram of the process of sending business data through the SysLog service in the related art is shown, such as Figure 1 As shown, the data collection process mainly includes the following steps:

[0040] First, each device needs to enable the SysLog outbound service and define the outbound log format. Common log formats can be JSON or vertical bar separator (|).

[0041] Second, after the log is sent to the message middleware, the user needs to parse and clean the original SysLog, and persist it to the relevant storage device;

[0042] Third, the business system can develop business based on the processed data in the storage device.

[0043] However, the above solution has the following drawbacks:

[0044] The local campus network and equipment are complex, and network disconnection and physical machine failure are very common. Therefore, this solution is very prone to data loss, and once lost, it is difficult to troubleshoot. In addition, the data sent by SysLog will contain a large amount of redundant protocol data that is not related to the business, causing a lot of bandwidth waste.

[0045] refer to Figure 2 , Figure 2 A schematic diagram showing a process of collecting business data by developing an agent in the related art mainly includes the following steps:

[0046] First, deploy an agent on the device side, which can access the data generated by the device;

[0047] Second, send designated business data to the server regularly. The server has an independent data receiving service. After receiving the message, the data receiving service puts it into middleware such as MQ for consumption by the business system.

[0048] However, the above solution has the following drawbacks:

[0049] Although this method can reduce the transmission bandwidth by extracting the specified business data and solve the impact of some network conditions by retrying after failure, the client proxy method is still unable to guarantee reliable data transmission due to operating system anomalies, device failures, and the fault light of the proxy itself. In addition, when business conditions change, the client often needs to be upgraded, which is costly and difficult to expand.

[0050] In the embodiments of the present disclosure, firstly, a data acquisition system is provided, which at least to some extent overcomes the defect of the related art that data loss is easily caused.

[0051] Figure 3 A schematic diagram showing the structure of a data acquisition system in an exemplary embodiment of the present disclosure is shown; Figure 3As shown, the data acquisition system 300 may include a task scheduling unit 310, a data acquisition unit 320 and a data writing unit 330. Among them:

[0052] The task scheduling unit 310 is used to allocate collection tasks to multiple collection nodes and adjust the collection tasks according to the status of the multiple collection nodes;

[0053] The multiple collection nodes included in the data collection unit 320 are used to collect business data from devices deployed in the isolated network to complete the collection task;

[0054] The data writing unit 330 is used to write the business data into the designated database, and, in the process of writing the business data into the designated database, record the writing progress of the business data in real time so as to write the data next time according to the writing progress.

[0055] exist Figure 3 In the technical solution provided by the illustrated embodiment, on the one hand, the task scheduling unit allocates collection tasks to multiple collection nodes, and adjusts the collection tasks according to the status of multiple collection nodes, and can timely switch the collection node that executes the collection task when the node status is abnormal, so as to avoid the collection task being shelved, thereby causing the problem of data loss. On the other hand, the data writing unit writes the business data collected by multiple collection nodes into the designated database, and in the process of writing the business data into the designated database, the writing progress of the business data is recorded in real time, so as to write the next data according to the writing progress, which can avoid data loss caused by a weak network environment or system failure, thereby ensuring the reliability of data synchronization.

[0056] The following Figure 3 The specific steps performed by each unit in the following are described in detail:

[0057] The task scheduling unit 310 is used to allocate collection tasks to multiple collection nodes and adjust the collection tasks according to the status of the multiple collection nodes.

[0058] In an exemplary embodiment of the present disclosure, the task scheduling unit may monitor the status of the above-mentioned multiple collection nodes.

[0059] Exemplarily, each collection node can send its own heartbeat data packet to the above-mentioned task scheduling unit according to a preset sending frequency (for example: 5 seconds each time, which can be set according to actual conditions, and the present disclosure does not make special limitations on this). The task scheduling unit can count the number of times each collection node times out of sending heartbeat data packets. Then, when the number of timed-out sending times is greater than or equal to a preset threshold value (for example: 3 times, which can be set according to actual conditions), it can be determined that the collection node is in an abnormal state. Otherwise, it can be determined that the above-mentioned collection node is in a normal state.

[0060] In an exemplary embodiment of the present disclosure, the task scheduling unit can also assign collection tasks to the above-mentioned multiple collection nodes, monitor the status of each collection task (for example: not started, started but not assigned, running, running completed, running failed, etc.), and receive the execution progress of the collection tasks reported by each collection node to perform life cycle management of the collection tasks.

[0061] Exemplary, reference Figure 4 , Figure 4 A schematic diagram of a process of allocating collection tasks to multiple collection nodes by a task scheduling unit in an embodiment of the present disclosure is shown, including steps S401 to S403:

[0062] In step S401, the probability value of allocating the current acquisition task to each acquisition node is determined according to the current state representation value of each acquisition node, the state parameter of each acquisition node and the comprehensive state parameters of multiple acquisition nodes.

[0063] In this step, the current state characterization value may be 0 or 1, 0 indicating that the acquisition node is currently in an abnormal state (eg, node failure, or insufficient remaining resources of the node), and 1 indicating that the acquisition node is currently in a normal state.

[0064] The above-mentioned state parameter may include at least one resource indicator value, wherein the resource indicator value may be: the amount of free memory, the amount of free disk and the number of free slots.

[0065] Among them, memory refers to random access memory RAM, and free memory refers to the amount of unoccupied memory.

[0066] Disk refers to a storage device that uses magnetic recording technology to store data. Free disk space refers to the amount of disk space that is not occupied.

[0067] A slot is the basic unit of data management and migration within a cluster. The number of free slots refers to the number of unoccupied slots.

[0068] The following is an example in which the at least one resource indicator value includes three resource indicator values, namely, the amount of free memory, the amount of free disk, and the number of free slots. The comprehensive status parameter of multiple acquisition nodes can be the sum of the free memory amounts of multiple acquisition nodes, the sum of the free disk amounts of multiple acquisition nodes, and the sum of the free slots of multiple acquisition nodes.

[0069] Furthermore, the proportion of the free memory of the collection node i to the total free memory of the n collection nodes, the proportion of the free disk of the collection node i to the total free disk of the n collection nodes, and the proportion of the number of free slots of the collection node i to the total free slots of the n collection nodes can be calculated, and then the cumulative value of the above three proportions is calculated. Furthermore, the probability value of allocating the current collection task j to any collection node i can be determined based on the product between the current state representation value of the collection node i and the above cumulative value.

[0070] Specifically, if the current state characterization value of the collection node i is 0, the above product is 0, that is, the probability value of assigning the current collection task j to the collection node i is 0, and if the current state characterization value of the above collection node i is 1, the above product is the above accumulated value, that is, the accumulated value can represent the probability value of assigning the current collection task j to the collection node i.

[0071] Similarly, the probability value of assigning each collection task to each collection node can be calculated to obtain multiple probability values.

[0072] In step S402, a maximum probability value is determined from multiple probability values.

[0073] In this step, the maximum probability value may be determined from the multiple probability values ​​obtained in the above step S401.

[0074] In step S403, the current collection task is assigned to the collection node corresponding to the maximum probability value.

[0075] In this step, the current collection task j may be assigned to the collection node corresponding to the above maximum probability value.

[0076] refer to Figure 5 , Figure 5 A schematic diagram of a process of allocating collection tasks to multiple collection nodes by another task scheduling unit in an embodiment of the present disclosure is shown, including steps S501 to S503:

[0077] In step S501, the probability value of allocating the current acquisition task to each acquisition node is determined based on the current state characterization value of each acquisition node, the state parameters of each acquisition node and the comprehensive state parameters of multiple acquisition nodes, the historical task execution data of each acquisition node and the comprehensive historical task execution data of multiple acquisition nodes.

[0078] In this step, the historical task execution data of each collection node may be: the number of times the historical collection tasks of the same type as the current collection task failed to execute on each collection node within the historical time period (for example, the past month, which can be set according to actual conditions). Correspondingly, the comprehensive historical task execution data of the above multiple collection nodes may be: the sum of the number of times the historical collection tasks of the same type as the current collection task failed to execute on multiple collection nodes within the historical time period.

[0079] Exemplarily, the at least one resource indicator value mentioned above still includes three resource indicator values, namely, the amount of free memory, the amount of free disk and the number of free slots. Then, the comprehensive status parameter of multiple acquisition nodes can be the sum of the free memory amounts of multiple acquisition nodes, the sum of the free disk amounts of multiple acquisition nodes and the sum of the number of free slots of multiple acquisition nodes.

[0080] Furthermore, referring to the relevant explanation of the above step S401, we can calculate the ratio of the free memory amount of collection node i to the sum of the free memory amounts of n collection nodes, the ratio of the free disk amount of collection node i to the sum of the free disk amounts of n collection nodes, the ratio of the number of free slots of collection node i to the sum of the number of free slots of n collection nodes, the ratio of the number of failed executions of historical collection tasks of the same type as collection task j on collection node i to the sum of the number of failed executions of historical collection tasks of the same type as collection task j on n collection nodes, and then calculate the cumulative value of the above four ratios.

[0081] Furthermore, the probability value of allocating the current collection task j to any collection node i may be determined according to the product of the current state representation value of the collection node i and the above-mentioned accumulated value.

[0082] Specifically, if the current state characterization value of the collection node i is 0, the above product is 0, that is, the probability value of assigning the current collection task j to the collection node i is 0, and if the current state characterization value of the above collection node i is 1, the above product is the above accumulated value, that is, the accumulated value can represent the probability value of assigning the current collection task j to the collection node i.

[0083] Similarly, the probability value of assigning each collection task to each collection node can be calculated to obtain multiple probability values.

[0084] In step S502, a maximum probability value is determined from multiple probability values.

[0085] In this step, the maximum probability value may be determined from the multiple probability values ​​obtained in the above step S501.

[0086] In step S503, the current collection task is assigned to the collection node corresponding to the maximum probability value.

[0087] In this step, the current collection task j may be assigned to the collection node corresponding to the above maximum probability value.

[0088] refer to Figure 6 , Figure 6 A schematic diagram of a process of allocating collection tasks to multiple collection nodes by a task scheduling unit in another embodiment of the present disclosure is shown, including steps S601 to S603:

[0089] In step S601, the probability value of allocating the current collection task to each collection node is determined based on the current state characterization value of each collection node, the state parameters of each collection node and the comprehensive state parameters of multiple collection nodes, the weight coefficients corresponding to the state parameters, the historical task execution data of each collection node and the comprehensive historical task execution data of multiple collection nodes, and the weight coefficients corresponding to the historical task execution data.

[0090] In this step, weight coefficients may be pre-assigned to the above-mentioned state parameters and the above-mentioned historical task execution data. Exemplarily, the above-mentioned at least one resource indicator value includes three resource indicator values, namely, free memory, free disk and free slot number, for example. Then, the comprehensive state parameter of multiple acquisition nodes may be the sum of free memory of multiple acquisition nodes, the sum of free disk of multiple acquisition nodes and the sum of free slot number of multiple acquisition nodes. The weight coefficient assigned to the free memory may be A, the weight coefficient assigned to the free disk may be B, the weight coefficient assigned to the free slot number may be C, and the weight coefficient assigned to the historical task execution data may be D. A, B, C and D may all be values ​​greater than 0 and less than 1, A+B+C+D=1, and the specific values ​​of A, B, C and D may be set according to business needs.

[0091] Furthermore, the probability value of assigning the current collection task j to any collection node i can be calculated based on the following formula 1:

[0092]

[0093] Where CM(j|i) represents the probability value of assigning the above acquisition task j to acquisition node i, P represents the current state representation value of acquisition node i, S i Indicates the amount of free memory of acquisition node i, M i Indicates the free disk space of the collection node i, D i Indicates the number of free slots of collection node i, F i Indicates the number of times that historical collection tasks of the same type as collection task j failed to execute on collection node i.

[0094] k represents the identification number of the above multiple collection nodes, and the value of k ranges from 1 to n. Represents the sum of the free memory of the above n acquisition nodes, It represents the sum of free disk space of the n acquisition nodes mentioned above. Represents the sum of the number of free slots of the above n collection nodes, It represents the sum of the number of failed executions of historical collection tasks of the same type as collection task j on n collection nodes.

[0095] A represents the weight coefficient corresponding to the amount of free memory, B represents the weight coefficient corresponding to the amount of free disk, C represents the weight coefficient corresponding to the number of free slots, and D represents the weight coefficient corresponding to the historical task execution data.

[0096] Similarly, the probability value of allocating each collection task to each collection node can be calculated based on the above formula 1 to obtain multiple probability values.

[0097] In step S602, a maximum probability value is determined from multiple probability values.

[0098] In this step, the maximum probability value may be determined from the multiple probability values ​​obtained in the above step S601.

[0099] In step S603, the current collection task is assigned to the collection node corresponding to the maximum probability value.

[0100] In this step, the current collection task j may be assigned to the collection node corresponding to the above maximum probability value.

[0101] Obviously, the above-mentioned task scheduling rules set in the present disclosure not only take into account the current state characterization value of each collection node (i.e., whether the node is faulty), but also take into account the state parameters of the collection node, and the proportion of the idle resource index value of the collection node to the comprehensive idle resource index value of multiple collection nodes. Thus, it can solve the problem of data collection failure caused by insufficient resources of the node itself, and ensure that the collection tasks can be reasonably allocated in the complex and changeable network environment of the local park.

[0102] After assigning collection tasks to the above-mentioned multiple collection nodes, the task scheduling unit 310 can also monitor the status of the above-mentioned multiple collection nodes. Further, when any collection node is in an abnormal state, the task scheduling unit can obtain the execution progress of the target collection task assigned to the collection node, and schedule the target collection task to other target collection nodes in a normal state, so that the target collection node continues to execute the above-mentioned target collection task based on the above-mentioned execution progress.

[0103] Therefore, the present invention can switch the collection node that executes the target collection task in time when an abnormality occurs in the node, so as to avoid the target collection task being shelved, thereby causing the problem of data loss. In addition, the target collection node can also execute the target collection task according to the previous collection progress, which can avoid the problem of repeated data collection and improve data collection efficiency.

[0104] Next reference Figure 3 The data collection unit 320 includes multiple collection nodes, which are used to collect business data from devices deployed in the isolated network to complete the collection task.

[0105] In the exemplary embodiments of the present disclosure, it should be noted that the data acquisition unit can be independently deployed on the device end in the isolated network, or can be mixed with other units of the above-mentioned data acquisition system, and can be set according to actual conditions. The present disclosure does not specifically limit this. The multiple acquisition nodes included in the data acquisition unit are independent of each other and do not affect each other.

[0106] The function of the collection node is to: regularly send heartbeat data packets to the above-mentioned task scheduling unit to report its own status; and after receiving the collection task assigned by the task scheduling unit, collect business data (which may be log data) from the devices deployed in the isolated network according to the above-mentioned collection task to complete the above-mentioned collection task.

[0107] After collecting the business data from the device, the collection node may first cache the business data locally, and then send the business data to the data processing unit in sequence according to the collection order.

[0108] In the process of sending business data to the data processing unit, the collection node can record the progress of business data sending in real time. Only when all the collected business data are completely sent to the data processing unit will the business data be cleared. In extreme cases, even if the business data is cleared locally, the collection node can still pull the business data from the device to the local again for the data processing unit to process it.

[0109] In the data collection solutions in the related technologies, the business parties are generally required to clean the data by themselves. Moreover, since different business parties require different data, each business party often needs to maintain the relevant data processing system by itself, which is very costly.

[0110] In the present disclosure, after receiving the above business data, the data processing unit can filter the business data according to preset business conditions to achieve automatic cleaning of the business data.

[0111] The preset business conditions include any one or more of the following combinations: preset structured query language SQL, preset regular expressions, and business conditions configured by users.

[0112] It can be seen that the data processing unit provided in the present disclosure supports SQL and regular expression methods to filter business data, provides a configuration as a service method, can realize the cleaning of business data to reduce unnecessary redundant data, and solves the technical problem that the data collected in related technologies will contain a large amount of business-irrelevant data, and users need to develop programs according to their own business needs to extract data, thereby improving users' data acquisition efficiency.

[0113] After sending the business data to the data processing unit, the collection node can also monitor the metadata change information of the device in real time and synchronize the metadata change information to the data processing node. Therefore, after the data processing unit receives the above business data and before completing the data processing, the metadata change information corresponding to the business data is obtained from the above collection node, and the above metadata change information can be synchronized to the business data.

[0114] Among them, metadata, also known as intermediary data or relay data, is data about data. It is mainly information describing data properties, used to support functions such as indicating storage location, historical data, resource search, and file records. Metadata is a kind of electronic catalog. In order to achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby achieving the purpose of assisting data retrieval. Metadata is information about the organization of data, data domains, and their relationships. In short, metadata is data about data.

[0115] For example, after the acquisition node has collected business data, the user performs a table structure change in the source data in the device, but the corresponding operation is not performed in the business data. At this time, the business data and the source data are inconsistent, resulting in an error. In order to improve the robustness of the system, the data processing unit in the present disclosure obtains the metadata change information monitored by the acquisition node and synchronizes it to the business data.

[0116] Continue to refer Figure 3 The data writing unit 330 is used to write the business data into the designated database, and in the process of writing the business data into the designated database, record the writing progress of the business data in real time so as to write the data next time according to the writing progress.

[0117] In an exemplary embodiment of the present disclosure, the data writing unit 330 can write business data into a designated database. The designated database can be Mysql (a relational database management system), ES (ElasticSearch, a search server based on Lucene), CK (Clickhouse, a column-based storage database), etc. It can be set according to actual conditions, and the present disclosure does not make any special limitations on this.

[0118] It should be noted that in the process of writing business data to the designated database, the data writing unit 330 can record the writing progress of the business data in real time. Therefore, when data writing fails in a weak network environment, or when a system failure causes the data writing operation to abnormally exit, the data writing status is recorded in time. Therefore, after the network condition improves or the system failure is repaired, the next data writing can be performed according to the above writing progress to avoid data loss caused by a weak network environment or system failure, thereby ensuring the reliability of data synchronization.

[0119] In an exemplary embodiment of the present disclosure, the data writing unit 330 can also count in real time the data writing rate of the specified database and the backlog of the processing queue used to temporarily store the written data, and determine the writing performance of the specified database based on the above writing rate and backlog. Furthermore, the amount of data written to the specified database at a single time (i.e., the Batch size) can be adjusted according to the writing performance, so that the Batch size is adapted to the current writing performance of the specified database, thereby avoiding the specified database from being overly stressed and causing a crash or downtime.

[0120] For example, when the data writing rate of the designated database is lower than the preset rate, or the amount of accumulated data is greater than the preset amount of data, the amount of data written to the execution database at a time can be reduced, that is, the Batch value can be reduced. Furthermore, the business data or the filtered business data can be written to the designated database in batches according to the adjusted data amount.

[0121] refer to Figure 7 , Figure 7 The overall architecture diagram of the data acquisition system in the embodiment of the present disclosure is shown as follows: Figure 7 As shown:

[0122] The data collection unit (including multiple collection nodes, such as collection node 1 and collection node 2 shown in the figure) and the device are deployed in the same network, and the data collection unit maintains communication with the remote task scheduling unit;

[0123] The acquisition node in the data acquisition unit is responsible for collecting business data from the device, and the data processing unit is responsible for filtering and processing the business data according to the specified business conditions, supporting user-defined business conditions;

[0124] The task scheduling unit assigns collection tasks to the collection nodes of the data collection unit, manages the life cycle of the collection tasks, monitors the status of the collection nodes and the execution progress of the collection tasks, and schedules tasks according to the node status;

[0125] The data writing unit monitors the performance of the specified database, dynamically adjusts the amount of data written to the specified database at a time according to its performance, and records the writing progress in real time during the batch writing process to ensure reliable data delivery;

[0126] The business system can query the required business data from the specified database (i.e. Mysql, ES, CK).

[0127] Based on the above technical solutions, the present disclosure has at least the following technical effects:

[0128] Better robustness: The data acquisition unit will start up when the data acquisition system starts up. Once the acquisition node crashes or goes down, the task scheduling unit can sense the disconnection and schedule the acquisition task through a certain scheduling strategy.

[0129] Robustness is the transliteration of Robust, which means strong and strong. It is also the ability of the system to survive abnormal and dangerous situations. For example, whether computer software can survive input errors, disk failures, network overloads or intentional attacks without freezing or crashing is the robustness of the software. The so-called "robustness" also refers to the characteristics of the control system maintaining certain other performances under certain (structure, size) parameter perturbations.

[0130] Ease of use: The entire data acquisition unit can be deployed independently on the device side of the same network as the device, or it can be mixed with other modules. The acquisition node itself is stateless, and the acquisition nodes are independent of each other. The acquisition tasks can be switched between different acquisition nodes.

[0131] Scalability: The collection nodes are decoupled from the data processing unit and data filtering rules in the system. These collection nodes can be dynamically scheduled through the task scheduling unit, and the data filtering rules support users to configure them according to their own needs.

[0132] The present disclosure can at least be applied to data collection scenarios of multiple systems such as logistics, including synchronization of small and large production to report data in WMS (Warehouse Management System), data collection of national report database, synchronization of order data on the e-commerce platform side by merchant workbench, etc. Relying on the data collection solution in the present disclosure, it can ensure extremely strong robustness in the local complex and changeable network environment (machine downtime, 4G weak network frequent packet loss, etc.), and is suitable for major promotion scenarios such as 618 and Double 11.

[0133] Figure 8 A flow chart of a data collection method in an embodiment of the present disclosure is shown, and the execution subject of the data collection method may be the above-mentioned data collection system.

[0134] refer to Figure 8 According to an embodiment of the present invention, a data collection method comprises the following steps:

[0135] Step S810, the task scheduling unit allocates collection tasks to multiple collection nodes, and adjusts the collection tasks according to the states of the multiple collection nodes;

[0136] Step S820, the multiple collection nodes collect business data from the devices deployed in the isolated network to complete the collection task;

[0137] Step S830: the data writing unit writes the business data into the designated database, and, in the process of writing the business data into the designated database, records the writing progress of the business data in real time, so as to write the data next time according to the writing progress.

[0138] In an exemplary embodiment of the present disclosure, a task scheduling unit assigns acquisition tasks to the multiple acquisition nodes, including: determining a probability value for assigning the current acquisition task to each of the acquisition nodes based on a current state characterization value of each of the acquisition nodes, a state parameter of each of the acquisition nodes, and a comprehensive state parameter of the multiple acquisition nodes; determining a maximum probability value from the multiple probability values; and assigning the current acquisition task to the acquisition node corresponding to the maximum probability value.

[0139] In an exemplary embodiment of the present disclosure, a task scheduling unit assigns acquisition tasks to the multiple acquisition nodes, including: determining a probability value for assigning a current acquisition task to each of the acquisition nodes based on a current state characterization value of each of the acquisition nodes, a state parameter of each of the acquisition nodes and a comprehensive state parameter of the multiple acquisition nodes, historical task execution data of each of the acquisition nodes and the comprehensive historical task execution data of the multiple acquisition nodes; determining a maximum probability value from the multiple probability values; and assigning the current acquisition task to the acquisition node corresponding to the maximum probability value.

[0140] In an exemplary embodiment of the present disclosure, a task scheduling unit assigns acquisition tasks to the multiple acquisition nodes, including: determining a probability value for assigning a current acquisition task to each of the acquisition nodes based on a current state characterization value of each of the acquisition nodes, a state parameter of each of the acquisition nodes and a comprehensive state parameter of the multiple acquisition nodes, a weight coefficient corresponding to the state parameter, historical task execution data of each of the acquisition nodes and the comprehensive historical task execution data of the multiple acquisition nodes, and a weight coefficient corresponding to the historical task execution data; determining a maximum probability value from the multiple probability values; and assigning the current acquisition task to the acquisition node corresponding to the maximum probability value.

[0141] In an exemplary embodiment of the present disclosure, the state parameter includes at least one resource indicator value; the resource indicator value includes: the amount of free memory, the amount of free disk and the number of free slots.

[0142] In an exemplary embodiment of the present disclosure, the task scheduling unit receives heartbeat data packets sent by the multiple collection nodes at a preset sending frequency; and evaluates the status of the multiple collection nodes according to the number of timeout sending times of the heartbeat data packets.

[0143] In an exemplary embodiment of the present disclosure, the task scheduling unit adjusts the acquisition task according to the status of the multiple acquisition nodes, including: when any acquisition node is evaluated to be in an abnormal state, obtaining the execution progress of the target acquisition task assigned to the acquisition node; scheduling the target acquisition task to other target acquisition nodes in a normal state, so that the target acquisition node continues to execute the target acquisition task based on the execution progress.

[0144] In an exemplary embodiment of the present disclosure, the data processing unit filters the business data according to preset business conditions; wherein the preset business conditions include any one or more combinations of the following forms: a preset structured query language, a preset regular expression, and business conditions configured by the user.

[0145] In an exemplary embodiment of the present disclosure, a data processing unit obtains metadata change information corresponding to the business data; and synchronizes the metadata change information to the business data.

[0146] In an exemplary embodiment of the present disclosure, a data writing unit adjusts the amount of data written to the designated database at a single time according to the writing performance of the designated database; the writing performance includes the data writing rate of the designated database; and according to the data amount, the business data or the business data after filtering is written in batches to the designated database.

[0147] The specific details of each step in the above data acquisition method have been described in detail in the corresponding data acquisition system, so they will not be repeated here.

[0148] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0149] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0150] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0151] The present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device.

[0152] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0153] Computer-readable storage media can send, propagate or transmit programs for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0154] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiments.

[0155] In addition, an electronic device capable of implementing the above method is also provided in an embodiment of the present disclosure.

[0156] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0157] Refer to the following Fig. 9 The electronic device 900 according to this embodiment of the present disclosure is described. Fig. 9 The electronic device 900 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0158] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.

[0159] The storage unit stores a program code, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 910 can perform the following steps: Figure 8As shown: Step S810, the task scheduling unit assigns collection tasks to multiple collection nodes, and adjusts the collection tasks according to the status of the multiple collection nodes; Step S820, the multiple collection nodes collect business data from devices deployed in the isolated network to complete the collection tasks; Step S830, the data writing unit writes the business data to the specified database, and, in the process of writing the business data to the specified database, records the writing progress of the business data in real time, so as to perform the next data writing according to the writing progress.

[0160] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0161] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0162] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0163] The electronic device 900 may also communicate with one or more external devices 1000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0164] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A data acquisition system, characterized in that: The data acquisition system includes a task scheduling unit, a data acquisition unit and a data writing unit. The data acquisition unit includes a plurality of acquisition nodes, wherein: The task scheduling unit is used to allocate collection tasks to the multiple collection nodes and adjust the collection tasks according to the states of the multiple collection nodes; The multiple collection nodes are used to collect business data from devices deployed in the isolated network to complete the collection task; The data writing unit is used to write the business data into a designated database, and, during the process of writing the business data into the designated database, record the writing progress of the business data in real time so as to write the data next time according to the writing progress.

2. The system according to claim 1, characterized in that The task scheduling unit allocates collection tasks to the multiple collection nodes, including: Determine the probability value of allocating the current acquisition task to each of the acquisition nodes according to the current state representation value of each of the acquisition nodes, the state parameter of each of the acquisition nodes and the comprehensive state parameters of the multiple acquisition nodes; Determining a maximum probability value from a plurality of said probability values; The current acquisition task is assigned to the acquisition node corresponding to the maximum probability value.

3. The system according to claim 1, characterized in that The task scheduling unit allocates collection tasks to the multiple collection nodes, including: Determine the probability value of allocating the current acquisition task to each of the acquisition nodes according to the current state representation value of each of the acquisition nodes, the state parameter of each of the acquisition nodes and the comprehensive state parameters of the multiple acquisition nodes, the historical task execution data of each of the acquisition nodes and the comprehensive historical task execution data of the multiple acquisition nodes; Determining a maximum probability value from a plurality of said probability values; The current acquisition task is assigned to the acquisition node corresponding to the maximum probability value.

4. The system according to claim 1, characterized in that The task scheduling unit allocates collection tasks to the multiple collection nodes, including: Determine the probability value of allocating the current acquisition task to each of the acquisition nodes according to the current state representation value of each of the acquisition nodes, the state parameter of each of the acquisition nodes and the comprehensive state parameter of the multiple acquisition nodes, the weight coefficient corresponding to the state parameter, the historical task execution data of each of the acquisition nodes and the comprehensive historical task execution data of the multiple acquisition nodes, and the weight coefficient corresponding to the historical task execution data; Determining a maximum probability value from a plurality of said probability values; The current acquisition task is assigned to the acquisition node corresponding to the maximum probability value.

5. The system according to any one of claims 2 to 4, characterized in that: The state parameter includes at least one resource indicator value; The resource indicator values ​​include: the amount of free memory, the amount of free disk, and the number of free slots.

6. The system according to claim 1, characterized in that The task scheduling unit is also used for: Receiving heartbeat data packets sent by the multiple acquisition nodes according to a preset sending frequency; The states of the multiple collection nodes are evaluated according to the number of timeout sending times of the heartbeat data packets.

7. The system according to claim 6, characterized in that The task scheduling unit adjusts the collection task according to the states of the multiple collection nodes, including: When any collection node is evaluated to be in an abnormal state, the execution progress of the target collection task assigned to the collection node is obtained; The target acquisition task is scheduled to other target acquisition nodes in a normal state, so that the target acquisition nodes continue to execute the target acquisition task based on the execution progress.

8. The system according to claim 1, characterized in that The data acquisition system further comprises a data processing unit, which is used for: Filtering the business data according to preset business conditions; The preset business conditions include any one or more of the following combinations: Preset structured query language, preset regular expressions, and user-configured business conditions.

9. The system according to claim 8, characterized in that The data processing unit is also used for: Obtaining metadata change information corresponding to the business data; The metadata change information is synchronized with the business data.

10. The system according to claim 1, characterized in that The data writing unit is also used for: adjusting the amount of data written to the designated database at a single time according to the writing performance of the designated database; the writing performance includes the data writing rate of the designated database; The business data or the business data after filtering is written into the designated database in batches according to the data volume.

11. A data collection method, characterized in that: include: The task scheduling unit allocates collection tasks to multiple collection nodes, and adjusts the collection tasks according to the states of the multiple collection nodes; The multiple collection nodes collect business data from devices deployed in the isolated network to complete the collection task; The data writing unit writes the business data into a designated database, and in the process of writing the business data into the designated database, records the writing progress of the business data in real time, so as to write the data next time according to the writing progress.

12. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data collection method according to claim 11 is implemented.

13. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to perform the data collection method of claim 11 by executing the executable instructions.

Images