Data collection method and device based on heterogeneous data source offline synchronization tool

Abstract

The application provides a data collection method and device based on an offline synchronization tool of a heterogeneous data source. On one hand, configuration information recorded in a configuration database is used as the default configuration of the synchronization tool, thereby effectively reducing the resources consumed for configuring the synchronization tool. On the other hand, the synchronization tool is configured according to the received data collection instruction, so that the synchronization tool can be adapted to the data collection instruction, and the data collection effect can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data collection, in particular, the present application relates to a data collection method and device based on offline synchronization tool of heterogeneous data source. BACKGROUND

[0002] With the development of information society, the rapid development of big data analysis and other technologies, the integration of a large number of information data of various types, the realization of information sharing and internal information integration has become more and more important. The informatization development of current enterprises mostly has the problem of difficult synchronization of heterogeneous data sources.

[0003] Generally speaking, the process of data collection needs to be executed by a synchronization tool. Before data collection is performed by using the synchronization tool, the synchronization tool needs to be configured according to the user's demand, the situation of the data source and other information. However, the configuration work for the synchronization tool is often highly professional and complex, which affects the data collection efficiency based on the synchronization tool. SUMMARY

[0004] The present application provides a data collection method and device based on offline synchronization tool of heterogeneous data source, which effectively reduces the configuration difficulty of the synchronization tool, effectively improves the data collection effect, and the technical scheme adopted by the present application is as follows:

[0005] In a first aspect, a data collection method based on offline synchronization tool of heterogeneous data source is provided. The method is based on a data collection system, which includes a background server, a configuration database and a plurality of synchronization tools. The configuration database is used to record the configuration information of the synchronization tools. The method is executed by the background server and includes the following steps:

[0006] For each synchronization tool, when the state of the synchronization tool is idle, the configuration information corresponding to the synchronization tool in the configuration database is determined as the default information of the synchronization tool.

[0007] According to the default information, the synchronization tool is configured.

[0008] When a data collection instruction is received, the synchronization tool corresponding to the data collection instruction is determined as the target synchronization tool.

[0009] According to the data collection instruction, the configuration information of the target synchronization tool is determined as the target information of the target synchronization tool.

[0010] According to the target information, the target synchronization tool is configured, and according to the target information, the configuration information corresponding to the synchronization tool in the configuration database is updated.

[0011] The configured synchronization tools are used to collect data.

[0012] In an optional embodiment, before the method, the method further comprises:

[0013] For each preset network segment, it is determined whether the network segment corresponds to a synchronization tool.

[0014] If not, among the synchronization tools that do not correspond to a network segment, it is determined that the network segment corresponds to a synchronization tool.

[0015] In an optional embodiment of the present specification, according to the data collection instruction, the configuration information of the target synchronization tool is determined as the target configuration information of the target synchronization tool, which comprises:

[0016] It is determined whether the current configuration of the target synchronization tool matches the data collection instruction.

[0017] If not, in the configuration database, the configuration information with the highest matching degree with the data collection instruction is determined as the target configuration information of the target synchronization tool.

[0018] In an optional embodiment of the present specification, the data collection system further comprises a front-end server; and according to the default information, the synchronization tool is configured, which comprises:

[0019] When the synchronization request sent by the front-end server is received, the synchronization tool is configured according to the default information.

[0020] In an optional embodiment of the present specification, the front-end server comprises an interactive interface, so that the user generates a data collection instruction according to the interactive interface.

[0021] In an optional embodiment of the present specification, the method further comprises:

[0022] The synchronization tool corresponding to the configuration adjustment request generated by the front-end server is determined as the tool to be adjusted.

[0023] According to the adjustment information corresponding to the configuration adjustment request, the configuration information corresponding to the tool to be adjusted in the configuration database is adjusted.

[0024] In an optional embodiment of the present specification, the method further comprises:

[0025] According to the query request generated by the front-end server, the information corresponding to the query request in the configuration database is obtained and returned to the front-end server.

[0026] In a second aspect, a data collection method and device based on an offline synchronization tool of heterogeneous data sources are provided. The device is based on a data collection system, which includes a background server, a configuration database and a plurality of synchronization tools. The configuration database is used to record configuration information of the synchronization tools. The device is used for the background server and includes:

[0027] A default information determining module is configured to, for each synchronization tool, determine configuration information corresponding to the synchronization tool in the configuration database as default information of the synchronization tool when a state of the synchronization tool is idle.

[0028] A first configuration module is configured to configure the synchronization tool according to the default information.

[0029] A target synchronization tool determining module is configured to determine a synchronization tool corresponding to a data collection instruction as a target synchronization tool when the data collection instruction is received.

[0030] A target information determining module is configured to determine configuration information of the target synchronization tool as target information of the target synchronization tool according to the data collection instruction.

[0031] A second configuration module is configured to configure the target synchronization tool according to the target information.

[0032] An updating module is configured to update the configuration information corresponding to the synchronization tool in the configuration database according to the target information.

[0033] A collection module is configured to collect data by using the configured synchronization tools.

[0034] In an optional embodiment of the present disclosure, the data collection device based on the offline synchronization tool of the heterogeneous data sources can further include a synchronization tool allocation module.

[0035] The synchronization tool allocation module is configured to determine whether a network segment corresponds to a synchronization tool for each preset network segment. If not, the synchronization tool corresponding to the network segment is determined from the synchronization tools not corresponding to any network segment.

[0036] In an optional embodiment of the present disclosure, the target information determining module is specifically configured to determine whether the current configuration of the target synchronization tool matches the data collection instruction. If not, configuration information with the highest matching degree to the data collection instruction is determined in the configuration database as the target information of the target synchronization tool. If matched, the configuration of the target synchronization tool is not modified.

[0037] In an optional embodiment of the present disclosure, the data collection system further comprises a front-end server. The first configuration module is specifically configured to configure the synchronization tool according to the default information when receiving a synchronization request sent by the front-end server.

[0038] In an optional embodiment of the present disclosure, the front-end server comprises an interactive interface, so that a user generates a data collection instruction according to the interactive interface.

[0039] In an optional embodiment of the present disclosure, the data collection device based on the offline synchronization tool of the heterogeneous data source can further comprise an adjustment module. The adjustment module is configured to determine the synchronization tool corresponding to a configuration adjustment request generated by the front-end server as an adjustment tool. According to the adjustment information corresponding to the configuration adjustment request, the configuration information corresponding to the adjustment tool in the configuration database is adjusted.

[0040] In an optional embodiment of the present disclosure, the data collection device based on the offline synchronization tool of the heterogeneous data source can further comprise a query module. The query module is configured to obtain information corresponding to a query request in the configuration database according to the query request generated by the front-end server, and return to the front-end server.

[0041] In a third aspect, an electronic device is provided, and the electronic device comprises:

[0042] one or more processors;

[0043] a memory;

[0044] one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the data collection method based on the offline synchronization tool of the heterogeneous data source shown in the first aspect.

[0045] In a fourth aspect, a computer readable storage medium is provided, and the computer storage medium is used to store computer instructions, when the computer instructions are executed on a computer, the computer can execute the data collection method based on the offline synchronization tool of the heterogeneous data source shown in the first aspect.

[0046] The present application provides a data collection method and device based on an offline synchronization tool of a heterogeneous data source, an electronic device and a computer readable storage medium. On the one hand, the configuration information recorded in the configuration database is used as the default configuration of the synchronization tool, which effectively reduces the resources consumed by the configuration of the synchronization tool. On the other hand, the synchronization tool is configured according to the received data collection instruction, which can make the synchronization tool fit the data collection instruction, and is conducive to realizing the data collection effect.

[0047] Additional aspects and advantages of the present application will be apparent from the following description of the application and as can be learned by practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0048] The above and / or additional aspects and advantages of the present application will become apparent and be more readily understood through consideration of the following description, taken in conjunction with the accompanying drawings, in which:

[0049] Figure 1 A schematic diagram of a data acquisition system architecture according to an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a data acquisition process based on a heterogeneous data source offline synchronization tool according to an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a data acquisition device based on a heterogeneous data source offline synchronization tool according to an embodiment of the present application;

[0052] Figure 4 A schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0053] Embodiments of the present application are described below in detail, examples of which are shown in the accompanying drawings, in which the same or similar elements are denoted throughout by the same or similar reference numerals, and in which the embodiments described below are examples only, and are not to be construed as limiting the present application.

[0054] In the description of embodiments of the present application, the words "exemplary", "for example", or "e.g." are used to mean an example, instance, or illustration. Any embodiment or design solution described as "exemplary", "for example", or "e.g." in the embodiments of the present application should not be construed as being more advantageous or superior in any way than other embodiments or design solutions. In fact, a word or phrase such as "exemplary", "for example", or "e.g." is intended to present one or more examples or instances in a particular way.

[0055] In the description of embodiments of the present application, the term "and / or", merely describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B, which can represent three cases of existence of A alone, existence of B alone, and existence of A and B simultaneously. In addition, unless otherwise specified, the term "a plurality of" means two or more. For example, a plurality of systems means two or more systems, and a plurality of terminals means two or more terminals.

[0056] In addition, the terms "first", "second", "third", etc. are used herein only to describe different instances, and are not used to imply or suggest relative importance or a number of the technical features indicated. Thus, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. The terms "include", "contain", "have" and their variants mean "including but not limited to", unless otherwise specifically emphasized.

[0057] It should be noted that only parts related to the application are shown in the drawings for the purpose of description. The embodiments in the specification and the features in the embodiments can be combined with each other without conflict.

[0058] Flowcharts are used in the specification to illustrate the operations performed by the system according to the embodiments of the specification. It should be understood that the preceding or subsequent operations are not necessarily performed in sequence. Instead, the steps can be processed in reverse order or simultaneously. Meanwhile, other operations can be added to these processes, or one or more steps can be removed from these processes.

[0059] The data collection process based on the offline synchronization tool of the heterogeneous data source in the specification is based on a data collection system for managing and controlling the data collection process, and the functions of the management and control are at least embodied in the configuration for the synchronization tool.

[0060] As shown in Figure 1 The data collection system in the specification includes a background server, a configuration database for recording the configuration information of the synchronization tool, and a plurality of synchronization tools (i.e., synchronization tool 1 to synchronization tool n), and the configuration process performed for the synchronization tool can be executed by the background server therein.

[0061] As shown in Figure 2 The data collection process based on the offline synchronization tool of the heterogeneous data source in the specification can include one or more of the following steps.

[0062] S200: For each synchronization tool, when the state of the synchronization tool is idle, determine the configuration information corresponding to the synchronization tool in the configuration database as the default information of the synchronization tool.

[0063] S202: Configure the synchronization tool according to the default information.

[0064] The synchronization tool in the specification is used to collect data from a preset data source. When there is no data update in the data source corresponding to the synchronization tool, or the data collection system does not receive a data collection instruction, the data collection system will not collect data from the data source. At this time, the synchronization tool will be in an idle state.

[0065] But in actual scenarios, the data collection system can be in a working environment with high real-time requirements, and can receive data collection instructions at any time. Therefore, the process in the specification configures the synchronization tool in an idle state through default information, so that the synchronization tool can enter the working state in time.

[0066] It should be noted that the specific type of the synchronization tool is not limited in the specification. In an optional embodiment, the synchronization tool can be dataX.

[0067] S204: When receiving the data collection instruction, determine the synchronization tool corresponding to the data collection instruction as the target synchronization tool.

[0068] In an optional embodiment of the specification, the data collection system can also include a front-end server. The front-end server at least has the function of interacting with the business personnel, and the front-end server can generate data collection instructions under the control of the business personnel.

[0069] Generally, the data collection instruction corresponds to the data source, and one data collection instruction is usually directed to one or several data sources. In order to facilitate the configuration of the synchronization tool, the synchronization tool can be made to correspond to a specified data source or a specified area of a data source, so that data collection transactions for the data source or the area are allocated to the synchronization tool as much as possible.

[0070] Then, when receiving the data collection instruction, the data source corresponding to the data collection instruction can be determined, and then the synchronization tool corresponding to the data source in each synchronization tool is determined as the target synchronization tool. The target synchronization tool is the synchronization tool for data collection according to the data collection instruction.

[0071] S206: According to the data collection instruction, determine the configuration information of the target synchronization tool as the target information of the configuration of the target synchronization tool.

[0072] In the foregoing step, the default configuration has been made for the synchronization tool. In the case where the default configuration meets the data collection instruction, the default configuration is the target information, and the target synchronization tool does not need to be further configured.

[0073] But generally, there is usually a certain difference between different data collection instructions. For example, the user's data collection instruction for a data source is for time period t, and the next data collection instruction of the user for the data source is more likely to be for time period t+1 rather than time period t. Therefore, there can be an adjustment of the configuration of the synchronization tool according to the data collection instruction. But the adjustment does not need to configure all parameters of the synchronization tool.

[0074] In an optional embodiment, the process of determining the target information can be: determining, according to the data collection instruction, reference information, the reference information showing configuration information that the synchronization tool should have under the condition of collecting information corresponding to the data collection instruction. Determining the difference between the reference information and the current configuration information of the target synchronization tool as the target information.

[0075] S208: configuring the target synchronization tool according to the target information; and updating the configuration information corresponding to the synchronization tool in the configuration database according to the target information.

[0076] Since the target information is not all parameters of the synchronization tool, only the part of the parameters corresponding to the target information of the synchronization tool needs to be configured.

[0077] In addition, the process in the specification also updates the configuration database according to the target information, so that the configuration information in the configuration database can be adjusted according to the user's demand.

[0078] S210: using the configured synchronization tools to collect data.

[0079] After the configuration of the synchronization tool is completed, the data collection system can be used to collect data.

[0080] The data collection process based on the heterogeneous data source offline synchronization tool provided by the embodiment of the specification can effectively reduce the resources consumed by the configuration of the synchronization tool by using the configuration information recorded in the configuration database as the default configuration of the synchronization tool. On the other hand, the configuration of the synchronization tool according to the received data collection instruction can make the synchronization tool fit the data collection instruction, which is conducive to the realization of the data collection effect.

[0081] The synchronization tools in the specification have weak coupling between them, and the arrangement of the synchronization tools has strong flexibility. It can be seen that through the process in the specification, the plug-in management of the synchronization tool can be realized, the configuration file can be quickly generated, and the large number of synchronization tasks have good coping ability.

[0082] In addition, the data collection system in the specification has high scalability. Before the implementation of the foregoing steps, it can be determined whether the synchronization tool corresponding to each preset network segment exists; if not, the synchronization tool corresponding to the network segment is determined in the synchronization tool that does not correspond to the network segment.

[0083] In an optional embodiment of the present specification, the process of determining the target information can be: determining whether the current configuration of the target synchronization tool matches the data collection instruction. If not, locating the parameters in the current configuration of the target synchronization tool that do not match the data collection instruction as the parameters to be adjusted; in the configuration database, determining the configuration information corresponding to the parameters to be adjusted and having the highest matching degree with the data collection instruction as the target information of the configuration of the target synchronization tool.

[0084] The timing of configuring the synchronization tool in the present specification is not specifically limited, and in an optional embodiment, the synchronization tool is configured according to the default information when the synchronization request sent by the front-end server is received.

[0085] The process in the present specification can not only realize automatic parameter configuration of the synchronization tool, but also configure the synchronization tool according to the instruction of the user. Specifically, the front-end server comprises an interactive interface, and the user can generate a configuration adjustment request through the interactive interface. The synchronization tool corresponding to the configuration adjustment request generated by the front-end server is determined as the tool to be adjusted. The configuration information corresponding to the tool to be adjusted in the configuration database is adjusted according to the adjustment information corresponding to the configuration adjustment request.

[0086] It can be seen that the interactive interface is involved, which is beneficial to improve the visualization of the configuration process of the synchronization tool, and facilitates the user to control and supervise the configuration process of the synchronization tool and the data collection process based on the configuration of the synchronization tool.

[0087] In addition, the user can also generate a query request through the interactive interface, and then the background server obtains the information corresponding to the query request in the configuration database according to the query request generated by the front-end server and returns to the front-end server.

[0088] As known from the foregoing, the number of synchronization tools of the data collection system in the present specification can not be unique, and the process in the present specification can be implemented for each synchronization tool. Moreover, the timing of configuring the parameters of different synchronization tools can be different or the same. It can be seen that the process in the present specification can realize unified and coordinated management of the synchronization tools, and has strong flexibility. The configuration mode and timing of the synchronization tool can be dynamically adjusted according to the actual demand, so that the data collection system in the present specification is suitable for enterprise-level large-scale data collection.

[0089] In an optional embodiment of the present specification, the front-end server communicates with the back-end server in an http+json manner. The front-end server can use a vue.js framework. The UI library used for the interactive interface style of the front-end server can be iview, the data display component library dataV, the vuex state management realizes the communication between multiple components, and the custom encapsulation axios request is used for data interaction between the front-end server and the back-end server. In an optional embodiment, the version of vue of the front-end server can be 2.5.2. The version of iview can be 3.5.1. The version of dataV can be 2.9.6.

[0090] The back-end server can use spring boot as a basic framework, and integrate spring mvc, spring, jpa, shiro and other components. When the synchronization tool is datax, the version of the synchronization tool can be 3.0. In an optional embodiment, the version of spring boot can be 1.5.12. The version of shiro can be 1.3.2. The version numbers of spring mvc, spring and jpa depend on the version number of spring boot, and are consistent with the version number of spring boot.

[0091] In addition, the configuration database in the present specification can be a relational database such as Mysql.

[0092] It can be seen that the data acquisition system in the present specification not only has strong functionality, but also provides conditions for subsequent secondary development, has good scalability, and is suitable for enterprise-level massive heterogeneous data synchronization.

[0093] The present specification further provides a data acquisition device based on an offline synchronization tool of a heterogeneous data source, which can execute the data acquisition process based on the offline synchronization tool of the heterogeneous data source provided in the above embodiments of the present application, as shown in the following. Figure 3 The device is applied to a back-end server of an acquisition system; and the device comprises:

[0094] A default information determination module 300 is configured to, for each synchronization tool, when the state of the synchronization tool is idle, determine the configuration information corresponding to the synchronization tool in the configuration database as the default information of the synchronization tool;

[0095] A first configuration module 302 is configured to configure the synchronization tool according to the default information;

[0096] A target synchronization tool determination module 304 is configured to, when a data acquisition instruction is received, determine the synchronization tool corresponding to the data acquisition instruction as a target synchronization tool;

[0097] The target information determining module 306 is configured to determine configuration information of the target synchronization tool as target information of the configuration of the target synchronization tool according to the data collection instruction.

[0098] The second configuration module 308 is configured to configure the target synchronization tool according to the target information.

[0099] The updating module 310 is configured to update the configuration information of the synchronization tool in the configuration database according to the target information.

[0100] The collection module 312 is configured to collect data by using the configured synchronization tools.

[0101] The data collection apparatus based on the heterogeneous data source offline synchronization tool provided by the embodiments of the present specification can effectively reduce the resources consumed for configuring the synchronization tool by using the configuration information recorded in the configuration database as the default configuration of the synchronization tool. In addition, the synchronization tool can be configured according to the received data collection instruction, so that the synchronization tool can be adapted to the data collection instruction, which is beneficial to the data collection effect.

[0102] In an optional embodiment of the present specification, the data collection apparatus based on the heterogeneous data source offline synchronization tool can further include a synchronization tool allocation module 314.

[0103] The synchronization tool allocation module 314 is configured to determine whether the synchronization tool corresponding to each preset network segment exists; if not, determine the synchronization tool corresponding to the network segment from the synchronization tools not corresponding to the network segments.

[0104] In an optional embodiment of the present specification, the target information determining module 306 is specifically configured to determine whether the current configuration of the target synchronization tool matches the data collection instruction; if not, determine the configuration information with the highest matching degree with the data collection instruction in the configuration database as the target information of the configuration of the target synchronization tool. If matched, the configuration of the target synchronization tool is not modified.

[0105] In an optional embodiment of the present specification, the data collection system further includes a front-end server. The first configuration module 302 is specifically configured to configure the synchronization tool according to the default information when receiving the synchronization request sent by the front-end server.

[0106] In an optional embodiment of the present specification, the front-end server includes an interactive interface, so that the user generates the data collection instruction according to the interactive interface.

[0107] In an optional embodiment of the present disclosure, the data collection apparatus based on the offline synchronization tool of the heterogeneous data source can further comprise an adjustment module 316. The adjustment module 316 is configured to determine the synchronization tool corresponding to the configuration adjustment request generated by the front-end server as a tool to be adjusted. According to the adjustment information corresponding to the configuration adjustment request, the configuration information corresponding to the tool to be adjusted in the configuration database is adjusted.

[0108] In an optional embodiment of the present disclosure, the data collection apparatus based on the offline synchronization tool of the heterogeneous data source can further comprise a query module 318. The query module 318 is configured to obtain the information corresponding to the query request in the configuration database according to the query request generated by the front-end server, and return to the front-end server.

[0109] For the apparatus embodiment, since it basically corresponds to the method embodiment, the related part can be referred to the part of the method embodiment. The apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of one or more embodiments of the present disclosure. Those skilled in the art can understand and implement it without creative labor.

[0110] Further, the embodiment of the present application provides an electronic device, as shown in Figure 4 , as shown in Figure 4 The electronic device 40 shown in the figure comprises a processor 401 and a memory storage device 403. Wherein, the processor 401 and the memory storage device 403 are connected, such as through a bus 402. Further, the electronic device 40 can further comprise a transceiver 404. It should be noted that in actual application, the transceiver 404 is not limited to one, and the structure of the electronic device 40 does not constitute a limitation to the embodiments of the present application. Wherein, the processor 401 is applied to the embodiments of the present application, and is used to realize the functions of various modules shown in Figure 3 The transceiver 404 comprises a receiver and a transmitter.

[0111] The processor 401 can be a CPU, a general processor, a DSP, an ASIC, an FPGA or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can realize or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 401 can also be a combination of computing functions, such as one or more microprocessor combinations, DSP and microprocessor combinations, etc.

[0112] Bus 402 can include a path for communicating information among the above components. Bus 402 can be a PCI bus or an EISA bus, etc. Bus 402 can be divided into an address bus, a data bus, a control bus, etc. For convenience, Figure 4 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0113] Storage device 403 can be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, an EEPROM, a CD-ROM or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but not limited to this.

[0114] Storage device 403 is used to store application program code for implementing the scheme of the present application, and is controlled by processor 401 to execute. Processor 401 is used to execute the application program code stored in storage device 403, and is used to implement the functions of each module shown in the figure. Figure 3

[0115] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The program is executed by a processor to implement the method shown in the above embodiment.

[0116] It should be understood that although each step in the flowchart of the figure is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified in this article, the execution of these steps is not strictly limited in sequence, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the figure can include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order is not necessarily sequential, but can be alternately executed with other steps or sub-steps or stages of other steps.

[0117] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the application be limited only by the scope of the claims, which follow, and that the application encompass all variations, uses, or adaptations of the application following in general the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the application being indicated by the following claims.​

[0118] It is to be understood that the application is not limited to the precise construction herein described and illustrated in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope thereof. The scope of the application is limited only by the claims that follow.

[0119] The above merely preferred embodiments of the application and it is to be understood that numerous improvements and modifications will occur to those skilled in the art, without departing from the scope of the application, which is defined only by the claims that follow.

Claims

1. A data collection method based on an offline synchronization tool of a heterogeneous data source, the method being based on a data collection system, the data collection system comprising a background server, a configuration database and a plurality of synchronization tools, the configuration database being used to record configuration information of the synchronization tools, the method being executed by the background server, the method comprising: for each synchronization tool, when the state of the synchronization tool is idle, determining configuration information corresponding to the synchronization tool in the configuration database as default information of the synchronization tool; configuring the synchronization tool in an idle state by the default information according to the default information; when a data collection instruction is received, determining a data source corresponding to the data collection instruction, and then determining a synchronization tool corresponding to the data source as a target synchronization tool according to the synchronization tools; determining configuration information of the target synchronization tool as target information of configuration of the target synchronization tool according to the data collection instruction; configuring the target synchronization tool according to the target information; and updating the configuration information corresponding to the synchronization tool in the configuration database according to the target information, so that the configuration information in the configuration database is adjusted according to the requirements of a user; configuring part of parameters of the synchronization tool corresponding to the target information; collecting data by using the configured synchronization tools; before the data collection, the method further comprises a process of determining the target information: determining whether the current configuration of the target synchronization tool matches the data collection instruction; if not, locating parameters in the current configuration of the target synchronization tool that do not match the data collection instruction as to-be-adjusted parameters; and determining configuration information corresponding to the to-be-adjusted parameters and having the highest matching degree with the data collection instruction in the configuration database as the target information of configuration of the target synchronization tool. Before the method, the method further comprises: for each preset network segment, determining whether the network segment corresponds to a synchronization tool; if not, determining the synchronization tool corresponding to the network segment among the synchronization tools not corresponding to any network segment. The process of determining the configuration information of the target synchronization tool as the target information of configuration of the target synchronization tool according to the data collection instruction comprises: determining whether the current configuration of the target synchronization tool matches the data collection instruction; and if not, determining configuration information having the highest matching degree with the data collection instruction in the configuration database as the target information of configuration of the target synchronization tool. The data collection system further comprises a front-end server; and the process of configuring the synchronization tool according to the default information comprises: when a synchronization request sent by the front-end server is received, configuring the synchronization tool according to the default information. The front-end server comprises an interactive interface, so that a user generates a data collection instruction according to the interactive interface. The method further comprises: determining a synchronization tool corresponding to a configuration adjustment request generated by the front-end server as a to-be-adjusted tool; and adjusting configuration information corresponding to the to-be-adjusted tool in the configuration database according to adjustment information corresponding to the configuration adjustment request. The method further comprises: ​ ​ 2. The method of claim 1, wherein, ​ ​ ​ 3. The method of claim 1, wherein, ​ ​ ​ 4. The method of claim 1, wherein, ​ ​ 5. The method of claim 4, wherein, ​ 6. The method of claim 4, wherein, ​ ​ ​ 7. The method of claim 4, wherein, ​ According to the query request generated by the front-end server, information corresponding to the query request in the configuration database is obtained and returned to the front-end server.

8. A data collection apparatus based on an offline synchronization tool of heterogeneous data sources, characterized in that, The device is based on a data acquisition system including a background server, a configuration database and a plurality of synchronization tools, the configuration database is used to record configuration information of the synchronization tools, and the device is used for the background server, and the device includes: A default information determination module configured to, for each synchronization tool, determine configuration information corresponding to the synchronization tool in the configuration database as default information of the synchronization tool when the state of the synchronization tool is idle; A first configuration module configured to configure the synchronization tool in the idle state by the default information according to the default information; A target synchronization tool determination module configured to, when receiving a data acquisition instruction, determine a data source corresponding to the data acquisition instruction, and then determine a synchronization tool corresponding to the data source as a target synchronization tool according to each synchronization tool; A target information determination module configured to determine configuration information of the target synchronization tool as target information of configuration of the target synchronization tool according to the data acquisition instruction; A second configuration module configured to configure the target synchronization tool according to the target information; An update module configured to update the configuration information corresponding to the synchronization tool in the configuration database according to the target information, so that the configuration information in the configuration database is adjusted according to the demand of a user; and configure part of parameters of the synchronization tool corresponding to the target information; An acquisition module configured to acquire data by using the configured synchronization tools; before data acquisition, the process of determining the target information is further included: determining whether the current configuration of the target synchronization tool matches the data acquisition instruction; If not, positioning parameters in the current configuration of the target synchronization tool that do not match the data acquisition instruction as to-be-adjusted parameters; in the configuration database, determining configuration information corresponding to the to-be-adjusted parameters and having the highest matching degree with the data acquisition instruction as target information of configuration of the target synchronization tool. 9.An electronic device, one or more processors; a memory; one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-7. The storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-7.

Images