Data synchronous acquisition system and method, electronic equipment and storage medium

Abstract

The invention provides a data synchronous acquisition system and method, electronic equipment and a storage medium, the data synchronous acquisition system comprises a data acquisition module and a unified data module, the data acquisition module uses a plurality of threads to acquire data in different data sources, the same data module performs format unification on the data in each data source, and the unified data is stored in the unified data module. The preset data template is filled with the target data with the unified format, the target data is stored in the database after filling is completed, multiple pieces of data source data are collected through the same data collection module, the data collection efficiency is improved, and the maintenance cost is reduced; furthermore, the data of different data sources are converted into a unified format by utilizing a unified data module according to a format conversion method preset for different data sources and are stored in a database, so that the problem of data inconsistency between different platforms can be found in time, and the data of different platforms can be quickly stored under the condition that the preset data template is completely filled. And the target data are stored in the database, so that the data integrity is improved, and the query reliability is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and in particular to a data synchronization acquisition system, method, electronic equipment and storage medium. Background Art

[0002] Enterprises usually have a variety of electronic devices, such as computers and servers. In order to effectively manage and schedule these electronic devices, multiple resource management platforms are usually used. Different resource management platforms may record the same machine data or different machines. In order to analyze the use of electronic devices, it is usually necessary to collect and associate the data recorded by multiple resource management platforms.

[0003] The related technology usually requires collecting data from different resource management platforms through different data collection services. For example, when the resource management platform includes Nova, noahee and bce, the related technology usually involves independently obtaining cloud base information based on the official API documentation of openstack, and the independent data acquisition solution provided by the enterprise using noahee, and the independent data acquisition solution provided by BCE.

[0004] This independent data collection method cannot guarantee unified data acquisition, has high maintenance costs, and has low data collection efficiency. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data synchronization acquisition system, method, electronic device and storage medium to achieve synchronous acquisition of data from different data sources and ensure data consistency.

[0006] According to one aspect of the present invention, a data synchronization acquisition system is provided, the system comprising a data acquisition module, a unified data module and a database, the data acquisition module comprising a plurality of threads;

[0007] The data collection module is used to collect data from each of the data sources using the multiple threads based on a data collection method preset for each of the data sources, and report each of the data to the unified data module;

[0008] The unified data module is used to convert the format of each data according to the format conversion method preset for each data source to obtain each target data that conforms to the preset data format; use each target data to fill a preset data template, and the preset data template contains the data content to be filled; when the preset data template is filled, the target data is stored in the database.

[0009] In a possible embodiment, the system further includes a synchronization lock, and the synchronization lock is used to record data information collected by each of the threads, wherein the data information includes data source information and data type information;

[0010] The data acquisition module is used to determine the data information to be collected; query the data information to be collected in the synchronization lock, and return to determine the data information to be collected when the data information to be collected is queried; if the data information to be collected is not queried, call the idle thread to collect data that conforms to the data type contained in the data information based on the data acquisition method preset for the data source contained in the data information.

[0011] In a possible embodiment, the unified data module is used to obtain the data reported by each of the threads when the preset data template is not filled in completely, until the preset data template is filled in completely.

[0012] In a possible embodiment, the data acquisition module is deployed in different machines in a distributed manner, and each of the machines includes multiple threads; the preset data acquisition method includes obtaining data from a database and obtaining data from a preset API.

[0013] In a possible embodiment, the data source includes multiple resource management platforms and server management platforms.

[0014] According to another aspect of the present invention, a data synchronization acquisition method is provided, the method is applied to the data synchronization system according to any one of claims 1 to 5, the method comprising:

[0015] Using multiple threads, collecting data from each of the data sources based on a data collection method preset for each data source;

[0016] According to the format conversion method preset for each data source, the format of each data is converted to obtain each target data conforming to the preset data format;

[0017] Filling a preset data template with each of the target data, wherein the preset data template contains data content to be filled;

[0018] When the preset data template is filled, each of the target data is stored in a database.

[0019] In a possible embodiment, the method further includes:

[0020] Determine the data information to be collected, wherein the data information to be collected includes data source information and data type information;

[0021] Querying the data information to be collected in the synchronization lock, and returning to the step of determining the data information to be collected when the data information to be collected is found; the synchronization lock is used to record the data information collected by each thread, the data information including data source information and data type information;

[0022] In the case that the data information to be collected is not found, an idle thread is called to collect data that conforms to the data type contained in the data information based on a data collection method preset for the data source contained in the data information.

[0023] In a possible embodiment, the method further includes: when the preset data template is not filled completely, obtaining data reported by each of the threads until the preset data template is filled completely;

[0024] The preset data collection method includes obtaining data from a database and obtaining data from a preset API.

[0025] According to another aspect of the present invention, there is provided an electronic device, comprising:

[0026] Processor; and

[0027] Memory for storing programs,

[0028] Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes any of the above-mentioned data synchronization acquisition methods.

[0029] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute any of the above-mentioned data synchronization acquisition methods.

[0030] One or more technical solutions provided in the embodiments of the present invention collect data from different data sources by utilizing multiple threads in a data acquisition module, and unify the format of the data in each data source by utilizing the same data module, fill the target data in the unified format into a preset data template, and store the target data in a database when the preset data template is filled. This enables the collection of data from multiple data sources by the same data acquisition module, improves data collection efficiency, reduces maintenance costs, and collects different data sources by multiple threads, further improving data collection efficiency. Furthermore, by utilizing a unified data module to convert data from different data sources into a unified format according to a format conversion method preset for different data sources and storing the data in a database, data inconsistency problems between different platforms can be discovered in a timely manner, thereby ensuring consistency of data responses between different platforms. In addition, when the preset data template is filled, each target data is stored in the database, thereby improving the integrity of the data in the database, thereby improving the query reliability of these data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0032] Figure 1 A structural schematic diagram of a data synchronization acquisition system provided by an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of another structure of the data synchronization acquisition system provided by an embodiment of the present invention;

[0034] Figure 3 A comparison diagram of the data synchronization acquisition system provided by an embodiment of the present invention and the data acquisition method in the related art;

[0035] Figure 4 A schematic diagram of a flow chart of a data synchronization acquisition method provided by an embodiment of the present invention;

[0036] Figure 5 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0037] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0038] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0039] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0040] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0041] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0042] In order to achieve unified data acquisition on different platforms, the embodiments of the present invention provide a data synchronization acquisition system, method, electronic device and storage medium. The data synchronization acquisition system provided by the embodiments of the present invention can be applied to any electronic device with a data acquisition function, which can be a server, a computer or a mobile terminal, etc. The scheme of the present invention is described below with reference to the accompanying drawings:

[0043] Figure 1 A structural diagram of a data synchronization acquisition system provided by an embodiment of the present invention, the data synchronization acquisition system 100 may include a data acquisition module 101, a unified data module 102 and a database 103, and the data acquisition module 101 includes multiple threads;

[0044] The data collection module 101 is used to collect data from each of the data sources using the multiple threads based on a data collection method preset for each of the data sources, and report each of the data to the unified data module;

[0045] The unified data module 102 is used to convert the format of each data according to the format conversion method preset for each data source to obtain each target data that conforms to the preset data format; use each target data to fill a preset data template, and the preset data template contains the data content to be filled; when the preset data template is filled, the target data is stored in the database 103.

[0046] By applying the embodiment of the present invention, data from different data sources are collected by using multiple threads in a data collection module, and the same data module is used to unify the format of the data in each data source, and the target data in the unified format is filled into a preset data template. When the preset data template is filled, the target data is stored in a database, so that data from multiple data sources can be collected by the same data collection module, the data collection efficiency is improved, and the maintenance cost is reduced. In addition, by collecting different data sources through multiple threads, the data collection efficiency is further improved. Furthermore, by using a unified data module to convert data from different data sources into a unified format according to a format conversion method preset for different data sources and storing it in a database, the problem of data inconsistency between different platforms can be discovered in time, and the consistency of data responses between different platforms is ensured. In addition, when the preset data template is filled, each target data is stored in the database, which improves the integrity of the data in the database, thereby improving the query reliability of these data.

[0047] The following is an exemplary description of the above data synchronization acquisition system:

[0048] The data synchronization acquisition system provided by the embodiment of the present invention can be deployed in different machines in a distributed deployment manner. Specifically, the data acquisition module can be deployed in multiple machines in a distributed deployment manner, and the machine is usually a server. Each server can include multiple data acquisition threads, and different data acquisition threads are used to collect different data types, which can be different data types in the same data source or different data types in different data sources.

[0049] The above data sources can be determined according to the actual application scenario. In a possible embodiment, the multiple data sources can be a resource management platform, a server management platform, etc. For example, the data source can include Nova, noahee, and bce, etc. For different data sources, each thread can collect data from the corresponding data source by using the data collection method preset for the data source.

[0050] The data type in each data source can be determined according to the function of the data source, and can include, for example, server information, virtual machine information, cabinet information, host information, instance information and other asset information. Based on the above examples, the data type can be nova's servers (servers), hypervisors (virtual machine management) data, noahee's asset_rack (cabinet assets), asset_host (host assets) data, bce's t_instance (instance), t_dedicated_hypervisor (dedicated virtual machine) data. In a possible embodiment, the cloud base information of each data source can be collected, wherein the cloud base information refers to the server information required for a private cloud deployment and construction, including host name, rack position, IP, computer room, operating system, vpc (Virtual Private Cloud), router and other information.

[0051] The above-mentioned data collection method may include obtaining data from a database and obtaining data from a preset API. For example, nova can send instance, image and other information through an API. For the platform, the platform data can be collected through the preset API in the platform. For noahee and bce, the data in the platform can be obtained through their databases. The specific data collection method can be flexibly set according to the actual application scenario, and the present invention does not specifically limit this. As a possible implementation method, the thread can send a data collection request to the API or the platform's database, and the corresponding platform can return the data in the platform through the API or the database based on the data collection request.

[0052] In a possible embodiment, the system further includes a synchronization lock, and the synchronization lock is used to record data information collected by each of the threads, wherein the data information includes data source information and data type information;

[0053] The data acquisition module is used to determine the data information to be collected; query the data information to be collected in the synchronization lock, and return to determine the data information to be collected when the data information to be collected is queried; if the data information to be collected is not queried, call the idle thread to collect data that conforms to the data type contained in the data information based on the data acquisition method preset for the data source contained in the data information.

[0054] The above-mentioned synchronization lock can specifically be a ZK (Zookeeper) synchronization lock. The ZK synchronization lock is a distributed lock. Specifically, the ZK synchronization lock is a file storage system, and the nodes created by each thread can be stored in the ZK synchronization lock. Before collecting data, each thread will create a node in the ZK synchronization lock. The node includes the data information that the thread needs to collect. The data information may include a data source identifier and a data type identifier. When other threads need to collect data of the same data type from the same data source, the node cannot be successfully created, so that only one thread collects data of data type A in data source A at the same time.

[0055] Multiple nodes can be created in the above ZK synchronization lock. The number of nodes, that is, the thread concurrency, can be set according to the actual application scenario, and the present invention does not make any specific limitation on this.

[0056] After data collection is completed, the corresponding node can be deleted. In a possible embodiment, the ZK synchronization lock may include a listener, which can monitor the node deletion event in the ZK synchronization lock. After the node is deleted, other threads can collect the same data type of the same data source again.

[0057] Different data sources usually store data through different data storage methods. For example, the same data may be stored through different storage identifiers, or the same data may be stored through different numerical types. The above-mentioned different storage identifiers may include storage identifiers with different case, different languages, different letters, etc., and the above-mentioned numerical types may include int, var, char, etc. In a possible embodiment, different format conversion methods may be set for different data sources. The format conversion method may include case conversion, identifier replacement, numerical conversion, etc., so as to convert the data in each data source into target data in a unified format. The unified format can be set according to the actual application scenario, and the present invention does not specifically limit this.

[0058] In a possible embodiment, a data template can be preset, and the preset data template includes data content that needs to be filled. The data content can be flexibly selected according to the actual application scenario, such as including server identification, total server resources, server resource usage, server-associated IP, domain name, etc. After the unified data module obtains the data reported by the data acquisition module and converts the data format to obtain the target data, the target data can be filled into the preset data template according to the data type of the target data and the corresponding data identification. Afterwards, it can be detected at a preset time interval whether the preset data template is filled, that is, whether there are numerical values ​​in the data content contained in the preset data template. If it is filled, the target data is stored in the database. As a possible implementation method, the target data can be filled into the preset data template to obtain the target data template, and the target data template is stored in the database.

[0059] The target data can be stored in the database in the form of Key-Value, where the key can be a server ID, a computer room ID, etc., and the value can be the corresponding server information, computer room information, etc. The database can be a relational database, such as MySQL, or a non-relational database, which can be selected according to the actual application scenario, and the present invention does not specifically limit this. After receiving the data request sent by the user, the data of each platform can be returned from the database to improve the data query efficiency and query reliability.

[0060] In a possible embodiment, when the preset data template is not completely filled, the data reported by each thread may be continuously obtained until the preset data template is completely filled.

[0061] like Figure 2 As shown, Figure 2 A structural diagram of a data synchronization acquisition system provided for an embodiment of the present invention, wherein the data synchronization acquisition system may include an acquisition service cluster, a ZK synchronization lock and a unified data module, wherein the acquisition service cluster includes multiple acquisition service servers, each acquisition service may include multiple acquisition threads, each acquisition thread is used to collect a type of data in a data source, and the data may include nova's servers, hypervisors data information, noahee's asset_rack, asset_host data information, bce's t_instance, t_dedicated_hypervisor data information, etc.

[0062] Each collection service in the collection service cluster is connected to the ZK synchronization lock. Before collecting data, each collection thread can create a node in the ZK synchronization lock. The node can include the thread and the data information to be collected. After a collection thread creates a node, other threads cannot create nodes containing the same data information to be collected, thereby achieving only one collection thread collecting a specific data type in a data source at the same time.

[0063] The collection thread in each collection service can report the collected data to the uniformization module. The uniformization module can determine the data conversion format of each data according to the data source, and convert the format of the corresponding data based on the corresponding data conversion format, so as to obtain the target data in a uniform format. After that, these target data can be filled into the preset data template until the preset data template is filled and the target data is stored in the database.

[0064] like Figure 3 As shown, Figure 3 A comparison chart of the data synchronization acquisition system provided by the embodiment of the present invention and the data acquisition solution in the related art. For Nova, noahee and bce platforms, the related art usually sets different acquisition services based on the data acquisition configuration of the three platforms, and the three acquisition services are isolated from each other and not synchronized. The data of the corresponding platform is collected by the three acquisition services and stored in the database corresponding to each data platform. Users may obtain data from the three databases. Due to inconsistent storage identifiers, the data returned by the three databases for the same user request may be inconsistent.

[0065] By applying the data synchronization collection method provided in the embodiment of the present invention, data from three platforms are collected through a data collection service, and the data of each platform is unified in format and stored in a database. Users can obtain data from each platform from the database, ensuring data consistency across different platforms and improving data query reliability and timeliness.

[0066] Based on the same inventive concept, an embodiment of the present invention further provides a data synchronization acquisition method according to another aspect of the present invention, and the method is applied to any of the above-mentioned data synchronization systems, such as Figure 4 As shown, the method may include:

[0067] S401, using multiple threads to collect data from each data source based on a data collection method preset for each data source;

[0068] S402, converting the format of each data according to the format conversion method preset for each data source to obtain each target data conforming to the preset data format;

[0069] S403, filling a preset data template with each of the target data, wherein the preset data template contains data content to be filled;

[0070] S404: When the preset data template is filled, each of the target data is stored in a database.

[0071] In a possible embodiment, the method further includes:

[0072] Determine the data information to be collected, wherein the data information to be collected includes data source information and data type information;

[0073] Querying the data information to be collected in the synchronization lock, and returning to the step of determining the data information to be collected when the data information to be collected is found; the synchronization lock is used to record the data information collected by each thread, the data information including data source information and data type information;

[0074] In the case that the data information to be collected is not found, an idle thread is called to collect data that conforms to the data type contained in the data information based on a data collection method preset for the data source contained in the data information.

[0075] In a possible embodiment, the method further includes: when the preset data template is not filled completely, obtaining data reported by each of the threads until the preset data template is filled completely;

[0076] The preset data collection method includes obtaining data from a database and obtaining data from a preset API.

[0077] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0078] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0079] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0080] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0081] refer to Figure 5 , a block diagram of an electronic device 500 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0082] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0083] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 may be any type of device capable of inputting information to the electronic device 500, and the input unit 506 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 507 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 may include, but is not limited to, a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0084] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the above-mentioned data synchronization acquisition method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 502 and / or the communication unit 509. In some embodiments, the computing unit 501 may be configured to perform the above-mentioned data synchronization acquisition method by any other appropriate means (e.g., by means of firmware).

[0085] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0086] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0087] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0088] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0089] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0090] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A data synchronization acquisition system, characterized in that: The system includes a data acquisition module, a unified data module and a database, and the data acquisition module includes multiple threads; The data collection module is used to collect data from each of the data sources using the multiple threads based on a data collection method preset for each of the data sources, and report each of the data to the unified data module; The unified data module is used to convert the format of each data according to the format conversion method preset for each data source to obtain each target data that conforms to the preset data format; and fill the preset data template with each target data, wherein the preset data template contains the data content to be filled; When the preset data template is filled, each of the target data is stored in a database.

2. The system according to claim 1, characterized in that The system further includes a synchronization lock, wherein the synchronization lock is used to record data information collected by each of the threads, wherein the data information includes data source information and data type information; The data collection module is used to determine the data information to be collected; Querying the data information to be collected in the synchronization lock, and returning to determine the data information to be collected when the data information to be collected is found; In the case that the data information to be collected is not found, an idle thread is called to collect data that conforms to the data type contained in the data information based on a data collection method preset for the data source contained in the data information.

3. The system according to claim 1, characterized in that The unified data module is used to obtain the data reported by each thread when the preset data template is not filled in completely, until the preset data template is filled in completely.

4. The system according to claim 1, characterized in that The data acquisition module is deployed in different machines in a distributed manner, and each of the machines includes multiple threads; the preset data acquisition method includes obtaining data from a database and obtaining data from a preset API.

5. The system according to claim 1, characterized in that The data sources include various resource management platforms and server management platforms.

6. A data synchronization acquisition method, characterized in that: The method is applied to the data synchronization system described in any one of claims 1 to 5, and the method comprises: Using multiple threads, collecting data from each of the data sources based on a data collection method preset for each data source; According to the format conversion method preset for each data source, the format of each data is converted to obtain each target data conforming to the preset data format; Filling a preset data template with each of the target data, wherein the preset data template contains data content to be filled; When the preset data template is filled, each of the target data is stored in a database.

7. The method according to claim 6, characterized in that The method further comprises: Determine the data information to be collected, wherein the data information to be collected includes data source information and data type information; Querying the data information to be collected in the synchronization lock, and returning to the step of determining the data information to be collected when the data information to be collected is found; the synchronization lock is used to record the data information collected by each thread, the data information including data source information and data type information; In the case that the data information to be collected is not found, an idle thread is called to collect data that conforms to the data type contained in the data information based on a data collection method preset for the data source contained in the data information.

8. The method according to claim 6, characterized in that The method further includes: when the preset data template is not filled, obtaining data reported by each of the threads until the preset data template is filled; The preset data collection method includes obtaining data from a database and obtaining data from a preset API.

9. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 6 to 8.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 6-8.

Images