A data collection method and device

Abstract

The present application provides a data acquisition method and device, based on a data acquisition system, the system includes a database synchronization component and a database synchronization agent. When the data acquisition system in this specification is in an operating state, the code inside the container component obtained according to the database synchronization component is not transparent to each end involved in the data acquisition process. For the container component, it is only necessary to configure its data synchronization interface to enable it to realize its function in the data acquisition process without adjusting the code in the database synchronization component, which effectively reduces the use threshold of the data acquisition system in this specification.

Description

Technical Field

[0001] The present application relates to the technical field of data acquisition, and in particular to a data acquisition method and device. Background Art

[0002] With the development of the information society, the processing of various businesses depends to a large extent on the collection of data.

[0003] The object targeted by the data collection process is called a database. Taking a relational database as an example, the way each database stores data is not exactly the same, which makes the data collection for the database face a more complex collection environment. In addition, data users have different needs for data, which also increases the complexity of data collection to a certain extent. This forces data users to maintain and manage the database synchronization components used for data collection according to their own business needs. However, database synchronization components are mostly code-based applications or devices with fixed applications. The maintenance and management of database synchronization components requires strong professional knowledge. It is difficult for users to know the code of database synchronization components, which raises the threshold for data collection. Summary of the invention

[0004] This application provides a data collection method and device, which effectively lowers the threshold for using the data collection system. The technical solution adopted by this application is as follows:

[0005] In a first aspect, a data collection method is provided. The method is based on a data collection system. The system includes a database synchronization component and a database synchronization agent. The method includes:

[0006] For each preset database, obtain metadata of the database;

[0007] Determine available information according to the metadata, wherein the available information shows a storage location of the data to be collected in the database;

[0008] Containerizing the database synchronization component according to the available information to obtain a container component;

[0009] Establishing a connection between the container component and the database synchronization agent according to the available information;

[0010] The connected container component and the database synchronization agent are used to collect data.

[0011] Optionally, before determining the available information according to the metadata, the method further includes:

[0012] According to each data acquisition request received by the data acquisition system within a preset time period in history, determining description information of each data collected by the data acquisition system within the preset time period;

[0013] Based on the metadata, available information is determined, including:

[0014] In the metadata, data matching the description information is determined as available information.

[0015] Optionally, containerizing the database synchronization component according to the available information to obtain a container component includes:

[0016] Processing is performed according to the storage location of the data corresponding to the available information in the database to obtain a plurality of fragments;

[0017] According to each obtained shard, the database synchronization component is containerized so that the obtained container component includes a plurality of data synchronization interfaces, and the data synchronization interfaces correspond to the shards one by one.

[0018] Optionally, establishing a connection between the container component and the database synchronization agent according to the available information includes:

[0019] A data synchronization interface obtained according to the available information is added to the database synchronization agent to establish a connection between the container component and the database synchronization agent.

[0020] Optionally, before data collection is performed using the connected container component and the database synchronization agent, the method further includes:

[0021] A designated interface of the database synchronization agent is generated according to the available information; the designated interface is used to obtain a data acquisition request corresponding to the available information.

[0022] Optionally, the connected container component and the database synchronization agent are used to collect data, including:

[0023] When a data acquisition request is detected, determining whether the data acquisition request matches the specified interface;

[0024] If so, the database synchronization agent generates a data acquisition instruction according to the data acquisition request and sends it to the container component, so that the container component acquires data from the database corresponding to the data acquisition request according to the acquisition instruction.

[0025] Optionally, before data collection is performed using the connected container component and the database synchronization agent, the method further includes:

[0026] generating a first test request according to the available information, and generating a second test request according to information other than the available information;

[0027] Inputting the first test request and the second test request into the designated interface;

[0028] When the state of the data collected according to the first test request is normal and the state of the data collected according to the second test request is abnormal, determining that the state of the data collection system is normal;

[0029] The connected container component and the database synchronization agent are used to collect data, including:

[0030] Under the condition that the state of the data acquisition system is normal, data acquisition is performed.

[0031] In a second aspect, a data acquisition device is provided, wherein the device is used in a data acquisition system, the system includes a database synchronization component and a database synchronization agent; the device includes:

[0032] A metadata acquisition module is configured to acquire metadata of each preset database;

[0033] An available information determination module, configured to determine available information according to the metadata, wherein the available information shows a storage location of the data to be collected in the database;

[0034] A container component generating module, configured to containerize the database synchronization component according to the available information to obtain a container component;

[0035] a connection module, configured to establish a connection between the container component and the database synchronization agent according to the available information;

[0036] The acquisition module is configured to use the connected container component and the database synchronization agent to perform data acquisition.

[0037] Optionally, the data collection system may further include a description information acquisition module 310 .

[0038] A description information acquisition module, configured to determine the description information of each data collected by the data acquisition system within the preset time period according to each data acquisition request received by the data acquisition system within the preset time period in history;

[0039] The available information determination module is specifically configured to determine data matching the description information in the metadata as available information.

[0040] Optionally, the container component generation module may include a sharding submodule and a data synchronization interface determination submodule.

[0041] Optionally, the connection module is specifically configured to add a data synchronization interface obtained according to the available information to the database synchronization agent to establish a connection between the container component and the database synchronization agent.

[0042] Optionally, the data acquisition system may further include a designated interface generation module. The designated interface generation module is configured to generate a designated interface of the database synchronization agent according to the available information; the designated interface is used to obtain a data acquisition request corresponding to the available information.

[0043] Optionally, the acquisition module is specifically configured to, when detecting a data acquisition request, determine whether the data acquisition request matches the specified interface. If so, the database synchronization agent generates a data acquisition instruction according to the data acquisition request and sends it to the container component, so that the container component acquires data from the database corresponding to the data acquisition request according to the acquisition instruction.

[0044] Optionally, the data acquisition system may further include a detection module, wherein the detection module is configured to generate a first test request according to the available information, and to generate a second test request according to information other than the available information. The first test request and the second test request are input into the designated interface. When the state of the data collected according to the first test request is normal, and the state of the data collected according to the second test request is abnormal, the state of the data acquisition system is determined to be normal.

[0045] Optionally, the acquisition module is specifically configured to perform data acquisition under the condition that the state of the data acquisition system is normal.

[0046] In a third aspect, an electronic device is provided, the electronic device comprising:

[0047] one or more processors;

[0048] Memory;

[0049] One or more application programs, wherein the one or more application programs are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: execute the data collection method shown in the first aspect.

[0050] In a fourth aspect, a computer-readable storage medium is provided, which is used to store computer instructions. When the computer-readable storage medium is run on a computer, the computer can execute the data collection method shown in the first aspect.

[0051] The present application provides a data acquisition method, device, electronic device and computer-readable storage medium, based on a data acquisition system, the system including a database synchronization component and a database synchronization agent. When the data acquisition system in this specification is in operation, the code inside the container component obtained according to the database synchronization component is not transparent to each end involved in the data acquisition process. For the container component, it is only necessary to configure its data synchronization interface to enable it to realize its function in the data acquisition process without adjusting the code in the database synchronization component, which effectively reduces the use threshold of the data acquisition system in this specification.

[0052] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0054] Figure 1 A schematic diagram of a data collection system architecture according to an embodiment of the present application;

[0055] Figure 2 A schematic diagram of a data collection process according to an embodiment of the present application;

[0056] Figure 3 A structural schematic diagram of a data acquisition device according to an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0058] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0059] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0060] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, and multiple terminals refers to two or more terminals.

[0061] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0062] It should be noted that, for the convenience of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments in this specification can be combined with each other.

[0063] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0064] In actual data collection scenarios, data users often face a more complex data collection environment. In order to meet the changing business needs of users and the different business needs of different users, so that the data collection effect can reach the expected level, users often need to maintain and manage the code of the database synchronization component according to their business needs, which makes data collection more difficult.

[0065] In view of this, the data collection process in this specification is proposed. The data collection process in this specification is based on a data collection system. For example, Figure 1 As shown, the data collection system includes a data source database synchronization component and a data source database synchronization agent. The data collection system is used to collect data from data source database 1 to data source database n, and then push the collected data to a distributed publish-subscribe message system for users to use the collected data.

[0066] like Figure 2 As shown, the process may include one or more of the following steps.

[0067] S200: For each preset data source database, obtain metadata of the data source database.

[0068] The database in this specification refers to a warehouse for storing data. It is a collection of large amounts of data that is stored in a computer for a long period of time, organized, shareable, and uniformly managed. This specification does not limit the type of database involved. In an optional embodiment, at least part of the database connected to the data acquisition system in this specification can store data in the form of tables. For example, the database in this specification can be a relational database.

[0069] In this specification, metadata, also known as intermediary data or relay data, is data about data, mainly information describing data properties, used to support functions such as indicating storage location, historical data, resource search, file records, etc. Metadata has the function of an electronic directory.

[0070] S202: Determine available information according to the metadata, where the available information shows a storage location of the data to be collected in the database.

[0071] After obtaining the metadata of each data source database, the metadata obtained from different data source databases can be selected to determine, for each data source database, the storage location, data identification and other information of the data in the data source database that can be used by the data acquisition system in this specification as available information.

[0072] It can be seen that the available information in this specification is used to obtain the data corresponding to the available information from the database in the subsequent steps. In addition, in addition to being able to show the storage location of the data to be collected in the database, the available information in this specification can be designed according to actual needs. For example, the description information of the data corresponding to the available information (for example, the aforementioned data attributes) can be used as the available information.

[0073] For example, in an optional business scenario, the data to be collected is the user's bank account usage records, such as transaction amounts, transaction times, etc. Then the storage location of the data indicating the bank account usage records stored in a database can be used as available information in this specification. At the same time, the database also stores information representing user attributes, such as the user's age, occupation, etc., but these data are not the objects of collection, so the storage location of these data and other information are not available information in this specification.

[0074] S204: Containerize the database synchronization component according to the available information to obtain a container component.

[0075] The purpose of containerizing the database synchronization component in this specification is to encapsulate the database synchronization component so that the database synchronization component only exposes several data synchronization interfaces. When the data acquisition system in this specification is in operation, the code inside the container component obtained according to the database synchronization component is opaque to all ends involved in the data acquisition process. For the container component, it only needs to configure its data synchronization interface to enable it to realize its function in the data acquisition process without adjusting the code in the database synchronization component, which effectively lowers the usage threshold of the data acquisition system in this specification.

[0076] S206: Establishing a connection between the container component and the database synchronization agent according to the available information.

[0077] The process of establishing the connection is to associate the container component with the database synchronization agent. When collecting data through the data collection system in this specification, the user of the data can personalize the container component through the database synchronization agent, which is convenient for the user to control and manage the data collection process.

[0078] As mentioned above, the containerization process of the database synchronization component is based on the available information. Therefore, when establishing the connection between the container component and the database synchronization agent, the available information is still used as the basis, which can effectively ensure the connection effect.

[0079] For example, in an optional embodiment, the data synchronization interface of the container component is determined by the database identifier in the available information (which can roughly show the location of the data to be collected), and then the database synchronization agent can establish a connection between the data synchronization interface and the database synchronization agent for each data synchronization interface of the container component based on the database identifier in the available information.

[0080] S208: Using the connected container component and the database synchronization agent to collect data.

[0081] After establishing the connection between the container component and the database synchronization agent, the user can configure the container component through the database synchronization component and control the data collection process to achieve the purpose of data collection.

[0082] As can be seen from the foregoing, the containerization process of the database synchronization component plays an important role in this specification. Now, how to implement the containerization process is described.

[0083] The containerization process may be: processing according to the storage location of the data corresponding to the available information in the database to obtain a plurality of shards. According to each of the obtained shards, the database synchronization component is containerized so that the obtained container component includes a plurality of data synchronization interfaces, and the data synchronization interfaces correspond to the shards one by one.

[0084] In the subsequent steps, when the container component and the database synchronization agent are established, the data synchronization interface obtained according to the available information can be added to the database synchronization agent to establish a connection between the container component and the database synchronization agent.

[0085] After the container component and the database synchronization agent are established, a data acquisition system with an available status is obtained. The data acquisition system with an available status can be used for data acquisition. The process of data acquisition can be to receive a data acquisition request sent by a user and perform data acquisition according to the data acquisition request.

[0086] The data collection process in this specification may be implemented under the triggering of a data acquisition request. In order to receive a data acquisition request, the data collection system in this specification is provided with a designated interface. Under the condition that the data is in an available state, the designated interface is an interface corresponding to the data that the data collection system can provide to the data user, and the designated interface is used to receive a data acquisition request corresponding to the available data.

[0087] The process of generating the designated interface may be: generating the designated interface of the database synchronization agent according to the available information; and the designated interface is used to obtain a data acquisition request corresponding to the available information.

[0088] The data acquisition process after generating the designated interface according to the available information may be: upon detecting a data acquisition request, determining whether the data acquisition request matches the designated interface (for example, the process may be determining whether the data acquisition request corresponds to the available information);

[0089] If yes, the database synchronization agent generates a data acquisition instruction according to the data acquisition request and sends it to the container component, so that the container component acquires data from the database corresponding to the data acquisition request according to the acquisition instruction. If no, a prompt is sent to the sender of the data acquisition request so that the sender regenerates the data acquisition request. Optionally, the prompt can be the available information, so that the sender can regenerate the data acquisition request according to the available information.

[0090] In an optional embodiment of the present specification, when the data acquisition request does not match the designated interface, information with the highest similarity to the data acquisition request may be determined from the available information, and the determined information may be displayed to the sender.

[0091] In the aforementioned embodiment, the transaction amount can be used as available data, but the user's attributes are not available data. In this case, the designated interface can be used to receive data acquisition requests for the transaction amount, but cannot receive data acquisition requests for the user's attributes.

[0092] In an optional embodiment of the present specification, the designated interface may be in the form of an interface to facilitate the operation of the data user. Specifically, the interface may include an area for displaying graphics and / or text, a text input area, buttons, etc.

[0093] Furthermore, the process in this specification can not only obtain a data acquisition system that can be conveniently used by data users, but also detect the data acquisition system to determine whether the data acquisition system is in a normal and usable state.

[0094] Specifically, the detection process may be: generating a first test request according to the available information, and generating a second test request according to information other than the available information; inputting the first test request and the second test request into the designated interface. When the state of the data collected according to the first test request is normal, and the state of the data collected according to the second test request is abnormal, it is determined that the state of the data collection system is normal.

[0095] The first test request is generated based on available information and has the properties of a positive example. If the data collection system is normal, data can be collected based on the first test request. The second test request is obtained based on information other than the available information and has the properties of a negative example. If the data collection system is normal, data cannot be collected based on the second test request.

[0096] If the data acquisition system does not acquire data according to the first test request, or if the data acquisition system acquires data according to the second test request, it indicates that some functions of the data acquisition system are abnormal.

[0097] After determining that the data acquisition system is abnormal, step S204 and subsequent steps may be re-executed.

[0098] Optionally, the aforementioned detection may be performed on the data acquisition system each time a connection is established between the container component and the database synchronization agent.

[0099] As can be seen from the foregoing, available information plays a key role in the data collection process of this specification. In an optional embodiment of this specification, the process of determining available information may be: according to each data acquisition request received by the data collection system in a preset time period in history, the description information of each data collected by the data collection system in the preset time period is determined. Afterwards, the data matching the description information is determined in the metadata as available information.

[0100] The data acquisition request may be sent by a data user. The data collection system in this specification may be oriented to multiple data users. For example, the data users may be multiple different companies, and each company may be a data user. For another example, the data users may be different departments of the same company, and any department may be a data user.

[0101] In order to meet the needs of different users, when determining the preset time period, the preset time period can be made to cover at least one data acquisition request sent by each user involved in the data acquisition system.

[0102] In an optional embodiment of the present specification, the preset time period can be determined according to the degree of demand for data by different users. The higher the user's demand for data, the higher the user's demand for data. Then, the weight of the user can be determined for each user, and the weight can be determined according to preset business rules. For each user, a data acquisition request sent by the user is received, and the number of data acquisition requests sent by each user is accumulated according to the weight of each user (for example, the number of data acquisition requests sent by each user is weighted and summed). If the accumulated result reaches a preset threshold, the available information is determined. At this time, the preset time period changes dynamically according to the accumulated result.

[0103] It should be noted that the data acquisition request used when determining the available information in this embodiment is not only used to determine the available information. When a data acquisition request is received in the history, the corresponding data acquisition processing is performed according to the data acquisition system before determining the available information. It can be seen that the process of determining the available information in this specification can be regarded as a process of intermittently resetting the data acquisition system.

[0104] This specification does not limit the specific type of the data source database synchronization component. In an optional embodiment, the data source database synchronization component may be a canal component, and the distributed publish-subscribe messaging system may be kafka. The data acquisition system may be a distributed data acquisition system. The platform used by the data acquisition system may be based on the OpenShift platform.

[0105] This specification does not limit the number of database synchronization components and the number of database synchronization agents included in the data acquisition system.

[0106] This specification further provides a data acquisition device that can execute a data acquisition process provided in the above embodiments of this application, such as Figure 3 As shown, the data acquisition device is applied to a data acquisition system; the data acquisition system includes a data source database synchronization component and a data source database synchronization agent, and the device includes:

[0107] The metadata acquisition module 300 is configured to acquire metadata of each preset data source database;

[0108] An available information determination module 302 is configured to determine available information according to the metadata, wherein the available information shows a storage location of the data to be collected in the database;

[0109] A container component generating module 304 is configured to containerize the database synchronization component according to the available information to obtain a container component;

[0110] A connection module 306, configured to establish a connection between the container component and the database synchronization agent according to the available information;

[0111] The collection module 308 is configured to use the connected container component and the database synchronization agent to collect data.

[0112] Optionally, the data collection system may further include a description information acquisition module 310 .

[0113] The description information acquisition module 310 is configured to determine the description information of each data collected by the data collection system within the preset time period according to each data acquisition request received by the data collection system within the preset time period in history;

[0114] The available information determination module 302 is specifically configured to determine data matching the description information in the metadata as available information.

[0115] Optionally, the container component generation module 304 may include a sharding submodule 3040 and a data synchronization interface determination submodule 3042 .

[0116] Optionally, the connection module 306 is specifically configured to add a data synchronization interface obtained according to the available information to the database synchronization agent to establish a connection between the container component and the database synchronization agent.

[0117] Optionally, the data acquisition system may further include a designated interface generation module 312. The designated interface generation module 312 is configured to generate a designated interface of the database synchronization agent according to the available information; the designated interface is used to obtain a data acquisition request corresponding to the available information.

[0118] Optionally, the acquisition module 308 is specifically configured to, when a data acquisition request is detected, determine whether the data acquisition request matches the specified interface. If so, the database synchronization agent generates a data acquisition instruction according to the data acquisition request and sends it to the container component, so that the container component acquires data from the database corresponding to the data acquisition request according to the acquisition instruction.

[0119] Optionally, the data acquisition system may further include a detection module 314, wherein the detection module 314 is configured to generate a first test request according to the available information, and to generate a second test request according to information other than the available information. The first test request and the second test request are input into the designated interface. When the state of the data collected according to the first test request is normal, and the state of the data collected according to the second test request is abnormal, the state of the data acquisition system is determined to be normal.

[0120] Optionally, the acquisition module 308 is specifically configured to perform data acquisition under the condition that the state of the data acquisition system is normal.

[0121] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of one or more embodiments of the present specification. A person of ordinary skill in the art can understand and implement them without creative work.

[0122] Furthermore, an embodiment of the present application provides an electronic device, such as Figure 4 As shown, Figure 4The electronic device 40 shown includes: a processor 401 and a memory storage device 403. The processor 401 and the memory storage device 403 are connected, such as through a bus 402. Further, the electronic device 40 may also include a transceiver 404. It should be noted that in actual applications, the transceiver 404 is not limited to one, and the structure of the electronic device 40 does not constitute a limitation on the embodiments of the present application. The processor 401 is used in the embodiments of the present application to implement Figure 3 The functions of the modules are shown in FIG. 4. The transceiver 404 includes a receiver and a transmitter.

[0123] Processor 401 may be a CPU, a general purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0124] The bus 402 may include a path to transmit information between the above components. The bus 402 may be a PCI bus or an EISA bus, etc. The bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0125] The storage device 403 can be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, CD-ROM or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0126] The storage device 403 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the storage device 403 to implement Figure 3 The functions of each module are shown.

[0127] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the method shown in the above embodiment is implemented.

[0128] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0129] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0130] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0131] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data collection method, the method is based on a data collection system, the system includes a database synchronization component and a database synchronization agent; the method includes: For each preset database, obtain metadata of the database; Determine available information according to the metadata, wherein the available information shows a storage location of the data to be collected in the database; Containerizing the database synchronization component according to the available information to obtain a container component; Establishing a connection between the container component and the database synchronization agent according to the available information; Generate a designated interface of the database synchronization agent according to the available information; the designated interface is used to obtain a data acquisition request corresponding to the available information; The connected container component and the database synchronization agent are used to collect data, including: when a data acquisition request is detected, determining whether the data acquisition request matches the specified interface; If so, the database synchronization agent generates a data acquisition instruction based on the data acquisition request and sends it to the container component, so that the container component obtains data from the database corresponding to the data acquisition request according to the acquisition instruction. When the data acquisition request does not match the specified interface, the information with the highest similarity to the data acquisition request is determined from the available information, and the determined information is displayed to the sender of the data acquisition request.

2. The method according to claim 1, characterized in that Before determining the available information according to the metadata, the method further includes: According to each data acquisition request received by the data acquisition system within a preset time period in history, determining description information of each data collected by the data acquisition system within the preset time period; Based on the metadata, available information is determined, including: In the metadata, data matching the description information is determined as available information.

3. The method according to claim 1, characterized in that According to the available information, the database synchronization component is containerized to obtain a container component, including: Processing is performed according to the storage location of the data corresponding to the available information in the database to obtain a plurality of fragments; According to each obtained shard, the database synchronization component is containerized so that the obtained container component includes a plurality of data synchronization interfaces, and the data synchronization interfaces correspond to the shards one by one.

4. The method according to claim 3, characterized in that Establishing a connection between the container component and the database synchronization agent according to the available information includes: A data synchronization interface obtained according to the available information is added to the database synchronization agent to establish a connection between the container component and the database synchronization agent.

5. The method according to claim 1, characterized in that Before collecting data using the connected container component and the database synchronization agent, the method further includes: generating a first test request according to the available information, and generating a second test request according to information other than the available information; Inputting the first test request and the second test request into the designated interface; When the state of the data collected according to the first test request is normal and the state of the data collected according to the second test request is abnormal, determining that the state of the data collection system is normal; The connected container component and the database synchronization agent are used to collect data, including: Under the condition that the state of the data acquisition system is normal, data acquisition is performed.

6. A data acquisition device, characterized in that: The device is used in a data acquisition system, the system includes a database synchronization component and a database synchronization agent; the device includes: A metadata acquisition module is configured to acquire metadata of each preset database; An available information determination module, configured to determine available information according to the metadata, wherein the available information shows a storage location of the data to be collected in the database; A container component generating module, configured to containerize the database synchronization component according to the available information to obtain a container component; a connection module, configured to establish a connection between the container component and the database synchronization agent according to the available information; A designated interface generation module generates a designated interface of the database synchronization agent according to the available information; the designated interface is used to obtain a data acquisition request corresponding to the available information; The acquisition module is configured to use the connected container component and the database synchronization agent to perform data acquisition, including: when a data acquisition request is detected, determining whether the data acquisition request matches the specified interface; if so, making the database synchronization agent generate a data acquisition instruction according to the data acquisition request, and sending it to the container component, so that the container component acquires data from the database corresponding to the data acquisition request according to the acquisition instruction; when the data acquisition request does not match the specified interface, determining the information with the highest similarity to the data acquisition request from the available information, and displaying the determined information to the sender of the data acquisition request.

7. An electronic device, one or more processors; Memory; One or more applications, wherein the one or more applications are stored in a memory and configured to be executed by one or more processors, and the one or more programs are configured to: execute the method described in any one of claims 1-5 above.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 5 is implemented.

Images