Method and device for synchronizing multi-source heterogeneous data with data lake and storage medium

Abstract

The invention discloses a method and device for synchronizing multi-source heterogeneous data with a data lake and a storage medium, and the method comprises the steps: obtaining a data interface specification and an interactive data task table of each system; obtaining original data of each system based on the data interface specification and the interaction data task table of each system; preprocessing the original data of each system to obtain preprocessed data of each system; and synchronizing the data preprocessed by each system to a data lake. Therefore, based on the data interface specification and the interactive data task table of each system, the multi-source heterogeneous data is imported into the data lake, the difficulty and the cost of integration of the multi-source heterogeneous data are reduced, and the consistency and the accuracy of the data are improved.

Description

Technical Field

[0001] The present invention relates to the field of data storage technology, and in particular to a method, device and storage medium for synchronizing a data lake with multi-source heterogeneous data. Background Art

[0002] Currently, the task data of each upstream system is stored in its own system, which makes it difficult to communicate between data, increasing the difficulty and cost of data integration. Moreover, when business needs change, each system needs to be reconfigured and adjusted, increasing maintenance costs. Based on this, it is necessary to uniformly manage multi-source heterogeneous data to improve data consistency and accuracy.

[0003] In the existing technology, data lakes can bring together different types of data and do not require predefined models for data analysis. Therefore, it is necessary to import multi-source heterogeneous data into data lakes to realize the collection and management of multi-source heterogeneous data and meet the various needs of organizational structure. Summary of the invention

[0004] The present invention provides a method and device for synchronizing a data lake with multi-source heterogeneous data to solve the technical problem of low response accuracy in the above-mentioned prior art.

[0005] To this end, the present invention proposes a method for synchronizing a data lake with multi-source heterogeneous data, which can obtain the user's conversation intention and key information based on the target intention recognition model, and obtain the target answer data through the user's conversation intention and key information, thereby accurately answering the user's questions and improving the accuracy of the answer data.

[0006] Another object of the present invention is to provide a device for synchronizing a data lake with multi-source heterogeneous data.

[0007] To achieve the above object, the present invention proposes a method for synchronizing a data lake with multi-source heterogeneous data, the method comprising:

[0008] Obtain data interface specifications and interactive data task tables for each system;

[0009] Based on the data interface specification and the interactive data task table of each system, obtaining the original data of each system;

[0010] Preprocessing the original data of each system to obtain preprocessed data of each system;

[0011] The pre-processed data of each system is synchronized to the data lake.

[0012] The method for synchronizing a data lake with multi-source heterogeneous data in an embodiment of the present invention may also have the following additional technical features:

[0013] In one embodiment of the present invention, the data interface specification includes: data format, transmission protocol and interface calling method.

[0014] In one embodiment of the present invention, the interactive data task table includes: primary key, source table name, source table corresponding id, business serial number, source system name, calling system name, calling system interface name, task status, calling time, interface return information, number of calls, remarks, insertion time, and update time.

[0015] In one embodiment of the present invention, preprocessing the raw data of each system to obtain the preprocessed data of each system includes:

[0016] Performing data cleaning on the original data of each system to obtain the first data of each system;

[0017] Performing format conversion processing on the first data of each system to obtain the second data of each system;

[0018] Error correction is performed on the second data of each system to obtain the data processed by each system.

[0019] In one embodiment of the present invention, the method further comprises:

[0020] Determine the alarm rules corresponding to each system;

[0021] Based on the alarm rules, the process of synchronizing the data lake with the system data is monitored and warned.

[0022] In one embodiment of the present invention, the obtaining of the alarm rules corresponding to each system includes: determining the monitoring indicators, alarm thresholds and alarm methods corresponding to each system.

[0023] To achieve the above object, the present invention further provides a device for synchronizing a data lake with multi-source heterogeneous data, the device comprising:

[0024] The first acquisition module is used to acquire data interface specifications and interactive data task tables of each system;

[0025] A second acquisition module, used to acquire the original data of each system based on the data interface specification and the interactive data task table of each system;

[0026] A processing module, used for preprocessing the raw data of each system to obtain the preprocessed data of each system;

[0027] The synchronization module is used to synchronize the data pre-processed by each system to the data lake.

[0028] Another object of the present invention is to provide an electronic device, comprising:

[0029] at least one processor; and

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods described in the above aspect.

[0032] Another object of the present invention is to provide a computer storage medium, wherein the computer storage medium stores computer executable instructions; after the computer executable instructions are executed by a processor, the computer executes any one of the methods described in the above aspect.

[0033] The method, device and storage medium for synchronizing a data lake with multi-source heterogeneous data in an embodiment of the present invention obtain question data input by a user; input the question data into a target intent recognition model to obtain the user's conversation intent and key information; determine the corresponding reply template based on the conversation intent; determine the initial answer data for the question data based on the key information and the knowledge graph; obtain the target answer data for the question data based on the reply template and the initial answer data. Thus, the present invention obtains the user's conversation intent and key information based on the target intent recognition model, and obtains the target answer data through the user's conversation intent and key information, thereby accurately answering the user's questions and improving the accuracy of the answer data.

[0034] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0036] Figure 1 is a flowchart of a method for synchronizing a data lake with multi-source heterogeneous data according to an embodiment of the present invention;

[0037] Figure 2 is a structural diagram of a device for synchronizing a data lake with multi-source heterogeneous data according to an embodiment of the present invention;

[0038] Figure 3 The present invention is a block diagram of an electronic device suitable for executing a method for synchronizing a data lake with multi-source heterogeneous data according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0039] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0041] The following describes a method and device for synchronizing a data lake with multi-source heterogeneous data according to an embodiment of the present invention with reference to the accompanying drawings.

[0042] Figure 1 It is a flowchart of a method for synchronizing a data lake with multi-source heterogeneous data in an embodiment of the present invention.

[0043] like Figure 1 As shown, the method includes:

[0044] S1, obtain data interface specifications and interactive data task tables of each system;

[0045] In one embodiment of the present invention, the execution subject of the method for synchronizing a data lake with multi-source heterogeneous data is a device for synchronizing a data lake with multi-source heterogeneous data, which can be implemented by software and / or hardware. The device for synchronizing a data lake with multi-source heterogeneous data in this embodiment can be configured in an electronic device. In this example embodiment, the electronic device may include devices such as terminal devices and servers, and this embodiment does not limit the electronic device. In one embodiment of the present invention, the above method can be applied to a unified data management platform.

[0046] In one embodiment of the present invention, the data interface specification may be negotiated and defined by each upstream system. Also, in one embodiment of the present invention, the data interface specification may include: data format, transmission protocol and interface calling method.

[0047] Also, in one embodiment of the present invention, the above-mentioned interactive data task table may include: primary key, source table name, source table corresponding id, business serial number, source system name, calling system name, calling system interface name, task status, calling time, interface return information, number of calls, remarks, insertion time, and update time.

[0048] In one embodiment of the present invention, a defined number may be used to represent the task status. For example, if the task status is 1, it means the task status is successful; if the task status is 0, it means the task status is failed; if the task status is 5, it means the task status is being pushed; if the task status is 9, it means the task status is to be pushed.

[0049] S2, based on the data interface specifications and the interactive data task table of each system, obtain the original data of each system;

[0050] In one embodiment of the present invention, after obtaining the data interface specification and the interactive data task table of each system through the above steps, the original data of each system can be obtained based on the data interface specification and the interactive data task table of each system.

[0051] Specifically, in one embodiment of the present invention, data in the interactive data task table can be periodically obtained from each upstream system through the data format, transmission protocol and interface calling method defined in the data interface specification, and the data in the interactive data task table can be determined as the original data of each system.

[0052] For example, the original data of interactive data task table 1 is obtained from upstream system 1 and written into interactive data task table 1; the original data of interactive data task table 2 is captured from upstream system 2 and written into interactive data task table 2.

[0053] S3, preprocessing the original data of each system to obtain preprocessed data of each system;

[0054] In one embodiment of the present invention, after the original data of each system is obtained through the above steps, the original data of each system can be preprocessed to obtain preprocessed data of each system to ensure the accuracy and consistency of the data.

[0055] In one embodiment of the present invention, the method of preprocessing the raw data of each system to obtain the preprocessed data of each system may include the following steps:

[0056] S31, performing data cleaning on the original data of each system to obtain the first data of each system;

[0057] S32, performing format conversion processing on the first data of each system to obtain the second data of each system;

[0058] S33, performing error correction on the second data of each system to obtain processed data of each system.

[0059] Among them, in one embodiment of the present invention, it is possible to check whether there are identical records in the original data of each system. If there are identical records, one of the duplicate records is deleted to complete data cleaning of the original data of each system to obtain the first data of each system. And, in one embodiment of the present invention, the first data of each system can be format converted to obtain the second data of each system, so that the data format in the second data is unified.

[0060] Furthermore, in one embodiment of the present invention, the above-mentioned method of performing error correction on the second data of each system to obtain the processed data of each system may include: performing error correction on the second data of each system by processing the missing values ​​of the second data of each system and correcting the abnormal values ​​of the original data to obtain the processed data of each system.

[0061] Furthermore, in one embodiment of the present invention, missing values ​​in the original data can be processed by filling (such as filling with the mean, median or mode) or deleting records containing missing values. For example, assuming that the "annual income" field of the target customer is empty, the mean of the annual income of the target customer with the same occupation and work experience as the target customer can be filled. In another embodiment of the present invention, outliers in the original data can be identified by statistical methods (such as the IQR rule or Z-score), and the outliers can be corrected. For example, assuming that the "age" field is identified as 300, the "age" field is corrected to a reasonable value.

[0062] S4, synchronizes the pre-processed data from each system to the data lake.

[0063] Among them, in one embodiment of the present invention, after obtaining the data pre-processed by each system through the above steps, the data pre-processed by each system can be synchronized to the data lake.

[0064] Furthermore, in one embodiment of the present invention, the monitoring and alarming in the existing systems are designed for specific scenarios, lacking flexibility and scalability. When business requirements change, each system needs to be reconfigured and adjusted, increasing maintenance costs. Based on this, in one embodiment of the present invention, in the process of synchronizing the multi-source heterogeneous data of the above-mentioned systems to the data lake, the above-mentioned method also includes: determining the alarm rules corresponding to each system, and monitoring and warning the process of synchronizing the data of each system to the data lake based on the alarm rules.

[0065] Specifically, in one embodiment of the present invention, the method for determining the alarm rules corresponding to each system may include: determining the monitoring indicators, alarm thresholds, and alarm methods corresponding to each system. In one embodiment of the present invention, the monitoring indicators, alarm thresholds, and alarm methods corresponding to each system may be determined according to user needs.

[0066] And, in one embodiment of the present invention, the monitoring indicators may include the interface task status as failed and the number of calls. In one embodiment of the present invention, the potential problems of each system can be discovered in time by real-time monitoring of the changes in the monitoring indicators.

[0067] Furthermore, in one embodiment of the present invention, the user can set the alarm threshold and alarm method (such as email or text message or internal communication software of the enterprise) according to actual needs. In one embodiment of the present invention, when the monitoring index exceeds the alarm threshold or is the same as the alarm threshold, the alarm method can be automatically triggered to notify the relevant personnel of the system. In one embodiment of the present invention, the alarm threshold can correspond to the monitoring index one by one.

[0068] For example, in one embodiment of the present invention, it is assumed that the monitoring indicators corresponding to the upstream system 1 are the task status and the number of calls, and the alarm threshold of the task status is failure, the alarm threshold of the number of calls is 5, and the alarm rule is to notify the person in charge of the above system 1 by email; the monitoring indicator corresponding to the upstream system 2 is the task status, and the alarm threshold of the task status is to be pushed, and the alarm rule is to notify the person in charge of the above system 2 by email and SMS.

[0069] It should be noted that in one embodiment of the present invention, the process of synchronizing data lakes of various system data is monitored and warned based on alarm rules, and the configuration of multiple different alarm rules is supported, which is convenient for users to issue warnings for various conditions according to actual needs, solves the lack of flexibility in monitoring and alarming of various systems, and reduces maintenance costs.

[0070] In the method for synchronizing multi-source heterogeneous data into a data lake proposed by the present invention, data interface specifications and interactive data task tables of each system are obtained; based on the data interface specifications and interactive data task tables of each system, the original data of each system is obtained; the original data of each system is preprocessed to obtain the preprocessed data of each system; and the preprocessed data of each system is synchronized to the data lake. Therefore, the present invention imports multi-source heterogeneous data into the data lake based on the data interface specifications and interactive data task tables of each system, reduces the difficulty and cost of integrating multi-source heterogeneous data, and improves the consistency and accuracy of the data.

[0071] In order to implement the above embodiment, Figure 2 As shown, this embodiment also provides a device 10 for synchronizing a data lake with multi-source heterogeneous data, the device comprising a first acquisition module 201, a second acquisition module 202, a processing module 203 and a synchronization module 204;

[0072] The first acquisition module 201 is used to acquire data interface specifications and interactive data task tables of each system;

[0073] The second acquisition module 202 is used to acquire the original data of each system based on the data interface specification and the interactive data task table of each system;

[0074] The processing module 203 is used to pre-process the raw data of each system to obtain the pre-processed data of each system;

[0075] The synchronization module 204 is used to synchronize the data pre-processed by each system to the data lake.

[0076] In one embodiment of the present invention, the above data interface specification includes: data format, transmission protocol and interface calling method.

[0077] Furthermore, in one embodiment of the present invention, the above-mentioned interactive data task table includes: primary key, source table name, source table corresponding id, business serial number, source system name, calling system name, calling system interface name, task status, calling time, interface return information, number of calls, remarks, insertion time, and update time.

[0078] Further, in one embodiment of the present invention, the processing module 203 is specifically used to:

[0079] Clean the original data of each system to obtain the first data of each system;

[0080] Performing format conversion processing on the first data of each system to obtain the second data of each system;

[0081] Error correction is performed on the second data of each system to obtain the data processed by each system.

[0082] Furthermore, in one embodiment of the present invention, the above device is also used for:

[0083] Determine the alarm rules corresponding to each system;

[0084] Monitor and issue early warnings on the process of data synchronization between each system and the data lake based on alarm rules.

[0085] Furthermore, in one embodiment of the present invention, the above-mentioned device is also used to: determine the monitoring indicators, alarm thresholds and alarm methods corresponding to each system.

[0086] In the device for synchronizing multi-source heterogeneous data into a data lake proposed by the present invention, data interface specifications and interactive data task tables of each system are obtained; based on the data interface specifications and interactive data task tables of each system, the original data of each system is obtained; the original data of each system is preprocessed to obtain the preprocessed data of each system; and the preprocessed data of each system is synchronized into the data lake. Therefore, the present invention imports multi-source heterogeneous data into the data lake based on the data interface specifications and interactive data task tables of each system, reduces the difficulty and cost of integrating multi-source heterogeneous data, and improves the consistency and accuracy of the data.

[0087] like Figure 3 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0088] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 308, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other electronic devices through a computer network such as the Internet and / or various telecommunication networks.

[0089] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as a method for synchronizing a data lake with multi-source heterogeneous data. For example, in some embodiments, the method for synchronizing a data lake with multi-source heterogeneous data may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the method for synchronizing a data lake with multi-source heterogeneous data described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to execute the method for synchronizing a data lake with multi-source heterogeneous data in any other appropriate manner (for example, by means of firmware).

[0090] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input system, and at least one output system, and transmit data and instructions to the storage system, the at least one input system, and the at least one output system.

[0091] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing system, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, system, or electronic device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or electronic device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage electronic device, a magnetic storage electronic device, or any suitable combination of the foregoing.

[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display system (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing system (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of systems can also be used to provide interaction with a user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0094] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0095] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0096] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0097] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for synchronizing a data lake with multi-source heterogeneous data, characterized in that: The method comprises: Obtain data interface specifications and interactive data task tables for each system; Based on the data interface specification and the interactive data task table of each system, obtaining the original data of each system; Preprocessing the original data of each system to obtain preprocessed data of each system; The pre-processed data of each system is synchronized to the data lake.

2. The method according to claim 1, characterized in that The data interface specification includes: data format, transmission protocol and interface calling method.

3. The method according to claim 1, characterized in that The interactive data task table includes: primary key, source table name, source table corresponding id, business serial number, source system name, calling system name, calling system interface name, task status, calling time, interface return information, number of calls, remarks, insertion time, and update time.

4. The method according to claim 1, characterized in that The preprocessing of the raw data of each system to obtain the preprocessed data of each system includes: Performing data cleaning on the original data of each system to obtain the first data of each system; Performing format conversion processing on the first data of each system to obtain the second data of each system; Error correction is performed on the second data of each system to obtain the data processed by each system.

5. The method according to claim 1, characterized in that The method further comprises: Determine the alarm rules corresponding to each system; Based on the alarm rules, the process of synchronizing the data lake with the system data is monitored and warned.

6. The method according to claim 1, characterized in that The obtaining of the alarm rules corresponding to each of the systems includes: determining the monitoring indicators, alarm thresholds and alarm methods corresponding to each of the systems.

7. A device for synchronizing data lakes with multi-source heterogeneous data, characterized in that: The device comprises: The first acquisition module is used to acquire data interface specifications and interactive data task tables of each system; A second acquisition module, used to acquire the original data of each system based on the data interface specification and the interactive data task table of each system; A processing module, used for preprocessing the raw data of each system to obtain the preprocessed data of each system; The synchronization module is used to synchronize the data pre-processed by each system to the data lake.

8. The device according to claim 7, characterized in that The interactive data task table includes: primary key, source table name, source table corresponding id, business serial number, source system name, calling system name, calling system interface name, task status, calling time, interface return information, number of calls, remarks, insertion time, and update time.

9. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

10. A computer storage medium, wherein: The computer storage medium stores computer executable instructions; after the computer executable instructions are executed by the processor, the method according to any one of claims 1 to 6 can be implemented.

Images