Heterogeneous data conversion method and device, equipment and medium

Abstract

The invention discloses a heterogeneous data conversion method, a heterogeneous data conversion device, heterogeneous data conversion equipment and a medium, the method is applied to a heterogeneous data system, the heterogeneous data system comprises a connector and a plurality of source databases, all the source databases are in interactive connection with the connector, library information of each source database is acquired by the heterogeneous data system, strategy decision is carried out on all the library information, and the source databases are connected with the connector. Obtaining a target connector strategy corresponding to each source database; obtaining source end data of the corresponding source database through the connector according to each target connector strategy; and performing data conversion processing on the corresponding source end data through the connector according to each target connector strategy to obtain conversion data corresponding to each piece of source end data. According to the method, the application range of heterogeneous data conversion can be effectively expanded, and the secondary development amount of a heterogeneous system and the cost required by heterogeneous data conversion are reduced. The invention relates to the technical field of data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method, apparatus, device, and medium for converting heterogeneous data. Background Art

[0002] With the expansion of the scale of enterprise informatization, business systems in different periods of an enterprise may be built based on different operating systems and database platforms, which leads to the need for the enterprise to perform heterogeneous data conversion and transmission between different systems, so as to achieve data interaction notification between heterogeneous systems.

[0003] Currently, existing methods for heterogeneous data conversion and transmission usually develop a corresponding data conversion tool for each heterogeneous database in a heterogeneous system, and implement data conversion and transmission between the corresponding heterogeneous database in the heterogeneous system and the target database through each data conversion tool. This method has a small scope of application, a large amount of secondary development for heterogeneous systems, and a high cost.

[0004] Therefore, the problems existing in the prior art still need to be solved and optimized urgently. Summary of the Invention

[0005] An object of the present invention is to solve at least to some extent one of the technical problems existing in the related art.

[0006] To this end, an object of an embodiment of the present invention is to provide a method, apparatus, device, and medium for converting heterogeneous data. Among them, the method can effectively improve the applicable range of converting heterogeneous data, and reduce the amount of secondary development for heterogeneous systems and the cost required for heterogeneous data conversion.

[0007] To achieve the above technical object, the technical solutions adopted in the embodiments of the present application include:

[0008] In a first aspect, an embodiment of the present application provides a method for converting heterogeneous data, which is applied to a heterogeneous data system. The heterogeneous data system includes a connector and a plurality of source databases, and all the source databases are interconnected with the connector. The method includes:

[0009] Obtain the library information of each source database, and perform policy decision on all the library information to obtain a target connector policy corresponding to each source database. The target connector policy is used to indicate the data collection policy that the connector needs to execute. The target connector policy includes at least one of a timed collection policy, a listening collection policy, an interface collection policy, or a queue collection policy;

[0010] Through the connector, obtain the source-side data of the corresponding source database according to each target connector policy;

[0011] The corresponding source - end data is subjected to data conversion processing according to each of the target connector policies through the connector, and conversion data corresponding to each of the source - end data is obtained.

[0012] In addition, according to the method of the above - mentioned embodiments of the present application, the following additional technical features may also be included:

[0013] Further, in an embodiment of the present application, obtaining the source - end data of the source database according to the target connector policy includes:

[0014] If the target connector policy is the timed acquisition policy, data of the source database is acquired at regular intervals according to the timed acquisition policy to obtain the source - end data;

[0015] Or,

[0016] If the target connector policy is the listening acquisition policy, data of the source database is acquired by listening according to the listening acquisition policy to obtain the source - end data;

[0017] Or,

[0018] If the target connector policy is the interface acquisition policy, data of the source database is acquired through the data interface according to the interface acquisition policy to obtain the source - end data;

[0019] Or,

[0020] If the target connector policy is the queue acquisition policy, data of the source database is acquired through the data queue according to the queue acquisition policy to obtain the source - end data.

[0021] Further, in an embodiment of the present application, the acquiring data of the source database at regular intervals according to the timed acquisition policy to obtain the source - end data includes:

[0022] Obtaining the acquisition interval time in the timed acquisition policy and the acquisition waiting time of the connector;

[0023] Performing timed verification on the acquisition interval time according to the acquisition waiting time to obtain a timed verification result;

[0024] If the timed verification result is that the acquisition waiting time reaches the acquisition interval time, the source database is arranged for acquisition to obtain the source - end data;

[0025] The acquiring data of the source database by listening according to the listening acquisition policy to obtain the source - end data includes:

[0026] According to the monitoring and collection strategy, obtain database monitoring information, where the database monitoring information is used to indicate whether the data in the source database has changed;

[0027] If the database monitoring information indicates that the data in the source database has changed, then according to the database monitoring information, obtain the corresponding changed data in the source database;

[0028] Perform data parsing on the changed data to obtain the source data.

[0029] Further, in an embodiment of the present application, the obtaining the source data by performing data interface collection on the source database according to the interface collection strategy includes:

[0030] According to the interface collection strategy, obtain the interface data at the message interface of the source database;

[0031] Perform data processing on the interface data to obtain the source data;

[0032] The obtaining the source data by performing data queue collection on the source database according to the queue collection strategy includes:

[0033] Obtain a preset standard parsing format;

[0034] According to the queue collection strategy, obtain the queue data at the message queue of the source database;

[0035] According to the standard parsing format, perform format parsing on the queue data to obtain the source data.

[0036] Further, in an embodiment of the present application,

[0037] The performing data conversion processing on the source data according to the target connector strategy to obtain conversion data corresponding to the source data includes:

[0038] Use the source data corresponding to the first intermediate strategy as the first data, and obtain the first mapping configuration table of the first data, where the first intermediate strategy is a timed collection strategy or a monitoring collection strategy;

[0039] Perform data cleaning on the first data to obtain second data;

[0040] According to the first mapping configuration table, perform data mapping configuration on the second data to obtain the conversion data.

[0041] Further, in an embodiment of the present application, the performing data conversion processing on the source data according to the target connector strategy to obtain conversion data corresponding to the source data includes:

[0042] Use the source data corresponding to the second intermediate policy as the third data, and obtain the second mapping configuration table of the third data, where the second intermediate policy is an interface collection policy or a queue collection policy;

[0043] According to the second mapping configuration table, perform data mapping adaptation on the third data to obtain the converted data.

[0044] Further, in an embodiment of the present application, the method further includes:

[0045] Obtain a routing configuration table corresponding to the converted data, where the routing configuration table records several topic queue information corresponding to the target database;

[0046] According to the routing configuration table, send the converted data to the corresponding topic queue, so that the target database consumes and subscribes to the converted data in the topic queue.

[0047] In a second aspect, an embodiment of the present application provides a heterogeneous data conversion device, which is applied to a heterogeneous data system. The heterogeneous data system includes a connector and several source databases, and all the source databases are interconnected with the connector. The device includes:

[0048] A first processing unit, configured to obtain the library information of each source database, and perform policy decision on all the library information to obtain a target connector policy corresponding to each source database. The target connector policy is used to indicate the data collection policy that the connector needs to execute, and the target connector policy includes at least one of a timing collection policy, a listening collection policy, an interface collection policy, or a queue collection policy;

[0049] A second processing unit, configured to obtain the source data of the corresponding source database through the connector according to each target connector policy;

[0050] A third processing unit, configured to perform data conversion processing on the corresponding source data through the connector according to each target connector policy to obtain converted data corresponding to each source data.

[0051] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0052] At least one processor;

[0053] At least one memory, configured to store at least one program;

[0054] When the at least one program is executed by the at least one processor, the at least one processor implements the method of the first aspect above.

[0055] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a program executable by a processor is stored. The program executable by the processor is used to implement the method of the first aspect above when executed by the processor.

[0056] The advantages and beneficial effects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood through the practice of the present application:

[0057] A method, device, equipment and medium for converting heterogeneous data disclosed in an embodiment of the present application. Among them, the method is applied to a heterogeneous data system, and the heterogeneous data system includes a connector and a plurality of source databases. All the source databases are interconnected with the connector. It obtains the library information of each source database, and makes a policy decision on all the library information to obtain a target connector policy corresponding to each source database. The target connector policy is used to indicate the data acquisition policy that the connector needs to execute. The target connector policy includes at least one of a timed acquisition policy, a listening acquisition policy, an interface acquisition policy or a queue acquisition policy; through the connector, according to each target connector policy, the source-end data of the corresponding source database is obtained; through the connector, according to each target connector policy, the source-end data is subjected to data conversion processing to obtain conversion data corresponding to each source-end data. This method uses the target connector policy to instruct a connector to implement the conversion of source-end data in multiple source databases, which can effectively improve the applicable range of converting heterogeneous data, reduce the secondary development volume of heterogeneous systems and the cost required for heterogeneous data conversion. Description of the Drawings

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following introduces the drawings of the related technical solutions in the embodiments of the present application or the prior art. It should be understood that the drawings introduced below are only for conveniently and clearly expressing some embodiments of the technical solutions in the present application. For those skilled in the art, without creative efforts, other drawings can also be obtained according to these drawings.

[0059] Figure 1 It is a schematic flowchart of a method for converting heterogeneous data provided by an embodiment of the present application;

[0060] Figure 2 It is a schematic structural diagram of a device for converting heterogeneous data provided by an embodiment of the present application;

[0061] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present application. Specific implementation manners

[0062] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals indicate the same or similar elements or elements with the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and should not be construed as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation and illustration, and no limitation is imposed on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0064] Currently, existing heterogeneous data conversion and transmission methods usually develop a corresponding data conversion tool for each heterogeneous database in a heterogeneous system, and realize the data conversion and transmission between the corresponding heterogeneous database in the heterogeneous system and the target database through each data conversion tool. However, the secondary development amount required for developing a corresponding data conversion tool for each heterogeneous database in the heterogeneous database is relatively large and the cost is relatively high. In addition, since each data conversion tool can only realize the data conversion between the corresponding heterogeneous database and the target database, it often still needs to re-develop the corresponding data conversion tool when facing different heterogeneous systems, and the applicable range of this method is relatively small.

[0065] In view of this, the embodiments of the present invention provide a method, device, equipment and medium for converting heterogeneous data. Among them, the method first determines the target connector policy corresponding to each source database in the heterogeneous system, and indicates a connector through the target connector policy to implement the conversion of the source-side data in multiple source databases. It can realize the data conversion between multiple heterogeneous databases and the target database through one connector, effectively improving the applicable range of converting heterogeneous data, and reducing the secondary development amount of the heterogeneous system and the cost required for heterogeneous data conversion.

[0066] Referring to Figure 1 , in the embodiments of the present application, a method for converting heterogeneous data is applied to a heterogeneous data system. The heterogeneous data system includes a connector and a plurality of source databases. All the source databases are interconnected with the connector. The method includes:

[0067] Step 110: Obtain the library information of each of the source databases, and perform policy decision-making on all the library information to obtain a target connector policy corresponding to each of the source databases. The target connector policy is used to indicate the data collection policy that the connector needs to execute, and the target connector policy includes at least one of a timed collection policy, a listening collection policy, an interface collection policy, or a queue collection policy;

[0068] In an embodiment of the present application, the source database may be the business database of one of the business systems in a heterogeneous data system. Among them, the source database may be any one of a MySQL database, a PostgreSQL database, a MongoDB database, an Oracle database, a SQL Server database, etc. The library information may be the database type of the source database or the behavior pattern corresponding to the source database, etc.; the connector may be a data conversion tool for converting all heterogeneous databases in a heterogeneous system, and this connector may be built based on Kafka middleware, Spring Boot, and Debezium. The examples in the present application are only for illustration.

[0069] It can be understood that for the library information of a certain source database, Step 110 may be to determine a target connector policy corresponding to the source database based on the library information. There are various specific ways to determine the target connector policy. For example, policy decision-making is performed on the library information based on a preset policy mapping table. This policy mapping table records multiple database types and the connector policies corresponding to each database type. When the library information meets one of the database types in the policy mapping table, the corresponding connector policy is determined as the target connector policy. The same applies to the remaining library information, and it can be simply inferred. In addition, the timed collection policy, the listening collection policy, the interface collection policy, and the queue collection policy may be preset data collection policies for the connector to execute.

[0070] Step 120: Through the connector, obtain the source-side data of the corresponding source database according to each target connector policy;

[0071] In an embodiment of the present application, the connector in the heterogeneous data system may, according to each obtained target connector policy, execute the corresponding target connector policy on each source database to obtain the source-side data of each source database.

[0072] In some embodiments, Step 120: Obtain the source-side data of the source database according to the target connector policy, including:

[0073] A1. If the target connector policy is the timed collection policy, then perform timed data collection on the source database according to the timed collection policy to obtain the source-side data;

[0074] Further, step A1, according to the timing collection strategy, perform data timing collection on the source database to obtain the source-side data, including:

[0075] A11. Obtain the collection interval time in the timing collection strategy and the collection waiting time of the connector;

[0076] A12. Perform timing verification on the collection interval time according to the collection waiting time to obtain a timing verification result;

[0077] A13. If the timing verification result is that the collection waiting time reaches the collection interval time, perform orchestrated collection on the source database to obtain the source-side data;

[0078] In the embodiment of the present application, if the target connector strategy corresponding to a certain source database is the timing collection strategy, the connector can first obtain the collection interval time indicated in the timing collection strategy. The specific value of this collection interval time can be set according to actual situations, and the current collection waiting time of the connector, which corresponds to this source database and is used to represent the time difference from the previous orchestrated collection moment to the current time.

[0079] It can be understood that step A12 can be to compare the magnitudes of the collection waiting time and the collection interval time. Specifically, if the collection waiting time is less than the collection interval time, a timing verification result indicating that the collection waiting time has not reached the collection interval time can be obtained, and at this time, step A11 can be returned for execution; or, if the collection waiting time is greater than or equal to the collection interval time, a timing verification result indicating that the collection waiting time has reached the collection interval time can be obtained, and at this time, orchestrated collection can be performed on this source database. Specifically, based on the Spring Boot technology, several data tables in the source database can be periodically extracted in the form of a time stamp through a cron expression, and then all the collected data tables can be aggregated to obtain the source-side data.

[0080] Or, A2. If the target connector strategy is the listening collection strategy, perform data listening collection on the source database according to the listening collection strategy to obtain the source-side data;

[0081] Further, the step A2, perform data listening collection on the source database according to the listening collection strategy to obtain the source-side data, including:

[0082] A21. Obtain database listening information according to the listening collection strategy, and the database listening information is used to indicate whether the data in the source database has changed;

[0083] A22. If the database monitoring information indicates that the data in the source database has changed, obtain the corresponding changed data in the source database according to the database monitoring information.

[0084] A23. Parse the changed data to obtain the source-side data.

[0085] In the embodiment of the present application, if the target connector strategy corresponding to a certain source database is the monitoring and collection strategy, the connector can first monitor whether the data in the source database has changed according to the monitoring and collection strategy. Specifically, it can monitor whether the data, memory cache, message queue, etc. in the source database have changed through Debezium, and generate corresponding database monitoring information. Specifically, if the database monitoring information indicates that the data in the source database has not changed, return to step A21 for execution; or, if the database monitoring information indicates that the data in the source database has changed, the connector can obtain the changed data of the source database based on the data indicated as changed in the database monitoring information, and then parse the changed data of the table type, resource type, activity type, etc. There are already many specific data parsing methods to obtain the source-side data.

[0086] Or, A3. If the target connector strategy is the interface collection strategy, perform data interface collection on the source database according to the interface collection strategy to obtain the source-side data.

[0087] Further, the step A3 of performing data interface collection on the source database according to the interface collection strategy to obtain the source-side data includes:

[0088] A31. Obtain the interface data at the message interface of the source database according to the interface collection strategy.

[0089] A32. Process the interface data to obtain the source-side data.

[0090] In the embodiment of the present application, the source database message interface can be the Rest interface of the source database. Specifically, if the target connector strategy corresponding to a certain source database is the interface collection strategy, step A31 can be that when the source database pushes the interface data to the message interface, the connector obtains and processes the interface data based on the RestTemplate tool class to obtain the source-side data of the source database.

[0091] Or, A4. If the target connector strategy is the queue collection strategy, perform data queue collection on the source database according to the queue collection strategy to obtain the source-side data.

[0092] Further, step A4, performing data queue collection on the source database according to the queue collection strategy to obtain the source-side data, includes:

[0093] A41. Obtaining a preset standard parsing format;

[0094] A42. Obtaining queue data at the message queue of the source database according to the queue collection strategy;

[0095] A43. Parsing the format of the queue data according to the standard parsing format to obtain the source-side data.

[0096] In the embodiment of the present application, if the target connector strategy corresponding to a certain source database is the queue collection strategy, the connector can first obtain the standard parsing format corresponding to the source database according to the queue collection strategy. The data formats indicated by the standard parsing format can be various, such as json format, XML format, CSV format, etc. The embodiment of the present application takes the standard parsing format as json format as an example; then, the connector obtains the queue data pushed by the source database to the message queue through the kafka middleware, and performs data parsing on the queue data, thereby parsing the queue data into json format data and determining the json format data as the source-side data.

[0097] Step 130. Performing data conversion processing on the corresponding source-side data according to each target connector strategy through the connector to obtain conversion data corresponding to each source-side data.

[0098] In the embodiment of the present application, after the connector obtains the source-side data in the corresponding source database based on each target connector strategy, the connector can also perform data conversion on the corresponding source-side data based on each target connector strategy, thereby obtaining conversion data corresponding to each source-side data. Specifically, the data conversion performed by the connector on the source-side data of different source databases can be executed in parallel or cascaded. For example, the source-side data of each source database can be distributed and divided into different execution columns, and then the source-side data in each execution column is parallelly subjected to data conversion according to the target connector strategy corresponding to each execution column; or based on the timestamp of the source-side data, the source-side data of each source database can be sequentially divided into the same execution column, and then the conversion data corresponding to each source-side data is sequentially obtained based on the target connector strategy corresponding to each source-side data.

[0099] In some embodiments, step 130, performing data conversion processing on the source-side data according to the target connector strategy to obtain conversion data corresponding to the source-side data, includes:

[0100] B1. Use the source data corresponding to the first intermediate policy as the first data, and obtain the first mapping configuration table of the first data, where the first intermediate policy is a timed collection policy or a listening collection policy;

[0101] B2. Clean the first data to obtain the second data;

[0102] B3. According to the first mapping configuration table, perform data mapping configuration on the second data to obtain the converted data.

[0103] In the embodiments of the present application, for a certain source data, if the target connector policy is a timed collection policy or a listening collection policy, first, the first mapping configuration table of the first data can be obtained, and the first mapping configuration table records the target topic configuration information of the first data; then, the first data is cleaned through the connector of the heterogeneous data system. Specifically, cleaning operations such as data desensitization and data fuzzification can be performed on the first data to obtain the second data.

[0104] It can be understood that step B3 can be that the connector in the heterogeneous data system reads the first mapping configuration table, and based on the target topic configuration information in the first mapping configuration table, maps the data format of the second data into the topic format required by the target topic, so as to obtain the converted data.

[0105] In some embodiments, step 130. According to the target connector policy, perform data conversion processing on the source data to obtain the converted data corresponding to the source data, including:

[0106] C1. Use the source data corresponding to the second intermediate policy as the third data, and obtain the second mapping configuration table of the third data, where the second intermediate policy is an interface collection policy or a queue collection policy;

[0107] C2. According to the second mapping configuration table, perform data mapping adaptation on the third data to obtain the converted data.

[0108] In the embodiments of the present application, for a certain source data, if the target connector policy is an interface collection policy or a queue collection policy, first, the second mapping configuration table of the third data can be obtained, and the second mapping configuration table is similar to the aforementioned first mapping configuration table and can be simply analogized. Step C2 can be that the connector reads the second mapping configuration table, and based on the target topic configuration information in the second mapping configuration table, maps the data format of the third data into the topic format required by the target topic, so as to obtain the converted data.

[0109] In some embodiments, the method further includes:

[0110] D1. Obtain a routing configuration table corresponding to the conversion data, where the routing configuration table records several topic queue information corresponding to the target database;

[0111] D2. According to the routing configuration table, send the conversion data to the corresponding topic queue, so that the target database consumes and subscribes to the conversion data in the topic queue.

[0112] In the embodiment of the present application, the heterogeneous data system can also obtain a routing configuration table corresponding to the conversion data, and then read the routing configuration table through the connector and send the conversion data to the corresponding topic queue according to the topic queue information in the routing configuration table. Specifically, for a certain conversion data, the topic queue information in the routing configuration table can be compared based on the topic format recorded in the conversion data, so as to determine the topic queue corresponding to the conversion data, and send the conversion data to the corresponding topic queue through the connector.

[0113] Next, a conversion device for heterogeneous data proposed according to an embodiment of the present application will be described in detail with reference to the accompanying drawings.

[0114] Refer to Figure 2 , a conversion device for heterogeneous data proposed in the embodiment of the present application is applied to a heterogeneous data system, and the heterogeneous data system includes a connector and several source databases, and all the source databases are interconnected with the connector, including:

[0115] The first processing unit 101 is used to obtain the library information of each source database, and perform policy decision on all the library information to obtain a target connector policy corresponding to each source database, where the target connector policy is used to indicate the data collection policy that the connector needs to execute, and the target connector policy includes at least one of a timing collection policy, a listening collection policy, an interface collection policy, or a queue collection policy;

[0116] The second processing unit 102 is used to obtain the source-side data of the corresponding source database through the connector according to each target connector policy;

[0117] The third processing unit 103 is used to perform data conversion processing on the corresponding source-side data through the connector according to each target connector policy to obtain conversion data corresponding to each source-side data.

[0118] It can be understood that the content in the above method embodiments is applicable to the device embodiments of the present application. The functions specifically implemented by the device embodiments of the present application are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0119] Refer to Figure 3, embodiments of the present application further provide an electronic device, including:

[0120] At least one processor 201;

[0121] At least one memory 202, configured to store at least one program;

[0122] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiments.

[0123] Similarly, it can be understood that the content in the above method embodiments is applicable to the embodiments of this device. The functions specifically implemented by the embodiments of this device are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0124] Embodiments of the present application further provide a computer-readable storage medium, in which a program executable by a processor 201 is stored. The program executable by the processor 201 is used to implement the above method embodiments when executed by the processor 201.

[0125] Similarly, the content in the above method embodiments is applicable to the embodiments of this computer-readable storage medium. The functions specifically implemented by the embodiments of this computer-readable storage medium are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those of the above method embodiments.

[0126] In some alternative embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are foreseeable, in which the order of various operations is changed and the sub-operations described as part of a larger operation are executed independently.

[0127] In addition, although the present application has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It should also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present application. Rather, given the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Thus, those skilled in the art can implement the present application as set forth in the claims without undue experimentation. It should also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0128] If a function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method according to the embodiments of the present application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0129] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions), or in conjunction with such instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0130] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable media can even be paper or other suitable media on which a program can be printed, as the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0131] It should be understood that various parts of the present application can be implemented by hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0132] In the foregoing description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example", or "certain embodiments / examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0133] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0134] The above has specifically described the preferred embodiments of the present application, but the present application is not limited to the embodiments. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present application, and these equivalent deformations or substitutions are all included within the scope defined by the claims of the present application.

Claims

1. A method for converting heterogeneous data, characterized in that: Applied to a heterogeneous data system, the heterogeneous data system includes a connector and a plurality of source databases, all of the source databases are interactively connected with the connector, and the method includes: Obtaining the library information of each of the source databases, and making a policy decision on all of the library information to obtain a target connector policy corresponding to each of the source databases, wherein the target connector policy is used to indicate the data collection policy to be executed by the connector, and the target connector policy includes at least one of a timing collection policy, a monitoring collection policy, an interface collection policy, or a queue collection policy; Obtaining the corresponding source data of the source database through the connector according to each target connector strategy; The connector performs data conversion processing on the corresponding source data according to each target connector strategy to obtain conversion data corresponding to each source data.

2. The method according to claim 1, characterized in that: According to the target connector strategy, the source end data of the source database is obtained, including: If the target connector strategy is the timed collection strategy, then according to the timed collection strategy, data is collected on the source database at a timed basis to obtain the source end data; or, If the target connector strategy is the monitoring collection strategy, data monitoring and collection are performed on the source database according to the monitoring collection strategy to obtain the source end data; or, If the target connector strategy is the interface acquisition strategy, data interface acquisition is performed on the source database according to the interface acquisition strategy to obtain the source end data; or, If the target connector strategy is the queue collection strategy, data queue collection is performed on the source database according to the queue collection strategy to obtain the source end data.

3. The method according to claim 2, characterized in that The step of performing data collection on the source database according to the timed collection strategy to obtain the source end data includes: Obtaining the collection interval time in the timing collection strategy and the collection waiting time of the connector; According to the collection waiting time, the collection interval time is regularly verified to obtain a regular verification result; If the timing verification result is that the collection waiting time reaches the collection interval time, the source database is arranged for collection to obtain the source end data; The step of performing data monitoring and collection on the source database according to the monitoring and collection strategy to obtain the source end data includes: According to the monitoring collection strategy, database monitoring information is obtained, where the database monitoring information is used to indicate whether the data in the source database has changed; If the database monitoring information indicates that data in the source database has changed, then obtaining corresponding changed data in the source database according to the database monitoring information; The changed data is parsed to obtain the source data.

4. The method according to claim 2, characterized in that: The step of performing data interface acquisition on the source database according to the interface acquisition strategy to obtain the source end data includes: According to the interface collection strategy, acquiring interface data at the source database message interface; Processing the interface data to obtain the source data; The step of performing data queue collection on the source database according to the queue collection strategy to obtain the source end data includes: Get the preset standard parsing format; According to the queue collection strategy, obtain queue data at the source database message queue; According to the standard parsing format, the queue data is format parsed to obtain the source data.

5. The method according to claim 1, characterized in that According to the target connector strategy, the source data is converted to obtain conversion data corresponding to the source data, including: Using source data corresponding to a first intermediate strategy as first data, and acquiring a first mapping configuration table of the first data, wherein the first intermediate strategy is a timing collection strategy or a monitoring collection strategy; Performing data cleaning on the first data to obtain second data; According to the first mapping configuration table, data mapping configuration is performed on the second data to obtain the converted data.

6. The method according to claim 1, characterized in that According to the target connector strategy, the source data is converted to obtain conversion data corresponding to the source data, including: Using source data corresponding to the second intermediate strategy as third data, and acquiring a second mapping configuration table of the third data, wherein the second intermediate strategy is an interface collection strategy or a queue collection strategy; According to the second mapping configuration table, data mapping adaptation is performed on the third data to obtain the converted data.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining a routing configuration table corresponding to the conversion data, wherein the routing configuration table records information of several subject queues corresponding to the target database; According to the routing configuration table, the converted data is sent to the corresponding subject queue, so that the target database consumes and subscribes to the converted data in the subject queue.

8. A heterogeneous data conversion device, characterized in that: Applied to a heterogeneous data system, the heterogeneous data system includes a connector and a plurality of source databases, all of the source databases are interactively connected with the connector, and the device includes: A first processing unit is used to obtain the library information of each of the source databases, and to make a policy decision on all the library information to obtain a target connector policy corresponding to each of the source databases, wherein the target connector policy is used to indicate the data collection policy to be executed by the connector, and the target connector policy includes at least one of a timing collection policy, a monitoring collection policy, an interface collection policy, or a queue collection policy; A second processing unit, configured to obtain source-end data of the corresponding source database through the connector according to each target connector strategy; The third processing unit is used to perform data conversion processing on the corresponding source data through the connector according to each target connector strategy to obtain conversion data corresponding to each source data.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images