Efficient data exchange method based on heterogeneous structured database

Abstract

The invention relates to an efficient data exchange method based on a heterogeneous structured database, and belongs to the technical field of data processing, and the method comprises the steps that a data migration instruction is acquired, and the data migration instruction at least comprises a data source library, a target database and target data content; acquiring reference data content corresponding to the target data content from the data source library, dividing the reference data content into a plurality of data packets, and establishing a data transmission channel which at least comprises a plurality of sub-channels corresponding to the data packets; transmitting the corresponding data packets from a data source library to a target database through the sub-channels; and integrating all the data packets transmitted to the target database to obtain target data content, and storing the target data content in the target database. The method has the effect of improving the data migration processing efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to an efficient data exchange method based on a heterogeneous structured database. Background Art

[0002] Big data is developing rapidly, and a large amount of data has been accumulated in various industries. Some of the data is stored in relational databases such as Oracle, SqlServer, and Mysql, some in non-relational databases such as redis and MongoDB, and some in files such as CSV and XML. In data processing, it is often necessary to migrate or convert data between heterogeneous databases, or save data from files or applications into databases.

[0003] However, the time required for data migration or conversion will increase with the increase in the amount of data to be migrated or converted. Therefore, when a large amount of data needs to be converted, the problem of low data migration efficiency will easily occur, which needs to be improved. Summary of the invention

[0004] In order to solve the technical problem of low data migration efficiency when a large amount of data needs to be converted, the present application provides an efficient data exchange method based on a heterogeneous structured database.

[0005] In the first aspect, the present application provides an efficient data exchange method based on a heterogeneous structured database, which adopts the following technical solution: Obtaining a data migration instruction, wherein the data migration instruction at least includes a data source library, a destination database, and target data content; Retrieving reference data content corresponding to the target data content from the data source library, dividing the reference data content into a plurality of data packets, establishing a data transmission channel, wherein the data transmission channel at least includes a plurality of sub-channels corresponding to the data packets; transmitting the corresponding data packets from the data source library to the destination database through the sub-channels; All data packets transmitted to the target database are integrated to obtain the target data content and then stored in the target database.

[0006] By adopting the above technical scheme, the target data content to be migrated and transmitted is divided into several data packets to reduce the amount of data transmitted in the data transmission channel, thereby improving the data transmission speed of a single data transmission channel, and then establishing several data transmission channels corresponding to the data packets, and transmitting the corresponding data packets through the data transmission channels respectively, so as to realize the synchronous and parallel transmission of the corresponding data packets by all data transmission channels, thereby improving the migration transmission efficiency of the target data content, and finally integrating the data packets transmitted to the target database to obtain the target data content.

[0007] Optionally, the data transmission channel further includes a transit database, and all sub-channels include a sub-channel for transmitting data packets to the transit database and a sub-channel for transmitting data packets output by the transit database; the sub-channel for transmitting data packets to the transit database is called an input channel, and the sub-channel for transmitting data packets output by the transit database is called an output channel; The transfer database corresponds to a transfer attribute, and the transfer attribute at least includes a data merging attribute; The method further comprises: If there is a first transit database whose transit attribute is a data merging attribute in the data transmission channel, the first input channel and the first output channel are determined, and the data packets corresponding to all the first input channels for transmitting the same target data content are merged to form a new data packet, and the merged new data packet is transmitted through the first output channel; wherein, the first input channel refers to an input channel among all sub-channels contained in the same data transmission channel that transmits data packets to the first transit database, and the first output channel refers to a sub-channel among all sub-channels contained in the same data transmission channel that is used to transmit data packets output by the first transit database.

[0008] By adopting the above technical solution, when using the data transmission channel to transmit data packets, several data packets can be merged and integrated in advance before being transmitted to the destination database. This operation can merge and integrate the data packets one by one during the transmission process, reducing the time consumed by centralized merging and integration when all data packets are transmitted to the target database. On the other hand, it can also enable some sub-channels to be used for the transmission of target data content contained in other data migration instructions through merging.

[0009] Optionally, the transfer attribute also includes a data diversion attribute; The method further comprises: If there is a second transit database whose transit attribute is a data diversion attribute in the data transmission channel, a second input channel and a second output channel are determined, and the data packets corresponding to the second input channel are split into several diversion data packets, and the diversion data packets are transmitted through the second output channel; wherein, the second input channel refers to an input channel of all sub-channels contained in the same data transmission channel, which transmits data packets to the second transit database, and the second output channel refers to a sub-channel of all sub-channels contained in the same data transmission channel, which is used to transmit data packets output by the second transit database; and the second output channel is set corresponding to the diversion data packets.

[0010] By adopting the above technical solution, the corresponding setting of the second output channel and the diverted data packet means that the number of the second output channels is consistent with the number of the diverted data packets and corresponds one to one, so that the corresponding diverted data packets can be transmitted through the second output channel; and the purpose of the second transit database is to further split the data packets and use multiple second output channels for transmission, so as to further improve the data transmission speed.

[0011] Optionally, the transfer attribute further includes a data backup attribute; the target data content is composed of the reference data content and the regular data content, and the regular data content is stored in a transfer database whose transfer attribute is a data backup attribute; The method further comprises: If there is a third transit database whose transit attribute is a data backup attribute in the data transmission channel, the third input channel and the third output channel are determined, the regular data content contained in the target data content is retrieved from the third transit database, the regular data content is merged with the data packet transmitted corresponding to the third input channel to form a new data packet, and the merged new data packet is transmitted through the third output channel; wherein the third input channel refers to an input channel among all sub-channels contained in the same data transmission channel that transmits data packets to the third transit database, and the third output channel refers to a sub-channel among all sub-channels contained in the same data transmission channel that is used to transmit data packets output by the third transit database.

[0012] By adopting the above-mentioned technical solution, the regular data content can be understood as the data that frequently appears in the target data content of different data migration instructions during the data exchange and migration process. Such data can be stored in the third transit database in advance. When the target data content needs to be migrated, the third transit database can be added to the data transmission channel. During the data transmission process, the regular data content and the benchmark data content can be merged to reduce the data transmission burden of some sub-channels.

[0013] Optionally, the transit attribute also includes a security verification attribute; The method further comprises: If there is a fourth transit database whose transit attribute is a data merging attribute in the data transmission channel, a fourth input channel and a fourth output channel are determined; whenever a data packet transmitted by a fourth input channel is added to the fourth transit database, a security analysis is performed on the data packet, and if the analysis result is normal, the data packet is transmitted through the fourth output channel; if the analysis result is abnormal, the transmission of the data packet is suspended, and data abnormality warning information is output; wherein, the fourth input channel refers to an input channel for transmitting data packets to the fourth transit database among all sub-channels contained in the same data transmission channel, and the fourth output channel refers to a sub-channel for transmitting data packets output by the fourth transit database among all sub-channels contained in the same data transmission channel.

[0014] By adopting the above technical solution, during the data transmission process, a fourth transit database can be set in the data transmission channel according to the importance of the data content, so as to perform security analysis on the data packet, such as analyzing whether the data content in the data packet is complete, whether there is any data loss, etc., and finally obtaining the analysis result. If the analysis result is normal, the subsequent transmission will continue. If the data result is abnormal, such as data loss, then it is necessary to output data abnormality warning information at this time, so as to promptly discover abnormal problems in the data transmission process.

[0015] Optionally, if the analysis result is abnormal, suspending the transmission of the data packet and outputting data abnormality warning information includes: If the analysis result is abnormal, the transmission of the data packet is suspended, the data packet with the abnormal analysis result is determined as an abnormal data packet, and a suspected abnormal channel is determined, and data warning information with the suspected abnormal channel is output; wherein the suspected abnormal channel refers to all sub-channels included in the data transmission channel that transmit the abnormal data packet to the fourth transit database.

[0016] By adopting the above technical solution, if there is an abnormal data packet with abnormal analysis results, then all sub-channels that have transmitted the abnormal data packet will have abnormal risks. Therefore, this application will screen out all suspected abnormal channels and add them to the data warning information to facilitate maintenance personnel to check the corresponding suspected abnormal channels.

[0017] Optionally, dividing the reference data content into a plurality of data packets includes: According to the data type of the benchmark data content, the data volume corresponding to each data type, and the pre-stored transmission speed corresponding to each data type, the benchmark data content is divided into a number of data packets, and it is satisfied that: the time difference between any two data packets transmitted to the destination database is within a specified difference range.

[0018] By adopting the above technical solution, the benchmark data content is divided into data packets according to the time when the data packets are transmitted to the destination database, so that the time when all data packets are transmitted to the destination database is close, and the specified difference range can be considered as the time spent on integrating the data packets, so as to achieve the purpose of improving the overall migration and transmission efficiency of the target data content.

[0019] In a second aspect, the present application provides an efficient data exchange system based on a heterogeneous structured database, comprising: A migration instruction acquisition module, used to acquire a data migration instruction, wherein the data migration instruction at least includes a data source library, a destination database, and target data content; A data parallel migration module is used to retrieve the reference data content corresponding to the target data content from the data source library, divide the reference data content into a plurality of data packets, establish a data transmission channel, and the data transmission channel includes at least a plurality of sub-channels corresponding to the data packets; and transmit the corresponding data packets from the data source library to the destination database through the sub-channels; The data integration storage module is used to integrate all data packets transmitted to the target database to obtain the target data content and then store it in the target database.

[0020] In a third aspect, the present application provides an efficient data exchange device based on a heterogeneous structured database, comprising a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute any one of the methods described in the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program that can be loaded by a processor and execute any one of the methods described in the first aspect.

[0022] In summary, this application includes the following beneficial technical effects: The target data content to be migrated and transmitted is divided into several data packets to reduce the amount of data transmitted in the data transmission channel, thereby improving the data transmission speed of a single data transmission channel, and then establishing several data transmission channels corresponding to the data packets. The corresponding data packets are transmitted through the data transmission channels respectively to achieve synchronous and parallel transmission of the corresponding data packets by all data transmission channels, thereby improving the migration transmission efficiency of the target data content. Finally, the data packets transmitted to the target database are integrated to obtain the target data content. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 It is a flow chart of an efficient data exchange method based on a heterogeneous structured database disclosed in an embodiment of the present application.

[0025] Figure 2 It is a schematic diagram of the relationship between the transit database and the sub-channels for reflecting different transit attributes in the embodiment of the present application.

[0026] Figure 3 It is a structural block diagram of an embodiment of the present application for embodying an efficient data exchange system based on a heterogeneous structured database.

[0027] Explanation of the accompanying drawings: 1. Migration instruction acquisition module; 2. Data parallel migration module; 3. Data integration storage module. DETAILED DESCRIPTION

[0028] The following is combined with Figure 1-3 This application is described in further detail.

[0029] The embodiment of the present application discloses an efficient data exchange method based on heterogeneous structured databases (hereinafter referred to as the data exchange method), which is used to realize data migration and transmission between different databases, and to improve the transmission efficiency during the data migration and transmission process; wherein the execution subject of the data exchange method is an efficient data exchange system based on heterogeneous structured databases (hereinafter referred to as the data exchange system), which will be described in conjunction with the attached Figure 1-2 The specific process steps of the data exchange method are specifically explained.

[0030] S101, obtaining a data migration instruction, wherein the data migration instruction at least includes a data source library, a destination database, and target data content.

[0031] In practice, the data exchange system can access all databases so as to view and retrieve data content from the corresponding databases. Users can access the data exchange system through a web page and click the data migration button preset on the data exchange system to trigger the data migration instruction. At this time, the data exchange system is used to display an input box for users to enter the data source library and the destination database, and after the user enters the database, the database is opened for the user to select the data to be migrated, that is, the target data content. Correspondingly, the data source library refers to the database where the target data content is originally located, that is, the database from which the data exchange system retrieves the target data content, and the destination database refers to the destination to which the target data content is ultimately transmitted.

[0032] S102, retrieve the benchmark data content corresponding to the target data content from the data source library, divide the benchmark data content into several data packets, establish a data transmission channel, and the data transmission channel includes at least several sub-channels corresponding to the data packets; transmit the corresponding data packets from the data source library to the destination database through the sub-channels.

[0033] S103, integrating all data packets transmitted to the target database to obtain target data content and storing it in the target database.

[0034] In implementation, the present application will split the benchmark data content to form several data packets. The data contents corresponding to all data packets are integrated to form the benchmark data content. In addition, the data exchange system will establish a data transmission channel between the data source library and the destination database. The data transmission channel may include several sub-channels corresponding to the data packets. The correspondence between the sub-channels and the data packets described in the embodiment of the present application is reflected in quantity, that is, the number of sub-channels is consistent with the number of data packets, and is a one-to-one correspondence, so that the corresponding data packets are transmitted separately through each sub-channel. By splitting the benchmark data content into multiple data packets for synchronous and parallel transmission, the problem of affecting the data transmission speed due to the large amount of data is solved, and the data transmission speed is improved.

[0035] Correspondingly, after the data packets are transmitted to the destination database, the complete target data content can be obtained again by integrating and merging all the data packets.

[0036] Optionally, “dividing the reference data content into a plurality of data packets” in S102 includes the following steps: According to the data type of the benchmark data content, the data volume corresponding to each data type, and the pre-stored transmission speed corresponding to each data type, the benchmark data content is divided into several data packets, and it meets the following requirements: the time difference between any two data packets transmitted to the destination database is within the specified difference range.

[0037] In implementation, the data exchange system stores the correspondence between the transmission speed and data volume of data of different data types. After receiving the target data content, the data exchange system will determine the transmission speed of each data type contained in the benchmark data content based on the aforementioned correspondence, as well as all data types contained in the benchmark data content and the data volume of each data type, and then divide the data packets according to the transmission speed. The division method can be: first determine whether the benchmark data content contains more than one data type. If not, based on the total data volume, divide the benchmark data content into more than two data packets, and the number of data packets is random and not less than 2.

[0038] If at least two data types are included, the common factor corresponding to the data volume of all data types is determined. If there is a common factor, the common factor is used as the data volume of a single data packet, and the data content of each data type is divided into a number of data packets with the common factor as the data volume; if there is no common factor, the data volume of the data type data is gradually accumulated by a preset unit data volume until a common factor exists, such as the data volumes of the two data types a1 and a2 are A1 and A2 respectively; let A1=A1+n*b; 1≤n≤N, b is the preset unit data volume; if there is no common factor between the accumulated A1 and the initial A2, then n+1, until the minimum common factor B1 is found when n≤N, at this time, the corresponding n is determined to be n', and the data packets are divided according to the common factor, wherein the number of data packets of a2 is a2 / B; and the data volume of each data packet divided by a2 is B; the number of data packets of a1 is (A1+ n'*b) / B, B is preferentially used as the data volume of a single data packet, and the data volume of the last data packet divided by a1 is B-n'*b.

[0039] If in the process of accumulating A1, the common factor B is not found when n≤N, then A2 is accumulated, and the lowest common factor B2 between the accumulated A2 and the initial A1 is found when n≤N. The number of data packets and the data volume of a single data packet are also determined according to the above method. In the embodiment of the present application, it is assumed that N can satisfy that in the process of accumulating A1 or A2, at least any one of the accumulation processes can find a common factor.

[0040] In summary, the above division is used to make the transmission speed of the data volume of each data packet approximate, and the present application assumes that the data transmission channel length of each data packet is approximate, so that the arrival time of each data packet when transmitted from the data source database to the destination database can be approximate, that is, the corresponding difference is within the specified difference range, so as to improve the overall transmission efficiency of the target data content.

[0041] It should be noted that whenever a data transmission channel is established, the data exchange system will store data migration instructions, each sub-channel in the corresponding data transmission channel, and the data packet transmitted by each sub-channel. The data packet refers to the data packet formed by retrieving the benchmark data content from the data original library and dividing it from the data exchange system.

[0042] Optionally, the “establishing a data transmission channel” in S102 further includes the following steps: The components of the target data content are determined, and if the target data content contains regular data content, a third transit database storing the regular data content is added to the data transmission channel.

[0043] In practice, after obtaining the target data content, the data exchange system will determine whether there is conventional data content in the target data content, where conventional data content refers to data content that has been retrieved from the data source library multiple times in the data migration instructions obtained in the historical period. If there is conventional data content, then the target data content is considered to be specifically composed of: benchmark data content + conventional data content, and the benchmark data content refers to the data content of non-conventional data content. If not, then the specific composition of the target data content is only the benchmark data content.

[0044] Correspondingly, the data exchange system also includes a transit database, and the transit database corresponds to a transit attribute, and the transit attribute includes at least a data backup attribute, that is, the data exchange system includes a transit database (hereinafter referred to as the third transit database) whose transit attribute is a data backup attribute. The third transit database stores all regular data, and the data exchange system can determine, at every specified period, data content that appears more than a specified number of times from the target data content corresponding to the data migration instruction obtained within the specified period, as regular data content, and add it to the third transit database for updating.

[0045] When establishing a data transmission channel, the data exchange system can make the data transmission channel include a sub-channel and a third transfer database, that is, Figure 2-1The data transmission channel shown can be considered as the transmission path of the data packet, and the transmission path is composed of a sub-channel, a third transit database, a data source library, and a destination database; and the third transit database is located between the data source library and the destination database, that is, during the data packet transmission process, the data packet will pass through the third transit database via the transmission of the sub-channel, and the present application proposes that when the data packet passes through the third transit database, the data exchange system will retrieve the regular data content contained in the target data content corresponding to the current data migration instruction from the third transit database, and merge and integrate the regular data content with the content in the data packet to form a new data packet, that is, during the transmission process, the regular data content is added to the data packet, rather than adding the corresponding regular data content to the data packet in the data source library, in this way to improve the transmission speed of the sub-channel before the regular data is merged into the data packet. Accordingly, this operation corresponds to the following steps: If there is a third transit database whose transit attribute is a data backup attribute in the data transmission channel, the third input channel and the third output channel are determined, the regular data content contained in the target data content is retrieved from the third transit database, the regular data content is merged with the data packet transmitted corresponding to the third input channel to form a new data packet, and the merged new data packet is transmitted through the third output channel; wherein, the third input channel refers to an input channel of all sub-channels contained in the same data transmission channel that transmits data packets to the third transit database, and the third output channel refers to a sub-channel of all sub-channels contained in the same data transmission channel that is used to transmit data packets output by the third transit database.

[0046] In implementation, among the sub-channels included in the data transmission channel, the sub-channel directly connected to the transit database and used to transmit data packets to the transit database is called an input channel, and the sub-channel directly connected to the transit database and used to transmit data packets output by the transit database is called an output channel. Figure 2-1 The input channel that is directly connected to the third transit database and the data packets transmitted belong to the same data migration instruction is the third input channel, and the output channel that is directly connected to the third transit database and the data packets transmitted belong to the same data migration instruction is the third output channel. Each data packet corresponds to a third input channel and a third output channel before and after being merged with the regular data content.

[0047] Optionally, the transfer attribute further includes a data merging attribute, that is, the data exchange system further includes a transfer database (hereinafter referred to as the first transfer database) whose transfer attribute is the data merging attribute; Accordingly, the “establishing a data transmission channel” in S102 also includes the following steps: If the number of sub-channels directly connected to the target database exceeds a preset number, the number of advance merges is determined based on a preset correspondence table, and a transit database consistent with the number of advance merges is added to the data transmission channel, and it satisfies: any transit database is connected to more than one sub-channel among the sub-channels directly connected to the target database; wherein the correspondence table includes at least the number of different sub-channels and the number of merges corresponding to each number of sub-channels.

[0048] Data exchange methods also include: If there is a first transit database whose transit attribute is a data merging attribute in the data transmission channel, the first input channel and the first output channel are determined, and the data packets corresponding to all the first input channels for transmitting the same target data content are merged to form a new data packet, and the merged new data packet is transmitted through the first output channel; wherein, the first input channel refers to an input channel among all sub-channels contained in the same data transmission channel that transmits data packets to the first transit database, and the first output channel refers to a sub-channel among all sub-channels contained in the same data transmission channel that is used to transmit data packets output by the first transit database.

[0049] In implementation, when a new data packet is added to the first transit database, the data exchange system will determine the first input channel according to the data migration instruction to which the data packet belongs and its corresponding data transmission channel, and when all data packets corresponding to the first input channel have reached the first transit database, the aforementioned data packets will be merged to form a new data packet, and then the corresponding first output channel will be used to transmit the data packet. Figure 2-2 The input channel that is directly connected to the first transit database and the data packets transmitted belong to the same data migration instruction is the first input channel, and the output channel that is directly connected to the first transit database and the data packets transmitted belong to the same data migration instruction is the first output channel.

[0050] Optionally, the transfer attribute further includes a data diversion attribute, that is, the data exchange system further includes a transfer database (hereinafter referred to as the second transfer database) whose transfer attribute is the data diversion attribute. Accordingly, the “establishing a data transmission channel” in S102 also includes the following steps: Based on the data volume of each data packet obtained by the division, if the data volume of the target data packet exceeds the data volume threshold, and the total amount of sub-channels contained in the data transmission channel corresponding to the received data migration instruction within the current specified time period does not exceed the preset channel volume threshold, the number of diversions is determined, and a second transit database with a number consistent with the number of diversions is added to the data transmission channel, and the second transit database is used to receive the target data packet and split the target data packet into a number of data packets with a number consistent with the number of diversions; wherein the number of diversions satisfies: the data volume of the target data packet / the number of diversions is less than or equal to the data threshold.

[0051] Data exchange methods also include: If there is a second transit database whose transit attribute is a data diversion attribute in the data transmission channel, a second input channel and a second output channel are determined, and the data packets corresponding to the second input channel are split into several diversion data packets, and the diversion data packets are transmitted through the second output channel; wherein the second input channel refers to an input channel of all sub-channels contained in the same data transmission channel, which transmits data packets to the second transit database, and the second output channel refers to a sub-channel of all sub-channels contained in the same data transmission channel, which is used to transmit data packets output by the second transit database; and the second output channel is set corresponding to the diversion data packets.

[0052] In implementation, after the target data packet is transmitted to the second transfer database, that is, when a new data packet is added to the second transfer database, the data exchange system splits the data packet according to the diversion quantity corresponding to the data packet to form diversion data packets of the same number as the diversion quantity. Figure 2-3 Before being diverted, each data packet corresponds to a second input channel, and corresponds to a second output channel whose number is consistent with the diversion number, so as to transmit the diverted data packets belonging to the same data packet through the aforementioned second output channels.

[0053] Optionally, the transit attribute further includes a security verification attribute, that is, the data exchange system further includes a transit database (hereinafter referred to as the fourth transit database) whose transit attribute is a security verification attribute.

[0054] Accordingly, the “establishing a data transmission channel” in S102 also includes the following steps: Determine whether there is data content with a confidentiality level in the target data content. If so, add a specified number of fourth transit databases, and make the sub-channel corresponding to the data packet containing the data content with the confidentiality level pass through all the fourth transit databases during the transmission process. The confidentiality level can be manually calibrated by the user in the process of triggering the data migration instruction and selecting the target data content.

[0055] Data exchange methods also include: If there is a fourth transit database whose transit attribute is a data merging attribute in the data transmission channel, the fourth input channel and the fourth output channel are determined; whenever a data packet transmitted by the fourth input channel is added to the fourth transit database, a security analysis is performed on the data packet, and if the analysis result is normal, the data packet is transmitted through the fourth output channel; if the analysis result is abnormal, the transmission of the data packet is suspended, and data abnormality warning information is output; wherein, the fourth input channel refers to an input channel for transmitting data packets to the fourth transit database among all sub-channels contained in the same data transmission channel, and the fourth output channel refers to a sub-channel for transmitting data packets output by the fourth transit database among all sub-channels contained in the same data transmission channel.

[0056] Among them, the step of "if the analysis result is abnormal, suspending the transmission of the data packet and outputting data abnormality warning information" includes: If the analysis result is abnormal, the transmission of the data packet is suspended, the data packet with the abnormal analysis result is determined as an abnormal data packet, and the suspected abnormal channel is determined, and data warning information with the suspected abnormal channel is output; wherein the suspected abnormal channel refers to all sub-channels included in the data transmission channel that transmit abnormal data packets to the fourth transit database.

[0057] In implementation, the fourth transit database can be evenly spaced on the sub-channels corresponding to the data packets containing data of confidentiality level, so that the aforementioned data packets can pass through a fourth transit database at regular intervals, and whenever a new data packet is added to the fourth transit database, the data exchange system will perform a security analysis on the data packet. The security analysis method in this embodiment is specifically to verify whether the data packet currently transmitted to the fourth transit database is consistent with the content of the data packet originally divided, that is, to determine whether there is any content missing. If not, the analysis result is normal, and the data packet is transmitted using the fourth output channel. If so, the analysis result is abnormal, and the subsequent transmission of the data packet (that is, the abnormal data packet) is suspended at this time, and according to the data transmission channel corresponding to the data packet, all sub-channels (that is, suspected abnormal channels) before the data packet is transmitted to the current fourth transit database are determined, and abnormal warning information with the abnormal data packet and all suspected abnormal channel data is generated, and the abnormal warning information is displayed. Refer to Figure 2 -4, each data packet transmitted to the fourth transit database corresponds to a fourth input channel and a fourth output channel.

[0058] The present application also discloses an efficient data exchange system based on a heterogeneous structured database. Figure 3 ,include: A migration instruction acquisition module 1 is used to acquire a data migration instruction, wherein the data migration instruction at least includes a data source library, a destination database, and target data content; The data parallel migration module 2 is used to retrieve the reference data content corresponding to the target data content from the data source library, divide the reference data content into several data packets, establish a data transmission channel, and the data transmission channel includes at least several sub-channels corresponding to the data packets; and transmit the corresponding data packets from the data source library to the destination database through the sub-channels; The data integration storage module 3 is used to integrate all data packets transmitted to the target database to obtain target data content and then store it in the target database.

[0059] Optionally, it also includes a data mid-way merging module, which is used to determine the first input channel and the first output channel if there is a first transit database whose transit attribute is a data merging attribute in the data transmission channel, merge the data packets corresponding to all the first input channels used to transmit the same target data content into a new data packet, and transmit the merged new data packet through the first output channel; wherein the first input channel refers to an input channel among all sub-channels contained in the same data transmission channel that transmits data packets to the first transit database, and the first output channel refers to a sub-channel among all sub-channels contained in the same data transmission channel that is used to transmit data packets output by the first transit database.

[0060] Optionally, it also includes a data mid-way diversion module, which is used to determine a second input channel and a second output channel if there is a second transit database with a transit attribute as a data diversion attribute in the data transmission channel, split the data packets corresponding to the second input channel into several diverted data packets, and transmit the diverted data packets through the second output channel; wherein the second input channel refers to an input channel of all sub-channels contained in the same data transmission channel that transmits data packets to the second transit database, and the second output channel refers to a sub-channel of all sub-channels contained in the same data transmission channel that is used to transmit data packets output by the second transit database; and the second output channel is set corresponding to the diverted data packets.

[0061] Optionally, it also includes a data mid-way adding module, which is used to determine the third input channel and the third output channel if there is a third transit database with a transit attribute as a data backup attribute in the data transmission channel, retrieve the regular data content contained in the target data content from the third transit database, merge the regular data content with the data packet transmitted corresponding to the third input channel to form a new data packet, and transmit the merged new data packet through the third output channel; wherein, the third input channel refers to an input channel of all sub-channels contained in the same data transmission channel that transmits data packets to the third transit database, and the third output channel refers to a sub-channel of all sub-channels contained in the same data transmission channel that is used to transmit data packets output by the third transit database.

[0062] Optionally, it also includes a data security verification module, which is used to determine the fourth input channel and the fourth output channel if there is a fourth transit database whose transit attribute is a data merging attribute in the data transmission channel; whenever the fourth transit database adds a new data packet transmitted by the fourth input channel, the data packet is subjected to a security analysis, and if the analysis result is normal, the data packet is transmitted through the fourth output channel; if the analysis result is abnormal, the transmission of the data packet is suspended, and data abnormality warning information is output; wherein the fourth input channel refers to an input channel for transmitting data packets to the fourth transit database among all sub-channels contained in the same data transmission channel, and the fourth output channel refers to a sub-channel for transmitting data packets output by the fourth transit database among all sub-channels contained in the same data transmission channel.

[0063] Optionally, the data security verification module is also used to suspend the transmission of data packets if the analysis result is abnormal, determine the data packets with abnormal analysis results as abnormal data packets, determine suspected abnormal channels, and output data warning information with suspected abnormal channels; wherein the suspected abnormal channels refer to all sub-channels included in the data transmission channel that transmit abnormal data packets to the fourth transit database.

[0064] Optionally, the data parallel migration module 2 is also used to divide the benchmark data content into a number of data packets according to the data type of the benchmark data content, the data volume corresponding to each data type, and the pre-stored transmission speed corresponding to each data type, and satisfying that: the time difference between any two data packets transmitted to the destination database is within a specified difference range.

[0065] An embodiment of the present application also discloses an efficient data exchange device based on a heterogeneous structured database. The efficient data exchange device based on a heterogeneous structured database includes a memory and a processor. The memory stores a computer program that can be loaded by the processor and executes the efficient data exchange method based on a heterogeneous structured database as described above.

[0066] An embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program that can be loaded by a processor and execute the above-mentioned efficient data exchange method based on a heterogeneous structured database. The computer-readable storage medium includes, for example: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0067] It should be noted that, in this document, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

[0068] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit the protection scope of the application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on these embodiments, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

Claims

1. An efficient data exchange method based on heterogeneous structured database, characterized in that: include: Obtaining a data migration instruction, wherein the data migration instruction at least includes a data source library, a destination database, and target data content; Retrieving reference data content corresponding to the target data content from the data source library, dividing the reference data content into a plurality of data packets, establishing a data transmission channel, wherein the data transmission channel at least includes a plurality of sub-channels corresponding to the data packets; transmitting the corresponding data packets from the data source library to the destination database through the sub-channels; All data packets transmitted to the target database are integrated to obtain the target data content and then stored in the target database.

2. The efficient data exchange method based on heterogeneous structured database according to claim 1 is characterized in that: The data transmission channel also includes a transit database, and all sub-channels include a sub-channel for transmitting data packets to the transit database and a sub-channel for transmitting data packets output by the transit database; the sub-channel for transmitting data packets to the transit database is called an input channel, and the sub-channel for transmitting data packets output by the transit database is called an output channel; The transfer database corresponds to a transfer attribute, and the transfer attribute at least includes a data merging attribute; The method further comprises: If there is a first transit database whose transit attribute is a data merging attribute in the data transmission channel, the first input channel and the first output channel are determined, and the data packets corresponding to all the first input channels for transmitting the same target data content are merged to form a new data packet, and the merged new data packet is transmitted through the first output channel; wherein, the first input channel refers to an input channel among all sub-channels contained in the same data transmission channel that transmits data packets to the first transit database, and the first output channel refers to a sub-channel among all sub-channels contained in the same data transmission channel that is used to transmit data packets output by the first transit database.

3. The efficient data exchange method based on heterogeneous structured database according to claim 2 is characterized in that: The transfer attribute also includes a data diversion attribute; The method further comprises: If there is a second transit database whose transit attribute is a data diversion attribute in the data transmission channel, a second input channel and a second output channel are determined, and the data packets corresponding to the second input channel are split into several diversion data packets, and the diversion data packets are transmitted through the second output channel; wherein, the second input channel refers to an input channel of all sub-channels contained in the same data transmission channel, which transmits data packets to the second transit database, and the second output channel refers to a sub-channel of all sub-channels contained in the same data transmission channel, which is used to transmit data packets output by the second transit database; and the second output channel is set corresponding to the diversion data packets.

4. The efficient data exchange method based on heterogeneous structured database according to claim 2 is characterized in that: The transfer attribute also includes a data backup attribute; the target data content is composed of the reference data content and the regular data content, and the regular data content is stored in the transfer database whose transfer attribute is the data backup attribute; The method further comprises: If there is a third transit database whose transit attribute is a data backup attribute in the data transmission channel, the third input channel and the third output channel are determined, the regular data content contained in the target data content is retrieved from the third transit database, the regular data content is merged with the data packet transmitted corresponding to the third input channel to form a new data packet, and the merged new data packet is transmitted through the third output channel; wherein the third input channel refers to an input channel among all sub-channels contained in the same data transmission channel that transmits data packets to the third transit database, and the third output channel refers to a sub-channel among all sub-channels contained in the same data transmission channel that is used to transmit data packets output by the third transit database.

5. The efficient data exchange method based on heterogeneous structured database according to claim 2, characterized in that: The transit attributes also include security verification attributes; The method further comprises: If there is a fourth transit database whose transit attribute is a data merging attribute in the data transmission channel, a fourth input channel and a fourth output channel are determined; whenever a data packet transmitted by a fourth input channel is added to the fourth transit database, a security analysis is performed on the data packet, and if the analysis result is normal, the data packet is transmitted through the fourth output channel; if the analysis result is abnormal, the transmission of the data packet is suspended, and data abnormality warning information is output; wherein, the fourth input channel refers to an input channel for transmitting data packets to the fourth transit database among all sub-channels contained in the same data transmission channel, and the fourth output channel refers to a sub-channel for transmitting data packets output by the fourth transit database among all sub-channels contained in the same data transmission channel.

6. The efficient data exchange method based on heterogeneous structured database according to claim 5, characterized in that: If the analysis result is abnormal, the transmission of the data packet is suspended, and data abnormality warning information is output, including: If the analysis result is abnormal, the transmission of the data packet is suspended, the data packet with the abnormal analysis result is determined as an abnormal data packet, and a suspected abnormal channel is determined, and data warning information with the suspected abnormal channel is output; wherein the suspected abnormal channel refers to all sub-channels included in the data transmission channel that transmit the abnormal data packet to the fourth transit database.

7. The efficient data exchange method based on heterogeneous structured database according to claim 1 is characterized in that: The step of dividing the reference data content into a plurality of data packets comprises: According to the data type of the benchmark data content, the data volume corresponding to each data type, and the pre-stored transmission speed corresponding to each data type, the benchmark data content is divided into a number of data packets, and it is satisfied that: the time difference between any two data packets transmitted to the destination database is within a specified difference range.

8. An efficient data exchange system based on heterogeneous structured database, characterized in that: include: A migration instruction acquisition module (1) is used to acquire a data migration instruction, wherein the data migration instruction at least includes a data source library, a destination database, and target data content; The data parallel migration module (2) is used to retrieve reference data content corresponding to the target data content from the data source library, divide the reference data content into a plurality of data packets, establish a data transmission channel, the data transmission channel at least comprising a plurality of sub-channels corresponding to the data packets; and transmit the corresponding data packets from the data source library to the destination database via the sub-channels; The data integration storage module (3) is used to integrate all data packets transmitted to the target database to obtain target data content and then store it in the target database.

9. An efficient data exchange device based on a heterogeneous structured database, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute any one of the methods according to claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute any one of the methods as claimed in claims 1 to 7.

Images