Method, apparatus, device, medium and program product for data transmission

Abstract

The invention relates to the technical field of big data, and discloses a method, device and equipment for data transmission, a medium and a program product, and the method comprises the following steps: obtaining an incremental log event stream of a source end; judging whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, and if so, splitting the incremental log event stream to obtain a plurality of sub-event streams; on the basis of the sub-event streams, a cache data row is constructed, the cache data row comprises sub-data rows, the cache data row is written into a data transmission service cache queue, if the value of the position mark in the initial sub-data row is the value representing the head of the incremental log event stream, a data transmission transaction is started, and if the value of the position mark in the initial sub-data row is not the value representing the head of the incremental log event stream, the and importing data in the cache data row corresponding to the initial sub-data row into the target end, if the value of the position mark in the target sub-data row is the value representing the tail of the incremental log event stream, importing data in the cache data row corresponding to the target sub-data row into the target end, and submitting a data transmission transaction.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method, apparatus, device, medium and program product for data transmission. Background Art

[0002] In some complex business systems, there may be large transaction statements; for example, in some business systems that process video, voice, image or text, the corresponding business database stores large field content. During the incremental synchronization of DTS, when encountering large transactions or large field synchronization, the total size of a batch of data RowBatch may exceed the total size of the system memory, causing memory overflow, causing the DTS incremental synchronization work node to exit, resulting in incremental synchronization business interruption. Summary of the invention

[0003] In view of this, the present invention provides a method, apparatus, equipment, medium and program product for data transmission to solve the problem that when performing large field synchronization, the total size of some data streams may exceed the total size of the system memory, resulting in memory overflow, causing the incremental synchronization working node of the data transmission service to exit, causing the incremental synchronization business to be interrupted.

[0004] In a first aspect, the present invention provides a method for data transmission, the method comprising: obtaining an incremental log event stream from a source end; determining whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, if the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, splitting the incremental log event stream to obtain multiple sub-event streams; constructing a cache data row based on the sub-event stream, the cache data row comprising a sub-data row, writing the cache data row into the data transmission service cache queue, if the value of the position mark in the initial sub-data row is a value representing the head of the incremental log event stream, starting a data transmission transaction, importing data in the cache data row corresponding to the initial sub-data row to the target end, if the value of the position mark in the target sub-data row is a value representing the tail of the incremental log event stream, importing data in the cache data row corresponding to the target sub-data row to the target end, and committing the data transmission transaction.

[0005] In an optional embodiment, the determination of whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue includes: if the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, waiting for the data transmission service cache queue to be cleared; if the data transmission service cache queue is cleared, determining again whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue; if the data volume of the incremental log event stream is less than the available memory of the data transmission service cache queue, parsing the incremental log event stream, constructing the cache data row based on the incremental log event stream, and performing the data transmission transaction.

[0006] In an optional embodiment, after writing the cache data line into the data transmission service cache queue, the method further includes: determining whether the data transmission service cache queue includes only one cache data line; if the data transmission service cache queue includes one cache data line, if the cache data line is incomplete, obtaining the current connection and the current data transmission transaction, and constructing and executing the operation of loading the data transmission service cache queue.

[0007] In an optional embodiment, after writing the cache data row into the data transmission service cache queue, the method further includes: if the data transmission service cache queue includes multiple cache data rows, obtaining a new connection and starting a data transmission transaction, constructing and executing an operation of loading the data transmission service cache queue.

[0008] In an optional embodiment, after writing the cache data line into the data transmission service cache queue, the method further includes: if the data transmission service cache queue includes a cache data line, when the cache data line is complete, obtaining a new connection and starting a data transmission transaction, constructing and executing the operation of loading the data transmission service cache queue.

[0009] In an optional embodiment, after loading the data transmission service cache queue, the method also includes: determining whether the current cache data row is the tail of the current sub-event stream; if the current cache data row is the tail of the current sub-event stream, committing the current data transmission transaction; if the current cache data row is not the tail of the current sub-event stream, re-acquiring the next batch of data transmission service cache queues.

[0010] In a second aspect, the present invention provides a device for data transmission, which includes: a log acquisition module, used to acquire an incremental log event stream from a source end; a log evaluation module, used to determine whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, and if the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, split the incremental log event stream to obtain multiple sub-event streams; a log processing module, used to construct a cache data row based on the sub-event stream, the cache data row including a sub-data row, write the cache data row into the data transmission service cache queue, if the value of the position mark in the initial sub-data row is a value representing the head of the incremental log event stream, start a data transmission transaction, import the data in the cache data row corresponding to the initial sub-data row to the target end, if the value of the position mark in the target sub-data row is a value representing the tail of the incremental log event stream, import the data in the cache data row corresponding to the target sub-data row to the target end, and commit the data transmission transaction.

[0011] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for data transmission of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0012] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for data transmission of the first aspect or any corresponding embodiment thereof.

[0013] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the method for data transmission according to the first aspect or any corresponding embodiment thereof.

[0014] The method for data transmission provided in this embodiment is used by the DTS service for synchronization. No matter the working node of the DTS service runs on a machine with any memory specification, when encountering a large field, the large field refers to an incremental log whose single incremental log exceeds the memory space of the data transmission service cache queue. The synchronization module will adaptively split the event stream containing the field into sub-event streams adapted to the current memory specification size, and still ensure data consistency; the incremental log synchronization containing large fields is completed in a streaming manner, avoiding the memory overflow problem caused by the synchronization link specification purchased by the user being too small; on the one hand, it solves the business risk caused by the interruption of the production synchronization link, and on the other hand, it also reduces the operation and maintenance cost caused by manual intervention due to the failure of the working node. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 The schematic diagram of the workflow of the incremental synchronization function module of the DTS system is shown;

[0017] Figure 2 A schematic flow chart of a method for data transmission provided by an embodiment of the present invention is shown;

[0018] Figure 3 Another schematic diagram showing the flow of a method for data transmission;

[0019] Figure 4 Another schematic diagram showing the process of the method for data transmission;

[0020] Figure 5 Another schematic diagram showing the process of the method for data transmission;

[0021] Figure 6 A schematic diagram of the workflow of the selector thread is shown;

[0022] Figure 7 A schematic diagram of the workflow of the ELT thread is shown;

[0023] Figure 8 A schematic diagram showing the structure of a device for data transmission is shown;

[0024] Fig. 9 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] The concepts involved in the embodiments of the present invention are introduced below. Data transmission service (DTS) is the foundation of data circulation, integrating data migration, real-time data synchronization, and data subscription. It is suitable for helping users to quickly and stably migrate databases to the cloud without interrupting business, build real-time data synchronization channels to achieve database disaster recovery / dual active, and provide real-time data streams for data lakes / warehouses (OLAP).

[0027] Out of Memory (OOM) occurs when the memory requested during the execution of a computer program exceeds the available memory space provided by the system or application. When a program needs to allocate more memory to store data or perform operations, but the available memory has been exhausted, an out of memory occurs.

[0028] Data Manipulation Language (DML), statements for inserting, deleting, and updating a database, is also a type of event stream in the database incremental change log.

[0029] Data Definition Language (DDL), a statement for creating and changing the structure of database objects, is also a type of event stream in the database incremental change log.

[0030] The Data Transmission Service (DTS) system includes: structure migration, full migration and incremental synchronization. The incremental synchronization function module parses the incremental logs of the source database, and then packages the parsed event stream into the DTS proprietary format. After data extraction and data conversion, the captured incremental data is sent to the target database.

[0031] Figure 1 The following is a schematic diagram showing the workflow of the incremental synchronization function module of the DTS system. Figure 1As shown, the workflow of the incremental synchronization function module (DTS-syncer) 11 of the DTS system includes: the source library 12 will generate two event streams during the writing process: data manipulation (Data Manipulation Language, referred to as DML) and data definition (Data Definition Language, referred to as DDL) event streams. The source library 12 includes Oracle, MySQL and PostgreSQL. The DTS selector 13 (Selector) module captures these events by parsing the incremental log, and then encapsulates each event into a DTS general format data row (RowData) to form a batch of data streams (RowBatch). After data extraction 14 (Extractor) and data transformation 15 (Transformer), the loader 16 (Loader) is called to insert the incremental data into the target library 17. This process is the processing flow query / extraction / transformation / loading (Select / Extract / Transform / Load, referred to as SETL) of the synchronization module of the data transmission service. The star architecture design of DTS can support synchronization between multiple source libraries and target libraries. SETL first queries the incremental change event stream from the source end, extracts it into the proprietary format of the DTS cache queue, goes through the data conversion process, and finally loads it to the target end.

[0032] In some complex business systems, there may be large transaction statements; for example, in some business systems that process video, voice, image or text, the corresponding business database stores large field content. During the incremental synchronization of DTS, when encountering large transactions or large field synchronization, the total size of a batch of data RowBatch may exceed the total size of the system memory, causing memory overflow, causing the DTS incremental synchronization work node to exit, resulting in incremental synchronization business interruption.

[0033] For the above-mentioned problems, the methods for processing large transactions in related technologies include:

[0034] The debezium connector provides a distributed storage option for transactions. When incremental capture of large transactions is required, the transaction can be placed in a distributed storage system and sent to downstream processing when the transaction is committed. If debezium is used as the Selector module of DTS, this method can be used to avoid memory overflow in the selection (S) stage.

[0035] Canal is also a connector that can be used as a Selector module in DTS. When capturing incremental logs, canal directly passes incremental data to the downstream without waiting for the transaction to end, and avoids overflow in the S phase when large transactions occur.

[0036] When a data synchronization tool in the related technology receives data in the S phase, such as Otter, it will cache the data in a circular queue. This queue also limits the number and total size of submitted data batches. In extreme cases, only the data of one incremental log will be cached, which avoids memory overflow in the ETL phase when large transactions occur.

[0037] When a user encounters memory overflow due to large field synchronization, the relevant technology usually tells the user to purchase a data synchronization worker node with higher memory specifications to solve the problem. However, this approach is a contingency plan. If the user does not pay attention to his or her business situation and orders a smaller synchronization node, the problem can only be repaired manually after it occurs.

[0038] According to an embodiment of the present invention, a method embodiment for data transmission is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] In this embodiment, a method for data transmission is provided, which can be used in a terminal, such as a mobile phone, a tablet computer, a desktop computer, or a server, etc. Figure 2 A schematic diagram of a method for data transmission provided by an embodiment of the present invention is shown. Figure 2 As shown, the process includes the following steps:

[0040] Step S201, obtaining the incremental log event stream of the source end.

[0041] In this step, you can directly collect incremental log event streams through database connections and query statements, or obtain incremental log data through an application programming interface.

[0042] Step S202, determine whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue. If the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, split the incremental log event stream to obtain multiple sub-event streams.

[0043] Figure 3 Another schematic diagram showing the process of the method for data transmission is shown as follows Figure 3 shown. Figure 4Another schematic diagram of the process of the method for data transmission is shown. In this step, the incremental log event stream (Entry) of the source end (Source) 30 is obtained. A new submodule, the log assessment module (Assessor) 32, is created in the selector (Selector) 31. The log assessment module 32 determines whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue 34, and if the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue 34, the incremental log event stream is split. The log assessment module has different implementations under different source library types.

[0044] When the log evaluation module 32 determines that a single incremental log event stream exceeds the memory limit of the data transmission service cache queue 34 , the incremental log event stream may be split into sub-event streams (EntryPieces) and imported into the target end 35 in multiple times.

[0045] Step S203, based on the sub-event stream, construct a cache data row, the cache data row includes a sub-data row, write the cache data row into the data transmission service cache queue, if the value of the position mark in the initial sub-data row is the value representing the head of the incremental log event stream, start the data transmission transaction, import the data in the cache data row corresponding to the initial sub-data row to the target end, if the value of the position mark in the target sub-data row is the value representing the tail of the incremental log event stream, import the data in the cache data row corresponding to the target sub-data row to the target end, and commit the data transmission transaction.

[0046] In this step, the DTS cache queue 35 includes cached data rows (RowData). The cached data rows (RowData) include sub-data rows (RowDataPiece). The loader 36 determines the timing of starting and submitting the data transmission transaction according to the value of the position mark (flag) of the sub-data row (RowDataPiece): Figure 3 As shown, when flag is the value of the header (Head) representing the incremental log event stream, a data transmission transaction is started to import the first part of the incremental log event stream in the current RowData; Figure 4 As shown, when flag is a value representing the tail of the incremental log event stream, the last part of the incremental log event stream in the current RowData is imported and the data transmission transaction is submitted.

[0047] The loader 36 will have different behaviors depending on whether RowData is RowDataPiece. If RowData is RowDataPiece, the loader will add the cached data row to the end in an appended manner each time it loops. The above loader has different implementations under different target library types, and can use the built-in functions of different databases to build SQL statements to complete this function.

[0048] The method for data transmission provided in this embodiment is used by the DTS service for synchronization. No matter the working node of the DTS service runs on a machine with any memory specification, when encountering a large field, the large field refers to an incremental log whose single incremental log exceeds the memory space of the data transmission service cache queue. The synchronization module will adaptively split the event stream containing the field into sub-event streams adapted to the current memory specification size, and still ensure data consistency; the incremental log synchronization containing large fields is completed in a streaming manner, avoiding the memory overflow problem caused by the synchronization link specification purchased by the user being too small; on the one hand, it solves the business risk caused by the interruption of the production synchronization link, and on the other hand, it also reduces the operation and maintenance cost caused by manual intervention due to the failure of the working node.

[0049] In some optional embodiments, Figure 5 Another schematic diagram of the process of the method for data transmission is shown. Figure 5 As shown in the figure, when the amount of data in the incremental log event stream is less than the available memory of the data transmission service cache queue, the incremental logs are pulled and submitted in batches. Figure 3 , Figure 4 and Figure 5 The ET phase in SETL is omitted. In this way, incremental logs of small fields can be pulled and submitted in batches, which improves the transmission efficiency of the data transmission service.

[0050] In some optional embodiments, determining whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue includes: if the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, waiting for the data transmission service cache queue to be cleared; if the data transmission service cache queue is cleared, determining again whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue; if the data volume of the incremental log event stream is less than the available memory of the data transmission service cache queue, parsing the incremental log event stream, constructing cache data rows based on the incremental log event stream, and performing data transmission transactions.

[0051] In this embodiment, Figure 6 FIG. 1 shows a schematic diagram of the workflow of the selector thread. Figure 6As shown, in step S601, the selector thread is started; in step S602, the event stream is started to be monitored. In this step, the incremental log of the source library is monitored to obtain the event stream; in step S603, it is determined whether the cache queue is full. If the cache queue is full, the data in the cache queue is continuously processed until there is free space in the cache queue. If the cache queue is not full and there is free space, the process proceeds to step S604; in step S604, it is determined whether a new incremental log is generated. If no new incremental log is generated, the process continues to monitor until a new incremental log is generated, and the process proceeds to step S605; in step S605, the incremental log event stream is obtained to determine whether the size of the incremental log event stream exceeds the remaining space in the cache queue. If so, the process waits for the DTS cache queue to be cleared by the ETL thread in SETL, and the process proceeds to step S606. If not, the process proceeds to step S609; 06, determine whether the DTS cache queue is empty. If the DTS cache queue is empty, proceed to step S607; step S607, again determine whether the data volume of the incremental log event stream is greater than the remaining space of the DTS cache queue. If so, proceed to step S608. If not, that is, the data volume of the incremental log event stream is less than the remaining space of the DTS cache queue, proceed to step S609; step S608, perform partial parsing of the incremental log event stream; step S609, perform overall parsing of the incremental log event stream; step S610, after parsing is completed, construct a cache data row; step S611, put the constructed cache data row into the DTS cache queue; step S612, determine whether the current incremental log event stream is parsed. If parsing is completed, proceed to step S602. If parsing is not completed, proceed to step S606.

[0052] In this way, in the selector thread, the incremental log event stream that exceeds the memory limit is split and parsed before being delivered to the DTS cache queue, ensuring that the data in the DTS cache queue does not exceed the memory size limit, avoiding memory overflow, and adaptively completing data synchronization without manual intervention when encountering large field synchronization.

[0053] In some optional embodiments, after writing the cache data line into the data transmission service cache queue, the aforementioned method for data transmission also includes: determining whether the data transmission service cache queue includes only one cache data line; if the data transmission service cache queue includes one cache data line, if the cache data line is incomplete, obtaining the current connection and the current data transmission transaction, and loading the data transmission service cache queue.

[0054] In this embodiment, checking whether the DTS cache queue contains only one cache data row can be achieved by querying the length of the DTS cache queue or traversing the DTS cache queue. If there is only one cache data row in the DTS cache queue, it is necessary to further determine whether the cache data row is complete. The integrity of the data row may be determined based on specific business rules, data formats, or verification mechanisms.

[0055] In the case of incomplete cached data rows, you can obtain the current database connection and data transfer transaction. You can obtain an available connection from the connection pool and join the existing data transfer transaction. Load the data transfer service cache queue, read more data rows from the data source to supplement the DTS cache queue, or merge the incomplete data rows in the DTS cache queue with other related data rows to form a complete data set.

[0056] In this way, the consistency and integrity of the data can be maintained, and the robustness and fault tolerance of the system can be improved.

[0057] In some optional embodiments, after writing the cache data line into the data transmission service cache queue, the aforementioned method for data transmission further includes: if the data transmission service cache queue includes multiple cache data lines, obtaining a new connection and starting a data transmission transaction, loading the data transmission service cache queue.

[0058] In this embodiment, when the data transmission service cache queue includes multiple cache data rows, a new connection can be obtained from the connection pool to facilitate subsequent data transmission operations. In this way, the stability and availability of the connection can be guaranteed based on the new connection, so that the data transmission process will not be interrupted due to connection problems.

[0059] In some optional embodiments, after writing the cache data row into the data transmission service cache queue, the aforementioned method for data transmission further includes: if the data transmission service cache queue includes a cache data row, when the cache data row is complete, obtaining a new connection and starting a data transmission transaction to load the data transmission service cache queue.

[0060] In this embodiment, data verification may be performed, such as checking whether the format, size or type of the data meets expectations, or verifying the integrity of the data through business logic.

[0061] In this way, the consistency and integrity of the data can be guaranteed.

[0062] In some optional embodiments, after loading the data transmission service cache queue, the aforementioned method for data transmission also includes: determining whether the current cache data row is the tail of the current sub-event stream; if the current cache data row is the tail of the current sub-event stream, committing the current data transmission transaction; if the current cache data row is not the tail of the current sub-event stream, reacquiring the next batch of data transmission service cache queues.

[0063] In this embodiment, by checking the mark of the current cache data row, it can be determined whether the current cache data row is the tail of the current sub-event stream. If the current cache data row is not the tail of the sub-event stream, continue to process the next batch of data, so as to ensure the integrity of the transmitted data.

[0064] Figure 7 FIG. 1 shows a schematic diagram of the workflow of the ELT thread. Figure 7 As shown, in this embodiment, step S701, start the ETL thread in the SETL process; step S702, obtain the DTS cache queue data; step S703, perform the extraction E thread in the SETL process; step S704, perform the conversion T thread in the SETL process;

[0065] Step S705, determine whether the DTS cache queue data contains only one cache data row, if so, proceed to step S706; Step S706, the cache queue data contains only one cache data row, determine whether the cache data row is a complete event stream, if so, proceed to step S707, if not, proceed to step S708; Step S707, if the cache data row is a complete event stream, obtain a new connection, and proceed to step S710; Step S708, if the cache data row is incomplete, obtain the current connection, and proceed to step S709; Step S709, determine whether a new transaction has been opened, if so, proceed to step S710. Enter step S711, if not, go to step S710; step S710, if no new data transmission transaction is opened, open a new data transmission transaction, and go to step S712; step S711, if a new transaction is opened, obtain the current data transmission transaction, and go to step S712; step S712, build a load statement and execute it, and go to step S713; step S713, determine whether the current cached data row is the tail of the current newly added log event stream, if not, go to step S702, if so, go to step S714; step S714, commit the transaction, and go to step S702.

[0066] In this way, streaming data transmission will decompose and synchronize the content of a large field in a row of records containing large fields into multiple rows of records according to different database types, and use the transaction characteristics of the target database to avoid the target database seeing an incomplete row of records during multiple synchronizations. In the field of data transmission, the risk of Out Of Memory (OOM) is avoided.

[0067] The following uses the disaster recovery scenario of synchronizing Oracle to MySQL as an example to exemplarily illustrate the method for data transmission provided by the embodiment of the present invention.

[0068] A business table in the Oracle database. The business table is used to store business-related information. The definition statement of the business table is as follows:

[0069]

[0070] The BLOG_DATA field is used to store business content. The field type is CLOB, which is used to store large text strings.

[0071] The target database uses MySQL as the disaster recovery database, and the definition statement of the business table on the target side is as follows:

[0072]

[0073]

[0074] The BLOG_DATA field is used to store business content, and the field type is TEXT, which is used to store large text strings.

[0075] Selector threads include:

[0076] First, the synchronization node of DTS uses the method of parsing Oracle redo logs and archive logs to track the event flow. In each cycle of the Selector thread, the Oracle transaction change point SCN is used as a measurement method to start the log parsing tool logminer to parse a batch of logs. For example, starting from the starting point STARTSCN and parsing to ENDSCN, the parsing results will be stored in the view V$LOGMNR_CONTENTS.

[0077] Second, evaluate the size of each log in the parsing result set. Specifically, execute the following SQL statement on the view V$LOGMNR_CONTENTS:

[0078] SELECT DBMS_LOB.GETLENGTH(SQL_REDO)FROM V$LOGMNR_CONTENT BLOGS;

[0079] The SQL_REDO field represents the SQL statement parsed from the corresponding log, and the DBMS_LOB.GETLENGTH function obtains the character length of the SQL_REDO field.

[0080] Third, if the character length of the SQL_REDO field of one of the result sets exceeds the available memory size, log splitting is required. Use the following SQL statement to find the starting position of BLOG_DATA in the SQL_REDO field:

[0081] SELECT DBMS_LOB.INSTR(SQL_REDO,'BLOG_DATA')AS KEY_START

[0082] FROM V$LOGMNR_CONTENT

[0083] WHERE DBMS_LOB.INSTR(SQL_REDO,'BLOG_DATA')>0;

[0084] Fourth, according to the starting position KEY_START of the BLOG_DATA keyword, find the starting position of the corresponding value of BLOG_DATA [value_start, value_end).

[0085] Fifth, according to the available memory size, find the truncation position value_middle of the corresponding value of BLOG_DATA and divide the value into two parts [value_start, value_middle) and [value_middle, value_end).

[0086] Sixth, in the first loop of the Selector thread, use the following statement to query the first half of BLOG_DATA:

[0087] SELECT DBMS_LOB.SUBSTR(BLOG_DATA,value_start,value_middle-value_start)AS BLOG_DATA_PARTIAL1

[0088] FROM BLOGS;

[0089] And use the same method to query the values ​​of other fields.

[0090] Seventh, in the second loop of the Selector thread, use the following statement to query the second half of BLOG_DATA:

[0091] SELECT DBMS_LOB.SUBSTR(BLOG_DATA,value_middle,value_end-value_middle)AS BLOG_DATA_PARTIAL2

[0092] FROM BLOGS;

[0093] Eighth, the RowData constructed twice by the Selector thread is shown in Table 1:

[0094] Table 1 Cache data rows

[0095]

[0096] ETL threads, including:

[0097] In the ETL thread loop, the first time RowData is taken from the cache queue RowBatch, if there is only one event in the RowBatch and the value of tail in the RowData is false, a new connection is obtained and a transaction is started, and an insert statement is constructed to execute the Load action without committing.

[0098] In the ETL thread loop, the RowData is taken from the cache queue RowBatch for the first time. If there is only one event in the RowBatch and the value of tail in the RowData is true, the previous connection is obtained and the following statement is executed to update the value of the BLOG_DATA field:

[0099] UPDATE BLOGS

[0100] SET BLOG_DATA=CONCAT(BLOG_DATA,'abc')

[0101] WHERE BLOG_ID=<id_value> ;

[0102] Then commit the transaction.

[0103] In this way, it can be ensured that when users run DTS incremental synchronization nodes of any specifications, even if they encounter large field synchronization, the DTS incremental synchronization node can still work normally without memory overflow. In addition, the system built based on this method ensures that large fields can perform adaptive data synchronization while still having the star architecture characteristics of the DTS service, that is, any new access to different source and target databases. This method is effective in the incremental synchronization module of DTS and has good scalability.

[0104] In the present embodiment, a device for data transmission is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions thereof will not be repeated. As used below, the term "module" may implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0105] This embodiment provides a device for data transmission. Figure 8 A schematic diagram of the structure of a device for data transmission is shown, Figure 8 As shown, including:

[0106] The log acquisition module 801 is used to acquire the incremental log event stream of the source end.

[0107] The log evaluation module 32 is used to determine whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue. If the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, the incremental log event stream is split to obtain multiple sub-event streams.

[0108] The log processing module 33 is used to construct a cache data row based on the sub-event stream, and the cache data row includes a sub-data row. The cache data row is written into the data transmission service cache queue. If the value of the position mark in the initial sub-data row is the value representing the head of the incremental log event stream, the data transmission transaction is started, and the data in the cache data row corresponding to the initial sub-data row is imported to the target end. If the value of the position mark in the target sub-data row is the value representing the tail of the incremental log event stream, the data in the cache data row corresponding to the target sub-data row is imported to the target end, and the data transmission transaction is submitted.

[0109] In some optional implementations, the log evaluation module includes:

[0110] The log evaluation unit is used to wait for the data transmission service cache queue to be cleared if the data volume of the incremental log event stream is larger than the available memory of the data transmission service cache queue; if the data transmission service cache queue is cleared, it is used to determine again whether the data volume of the incremental log event stream is larger than the available memory of the data transmission service cache queue; if the data volume of the incremental log event stream is smaller than the available memory of the data transmission service cache queue, it parses the incremental log event stream, constructs cache data rows based on the incremental log event stream, and performs data transmission transactions.

[0111] In some optional implementations, the log processing module includes:

[0112] The first processing unit is used to determine whether the data transmission service cache queue includes only one cache data row; if the data transmission service cache queue includes one cache data row, when the cache data row is incomplete, obtain the current connection and the current data transmission transaction, and construct and execute the operation of loading the data transmission service cache queue.

[0113] In some optional implementations, the log processing module further includes:

[0114] The second processing unit is configured to obtain a new connection and start a data transmission transaction if the data transmission service cache queue includes multiple cache data rows, and to construct and execute an operation of loading the data transmission service cache queue.

[0115] In some optional implementations, the log processing module further includes:

[0116] The third processing unit is used to, if the data transmission service cache queue includes a cache data row and the cache data row is complete, obtain a new connection and start a data transmission transaction, and construct and execute an operation of loading the data transmission service cache queue.

[0117] In some optional implementations, the log processing module further includes:

[0118] The fourth processing unit is used to determine whether the current cache data line is the tail of the current sub-event stream; if the current cache data line is the tail of the current sub-event stream, commit the current data transmission transaction; if the current cache data line is not the tail of the current sub-event stream, re-acquire the next batch of data transmission service cache queues.

[0119] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0120] The device for data transmission in this embodiment is presented in the form of a functional unit, where the unit refers to an application specific integrated circuit (ASIC) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0121] The embodiment of the present invention also provides a computer device having the above Figure 8 An apparatus for data transmission is shown.

[0122] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig. 9As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.

[0123] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0124] The aforementioned memory 20 stores instructions executable by at least one processor 10, so that the aforementioned at least one processor 10 executes the method shown in the above embodiment.

[0125] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0126] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0127] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0128] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (such as a light emitting diode) and a tactile feedback device (such as a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0129] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0130] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0131] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for data transmission, characterized in that The method comprises: Get the incremental log event stream from the source; Determine whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue; if the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, split the incremental log event stream to obtain multiple sub-event streams; Based on the sub-event stream, a cache data row is constructed, and the cache data row includes sub-data rows. The cache data row is written into the data transmission service cache queue. If the value of the position mark in the initial sub-data row is the value representing the head of the incremental log event stream, a data transmission transaction is started, and the data in the cache data row corresponding to the initial sub-data row is imported to the target end. If the value of the position mark in the target sub-data row is the value representing the tail of the incremental log event stream, the data in the cache data row corresponding to the target sub-data row is imported to the target end, and the data transmission transaction is committed.

2. The method according to claim 1, characterized in that The determining whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue includes: If the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, wait for the data transmission service cache queue to be cleared; If the data transmission service cache queue is cleared, determining again whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue; If the data volume of the incremental log event stream is smaller than the available memory of the data transmission service cache queue, the incremental log event stream is parsed, the cache data row is constructed based on the incremental log event stream, and the data transmission transaction is performed.

3. The method according to claim 1 or 2, characterized in that: After writing the cache data line into the data transmission service cache queue, the method further includes: Determining whether the data transmission service cache queue includes only one cache data row; If the data transmission service cache queue includes one cache data row, and the cache data row is incomplete, the current connection and the current data transmission transaction are acquired, and an operation of loading the data transmission service cache queue is constructed and executed.

4. The method according to claim 3, characterized in that After writing the cache data line into the data transmission service cache queue, the method further includes: If the data transmission service cache queue includes a plurality of cache data rows, a new connection is acquired and a data transmission transaction is started, and an operation of loading the data transmission service cache queue is constructed and executed.

5. The method according to claim 3, characterized in that: After writing the cache data line into the data transmission service cache queue, the method further includes: If the data transmission service cache queue includes a cache data row, and the cache data row is complete, a new connection is acquired and a data transmission transaction is started, and an operation of loading the data transmission service cache queue is constructed and executed.

6. The method according to claim 3, characterized in that After loading the data transmission service cache queue, the method further includes: Determine whether the current cached data row is the tail of the current sub-event stream; If the current cached data row is the tail of the current sub-event stream, commit the current data transmission transaction; If the current cached data row is not the tail of the current sub-event stream, the next batch of data transmission service cache queues is re-acquired.

7. A device for data transmission, characterized in that The device comprises: The log acquisition module is used to obtain the incremental log event stream from the source end; A log evaluation module, used to determine whether the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, and if the data volume of the incremental log event stream is greater than the available memory of the data transmission service cache queue, split the incremental log event stream to obtain multiple sub-event streams; A log processing module is used to construct a cache data row based on the sub-event stream, wherein the cache data row includes a sub-data row, write the cache data row into the data transmission service cache queue, start a data transmission transaction if the value of the position mark in the initial sub-data row is a value representing the head of the incremental log event stream, import the data in the cache data row corresponding to the initial sub-data row to the target end, and if the value of the position mark in the target sub-data row is a value representing the tail of the incremental log event stream, import the data in the cache data row corresponding to the target sub-data row to the target end, and commit the data transmission transaction.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for data transmission according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for data transmission according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method for data transmission according to any one of claims 1 to 6.

Images