Breakpoint resuming method and device for data and electronic equipment

Abstract

The invention provides a data breakpoint resuming method and device and electronic equipment, and relates to the technical field of big data, cloud computing and the like. According to the specific implementation scheme, in response to synchronous interruption of a target data block of a data table, a flow check point corresponding to the target data block is obtained; receiving a synchronization request of the target data block, and obtaining a synchronization scene field from the stream check point; determining a target site storage field according to the synchronous scene field; and based on the target site storage field, carrying out breakpoint resume on the target data block. Therefore, the breakpoint resuming is performed on the target data block, so that the data synchronization efficiency is improved, the data synchronization consistency is ensured, the operation resources are saved, the stability and the high efficiency of the data service are improved, and the user experience is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and more specifically to the field of big data and cloud computing, and more particularly to a method, device and electronic device for resuming data transmission from a breakpoint. Background Art

[0002] In the information age, large amounts of data are generated every day and stored in different systems. For example, relational databases need to synchronize large amounts of data to other systems or engines after the original data is stored for further development or analysis to mine the value of the data. At the same time, since the original data is continuously generated in real time, that is, relational databases are constantly writing, updating, and deleting data. For data synchronization, if the data synchronization task is paused or failed for some reason, the synchronization can only be re-executed from the beginning. Repeated reading leads to very low overall synchronization efficiency and wastes computing resources. Summary of the invention

[0003] The present invention provides a method and device for resuming data transmission from a breakpoint, an electronic device, a storage medium and a program product.

[0004] According to one aspect of the present disclosure, a method for resuming data transmission from a breakpoint is provided, comprising: in response to a synchronization interruption of a target data block of a data table, obtaining a stream checkpoint corresponding to the target data block; receiving a synchronization request for the target data block, and obtaining a synchronization scenario field from the stream checkpoint; determining a target site storage field according to the synchronization scenario field; and resuming transmission of the target data block based on the target site storage field.

[0005] According to another aspect of the present disclosure, a data breakpoint-resume transmission device is provided, comprising: a first acquisition module, for acquiring a stream checkpoint corresponding to a target data block in response to a synchronization interruption of the target data block of a data table; a second acquisition module, for receiving a synchronization request for the target data block, and acquiring a synchronization scenario field from the stream checkpoint; a determination module, for determining a target site storage field according to the synchronization scenario field; and a breakpoint-resume transmission module, for performing breakpoint-resume transmission on the target data block based on the target site storage field.

[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the breakpoint-resume data transmission method described in one aspect of the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the breakpoint-resume data transmission method according to one aspect of the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method for breakpoint-resume data transmission according to one aspect of the present disclosure.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a flow chart of a method for resuming data transmission from a breakpoint according to the first embodiment of the present disclosure;

[0012] Figure 2 is a flow chart of a method for resuming data transmission from a breakpoint according to a second embodiment of the present disclosure;

[0013] Figure 3 A block diagram of a data breakpoint resuming transmission device for implementing the data breakpoint resuming transmission method of the embodiment of the present disclosure;

[0014] Figure 4 The present invention is a block diagram of an electronic device for implementing the method for resuming data transmission from a breakpoint according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0016] Data processing, including data collection, storage, retrieval, processing, change and transmission, aims to extract and derive valuable and meaningful data for certain specific users from large amounts of data that may be disorganized and difficult to understand.

[0017] Big Data, or massive data, refers to the amount of data involved that is so large that it cannot be captured, managed, processed, and organized into information that helps companies make more proactive business decisions within a reasonable time through mainstream software tools.

[0018] Cloud computing is a type of distributed computing. It refers to breaking down huge data computing programs into countless small programs through the network "cloud". These small programs are then processed and analyzed by a system composed of multiple servers to obtain results and return them to users.

[0019] The breakpoint-resume data transmission method of the embodiment of the present disclosure is described below with reference to the accompanying drawings.

[0020] Figure 1 It is a flowchart of the method for resuming data transmission from breakpoint to breakpoint according to the first embodiment of the present disclosure. It should be noted that the execution subject of the method for resuming data transmission from breakpoint to breakpoint in the present embodiment is a device for resuming data transmission from breakpoint to breakpoint, and the device for resuming data transmission from breakpoint to breakpoint can be a hardware device, or software in a hardware device, etc. The hardware device is, for example, a terminal device, a server, etc.

[0021] like Figure 1 As shown, the method for resuming data transmission from breakpoint provided by the embodiment of the present disclosure may include the following steps:

[0022] S101, in response to a synchronization interruption of a target data block of a data table, obtaining a stream checkpoint corresponding to the target data block.

[0023] It should be noted that the data breakpoint-resume transmission method proposed in the present disclosure performs breakpoint-resume transmission in units of data blocks, and after synchronization of each data block is completed, a stream checkpoint (StreamCheckpoint) corresponding to the data block is recorded.

[0024] The stream checkpoint StreamCheckpoint at least stores the real-time synchronization breakpoint resume value and the synchronization scenario field, and inherits the Checkpoint class.

[0025] It should be noted that, in response to the synchronization interruption of the target data block of the data table, the stream checkpoint corresponding to the data block of the data table may be queried to obtain the stream checkpoint corresponding to the target data block.

[0026] S102, receiving a synchronization request for a target data block, and obtaining a synchronization scenario field from a stream checkpoint.

[0027] Optionally, after synchronization of a target data block of a data table is interrupted, a synchronization instruction of the target data block may be automatically triggered based on pre-configuration in the data synchronization task, and a synchronization request may be generated according to the synchronization instruction to receive the synchronization request of the target data block.

[0028] Optionally, after synchronization of a target data block of a data table is interrupted, a synchronization instruction of the target data block may be generated based on a user operation, and a synchronization request may be generated according to the synchronization instruction to receive the synchronization request of the target data block.

[0029] The synchronization scene field (isSnapshotFinished) is a Boolean field, and the flag bit of the synchronization scene field is usually only true or false.

[0030] In the disclosed embodiment, the target synchronization scene can be determined based on the flag bit of the synchronization scene field (isSnapshotFinished), and based on the target synchronization scene, the target site storage field can be determined from the first site storage field (splitUpperBound) and the second site storage field (commitOffset) in the stream checkpoint.

[0031] S103, determining the target site storage field according to the synchronization scene field.

[0032] In an embodiment of the present disclosure, in response to the target synchronization scenario being a full synchronization scenario, the first point storage field (splitUpperBound) and the second point storage field (commitOffset) in the stream checkpoint are determined to be the target point storage fields.

[0033] In an embodiment of the present disclosure, in response to the target synchronization scenario being an incremental synchronization scenario, a second position storage field (commitOffset) in the stream checkpoint is determined to be a target position storage field.

[0034] S104, based on the target site storage field, performing breakpoint-resume transmission on the target data block.

[0035] Breakpoint resume means that during the data synchronization process, if the synchronization is paused or fails for any reason, the latest read location can be recorded in real time, and reading can continue from this location when synchronization is re-executed, without having to read all the data from the beginning every time synchronization is executed.

[0036] For example, for the DBLog algorithm, the data table of the relational database needs to have a primary key or a unique index. The DBLog algorithm first divides the entire data table into data blocks based on the primary key according to the specified size, and obtains data blocks. The same reading strategy is executed for each data block to read offline and real-time data together. For each data block, the current time point is recorded as the low point (LowWatermark), for example: the binlog point in the relational database, and then the offline data of the data block is read. After reading, the current time point is recorded as the high point (HighWatermark). By reading the real-time data between the low point and the high point, that is, while reading the offline data of this chunk in the previous step, the data in the data block is added, deleted, and modified. The change data is finally merged with the offline data read previously. In this case, the latest data has been obtained for the data block.

[0037] It should be noted that when the same operation is performed on each data block chunk, the latest real-time data of all data block chunks is obtained, and finally all the latest data of a table with continuous data changes is obtained. The data is pushed to the data warehouse (such as Doris and Hive, etc.) or the data lake to complete the data synchronization of the relational database, realize the real-time reading of the relational database without lock, and ensure the data consistency of offline and real-time data. It is worth noting that the core of the above DBLog algorithm is to maintain the data consistency of offline data and real-time data for each data block chunk. After all data block chunks are read, the entire data table has been fully synchronized. Then it enters the incremental stage, that is, only real-time data is read continuously, to ensure that if there are additions, deletions, updates, etc. in the data table after the reading is completed, they can also be read in time.

[0038] For example, in the breakpoint resume method in the application scenario of the DBLog algorithm, during the data synchronization process, if the synchronization is paused or fails for any reason, the latest read location can be recorded in real time, and reading can be continued from this location when the synchronization is re-executed. There is no need to read all the data from the beginning every time the synchronization is executed. The breakpoint resume function is provided by default, which can better support breakpoint resume.

[0039] In the disclosed embodiment, the upper boundary of the partition of the previous data block adjacent to the target data block can be determined from the first bit storage field, and the next data block to be synchronized is determined based on the upper boundary as the target data block, and the target data block is synchronized in a full synchronization scenario.

[0040] In the disclosed embodiment, the high bit position of the previous data block can be determined from the second bit storage field, and the high bit position of the previous data block can be used as the low bit position of the target data block for incremental transmission.

[0041] In summary, the data breakpoint-resume transmission method of the disclosed embodiment responds to the synchronization interruption of the target data block of the data table, obtains the stream checkpoint corresponding to the target data block, receives the synchronization request of the target data block, and obtains the synchronization scene field from the stream checkpoint, determines the target site storage field according to the synchronization scene field, and performs breakpoint-resume transmission on the target data block based on the target site storage field. Therefore, by performing breakpoint-resume transmission on the target data block, the efficiency of data synchronization is improved, the consistency of data synchronization is ensured, computing resources are saved, the stability and efficiency of data services are improved, and the user experience is enhanced.

[0042] Figure 2 FIG. 1 is a flow chart of a method for resuming data transmission from a breakpoint according to the second embodiment of the present disclosure. Figure 2 As shown, based on the above embodiment, the method for resuming data transmission from breakpoint in the embodiment of the present disclosure may specifically include the following steps:

[0043] S201, in response to a synchronization interruption of a target data block of a data table, obtaining a stream checkpoint corresponding to the target data block.

[0044] In an embodiment of the present disclosure, in response to the target data block being interrupted in a full synchronization scenario, a first flag bit (false) is written in the synchronization scenario field, the upper bound (UpperBound) of the previous data block adjacent to the target data block and the high watermark (HighWatermark) of the previous data block are determined, and the upper bound of the previous data block is stored in the first watermark storage field in the stream checkpoint, and the high watermark of the previous data block is stored in the second watermark storage field.

[0045] Optionally, the data table can be divided into blocks according to a set data block size to obtain multiple data blocks (chunks), determine the identification field of the data table, and determine the division boundary of each data block based on the identification field and the data block size, wherein the division boundary includes the lower boundary and the upper boundary of the data block.

[0046] The identification field of the data table may be a primary key or a unique index of the data table.

[0047] It should be noted that for a relational database, a data table of the relational database has a primary key or a unique index. The data table can be divided into blocks based on the primary key according to a set data block size (chunk Size) to determine the division boundary of each data block.

[0048] For example, the primary key of a data table in the relational database MySQL is the identity document (ID) field. The ID field has a total of nine values ​​from 1 to 9. If the data block size chunkSize is 3, the data table is divided according to the ID field, then the ID field from 1 to 3 is the first data block chunk, where the lower bound LowerBound of the first data block is 1, and the upper bound UpperBound is 3. From 4 to 6 is the second data block chunk, where the lower bound LowerBound of the second data block is 4, and the upper bound UpperBound is 6. From 7 to 9 is the third data block chunk, where the lower bound LowerBound of the third data block is 7, and the upper bound UpperBound is 9.

[0049] In an embodiment of the present disclosure, in response to a target data block being interrupted in an incremental synchronization scenario, a second flag bit (true) is written in the synchronization scenario field, the location of the target change log of the target data block is determined as the location for breakpoint resumption of the target data block, and the location of the target change log is stored in the second location storage field in the stream checkpoint.

[0050] Optionally, the offset map may be read to obtain the location of the candidate change log from the offset map, and the location of the target change log may be determined from the location of the candidate change log based on the time information and the change log type.

[0051] For example, by reading the offset mapping map, the location of the candidate change log that matches the query requirement can be obtained, and the location of the change log with the closest time is selected from the locations of the candidate change logs based on the time information, and based on the change log type, for example: the binlog location of the relational database MySQL and the redolog location of the relational database Oracle, to determine the location of the final target change log.

[0052] S202, receiving a synchronization request for a target data block, and obtaining a synchronization scenario field from a stream checkpoint.

[0053] The specific process of S202 can be found in the above embodiment and will not be described again here.

[0054] In the above embodiment, step S103 “determine the target site storage field according to the synchronization scene field” may specifically include the following steps S203 and S204.

[0055] S203: Determine a target synchronization scenario according to a flag bit of the synchronization scenario field.

[0056] It should be noted that for the full synchronization scenario, when a data block is read, it is paused or fails for some reason. Since all data blocks have not been read completely and enter the incremental phase, the first flag bit (false) is written in the synchronization scenario field.

[0057] In the embodiment of the present disclosure, in response to the synchronization scenario field (isSnapshotFinished) carrying the first flag bit (false), it is determined that the target synchronization scenario is a full synchronization scenario.

[0058] It should be noted that for the incremental synchronization scenario, that is, the full amount of data in the data table has been synchronized, and currently only the latest data of the addition, deletion, and update operations needs to be read, and the second flag bit (true) is written in the synchronization scenario field.

[0059] In the embodiment of the present disclosure, in response to the synchronization scenario field (isSnapshotFinished) carrying the second flag bit (true), it is determined that the target synchronization scenario is an incremental synchronization scenario.

[0060] S204: Determine a target location storage field from the first location storage field and the second location storage field in the stream checkpoint according to the target synchronization scenario.

[0061] In the disclosed embodiment, the target synchronization scenario is determined to be a full synchronization scenario, and the first point storage field (splitUpperBound) and the second point storage field (commitOffset) in the stream checkpoint are determined to be the target point storage fields.

[0062] In the embodiment of the present disclosure, it is determined that the target synchronization scenario is an incremental synchronization scenario, and the second site storage field (commitOffset) in the stream checkpoint is determined to be the target site storage field.

[0063] In the above embodiment, step S104 “resume the target data block based on the target site storage field” may specifically include the following steps S205 and S206 .

[0064] S205, determining the partition upper boundary from the first bit storage field, and determining the next data block to be synchronized based on the partition upper boundary as the target data block, and performing data synchronization on the target data block in a full synchronization scenario.

[0065] In the embodiment of the present disclosure, the upper boundary of the division is determined from the first bit point storage field, and the next data block chunk to be synchronized is determined based on the upper boundary of the division, and the target data block is fully synchronized.

[0066] S206, determining the high bit position of the previous data block from the second bit storage field, and using the high bit position of the previous data block as the low bit position of the target data block for incremental transmission.

[0067] In the disclosed embodiment, the second location storage field may be read to obtain the location of the recorded target change log, and incrementally resume the target data block in an incremental synchronization scenario starting from the location of the target change log.

[0068] For example, if the high bit position of the previous data block is 100, the full stage is skipped and incremental reading is performed directly. The high bit position of the previous data block is used as the low bit position of the target data block for incremental transmission, that is, the new data is read starting from position 101, which ensures high efficiency and data consistency and avoids repeated reading of data that has been read during the last synchronization process.

[0069] In summary, the data breakpoint-resume transmission method of the disclosed embodiment responds to the synchronization interruption of the target data block of the data table, obtains the stream checkpoint corresponding to the target data block, receives the synchronization request of the target data block, and obtains the synchronization scenario field from the stream checkpoint, determines the target synchronization scenario according to the flag bit of the synchronization scenario field, determines the target site storage field from the first site storage field and the second site storage field in the stream checkpoint according to the target synchronization scenario, determines the upper boundary of the division from the first site storage field, and determines the next data block to be synchronized as the target data block based on the upper boundary of the division, and performs data synchronization on the target data block in the full synchronization scenario, and determines the previous site storage field from the second site storage field. The high bit position of a data block is used, and the high bit position of the previous data block is used as the low bit position of the target data block for incremental continuation. The present disclosure improves the synchronization efficiency and service quality of relational databases. The data breakpoint continuation method ensures data consistency, and users do not need to intervene without perception, that is, it is a pure back-end optimization. Users can create data integration tasks on the front-end page as usual. If the task fails due to various reasons, the integration service will automatically record and store the latest bit position on the back-end. When the user clicks to start the task again, the data can be synchronized from the latest bit position. Users can experience more efficient and stable data services, which have the advantages of high efficiency, convenience, stability and high fault tolerance, and greatly improve the user experience.

[0070] Figure 3 The data breakpoint resuming device 300 proposed in the present disclosure includes: a first acquisition module 301, a second acquisition module 302, a determination module 303 and a breakpoint resuming module 304.

[0071] A first acquisition module 301 is used to obtain a stream checkpoint corresponding to a target data block in response to a synchronization interruption of the target data block in the data table;

[0072] A second acquisition module 302 is used to receive a synchronization request for the target data block and obtain a synchronization scenario field from the stream checkpoint;

[0073] A determination module 303, configured to determine a target location storage field according to the synchronization scene field;

[0074] The breakpoint resume module 304 is used to resume the breakpoint transmission of the target data block based on the target location storage field.

[0075] Furthermore, the determination module 303 is also used to: determine the target synchronization scene according to the flag bit of the synchronization scene field; and determine the target site storage field from the first site storage field and the second site storage field in the stream checkpoint according to the target synchronization scene.

[0076] Furthermore, the device 300 is also used to: in response to the synchronization scene field carrying a first flag bit, determine that the target synchronization scene is a full synchronization scene; determine the first point storage field and the second point storage field in the stream checkpoint as the target point storage field.

[0077] Furthermore, the device 300 is also used to: in response to the synchronization scene field carrying a second flag bit, determine that the target synchronization scene is an incremental synchronization scene; determine that the second site storage field in the stream checkpoint is the target site storage field.

[0078] Furthermore, the second acquisition module 302 is also used to: in response to the target data block being interrupted in the full synchronization scenario, write a first flag bit in the synchronization scenario field; determine the upper boundary of the division of the previous data block adjacent to the target data block and the high bit point of the previous data block; store the upper boundary of the division of the previous data block in the first bit point storage field in the stream checkpoint, and store the high bit point of the previous data block in the second bit point storage field.

[0079] Furthermore, the breakpoint resume module 304 is also used to: determine the upper boundary of the division from the first site storage field, and determine the next data block to be synchronized based on the upper boundary of the division as the target data block, and perform data synchronization on the target data block in a full synchronization scenario; determine the high bit point of the previous data block from the second site storage field, and use the high bit point of the previous data block as the low bit point of the target data block for incremental resumption.

[0080] Furthermore, the device 300 is also used to: divide the data table into blocks according to a set data block size to obtain multiple data blocks; determine an identification field of the data table, and determine a division boundary of each data block based on the identification field and the data block size, wherein the division boundary includes a lower division boundary and an upper division boundary of the data block.

[0081] Furthermore, the second acquisition module 302 is also used to: in response to the target data block being interrupted in the incremental synchronization scenario, write a second flag bit in the synchronization scenario field; determine the location of the target change log of the target data block as the location for breakpoint resumption of the target data block; and store the location of the target change log in the second location storage field in the stream checkpoint.

[0082] Furthermore, the breakpoint resume module 304 is also used to: read the second location storage field to obtain the location of the target change log; and perform incremental resume of the target data block in the incremental synchronization scenario starting from the location of the target change log.

[0083] Furthermore, the device 400 is also used to: read the offset map to obtain the location of the candidate change log from the offset map; and determine the location of the target change log from the location of the candidate change log based on time information and change log type.

[0084] In summary, the data breakpoint-resume transmission device of the embodiment of the present invention responds to the synchronization interruption of the target data block of the data table, obtains the stream checkpoint corresponding to the target data block, receives the synchronization request of the target data block, and obtains the synchronization scene field from the stream checkpoint, determines the target site storage field according to the synchronization scene field, and performs breakpoint-resume transmission on the target data block based on the target site storage field. Therefore, by performing breakpoint-resume transmission on the target data block, the efficiency of data synchronization is improved, the consistency of data synchronization is guaranteed, computing resources are saved, the stability and efficiency of data services are improved, and the user experience is enhanced.

[0085] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0086] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0087] Figure 4A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0088] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0089] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0090] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as Figures 1 to 3The breakpoint-resume data transmission method shown. For example, in some embodiments, the breakpoint-resume data transmission method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the breakpoint-resume data transmission method described above can be performed. Alternatively, in other embodiments, the computing unit 401 can be configured to execute the breakpoint-resume data transmission method by any other appropriate means (for example, by means of firmware).

[0091] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0092] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data breakpoint resume transmission device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0093] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0094] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0095] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0096] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0097] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the steps of the breakpoint-resume data transmission method shown in the above embodiment of the present disclosure are implemented.

[0098] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0099] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for resuming data transmission from a breakpoint, wherein: The method comprises: In response to a synchronization interruption of a target data block of the data table, obtaining a stream checkpoint corresponding to the target data block; receiving a synchronization request for the target data block, and obtaining a synchronization scenario field from the stream checkpoint; Determine a target site storage field according to the synchronization scene field; Based on the target site storage field, the target data block is resumably transmitted.

2. The method according to claim 1, wherein: The step of determining the target site storage field according to the synchronization scenario field includes: Determining a target synchronization scenario according to a flag bit of the synchronization scenario field; According to the target synchronization scenario, the target location storage field is determined from the first location storage field and the second location storage field in the stream checkpoint.

3. The method according to claim 2, wherein: The method further comprises: In response to the synchronization scenario field carrying a first flag bit, determining that the target synchronization scenario is a full synchronization scenario; A first location storage field and a second location storage field in the stream checkpoint are determined to be the target location storage field.

4. The method according to claim 2, wherein: The method further comprises: In response to the synchronization scenario field carrying a second flag bit, determining that the target synchronization scenario is an incremental synchronization scenario; A second location storage field in the stream checkpoint is determined to be the target location storage field.

5. The method according to claim 2 or 3, wherein: The obtaining of the stream checkpoint corresponding to the target data block includes: In response to the target data block being interrupted in a full synchronization scenario, writing a first flag bit in the synchronization scenario field; Determine the partition upper bound of a previous data block adjacent to the target data block and a high bit position of the previous data block; The upper boundary of the partition of the previous data block is stored in the first bit storage field in the stream checkpoint, and the high bit of the previous data block is stored in the second bit storage field.

6. The method according to claim 5, wherein: The step of resuming the breakpoint transmission of the target data block based on the target location storage field includes: Determine the partition upper bound from the first location storage field, determine the next data block to be synchronized based on the partition upper bound as the target data block, and perform data synchronization on the target data block in a full synchronization scenario; The high bit position of the previous data block is determined from the second bit position storage field, and the high bit position of the previous data block is used as the low bit position of the target data block for incremental transmission.

7. The method according to claim 5, wherein: The method further comprises: According to the set data block size, the data table is divided into blocks to obtain multiple data blocks; An identification field of the data table is determined, and based on the identification field and the data block size, a partition boundary of each data block is determined, wherein the partition boundary includes a partition lower boundary and a partition upper boundary of the data block.

8. The method according to claim 2 or 4, wherein: The obtaining of the stream checkpoint corresponding to the target data block further includes: In response to the target data block being interrupted in the incremental synchronization scenario, writing a second flag bit in the synchronization scenario field; Determine the location of the target change log of the target data block as the location for breakpoint resume transmission of the target data block; The location of the target change log is stored in a second location storage field in the stream checkpoint.

9. The method according to claim 8, wherein: The step of resuming the breakpoint transmission of the target data block based on the target location storage field includes: Reading the second location storage field to obtain the location of the target change log; Starting from the location of the target change log, the target data block is incrementally continued in the incremental synchronization scenario.

10. The method according to claim 8, wherein: Determining the location of the target change log of the target data block includes: Reading the offset map, and obtaining the location of the candidate change log from the offset map; The location of the target change log is determined from the locations of the candidate change logs based on time information and change log type.

11. A device for resuming data transmission from a breakpoint, wherein: The device comprises: A first acquisition module, configured to acquire a stream checkpoint corresponding to a target data block in response to a synchronization interruption of the target data block of the data table; A second acquisition module, configured to receive a synchronization request for the target data block and acquire a synchronization scenario field from the stream checkpoint; A determination module, used to determine a target site storage field according to the synchronization scene field; The breakpoint resume module is used to resume the breakpoint transmission of the target data block based on the target site storage field.

12. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

14. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images