Data synchronization method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data synchronization method and device, electronic equipment and a storage medium, belongs to the technical field of computers, and is suitable for the fields of financial science and technology and digital medical treatment. The method comprises the following steps: in a target file system, in response to a data transmission request of a source file system, obtaining a candidate operation log marked with a candidate log identifier from the source file system; wherein the first file is the same as a second file of the source file system; then, according to the candidate log identifier, performing log identification on the candidate operation log to obtain a target operation log; wherein the target operation log is used for indicating that the second file is updated into the third file. And finally, performing operation playback on the first file according to the target operation log to obtain a synchronous file which is the same as the third file. Therefore, the file modification operation in the source file system can be played back in the target file system, the data transmission quantity is reduced, and the data synchronization efficiency between the file systems is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology and is applicable to the fields of financial technology and digital medicine, and in particular to a data synchronization method and device, an electronic device, and a storage medium. Background Art

[0002] Currently, in order to achieve multi-task collaboration, task-related data needs to be synchronized between different file systems. For example, hybrid cloud data migration, multi-platform application development, and cross-institutional data sharing. In the field of financial technology, taking insurance product recommendations as an example, different clients need to obtain the insurance product data purchased by users, and then synchronize the insurance product data to the file system of the data analysis platform to analyze the insurance product data and obtain insurance product recommendation data. In the field of digital medicine, in order to achieve joint research on rare cases, for cases from different hospitals and research institutions, it is necessary to synchronize their case-related information stored in the file system, such as disease diagnosis information and treatment status information. However, for data synchronization between file systems, existing data synchronization methods only support data synchronization with files as the smallest unit, such as synchronization of files, synchronization of directories, and full scanning and synchronization of file systems. There is a large amount of invalid input / output (IO) overhead and IO amplification, resulting in low data synchronization efficiency. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a data synchronization method and device, an electronic device and a storage medium, aiming to reduce invalid IO overhead and IO amplification and improve the data synchronization efficiency between file systems.

[0004] To achieve the above-mentioned object, a first aspect of an embodiment of the present application provides a data synchronization method, which is applied to a target file system, wherein the target file system stores a first file. The method includes:

[0005] In response to a data transmission request from a source file system, obtaining a candidate operation log marked with a candidate log identifier from the source file system; wherein the first file is the same as a second file stored in the source file system;

[0006] performing log identification on the candidate operation log according to the candidate log identifier to obtain a target operation log; wherein the target operation log is used to indicate that the second file is updated to a third file, and the target operation log includes a candidate log entry having an entry generation time;

[0007] Selecting a target log entry from the candidate log entries according to the generation time of the entry;

[0008] The first file is replayed according to the target log entry to obtain a synchronized file; wherein the synchronized file is the same as the third file.

[0009] In some embodiments, the data transmission request includes a data transmission mode, and the data transmission mode includes an asynchronous transmission mode. Responding to the data transmission request from the source file system, obtaining a candidate operation log marked with a candidate log identifier includes:

[0010] Sending a log acquisition request to a message queue of a shared storage according to the asynchronous transmission mode; wherein the candidate operation log is written into the message queue by the source file system, and the shared storage is shared by the source file system and the target file system;

[0011] The candidate operation log shared by the shared storage message queue according to the log acquisition request is received.

[0012] In some embodiments, the candidate operation log has a log sharing time, and performing log identification on the candidate operation log according to the candidate log identifier to obtain the target operation log includes:

[0013] Performing playback identification on the candidate log according to the candidate log identifier, and screening out intermediate operation logs from the candidate log; wherein the intermediate operation logs are operation logs that have not been played back;

[0014] Sorting the intermediate operation logs to obtain an operation log sequence;

[0015] The intermediate operation logs are selected in sequence from the operation log sequence according to the log sharing time as the target operation logs.

[0016] In some embodiments, the target log entry includes: operation type, full file path, and operation data;

[0017] The operating and replaying the first file according to the target log entry to obtain a synchronized file includes:

[0018] Performing a path check on the first file according to the full path of the file to obtain path check information;

[0019] In a case where the path verification information indicates that the first file exists under the full file path, calling a modification operation according to the operation type;

[0020] Call the first file according to the full path of the file;

[0021] The operation data is written into the first file according to the modification operation to obtain the synchronization file.

[0022] In some embodiments, the data transmission mode further includes a synchronous transmission mode, and the step of obtaining the candidate operation log marked with a candidate log identifier in response to the data transmission request of the source file system further includes:

[0023] Calling a synchronous receiving interface according to the synchronous transmission mode;

[0024] The candidate operation log sent from the synchronous sending interface of the source file system is received through the synchronous receiving interface.

[0025] In some embodiments, the data synchronization method provided in the first aspect of the embodiments of the present application is applied to a source file system, where the source file system stores a second file. The method includes:

[0026] In response to receiving a file operation request, generating a candidate operation log marked with a log identifier according to the received file operation request; wherein the candidate operation log has a log data volume, and the file operation request includes a data synchronization priority and a data synchronization time window;

[0027] The second file is updated according to the candidate operation log to obtain a third file; wherein the target file system performs log identification on the candidate operation log to obtain a target operation log, and the target operation log is used to indicate that the second file is updated to the third file;

[0028] Obtaining network transmission parameters and a target system load of the target file system;

[0029] Selecting a data transmission mode according to the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters, and the target system load;

[0030] A data transmission request is generated according to the data transmission mode and the candidate operation log, and the data transmission request is sent to the target file system; wherein the data transmission request is used by the target file system to replay the operation of the first file stored in the target file system to obtain a synchronization file, the second file is the same as the first file, and the third file is the same as the synchronization file.

[0031] In some embodiments, selecting the data transmission mode according to the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters, and the target system load includes:

[0032] Calculating a synchronization transmission probability according to the data synchronization priority, the data synchronization time window, the log data volume, and the network transmission parameters;

[0033] Taking a weighted average of the target system load and the synchronous transmission probability to obtain a total synchronization probability;

[0034] Comparing the total synchronization probability with a preset synchronization transmission threshold to obtain synchronization transmission comparison information;

[0035] If the synchronous transmission comparison information indicates that the total synchronization probability is greater than the synchronous transmission threshold, determining the data transmission mode as the synchronous transmission mode;

[0036] When the synchronous transmission comparison information indicates that the synchronization success probability is less than the synchronous transmission threshold, the data transmission mode is determined to be an asynchronous transmission mode.

[0037] To achieve the above-mentioned object, a second aspect of an embodiment of the present application provides a data synchronization device, characterized in that it is applied to a target file system, where the target file system stores a first file, and the device includes:

[0038] a log acquisition module, configured to, in response to a data transmission request from a source file system, acquire a candidate operation log marked with a candidate log identifier from the source file system; wherein the first file is the same as a second file stored in the source file system;

[0039] a log identification module, configured to perform log identification on the candidate operation log according to the candidate log identifier to obtain a target operation log; wherein the target operation log is used to indicate that the second file is updated to a third file, and the target operation log includes a candidate log entry having an entry generation time;

[0040] a data selection module, configured to select a target log entry from the candidate log entries according to the generation time of the entry;

[0041] An operation playback module is used to perform operation playback on the first file according to the target log entry to obtain a synchronized file; wherein the synchronized file is the same as the third file.

[0042] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0043] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0044] The data synchronization method and apparatus, electronic device, and storage medium proposed in this application first respond to a data transmission request from a source file system on a target file system storing a first file and obtain a candidate operation log labeled with a candidate log identifier from the source file system; wherein the first file is identical to a second file stored in the source file system. Then, log identification is performed on the candidate operation log based on the candidate log identifier to obtain a target operation log; wherein the target operation log is used to indicate that the second file is updated to a third file, and the target operation log includes a candidate log entry with an entry generation time. Finally, a target log entry is selected from the candidate log entries based on the entry generation time, and operations on the first file are replayed based on the target log entry to obtain a synchronized file; wherein the synchronized file is identical to the third file. Therefore, the data synchronization method and device, electronic device, and storage medium proposed in the present application, when the target file system stores a first file and the source file system stores a second file, and the first and second files are identical, can record the modification operation on the second file in a candidate operation log after modifying the second file to obtain a third file, and transmit the candidate operation log to the target file system, so that the target file system can make the same modification to the first file as the second file based on the modification operation recorded in the candidate operation log, ultimately making the first file and the third file identical, thereby achieving data synchronization between file systems. Since the source file system only needs to transmit the candidate operation log to the second file, the amount of data transmitted between the source and target file systems can be reduced, that is, the IO overhead is reduced, and thus the IO amplification is reduced, so that the efficiency of data synchronization between file systems is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is an optional flowchart of the data synchronization method provided in an embodiment of the present application;

[0046] Figure 2 yes Figure 1 Step S101 in an optional flowchart;

[0047] Figure 3 yes Figure 1 Another optional flowchart of step S101;

[0048] Figure 4 yes Figure 1 Flowchart of step S102 in FIG.

[0049] Figure 5 yes Figure 1 Flowchart of step S104 in FIG.

[0050] Figure 6 This is another optional flowchart of the data synchronization method provided in an embodiment of the present application;

[0051] Figure 7 yes Figure 6 Flowchart of step S604 in FIG.

[0052] Figure 8 This is a flow chart of the data synchronization method provided in an embodiment of the present application for communication between a source file system and a target file system;

[0053] Figure 9A 1 is a structural diagram of a data synchronization system when a data sending device according to an embodiment of the present application transmits data to a data synchronization device through a synchronous transmission method;

[0054] Figure 9B 2. It is a structural diagram of a data synchronization system in which a data sending device according to an embodiment of the present application transmits data to a data synchronization device via asynchronous transmission;

[0055] Figure 10 is a structural diagram of a data sending device provided in an embodiment of the present application;

[0056] Figure 11 is a structural diagram of a data synchronization device provided in an embodiment of the present application;

[0057] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0059] It should be noted that although the system and device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the system and device, or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0061] First, let’s analyze some of the terms used in this application:

[0062] Input / Output Overhead (IO Overhead): The additional resource consumption and time delay caused by hardware limitations, protocol processing, resource competition, etc. when a computer system performs data input or output operations. Among them, data input operations can be reading files or receiving network data packets, and data output operations can be writing to disk or sending network responses. IO overhead is one of the important indicators for measuring system performance because it directly affects the response time and overall efficiency of the application. At the hardware level, IO overhead affects disk seek time, network transmission delay, and memory copy overhead. At the software level, IO overhead affects system call context switching, data serialization / deserialization, and cache miss penalties. For example, in a file system, when multiple files are written at a high frequency, the frequent movement of the disk head will lead to a decrease in throughput.

[0063] Input / output amplification (I / O Amplification, IO amplification): In a computer system, due to improper input and output operations, or improperly designed logic for performing file operations, the actual IO operation volume is far greater than the minimum operation volume required by the business logic, which in turn leads to degraded system performance, reduced resource utilization, and extended response time. IO amplification includes: write amplification, read amplification, and network transmission amplification. Taking write amplification as an example, the solid-state drive in the file system, due to the flash memory block erase mechanism, causes the actual amount of physical data written to be far greater than the logical data volume. Taking read amplification as an example, in order to read a file in the file system, it is necessary to read the entire data block where the file is located. Taking network transmission amplification as an example, IO amplification can be the frequent sending of small data packets, resulting in an excessively high proportion of protocol header overhead.

[0064] Operation Replay: Records and re-executes historical operation sequences to reproduce system state changes or verify logical correctness. Operation replay is commonly used for system failure recovery, program debugging, and testing.

[0065] Full Sync: Copies all data from the source file system to the target file system, regardless of whether the data already exists or has been updated. Full Sync ensures data consistency between file systems, but it consumes significant storage and computing resources and has a long synchronization cycle. Full Sync is suitable for file system initialization or recovery from data corruption.

[0066] Incremental Sync: Synchronizes only the differences between the source and target file systems, significantly reducing data transfer. Differential data refers to data that is generated after adding, updating, or deleting the original data, and is partially or completely different from the original data.

[0067] To achieve multi-task collaboration, task-related data needs to be synchronized between different file systems. For example, hybrid cloud data migration, multi-platform application development, and cross-institutional data sharing. In the field of financial technology, taking insurance product recommendations as an example, different clients need to obtain the insurance product data purchased by users, and then synchronize the insurance product data to the file system of the data analysis platform to analyze the insurance product data and obtain recommended insurance product data. In the field of digital medicine, in order to achieve joint research on rare cases, for cases from different hospitals and research institutions, it is necessary to synchronize case-related information stored in the file system, such as disease diagnosis information and treatment status information.

[0068] However, in order to protect the privacy, security and integrity of the information within the file, when data is synchronized between file systems, the data within the file will not be obtained and transmitted, but the entire file will be transmitted. That is, the existing data synchronization method only supports data synchronization with files as the smallest unit. For example, after a file in the source file system is modified, added or deleted, the source file system will transfer the file to the target file system so that the file replaces, adds or deletes the file in the target file system. Similarly, after a file is added or deleted in the source file system, the file will be transferred to the department file system so that the corresponding incremental synchronization such as addition or deletion of the file will be performed in the target file system. For another example, after multiple files in the source system are modified, all files in the source file system are transferred to the target file system and all files in the target file system are replaced to achieve a full scan and full synchronization of the file system. After some files in the source file system are modified, a full scan of the target file system and full synchronization between file systems will result in huge invalid IO overhead and IO amplification due to the transfer and synchronization of unmodified files. If some files in the source file system are modified, even if only the modified files are recorded and synchronized, and these modified files are synchronized between file systems, due to the large amount of data stored in some file systems, full data synchronization between file systems using files as the minimum unit will still result in significant inefficient I / O overhead and I / O amplification. Therefore, full data synchronization between file systems using files as the minimum unit will result in low data synchronization efficiency.

[0069] Based on this, embodiments of the present application provide a data synchronization method and device, an electronic device, and a storage medium, which aim to reduce invalid IO overhead and IO amplification, thereby improving the efficiency of data synchronization.

[0070] The data synchronization method and device, electronic device, and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the data synchronization method in the embodiments of the present application is described.

[0071] The data synchronization method provided in the embodiment of the present application relates to the field of computer technology and is applicable to the fields of financial technology and digital medicine. The data synchronization method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the data synchronization method, etc., but is not limited to the above forms.

[0072] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0073] Figure 1 This is an optional flowchart of the data synchronization method provided in the embodiment of the present application. Figure 1 The method described in this article is applied to a target file system, which is a file system that receives and synchronizes data based on the received data. The data received by the target file system is sent by a source file system. The source file system is the file system that modifies the data stored in the system. This modification can be performed by an application external to the file system by calling a file, or by modifying a pre-stored script within the file system, rather than by transmitting the data from the file system.

[0074] It should be noted that since data synchronization between file systems can be mutual, the source file system and the target file system are determined only according to the direction of data transmission. In actual application scenarios, any file system can be used only as a source file system, only as a target file system, or as both a source file system and a target file system.

[0075] It should be noted that the number of file systems involved in data synchronization can be more than two. When there are at least three file systems requiring data synchronization, one of the file systems can be used as the source file system, and the others as the target file systems. Alternatively, multiple file systems can be used as both the source and target file systems, with the remaining file systems as target file systems. Alternatively, each file system can be used as both the source and target file systems. For example, if there are three file systems in total, the three file systems can be named the first, second, and third file systems. Based on this, only the first file system can be used as the source file system, and the second and third file systems can be used as the target file systems. Alternatively, both the first and second file systems can be used as the source and target file systems, with only the third file system used as the target file system. In this case, if a file in the first file system is modified, the first file system can be treated as the source file system, and the second and third file systems can be treated as the target file systems. Operation logs can be sent from the first file system to the second and third file systems. Similarly, if a file in the second file system is modified, the second file system can be treated as the source file system, and the first and third file systems can be treated as the target file systems. Operation logs corresponding to the modification operations can be sent from the second file system to the first and third file systems. You can also use all three file systems as source and target file systems. When a file in any file system is modified, that file system will be considered the source file system, and the other file systems will be considered the target file systems, so that operation logs can be sent from one source file system to multiple target file systems.

[0076] It should be noted that the source file system proposed in the embodiment of the present application stores a second file, and the target file system stores a first file. Moreover, before the second file in the source file system is modified, the second file and the first file, their file names and internal data are the same. Therefore, when the second file is modified into the third file, the embodiment of the present application can record the operation of modifying the second file as an operation log, transfer the operation log between the source file system and the target file system, and replay the operation of the first file in the target file system according to the operation log, so that the first file is the same as the modified file. In the field of financial technology, taking the recommendation of insurance products as an example, the first file, the second file, and the third file can store the user's purchased product data or insurance product recommendation data. Taking the reimbursement of insurance products as an example, the first file, the second file, and the third file can store the user's purchased insurance product policy data, reimbursement form data, or remaining reimbursable amount data. In the field of digital medicine, taking the joint research on rare cases as an example, the first file, the second file, and the third file can store aggregated patient personal information, patient disease diagnosis information, or treatment status information.

[0077] Figure 1 The method shown may include but is not limited to steps S101 to S104.

[0078] Step S101: In response to a data transmission request from a source file system, a candidate operation log marked with a candidate log identifier is obtained from the source file system; wherein the first file is the same as the second file stored in the source file system;

[0079] Step S102: performing log identification on the candidate operation log according to the candidate log identifier to obtain a target operation log; wherein the target operation log is used to indicate that the second file is updated to the third file, and the target operation log includes a candidate log entry having an entry generation time;

[0080] Step S103, selecting a target log entry from the candidate log entries according to the entry generation time;

[0081] Step S104 , replaying the first file according to the target log entry to obtain a synchronized file; wherein the synchronized file is the same as the third file.

[0082] In steps S101 to S104 shown in the embodiment of the present application, in the target file system storing the first file, first, in response to the data transmission request of the source file system, a candidate operation log marked with a candidate log identifier is obtained from the source file system; wherein the first file is the same as the second file stored in the source file system. Then, log identification is performed on the candidate operation log based on the candidate log identifier to obtain a target operation log; wherein the target operation log is used to indicate that the second file is updated to the third file, and the target operation log includes a candidate log entry with an entry generation time. Finally, a target log entry is selected from the candidate log entries based on the entry generation time, and the operation of the first file is replayed based on the target log entry to obtain a synchronized file; wherein the synchronized file is the same as the third file. Therefore, the data synchronization method proposed in this application, when the target file system stores a first file and the source file system stores a second file, and the first and second files are identical, can record the modification operation on the second file in a candidate operation log after modifying the second file to obtain a third file, and transmit the candidate operation log to the target file system, so that the target file system can make the same modification to the first file as the second file based on the modification operation recorded in the candidate operation log, ultimately making the first and third files identical, thereby achieving data synchronization between the file systems. Because the source file system only needs to transmit the candidate operation log to the second file, the amount of data transmitted between the source and target file systems can be reduced, that is, IO overhead is reduced, IO amplification is reduced, and the efficiency of data synchronization between the file systems is improved.

[0083] In step S101 of some embodiments, the candidate operation log refers to the operation log sent from the source file system to the target file system, and the candidate operation log is marked with a candidate log identifier. The candidate log identifier is used to uniquely identify the candidate log and is stored separately by the source file system and the target file system, so that the target file system can identify whether the candidate operation log has been replayed and re-acquire the candidate operation log based on the candidate log identifier when the replay of the candidate operation log fails. The data transfer request refers to an operation initiated by the source file system to the target file system for sending the candidate operation log from the source file system to the target file system. On this basis, the target file system can respond to the data transfer request of the source file system and obtain the candidate operation log marked with the candidate log identifier from the source file system.

[0084] In some embodiments, the data transmission request includes a data transmission mode, which includes an asynchronous transmission mode and a synchronous transmission mode. The asynchronous transmission mode refers to the case where the source file system transmits the candidate operation log to the target file system without strict time synchronization requirements. That is, the time when the source file system sends the candidate operation log and the time when the target file system receives the candidate operation log may not be the same.

[0085] For example, in an insurance product recommendation system in the fintech sector, files that record user personal information or detailed information about purchased insurance products, which have low real-time synchronization requirements, can be modified to generate candidate operation logs that can be transferred from the source file system to the target file system via asynchronous transmission. In the digital healthcare sector, for example, in collaborative research on treatments for rare diseases, candidate operation logs can be transferred via asynchronous transmission. These files can include medication records or medical history information, which have low real-time synchronization requirements.

[0086] Synchronous transmission means that the source and target file systems transmit candidate operation logs under time synchronization and use the same clock signal to ensure data synchronization. That is, the source file system sends candidate operation logs at the same time as the target file system receives them.

[0087] For example, in the insurance product system of the fintech sector, for insurance product renewal and policy reimbursement services, after a user's purchased insurance product is renewed, the remaining reimbursable amount and policy period of the purchased insurance product change. The file recording the remaining reimbursable amount and policy period of the purchased insurance product needs to be modified and synchronized in real time to facilitate the smooth reimbursement of the purchased insurance product in subsequent policy reimbursement services. In the field of digital medicine, taking the joint research on treatments for rare diseases as an example, the files that need to be transmitted through synchronous transmission for candidate operation logs can be files that record case diagnosis information, treatment plans, etc., so that researchers can promptly understand the case's condition and study treatment plans.

[0088] It should be noted that the "same time" here refers to the short period between when the source file sends the candidate operation log and when the target file system receives it, which can be considered the same. This does not mean that the moments are exactly the same. For example, the source file system sends the candidate operation log at 13:14:52 on January 2, 2024, and the target file system receives it at 13:14:54 on January 2, 2024.

[0089] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S202:

[0090] Step S201: Send a log acquisition request to a message queue of a shared storage in an asynchronous transmission mode; wherein the candidate operation log is written into the message queue by the source file system, and the shared storage is shared by the source file system and the target file system;

[0091] Step S202: receiving a candidate operation log shared by a shared storage message queue according to a log acquisition request.

[0092] In step S201 of some embodiments, during data transmission between file systems, if a large amount of data is transmitted in a short period of time, the source file system and the target file system will allocate a large amount of resources to receive the data, reducing the resources allocated by the source file system and the target file system for other tasks, thereby reducing the overall work efficiency of the source file system and the target file system. Therefore, the source file system needs to transmit candidate operation logs to the target file system via asynchronous transmission. Specifically, the source file system first sends the candidate operation logs to shared storage, and then the target file system sends a log acquisition request to the shared storage according to a specific log acquisition rule. The log acquisition rule can be at least one of the following: sending a log acquisition request to the shared storage according to a specific time period, which can be once an hour, once a day, or another time period; comparing the target system load of the target file system with a preset system load threshold to obtain load comparison information; if the load comparison information indicates that the target system load is lower than the system load threshold, obtaining the candidate operation log from the shared storage; the system load threshold can be 80% or other specific value; and immediately sending a log acquisition request to the shared storage in response to the data transmission request from the source file system.

[0093] It should be noted that, since the candidate operation log is shared between the source file system and the target file system only through shared storage, when multiple target file systems read the same candidate operation log, the target file system may read the candidate operation log repeatedly or miss the candidate operation log. Moreover, it is difficult to ensure that the order in which the target file system obtains the candidate operation log from the shared storage is consistent with the order in which the source file system sends the candidate operation log to the shared storage. This may cause the order in which the target file system replays operations based on the candidate operation log to be different from the order in which the source file system modifies the files, thereby making it impossible to synchronize data between the file systems. Therefore, it is necessary to set up a message queue in the shared storage. The message queue is used to store the candidate operation logs shared by the source file system to the shared storage through asynchronous transmission, and to share the candidate operation logs to the target file system according to the order in which the source file system shares the candidate operation logs.

[0094] In step S202 of some embodiments, the target file system receives candidate operation logs shared by a message queue on a shared storage in response to a log retrieval request. Specifically, the target file system may pre-subscribe to the message queue and then pull the candidate operation logs from the message queue in response to the log retrieval request. Alternatively, the shared storage may push the candidate operation logs to the target file system in response to the log retrieval request.

[0095] It should be noted that the message queue contains at least one topic, which corresponds to a source file system and at least one target file system, and is used to share candidate operation logs from the source file system to the target file system that subscribes to the topic. Specifically, when the number of file systems participating in data synchronization is at least three, and there are at least two target file systems corresponding to one source file system, at least two target file systems jointly subscribe to a topic initiated by the source file system for sharing candidate operation logs, and all target file systems in the topic are a target system group. The candidate operation logs shared by the source file system to the topic will eventually be shared to all target file systems in the target system group.

[0096] In the steps S201 to S202 shown in the embodiment of the present application, a log acquisition request is first sent to a message queue of shared storage according to an asynchronous transmission method; wherein, the candidate operation log is written to the message queue by the source file system, and the shared storage is shared by the source file system and the target file system. Then, the candidate operation log shared by the message queue of the shared storage according to the log acquisition request is received. Therefore, the candidate operation log can be asynchronously transmitted through the shared storage, so that the candidate operation log sending operation of the source file system and the candidate operation log reception of the target file system do not have to be performed at the same time, that is, the real-time transmission requirement of the candidate operation log is reduced, thereby reducing the resource allocation amount of the source file system and the target file system when transmitting the candidate operation log, and ultimately improving the overall work efficiency of the source file system and the target file system. Moreover, by setting up a message queue in the shared storage to store and share the candidate operation logs, it can ensure that the target file system can read the candidate operation logs and replay the operations in the order in which the source file system sends the candidate operation logs to the shared storage, thereby improving the accuracy of data synchronization and ensuring that all target file systems receive and only receive the candidate operation logs once when there are multiple target file systems, thus avoiding repeated reading or omissions. While ensuring that data synchronization can be achieved, it reduces the IO overhead and IO amplification caused by repeated acquisition of candidate logs.

[0097] See also Figure 3 In some embodiments, the data transmission mode further includes a synchronous transmission mode, and step S101 may further include but is not limited to steps S301 to S302:

[0098] Step S301, calling a synchronous receiving interface according to a synchronous transmission mode;

[0099] Step S302: receiving the candidate operation log sent from the synchronous sending interface of the source file system through the synchronous receiving interface.

[0100] In step S301 of some embodiments, during data transmission between file systems, if all candidate operation logs are transmitted asynchronously, files with high real-time data synchronization requirements will be difficult to transfer from the source file system to the target file system in a timely manner. Therefore, for candidate operation logs with high real-time synchronization requirements, they need to be transmitted using a synchronous transmission method. For example, in the field of financial technology, when it is necessary to synchronize product resource transfer data in real time to conduct product risk assessments, obtain product risk assessment data, and then make real-time product recommendations based on the product risk assessment data, a synchronous transmission method can ensure that the resource data in the target file system remains highly consistent with the resource data in the source file system, thereby providing timely and accurate data support for real-time product recommendations. In addition, in the field of digital healthcare, when hospitals need to share patients' emergency medical information in real time to promptly access medical resources for patients based on emergency medical information, a synchronous transmission method can ensure that emergency medical information and the allocation status of medical resources are instantly transmitted and synchronized between the medical resource allocation systems of each hospital. On this basis, a synchronous reception interface can be called according to the synchronous transmission method. Specifically, when the target file system responds to a data transmission request from the source file system, it will call the synchronous reception interface according to the synchronous transmission method in the data transmission request. The synchronous receiving interface is used to receive candidate operation logs sent by the source file system through synchronous transmission. A communication channel is established in advance with the synchronous sending interface connected to the source file system.

[0101] In step S302 of some embodiments, the synchronous sending interface of the source file system refers to an interface for the source file system to send the candidate operation log to the target file system through synchronous transmission.

[0102] In steps S301 to S302 shown in the embodiment of the present application, the synchronous receiving interface is first called according to the synchronous transmission method, and then the candidate operation log sent from the synchronous sending interface of the source file system is received through the synchronous receiving interface. Therefore, the candidate operation log can be actually transmitted for files with real-time synchronization requirements through the synchronous transmission method, thereby ensuring the real-time and consistency of file synchronization between file systems.

[0103] In some embodiments, in step S102, due to network delays or loss, the source file system mistakenly believes that a previously sent candidate operation log has failed to reach the target file system, and therefore sends the same candidate operation log again, causing the target file system to receive duplicate candidate operation logs. Alternatively, the source file system encounters a failure or anomaly when generating the candidate operation log, resulting in duplicate candidate operation logs being sent to the target file system multiple times. Therefore, it is necessary to perform log identification on the candidate operation log based on the candidate log identifier to obtain the target operation log, so as to ensure that the target file system is synchronized only once based on a candidate operation log, and to ensure that the target file system is synchronized according to the order in which the source file system updates the files.

[0104] The target operation log is selected based on the chronological order of the candidate operation logs shared by the source file system. Furthermore, the target operation log is unique among all candidate operation logs received by the target file system. Furthermore, because the target operation log is used in the source file system to indicate that the second file has been updated to the third file, the target file system can update the first file, which is identical to the second file, based on the target operation log to obtain a synchronized file identical to the third file. Therefore, the target file system will only update the first file based on the target operation log after it has been selected as the target operation log. In other words, the target operation log must not have been updated by the target file system.

[0105] In some embodiments, the target operation log includes candidate log entries with entry generation times. A candidate log entry is a log entry that is the smallest unit recording a source file operation. Each candidate log entry details a specific file modification operation, indicating updates to files in both the source and target file systems. The entry generation time is the time when the candidate log entry was generated and can describe the time when the source file system performed the file operation. Since the source file system can only perform one operation on the same file at a time, the order of file operations can be determined based on the entry generation time, and the file operations can then be replayed based on this order.

[0106] See also Figure 4In some embodiments, the candidate operation log has a log sharing time, which is the time when the source file system sends the candidate operation log. The specific method for determining the log sharing time may be: the source file system timestamps the candidate operation log before sending the candidate operation log. Since the source file system will send the candidate operation log immediately after timestamp the candidate operation log with the log sharing time, the timestamp can be regarded as the log sharing time. In addition, since the time when the source file system generates the candidate operation log and the time when the candidate operation log is sent are very close, the log generation time of the candidate operation log can also be regarded as the log sharing time. Step S102 of the data synchronization method may include but is not limited to steps S401 to S403:

[0107] Step S401: replay and identify candidate logs based on candidate log identifiers, and select intermediate operation logs from the candidate logs; wherein the intermediate operation logs are operation logs that have not been replayed;

[0108] Step S402: sort the intermediate operation logs to obtain an operation log sequence;

[0109] Step S403 : Select intermediate operation logs from the operation log sequence in sequence according to the log sharing time as target operation logs.

[0110] In step S401 of some embodiments, the intermediate operation log is an operation log that has not been replayed. On this basis, the candidate log is replayed and identified according to the candidate log identifier, and the intermediate operation log can be screened out from the candidate log. Specifically, the candidate log is replayed and identified according to the candidate log identifier, including: first, obtaining the historical log identifier; wherein the historical log identifier is: the log identifier of the target operation log after the target file system updates the file according to the received target operation log before obtaining the candidate operation log this time; then, the historical log identifier and the candidate log identifier are compared to obtain the first identifier comparison information, and when the first identifier comparison information indicates that each historical log identifier is different from the candidate log identifier, the candidate log identifier is determined to be the intermediate operation log, or when the first identifier comparison information indicates that any historical log identifier is the same as the candidate log identifier, the candidate operation log is deleted.

[0111] After step S401 in some embodiments, it is further necessary to determine if the intermediate operation logs are duplicated. Specifically, a selected operation log is selected from the intermediate operation logs, and each intermediate operation log other than the selected operation log is used as an identification operation log. The selected operation log and the identification operation log are different, and the selected operation log is annotated with a selected log identifier, while the identification operation log is annotated with an identification log identifier. The selected log identifier and the identification log identifier are compared to obtain second identifier comparison information. If the second identifier comparison information indicates that the selected log identifier is identical to any other log identifier, the selected operation log is deleted.

[0112] In step S402 of some embodiments, to ensure that the order in which the target file system replays file operations is consistent with the order in which the source file system updates the files, the intermediate operation logs need to be sorted to obtain an operation log sequence. The operation log sequence is a dynamically changing sequence. Specifically, each time the target file system receives a new candidate operation log, the new candidate operation log is added to the operation log sequence. After the target file system replays operations based on the target operation log, the target operation log is removed from the operation log sequence.

[0113] In some embodiments, in step S403, after the intermediate operation logs are placed in the operation log sequence, intermediate operation logs can be sequentially selected from the operation log sequence based on the log sharing time, and used as target operation logs. This ensures that the target file system replays the operation logs according to the order in which the source file system updated the files, thereby ensuring that the target file system and the source file system update the files in the same order. Specifically, the intermediate operation log with the earliest log sharing time can be selected as the target operation log by comparing the log sharing times.

[0114] In steps S401 to S403 shown in the embodiment of the present application, the candidate logs are first replayed and identified based on the candidate log identifier to filter out the operation logs that have not been replayed as intermediate operation logs, and then the intermediate operation logs are selected from the operation log sequence in sequence according to the log sharing time as the target operation logs. Therefore, the data synchronization method shown in the embodiment of the present application can select the operation logs that have not been replayed as intermediate operation logs based on the candidate log identifier to ensure that each candidate operation log will only be replayed once by the target file system, and by selecting the target operation log based on the log sharing time, it can be ensured that the order in which the target file system and the source file system update files is the same, thereby ensuring the synchronization of file updates.

[0115] In step S103 of some embodiments, since the order in which the source file system operates on the file can be determined based on the entry time, and the same files exist in both the source file system and the target file system, the operation order can be determined based on the entry generation time, and then the candidate log entries are selected one by one as the target log entries according to the operation order to replay the operations on the files in the target file system according to the target log entries.

[0116] In step S104 of some embodiments, the synchronization file is the file obtained after the target file system updates the first file. Since the update of the first file is implemented according to the target operation log, and the target operation log is used to guide the source file system to update the second file to the third file, the synchronization file and the third file are the same. Operation playback refers to recording the operation of updating the second file in the source file system and re-executing the operation in the target file system to reproduce the state change of the source file system in the target file system. Therefore, the data synchronization method proposed in the embodiment of the present application selects the target operation log according to the log sharing time, and updates the first file in the target file system according to the target operation log. It can perform the playback operation in chronological order, thereby reducing the operation conflicts caused by improper order of operation playback, that is, realizing automatic conflict resolution.

[0117] It should be noted that the semantics of the operation logs recorded in this application are standard file interface semantics, which can be universal across platforms and operating systems, such as the Portable Operating System Interface (POSIX) standard semantics. Therefore, it is possible to support synchronization of candidate operation logs between file systems on different platforms and different operating systems. Among them, different platforms can be web and desktop apps, etc., and different operating systems can be Windows systems, Linux systems, and macOS systems, etc.

[0118] In some embodiments, the target log entry includes the operation type, the full file path, and the operation data. The operation type refers to the type of modification operation performed by the file system on the file information, such as setattr and create. The semantics of the operation type conform to the POSIX standard, facilitating data transmission and operation playback between file systems, thereby ensuring data consistency between file systems.

[0119] The full file path indicates the location of the file being operated on in both the source and target file systems. It's important to note that in different file systems, especially those running different operating systems, the inodes for the same file may differ. Therefore, when synchronizing data between file systems running different operating systems, it's not possible to find the first file based on the inode of the second file. Therefore, it's necessary to set the same full file path for the file systems involved in data synchronization—that is, to set the same full file path for the first and second files—so that the second file can be found in the target file system based on the full file path of the first file.

[0120] Operational data refers to data used to modify the first file, and is used to modify the original data of the second file in the target file system so that the synchronized file obtained after the modification is identical to the third file. It should be noted that the operational data in this application refers to the portion of the file's internal data that has changed after the modification, rather than all the data within the file. Operational data can be either file metadata or data stored within the file, such as text, structured data, log data, encoded data, or markup language.

[0121] It should be noted that the target log entry can be recorded in the target operation log in the form of {op, fullpath, data}, where op is the operation type, fullpath is the full file path, and data is the operation data.

[0122] See also Figure 5 In some embodiments, step S104 may include but is not limited to steps S501 to S504:

[0123] Step S501, performing a path check on the first file according to the full path of the file to obtain path check information;

[0124] Step S502: If the path verification information indicates that the first file exists under the full file path, a modification operation is called according to the operation type;

[0125] Step S503, calling the first file according to the full file path;

[0126] Step S504: write the operation data into the first file according to the modification operation to obtain a synchronization file.

[0127] In step S501 of some embodiments, the target file system may mistakenly delete the first file or perform other reasons, resulting in the first file not existing under the full file path. Therefore, before replaying the first file, it is necessary to first perform a path check on the first file based on the full file path to obtain path check information. The path check information is used to describe whether the full file path exists in the target file system. The path check information includes path existence information and path error information. Specifically, the full file path includes a second file directory and a second file name. The second file directory indicates the directory where the second file is located, and the second file name is the name of the second file. Based on this, the target file system is first searched for a directory identical to the second file directory to obtain directory check information. If the directory check information indicates that a directory identical to the second file directory exists in the target file system, the directory is determined to be the first file directory. Then, the first file directory is searched for a file name identical to the second file name to obtain name check information. If the name check information indicates that a file name identical to the second file name exists in the first file directory, the file name is determined to be the first file name, and the path check information is determined to be path existence information. The path existence information indicates that the first file exists under the full file path. Otherwise, it is determined that the path verification information is path error information, and the path error information indicates that the first file directory or the first file does not exist under the full file path.

[0128] In step S502 of some embodiments, when the path verification information indicates that the first file exists under the full path of the file, a modification operation can be called according to the operation type. The modification operation is used to modify the first file. Specifically, if the semantics of the operation in the target file system are POSIX standard semantics, the modification operation corresponding to the operation type can be directly called. If the semantics of the operation in the target file system are not POSIX standard semantics, it is necessary to preset a semantic conversion table and search the semantic conversion table for the corresponding modification operation according to the operation type to achieve operation playback of the files of the target file system.

[0129] In step S504 of some embodiments, the operation data can be written to the first file based on the modification operation to obtain a synchronized file. Specifically, if the operation data is metadata, the operation data includes metadata attributes and metadata synchronization values. First, an attribute search is performed on the first file based on the metadata attributes to obtain initial attributes; the initial attributes correspond to metadata initial values. Then, the initial metadata values are overwritten based on the modification operation and the metadata synchronization value to obtain synchronized attributes, and the first file containing the synchronized attributes is used as the synchronized file. For example, the target log entry is {setattr, "D:\data\filea.txt", "timestamp", "2025-3-18 00:00"}, where setattr is the operation type, the corresponding modification operation is setting file attributes, "D:\data\filea.txt" is the full file path of the second file, "D:\data\" is the second file directory, "filea.txt" is the second file name, "timestamp" is the timestamp, which is the metadata attribute of the second file, and "2025-3-18 00:00" is the metadata synchronization value. On this basis, first find the "timestamp" attribute of the file filea.txt, and then replace the value of "timestamp" with "2025-3-18 00:00".

[0130] If the operation data is data stored within a file, for structured data, log data, and markup language, the method for writing the operation data to the first file based on the modification operation can refer to the embodiment described above for writing operation data that is metadata. For unstructured data such as text and encoded data, the operation data may include, but is not limited to, a data offset or a synchronization data value. Based on this, the method for writing the operation data to the first file based on the modification operation can be to modify the initial data to the synchronization data value based on the modification operation, obtain the synchronization data, and use the first file containing the synchronization data as the synchronization file. For example, the target log entries are {lseek, "E:\data\fileb.txt", 20} and {write, "E:\data\fileb.txt", datab}, where lseek and write are operation types, lseek corresponds to a modification operation for locating the data offset, and write corresponds to a modification operation for writing the file. "D:\data\fileb.txt" is the full file path of the second file, "D:\data\" is the second file directory, "fileb.txt" is the second file name, "20" is the data offset, and "datab" is the synchronization data value. On this basis, we can first use the lseek operation to locate the 20th byte of the file fileb.txt, and then use the write operation to replace the data starting from the 20th byte with datab, where the length of the replacement data is the length of datab, to obtain the synchronization data, and save the parsed file containing the synchronization data as a synchronization file.

[0131] In the steps S501 to S504 shown in the embodiment of the present application, a path check is first performed on the first file according to the full path of the file to obtain path check information, and then, when the path check information indicates that the first file exists under the full path of the file, a modification operation is called according to the operation type. Then, the first file is called according to the full path of the file, and the operation data is written to the first file according to the modification operation to obtain a synchronization file. Therefore, the data synchronization method shown in the embodiment of the present application performs a path check on the first file according to the full path of the file recorded in the target log entry, so that when the first file exists under the full path of the file, the first file is called, and the operation data is written to the first file according to the modification operation corresponding to the operation type to obtain a synchronization file, so that the first file in the target file system can correspond to the synchronization modification operation of the second file in the source file system, and finally obtain a synchronization file that is the same as the third file. In addition, since the target log entry of the target operation log only records the operation type, the full path of the file and the operation data, and the operation data is only the data that has changed in the second file, the data volume of the target operation log is much smaller than the data volume of the second file. Therefore, compared with transmitting the second file, transmitting the target operation log can reduce the data transmission time between file systems, thereby improving the efficiency of data synchronization between file systems.

[0132] See also Figure 6 In some embodiments, the data synchronization method is applied to the source file system. The method may also include but is not limited to steps S601 to S605:

[0133] Step S601: In response to receiving a file operation request, generating a candidate operation log marked with a log identifier according to the received file operation request; wherein the candidate operation log has a log data volume, and the file operation request includes a data synchronization priority and a data synchronization time window;

[0134] Step S602: updating the second file according to the candidate operation log to obtain a third file; wherein the target file system performs log identification on the candidate operation log to obtain a target operation log, and the target operation log is used to indicate that the second file is updated to the third file;

[0135] Step S603, obtaining network transmission parameters and target system load of the target file system;

[0136] Step S604: selecting a data transmission method based on the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters, and the target system load;

[0137] Step S605, generate a data transmission request according to the data transmission mode and the candidate operation log, and send the data transmission request to the target file system; wherein, the data transmission request is used by the target file system to replay the operation of the first file stored in the target file system to obtain a synchronized file, the second file is the same as the first file, and the third file is the same as the synchronized file.

[0138] In step S601 of some embodiments, a file operation request is a method of interaction between a program and a source file system. It is used to perform operations on a specified file in the source file system, such as reading, writing, creating, deleting, and renaming. It can be issued by an application or a user. The file operation request includes a data synchronization priority and a data synchronization time window. The data synchronization priority is used to determine the importance and synchronization urgency of the data synchronization task and can be determined based on the business corresponding to the file operation request. For example, data synchronization priorities can be divided into levels 1 to 5 based on importance and synchronization urgency. Data synchronization tasks at level 1 have the highest importance and synchronization urgency. Data synchronization tasks at level 1 may include synchronizing urgent medical information and the allocation status of medical resources, or resource transfer data used for product risk assessment. Data synchronization tasks at level 5 have the lowest importance and synchronization urgency and may include synchronizing modified personal information. The data synchronization time window refers to the period within a specific time period during which data synchronization is allowed or scheduled. Different files have different data synchronization time windows. For example, for a file containing medical resource allocation status information, the data synchronization time window may start at the time the file operation request is received and last for 1 minute. For files containing medical imaging data, the data synchronization window can be from 10:00 PM to 12:00 AM daily. A candidate operation log has a log data volume, which describes the size of the candidate operation log and can be used to determine the data transmission method. Therefore, in response to receiving a file operation request, a candidate operation log labeled with a log identifier can be generated based on the received file operation request.

[0139] In step S602 of some embodiments, the specific implementation of updating the second file to obtain the third file based on the candidate operation log may include: first, calling a modification operation based on the operation type in the candidate operation log, then calling the second file based on the full file path in the candidate operation log, and writing the operation data to the second file based on the modification operation in the candidate operation log to obtain the third file. A more specific implementation is similar to the specific embodiment of replaying the operation of the first file based on the target log entry to obtain the synchronized file. Please refer to the specific embodiments of steps S401 to S403, and this application will not elaborate on them.

[0140] In step S603 of some embodiments, since there may be network delays in the communication network between the source file system and the target file system, resulting in low data transmission efficiency between the source file system and the target file system, and thus low data synchronization efficiency between the source file system and the target file system, it is necessary to determine the data transmission method based on the real-time network status between the source file system and the target file system. Moreover, since the load of the target file system will affect the duration of the target file system to replay the operation of the candidate operation log, thereby affecting the low data synchronization efficiency, it is also necessary to determine the data transmission method based on the system load. Therefore, it is necessary to obtain network transmission parameters and target system load, wherein the network transmission parameters are used to represent the real-time network status, and the target system load is used to describe the load of the target file system, wherein the value range of the target system load is [0,1], for example 0.85.

[0141] In step S604 of some embodiments, since the data synchronization priority and the data synchronization time window can reflect the real-time requirements for synchronizing the candidate operation logs, the log data volume and network transmission parameters can be used to determine the data transmission duration for synchronizing the candidate operation logs. The target system load will affect the duration of the target file system's operation playback on the candidate operation logs. Therefore, the data transmission method can be selected based on the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters and the target system load.

[0142] In step S605 of some embodiments, after determining the data transmission method, a data transmission request can be generated based on the data transmission method and the candidate operation log, and the data transmission request can be sent to the target file system, so that the target file system can replay the operation on the first file according to the data transmission request to obtain a synchronized file, and the third file is the same as the synchronized file. For specific implementation methods, please refer to the aforementioned specific embodiment of applying the data synchronization method of the present application to the target file system, and this application will not repeat them here.

[0143] In steps S601 to S605 shown in the embodiment of the present application, in response to receiving a file operation request, a candidate operation log marked with a log identifier is first generated according to the received file operation request; wherein, the candidate operation log has a log data volume, the file operation request includes a data synchronization priority and a data synchronization time window, and the second file is the same as the first file stored in the target file system. Then, the second file is updated to obtain a third file according to the candidate operation log; wherein, the target file system performs log identification on the candidate operation log to obtain a target operation log, and the target operation log is used to indicate that the second file is updated to the third file. Thereafter, the network transmission parameters and the target system load of the target file system are obtained, and a data transmission method is selected according to the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters and the target system load. Finally, a data transmission request is generated according to the data transmission method and the candidate operation log, and the data transmission request is sent to the target file system, so that the target file system replays the operation of the first file according to the data transmission request to obtain a synchronized file that is the same as the third file. Therefore, the data synchronization method illustrated in the embodiment of the present application generates a candidate operation log according to a file operation request in the source file system, and after updating the second file according to the candidate operation log, by selecting the data transmission method according to the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters and the target system load, it can determine the data transmission method in real time, and transmit the candidate operation log to the target file system according to the data transmission method, thereby improving the efficiency of data transmission between the source file system and the target file system, thereby effectively reducing the invalid IO overhead and IO amplification caused by repeated transmission of the candidate operation log due to network delay, and reducing the data synchronization time caused by the blockage of the data transmission channel of the candidate operation log and the excessive load of the target system, thereby ultimately improving the data synchronization efficiency between file systems.

[0144] See also Figure 7 In some embodiments, step S604 may include but is not limited to steps S701 to S705:

[0145] Step S701, calculating the synchronization transmission probability according to the data synchronization priority, the data synchronization time window, the log data volume and the network transmission parameters;

[0146] Step S702: performing weighted averaging on the target system load and the synchronization transmission probability to obtain a total synchronization probability;

[0147] Step S703: comparing the total synchronization probability with a preset synchronization transmission threshold to obtain synchronization transmission comparison information;

[0148] Step S704: if the total synchronization probability represented by the synchronization transmission comparison information is greater than the synchronization transmission threshold, the data transmission mode is determined to be the synchronous transmission mode;

[0149] Step S705 : when the synchronous transmission comparison information indicates that the synchronization success probability is less than the synchronous transmission threshold, the data transmission mode is determined to be the asynchronous transmission mode.

[0150] In step S701 of some embodiments, the synchronous transmission probability can be calculated based on the data synchronization priority, the data synchronization time window, the log data volume and the network transmission parameters. Specifically, the network transmission parameters include: network delay and network bandwidth. On this basis, first, multiplication operation is performed according to the preset transmission attenuation coefficient, the operation log data volume and the unit data transmission time to obtain the transmission time pressure, and the network delay and the preset delay attenuation coefficient are multiplied to obtain the network delay pressure; then, according to the data synchronization priority, the target priority weight of the data synchronization priority is found from the preset priority weight table, and the time window length is determined according to the data synchronization time window, so as to multiply the target priority weight and the time window length to obtain the transmission pressure weight; finally, the exponential distribution probability is calculated according to the inverse of the transmission pressure weight, the network delay pressure and the transmission time pressure to obtain the synchronous transmission probability. The above steps can be expressed by the following formula:

[0151]

[0152] Among them, P 同步传输 is the synchronization transmission probability, λ is the transmission attenuation coefficient, D is the amount of operation log data, B is the network bandwidth, μ is the delay attenuation coefficient, L is the network delay, w is the target priority weight, T req is the time window length.

[0153] In step S702 of some embodiments, a weighted average of the target system load and the synchronization transmission probability may be performed to obtain a total synchronization probability. Specifically, the specific implementation of the weighted average of the target system load and the synchronization transmission probability may be expressed by the following formula:

[0154] P 总 =(1-a)P 同步传输 +a·TSL;

[0155] Among them, P 总 is the total synchronization probability, TSL is the target system load, and a is the weighting coefficient of the target system load.

[0156] In step S703 of some embodiments, a preset synchronous transmission threshold is used together with the total synchronization probability to determine the data transmission mode. Specifically, the total synchronization probability and the preset synchronous transmission threshold can be compared to obtain synchronous transmission comparison information, which describes whether the synchronous transmission threshold is greater than the total synchronization probability. For example, the synchronous transmission threshold can be set to 0.9 or other values between [0, 1].

[0157] In step S704 of some embodiments, if the synchronization comparison information indicates that the total synchronization probability is greater than the synchronization transmission threshold, the data transmission mode may be determined as the synchronization transmission mode. For example, if the synchronization transmission threshold is set to 0.8 and the total synchronization probability is 0.91, the data transmission mode may be determined as the synchronization transmission mode.

[0158] In step S705 of some embodiments, if the synchronous transmission comparison information indicates that the synchronization pass probability is less than the synchronous transmission threshold, the data transmission mode may be determined to be asynchronous transmission mode. For example, if the synchronous transmission threshold can be set to 0.9 and the total synchronization probability is 0.88, the data transmission mode is determined to be asynchronous transmission mode. When the data transmission mode is asynchronous transmission mode, the source file system writes the candidate operation log to the shared storage via RPC calls or TCP transmission.

[0159] It should be noted that if the synchronization pass probability is equal to the synchronous transmission threshold, the data transmission mode can be determined as either synchronous transmission mode or asynchronous transmission mode, which can be determined by those skilled in the art according to actual application conditions.

[0160] In steps S701 to S705 shown in the embodiment of the present application, the synchronization transmission probability is first calculated based on the data synchronization priority, the data synchronization time window, the log data volume and the network transmission parameters, and the target system load and the synchronization transmission probability are weighted averaged to obtain the total synchronization probability. Then, the total synchronization probability is compared with the preset synchronization transmission threshold to obtain the synchronization transmission comparison information. Among them, if the synchronization transmission comparison information indicates that the total synchronization probability is greater than the synchronization transmission threshold, the data transmission mode is determined to be the synchronous transmission mode; if the synchronization transmission comparison information indicates that the synchronization pass probability is less than the synchronization transmission threshold, the data transmission mode is determined to be the asynchronous transmission mode. Therefore, the data synchronization method shown in the embodiment of the present application can calculate the total synchronization probability based on the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters and the target system load, and determine the data transmission mode based on the total synchronization probability and the synchronization transmission threshold, thereby ensuring that the data transmission mode can adapt to different target file system loads, network status between file systems and business needs, thereby improving the efficiency of data synchronization between file systems.

[0161] See also Figure 8In an embodiment of the present application, the source file system first responds to a received file operation request and generates a candidate operation log marked with a log identifier according to the received file operation request; wherein the candidate operation log has a log data volume, and the file operation request includes a data synchronization priority and a data synchronization time window; then, the second file is updated to obtain a third file according to the candidate operation log; wherein, the target file system performs log identification on the candidate operation log to obtain a target operation log, and the target operation log is used to indicate that the second file is updated to the third file; then, the network transmission parameters and the target system load of the target file system are obtained, and a data transmission method is selected according to the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters and the target system load, so as to generate a data transmission request according to the data transmission method and the candidate operation log, and send the data transmission request to the target file system. In the target file system, first, in response to the data transmission request of the source file system, a candidate operation log marked with a candidate log identifier is obtained from the source file system; wherein the first file is the same as the second file stored in the source file system; then, a log acquisition request is sent to a message queue of a shared storage according to an asynchronous transmission method; wherein, the candidate operation log is written to the message queue by the source file system, and the shared storage is shared by the source file system and the target file system. Then, the candidate operation log is identified according to the candidate log identifier to obtain a target operation log; wherein, the target operation log is used to indicate that the second file is updated to the third file, and the target operation log includes a candidate log entry with an entry generation time. Then, the target log entry is selected from the candidate log entries according to the entry generation time, and the operation of the first file is replayed according to the target log entry to obtain a synchronization file; wherein, the synchronization file is the same as the third file. Therefore, the data synchronization method proposed in the embodiment of the present application, when the target file system stores the first file and the source file system stores the second file, and the first file and the second file are the same, after the second file is modified to obtain the third file, the modification operation of the second file can be recorded in the candidate operation log, and the candidate operation log is transmitted to the target file system, so that the target file system can perform the same modification as the second file on the first file according to the modification operation recorded in the candidate operation log, and finally make the first file and the third file the same, thereby achieving data synchronization between the file systems. Since the source file system only needs to transmit the candidate operation log to the second file, the amount of data transmitted between the source file system and the target file system can be reduced, that is, the IO overhead is reduced, and the IO amplification is reduced, so that the data synchronization efficiency between the file systems is improved.Moreover, in the source file system, a candidate operation log is generated according to the file operation request, and after the second file is updated according to the candidate operation log, the data transmission method is selected according to the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters and the target system load. The data transmission method can be determined in real time, and the candidate operation log is transmitted to the target file system according to the data transmission method, thereby improving the efficiency of data transmission between the source file system and the target file system, thereby effectively reducing the invalid IO overhead and IO amplification caused by repeated transmission of the candidate operation log due to network delay, and reducing the data synchronization time caused by the blockage of the data transmission channel of the candidate operation log and the excessive load of the target system, thereby improving the data synchronization efficiency between file systems.

[0162] See also Figure 9A The present application also provides a data synchronization system that implements the aforementioned data synchronization method. The system includes a data sending device 910 and a data synchronization device 920. The data sending device 910 is applied to the source file system, and the data synchronization device 920 is applied to the target file system. When the data transmission mode is synchronous, the data sending device 910 directly transmits the candidate operation log to the data synchronization device 920.

[0163] See also Figure 9B The data synchronization system provided in the embodiment of the present application also includes shared storage 930. When the data sending device 910 transmits the candidate operation log to the data synchronization device 920 through asynchronous transmission, the message queue of the shared storage 930 receives the candidate operation log shared by the data sending device 910 and shares the candidate operation log to the data synchronization device 920.

[0164] See also Figure 10 , the source file system stores a second file, and the data sending device 910 provided in this embodiment of the application includes:

[0165] The log generation module 911 is configured to generate a candidate operation log marked with a log identifier according to the received file operation request in response to receiving the file operation request; wherein the candidate operation log has a log data volume, and the file operation request includes a data synchronization priority and a data synchronization time window;

[0166] A file update module 912 is configured to update the second file according to the candidate operation log to obtain a third file; wherein the target file system performs log identification on the candidate operation log to obtain a target operation log, and the target operation log is used to indicate that the second file is updated to the third file;

[0167] Data acquisition module 913, used to obtain network transmission parameters and target system load of the target file system;

[0168] The transmission mode selection module 914 is used to select a data transmission mode based on the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters and the target system load;

[0169] The request sending module 915 is used to generate a data transmission request based on the data transmission method and the candidate operation log, and send the data transmission request to the target file system; wherein, the data transmission request is used by the target file system to replay the operation of the first file stored in the target file system to obtain a synchronized file, the second file is the same as the first file, and the third file is the same as the synchronized file.

[0170] The specific implementation of the data sending device 910 is basically the same as the specific embodiment of the above-mentioned data synchronization method, and will not be repeated here.

[0171] See also Figure 11 In some embodiments, the target file system stores a first file. The data synchronization device 920 provided in the embodiment of the present application includes:

[0172] The log acquisition module 921 is configured to, in response to a data transmission request from the source file system, acquire a candidate operation log marked with a candidate log identifier from the source file system; wherein the first file is the same as the second file stored in the source file system;

[0173] a log identification module 922 configured to perform log identification on the candidate operation log according to the candidate log identifier to obtain a target operation log; wherein the target operation log is used to indicate that the second file is updated to the third file, and the target operation log includes a candidate log entry having an entry generation time;

[0174] A data selection module 923 is configured to select a target log entry from candidate log entries based on the time at which the entry was generated;

[0175] The operation playback module 924 is configured to perform operation playback on the first file according to the target log entry to obtain a synchronized file; wherein the synchronized file is identical to the third file.

[0176] The specific implementation of the data synchronization device 920 is basically the same as the specific embodiment of the above-mentioned data synchronization method, and will not be repeated here.

[0177] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned data synchronization method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0178] See also Figure 12 , Figure 12The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0179] The processor 1201 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0180] The memory 1202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called by the processor 1201 to execute the data synchronization method of the embodiments of this application.

[0181] Input / output interface 1203, used to implement information input and output;

[0182] Communication interface 1204, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0183] Bus 1205 , which transmits information between various components of the device (e.g., processor 1201 , memory 1202 , input / output interface 1203 , and communication interface 1204 );

[0184] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0185] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data synchronization method is implemented.

[0186] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0187] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0188] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0189] The system and device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of these modules may be selected based on actual needs to achieve the objectives of the present embodiments.

[0190] Those skilled in the art will appreciate that all or some of the steps, systems, devices, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the system embodiments and device embodiments described above are merely schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system or device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system, device or unit can be electrical, mechanical or other forms.

[0192] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0193] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A data synchronization method, characterized in that: Applied to a target file system, the target file system storing a first file, the method includes: In response to a data transmission request from a source file system, obtaining a candidate operation log marked with a candidate log identifier from the source file system; wherein the first file is the same as a second file stored in the source file system; performing log identification on the candidate operation log according to the candidate log identifier to obtain a target operation log; wherein the target operation log is used to indicate that the second file is updated to a third file, and the target operation log includes a candidate log entry having an entry generation time; Selecting a target log entry from the candidate log entries according to the generation time of the entry; The first file is replayed according to the target log entry to obtain a synchronized file; wherein the synchronized file is the same as the third file.

2. The method according to claim 1, characterized in that The data transmission request includes a data transmission mode, and the data transmission mode includes an asynchronous transmission mode. Responding to the data transmission request of the source file system, obtaining a candidate operation log marked with a candidate log identifier includes: Sending a log acquisition request to a message queue of a shared storage according to the asynchronous transmission mode; wherein the candidate operation log is written into the message queue by the source file system, and the shared storage is shared by the source file system and the target file system; The candidate operation log shared by the shared storage message queue according to the log acquisition request is received.

3. The method according to claim 2, characterized in that The candidate operation log has a log sharing time, and performing log identification on the candidate operation log according to the candidate log identifier to obtain a target operation log includes: Performing playback identification on the candidate log according to the candidate log identifier, and screening out intermediate operation logs from the candidate log; wherein the intermediate operation logs are operation logs that have not been played back; Sorting the intermediate operation logs to obtain an operation log sequence; The intermediate operation logs are selected in sequence from the operation log sequence according to the log sharing time as the target operation logs.

4. The method according to claim 1, wherein The target log entry includes: operation type, full file path and operation data; The operating and replaying the first file according to the target log entry to obtain a synchronized file includes: Performing a path check on the first file according to the full path of the file to obtain path check information; In a case where the path verification information indicates that the first file exists under the full file path, calling a modification operation according to the operation type; Call the first file according to the full path of the file; The operation data is written into the first file according to the modification operation to obtain the synchronization file.

5. The method according to claim 2, characterized in that The data transmission mode also includes a synchronous transmission mode, and the step of obtaining the candidate operation log marked with the candidate log identifier in response to the data transmission request of the source file system further includes: Calling a synchronous receiving interface according to the synchronous transmission mode; The candidate operation log sent from the synchronous sending interface of the source file system is received through the synchronous receiving interface.

6. A data synchronization method, characterized in that: Applied to a source file system, where the source file system stores a second file, the method includes: In response to receiving a file operation request, generating a candidate operation log marked with a log identifier according to the received file operation request; wherein the candidate operation log has a log data volume, and the file operation request includes a data synchronization priority and a data synchronization time window; The second file is updated according to the candidate operation log to obtain a third file; wherein the target file system performs log identification on the candidate operation log to obtain a target operation log, and the target operation log is used to indicate that the second file is updated to the third file; Obtaining network transmission parameters and a target system load of the target file system; Selecting a data transmission mode according to the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters, and the target system load; A data transmission request is generated according to the data transmission mode and the candidate operation log, and the data transmission request is sent to the target file system; wherein the data transmission request is used by the target file system to replay the operation of the first file stored in the target file system to obtain a synchronization file, the second file is the same as the first file, and the third file is the same as the synchronization file.

7. The method according to claim 6, characterized in that The selecting of the data transmission mode according to the data synchronization priority, the data synchronization time window, the log data volume, the network transmission parameters, and the target system load includes: Calculating a synchronization transmission probability according to the data synchronization priority, the data synchronization time window, the log data volume, and the network transmission parameters; Taking a weighted average of the target system load and the synchronous transmission probability to obtain a total synchronization probability; Comparing the total synchronization probability with a preset synchronization transmission threshold to obtain synchronization transmission comparison information; If the synchronous transmission comparison information indicates that the total synchronization probability is greater than the synchronous transmission threshold, determining the data transmission mode as the synchronous transmission mode; When the synchronous transmission comparison information indicates that the synchronization success probability is less than the synchronous transmission threshold, the data transmission mode is determined to be an asynchronous transmission mode.

8. A data synchronization device, characterized in that: Applied to a target file system, the target file system stores a first file, the apparatus comprising: a log acquisition module, configured to, in response to a data transmission request from a source file system, acquire a candidate operation log marked with a candidate log identifier from the source file system; wherein the first file is the same as a second file stored in the source file system; a log identification module, configured to perform log identification on the candidate operation log according to the candidate log identifier to obtain a target operation log; wherein the target operation log is used to indicate that the second file is updated to a third file, and the target operation log includes a candidate log entry having an entry generation time; a data selection module, configured to select a target log entry from the candidate log entries according to the generation time of the entry; An operation playback module is used to perform operation playback on the first file according to the target log entry to obtain a synchronized file; wherein the synchronized file is the same as the third file.

9. An electronic device, characterized in that The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the data synchronization method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data synchronization method according to any one of claims 1 to 7 is implemented.

Images