Data synchronization methods, devices, equipment and storage media

Abstract

This application belongs to the field of data processing and relates to a data synchronization method, including acquiring a document dataset to be synchronized and a data synchronization target library; performing modeling operations on the document dataset to obtain a standardized document dataset; transmitting the standardized document dataset to a preset business database according to a preset format to obtain a business document dataset; upon receiving a data synchronization request for the business document dataset; performing a segmentation operation on the business document dataset according to the data synchronization request to obtain a segmented document dataset; performing data preprocessing on the segmented document dataset to obtain a processed document dataset; parsing the processed document dataset to obtain a parsed document dataset, and synchronizing the parsed document dataset to the data synchronization target library. This application also provides a data synchronization device, computer equipment, and storage medium. This application can improve the efficiency and accuracy of data synchronization.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data synchronization method, apparatus, device and storage medium. Background Technology

[0002] Data synchronization refers to the process of extracting data from a business database and transferring it to a data warehouse. Traditional data synchronization methods typically involve using a Data Management Service (DMS) to input and transfer data to the target database. Existing data management systems usually include open-source document systems (such as LogicalDoc and OpenKM) and commercially available spreadsheets (such as Vig, Google Sheets, and Office 360).

[0003] However, on the one hand, open-source document systems require a large amount of manpower for data entry, and the entered data may be inconsistent with the data in the business database. Data synchronization may also be interrupted, leading to data loss and making it impossible to guarantee the integrity of the synchronized files, resulting in low data synchronization efficiency and accuracy. On the other hand, commercial maintenance forms are stored by using offline document transmission, which is prone to data leakage and errors during manual operation can cause data corruption, resulting in low data synchronization accuracy.

[0004] In summary, improving the efficiency and accuracy of data synchronization has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a data synchronization method, apparatus, device, and storage medium, the main purpose of which is to improve the efficiency and accuracy of data synchronization.

[0006] To address the aforementioned technical problems, this application provides a data synchronization method, employing the following technical solution:

[0007] Obtain the document dataset to be synchronized and the data synchronization target library, perform modeling operations on the document dataset to obtain a standardized document dataset;

[0008] The standardized document dataset is transmitted to a preset business database according to a preset format to obtain a business document dataset.

[0009] Upon receiving a data synchronization request for the business document dataset;

[0010] The business document dataset is segmented according to the data synchronization request to obtain a segmented document dataset.

[0011] The segmented document dataset is preprocessed to obtain a processed document dataset;

[0012] The processed document dataset is parsed to obtain a parsed document dataset, and the parsed document dataset is synchronized to the data synchronization target library.

[0013] Furthermore, the step of performing preprocessing on the segmented document dataset according to the data synchronization request to obtain a processed document dataset includes:

[0014] Construct the data table fields and primary key of the pre-stored data of the segmented document dataset, and generate the index of the data table;

[0015] Create a split document data table based on the data table fields, the primary key, and the index;

[0016] The segmented document data table is loaded, inserted, updated, and transformed using preset rules to obtain the processed document dataset.

[0017] Furthermore, the step of parsing the processed document dataset to obtain a parsed document dataset includes:

[0018] Obtain the parsing object of the processed document dataset, and convert the processed document dataset into a string based on the parsing object;

[0019] Identify the string format of the string, and convert the string into a parsed document dataset of the corresponding format according to the string format.

[0020] Furthermore, the modeling operation on the document dataset to obtain a standardized document dataset includes:

[0021] Obtain the business requirements and data types of the document dataset;

[0022] Identify the data structure of the document dataset based on the data type described;

[0023] Based on the business requirements and the data structure, the document dataset is mapped to a preset document template to obtain the standardized document dataset.

[0024] Furthermore, the step of transmitting the standardized document dataset to a preset business database according to a preset format to obtain a business document dataset includes:

[0025] Extract field information from the specified document dataset;

[0026] The field information is marked according to the preset format to obtain the marked field;

[0027] The tag fields are mapped to the data tables in the business database to obtain the business document dataset.

[0028] Furthermore, the step of segmenting the business document dataset according to the data synchronization request to obtain a segmented document dataset includes:

[0029] Determine whether the data synchronization request has the necessary business permissions;

[0030] If the business permission exists for the data synchronization request, the business document dataset is extracted from the business database using a preset interface according to the data synchronization request.

[0031] The business database is sharded to obtain multiple data shards;

[0032] The table partitioning operation is performed in multiple data shards to obtain multiple data shards;

[0033] The business document dataset is segmented based on multiple data partition tables to obtain the segmented document dataset.

[0034] Furthermore, after receiving the data synchronization request for the business document dataset, and before performing a segmentation operation on the business document dataset according to the data synchronization request to obtain a segmented document dataset, the method further includes:

[0035] The data synchronization request is parsed to obtain multiple data synchronization tasks. The multiple data synchronization tasks are recombined according to the preset task concurrency to obtain a data synchronization request including multiple recombined tasks.

[0036] The step of segmenting the business document dataset according to the data synchronization request to obtain a segmented document dataset includes the following steps:

[0037] The business document dataset is segmented based on the data synchronization request that includes multiple reorganization tasks, resulting in a segmented document dataset.

[0038] To address the aforementioned technical problems, this application also provides a data synchronization device, which employs the following technical solution:

[0039] The modeling module is used to acquire the document dataset to be synchronized and the data synchronization target library, and to perform modeling operations on the document dataset to obtain a standardized document dataset.

[0040] The transmission module is used to transmit the standardized document dataset to a preset business database according to a preset format to obtain a business document dataset;

[0041] The receiving module is used to receive a data synchronization request for the business document dataset;

[0042] The segmentation module is used to perform segmentation operations on the business document dataset according to the data synchronization request to obtain a segmented document dataset.

[0043] The processing module is used to perform preprocessing on the segmented document dataset to obtain a processed document dataset; and

[0044] The synchronization module is used to parse the processed document dataset to obtain a parsed document dataset, and synchronize the parsed document dataset to the data synchronization target library.

[0045] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0046] Memory, storing at least one computer program; and

[0047] The processor executes the computer program stored in the memory to achieve the data synchronization described above.

[0048] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0049] The computer-readable storage medium stores at least one computer program, which is executed by a processor in an electronic device to achieve the data synchronization described above.

[0050] Compared with the prior art, this application has the following main advantages:

[0051] In this embodiment of the invention, after obtaining the document dataset to be synchronized and the data synchronization target library, the document dataset is first modeled to obtain a standardized document dataset. This standardized document dataset is then transmitted to a preset business database according to a preset format. This solves the problem of inconsistencies between the entered data and the data in the business database, standardizes the data structure, and improves the accuracy of subsequent data synchronization. Secondly, when receiving a data synchronization request for the standardized document dataset, the data synchronization request performs a segmentation operation on the business document dataset to obtain a segmented document dataset. This ensures the integrity of the synchronized data files even if data synchronization is interrupted, facilitating subsequent improvements in data synchronization accuracy. Furthermore, by performing pre-processing on the segmented document dataset, automated data processing is achieved, eliminating the need for manual retrieval of document data from the business database. This improves data synchronization efficiency and reduces the possibility of data corruption due to human error, thereby increasing the accuracy of data synchronization. Finally, by parsing the processed document dataset to obtain a parsed document dataset, and synchronizing the parsed document dataset to the data synchronization target library, adaptation to different document data formats is achieved, further improving the accuracy of data synchronization. Therefore, the data synchronization method, apparatus, device, and storage medium proposed in this invention can improve the efficiency and accuracy of data synchronization. Attached Figure Description

[0052] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0054] Figure 2 This is a flowchart of one embodiment of the data synchronization method according to this application;

[0055] Figure 3 This is a structural diagram of an embodiment of the control terminal in the data synchronization system according to this application;

[0056] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0057] The data format determination method provided in this invention is applied to a data processing system. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the application. The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or accompanying drawings of this application are used to distinguish different objects, not to describe a specific order.

[0058] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0059] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0060] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0061] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social online platform software.

[0062] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0063] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0064] It should be noted that the data synchronization method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the data synchronization system is generally set in the server / terminal device.

[0065] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0066] Continue to refer to Figure 2 A flowchart of an embodiment of the data synchronization method according to this application is shown. The data synchronization method includes the following steps:

[0067] S210. Obtain the document dataset to be synchronized and the data synchronization target library, perform modeling operations on the document dataset to obtain a standardized document dataset.

[0068] In this embodiment of the invention, the document dataset to be synchronized refers to the data set from which data in the business database needs to be synchronized to the data synchronization target database. For example, synchronizing data from a database storing collaborative documents from multiple users to ClickHouse. The documents to be synchronized can be Excel documents and collaborative Markdown documents, etc. The data synchronization target database refers to the database storing the document dataset to be synchronized, such as the ClickHouse data warehouse. The standardized document dataset refers to a data set with a consistent data structure.

[0069] In this embodiment of the invention, by acquiring the document dataset to be synchronized and the data synchronization target library, a modeling operation is performed on the document dataset to obtain a standardized document dataset. This standardizes the data structure and ensures that the entered data is consistent with the data structure of the business database when it is stored in the business database.

[0070] As an embodiment of the present invention, the step of performing a modeling operation on the document dataset to obtain a standardized document dataset includes:

[0071] Obtain the business requirements and data types of the document dataset;

[0072] Identify the data structure of the document dataset based on the data type described;

[0073] Based on the business requirements and the data structure, the document dataset is mapped to a preset document template to obtain the standardized document dataset.

[0074] The business requirements mentioned here refer to user-defined storage requirements. For example, user-defined fields are mapped to database fields.

[0075] In one embodiment of the present invention, the data type is data that represents information of a document dataset; the data structure refers to the structure that maps related data into a fixed format when storing a document dataset.

[0076] In this embodiment of the invention, the preset document template includes a map format structure template and a table format structure template. The map format template is typically used when mapping user structures and is managed in map format. The field mapping in the map format is used to map fields in the document dataset to key-value pairs, desc fields, etc. The table format template is commonly used to map user-defined fields to corresponding database fields for subsequent data storage, including multiple column_name fields.

[0077] Furthermore, in this embodiment of the invention, mapping the document dataset to a preset document template serves to resolve the issue of inconsistencies between the entered data and the data in the business database, ensuring data consistency and standardization, and facilitating subsequent improvements in the accuracy of data synchronization.

[0078] S220. The standardized document dataset is transmitted to a preset business database according to a preset format to obtain a business document dataset.

[0079] In this embodiment of the invention, the preset business database refers to a database that stores standardized document datasets, that is, the source database of the document datasets, such as a database that stores collaborative document data from multiple users.

[0080] In this embodiment of the invention, by transmitting the standardized document dataset to a preset business database according to a preset format to obtain a business document dataset, the problem of inconsistency between the entered data and the data in the business database can be solved, thereby improving the accuracy of subsequent data synchronization.

[0081] As an embodiment of the present invention, the step of transmitting the standardized document dataset to a preset business database according to a preset format to obtain a business document dataset includes:

[0082] Extract field information from the specified document dataset;

[0083] The field information is marked according to the preset format to obtain the marked field;

[0084] The tag fields are mapped to the data tables in the business database to obtain the business document dataset.

[0085] The field information is used to represent the specific meaning of the standard document dataset; the tag field refers to the tag information in the standard document dataset, and the tag field is mapped to the data table in a key-value manner.

[0086] For example, when modeling data using the map format, the Excel document data of the map class in the Object list is first extracted, and the table_name field of each row of data in ref_map is marked.

[0087] In this embodiment of the invention, by mapping the marker field to the data table of the business database, it can be ensured that different business personnel will not cause data errors when defining their own mapping data, thereby improving the accuracy of subsequent data synchronization.

[0088] S230, When receiving a data synchronization request for the business document dataset.

[0089] In this embodiment of the invention, the data synchronization request refers to the task of synchronizing a business document dataset to a data synchronization target library. For example, if the business document dataset contains update information for all products, then the data synchronization request can be a task for updating all products.

[0090] In this embodiment of the invention, after receiving the data synchronization request for the business document dataset, and before performing a segmentation operation on the business document dataset according to the data synchronization request to obtain a segmented document dataset, the method further includes:

[0091] The data synchronization request is parsed to obtain multiple data synchronization tasks. The multiple data synchronization tasks are recombined according to the preset task concurrency to obtain a data synchronization request including multiple recombined tasks.

[0092] The step of segmenting the business document dataset according to the data synchronization request to obtain a segmented document dataset includes the following steps:

[0093] The business document dataset is segmented based on the data synchronization request that includes multiple reorganization tasks, resulting in a segmented document dataset.

[0094] The data synchronization request is parsed to obtain a data synchronization task, which is then split into multiple data synchronization tasks to be executed in parallel, thereby improving the efficiency of data synchronization. For example, if a data synchronization request is for updating all products, the split data synchronization tasks can be product A update task, product B update task, product C update task, etc.

[0095] In one embodiment of the present invention, the preset task concurrency number can be a user-defined concurrency number based on actual business operations. For example, the maximum concurrency number is 10. If there are 15 product A update tasks and 5 product B update tasks, then the 5 product A update tasks and product B update tasks will be recombined. By recombining multiple data synchronization tasks according to the preset task concurrency number, the efficiency of task execution can be further improved.

[0096] S240. Perform a segmentation operation on the business document dataset according to the data synchronization request to obtain a segmented document dataset.

[0097] In this embodiment of the invention, by performing a segmentation operation on the business document dataset according to the data synchronization request to obtain a segmented document dataset, it is possible to avoid data loss when data synchronization is interrupted during the data synchronization process, thus ensuring the integrity of the synchronized documents and improving the accuracy of subsequent data synchronization.

[0098] As an embodiment of the present invention, the step of performing a segmentation operation on the business document dataset according to the data synchronization request to obtain a segmented document dataset includes:

[0099] Determine whether the data synchronization request has the necessary business permissions;

[0100] If the business permission exists for the data synchronization request, the business document dataset is extracted from the business database using a preset interface according to the data synchronization request.

[0101] The business database is sharded to obtain multiple data shards;

[0102] The table partitioning operation is performed in multiple data shards to obtain multiple data shards;

[0103] The business document dataset is segmented based on multiple data partition tables to obtain the segmented document dataset.

[0104] The business permissions mentioned here refer to the permissions between different business departments during the data synchronization process, including document read / write permissions, document encryption permissions, and document viewing permissions. For example, during data synchronization, the product department can only view documents related to its own product and cannot view documents from other departments. Furthermore, determining whether the data synchronization request has the necessary business permissions effectively establishes departmental and personnel separation, ensuring the security of data synchronization. Encrypting documents protects the security and confidentiality of enterprise document data, preventing data leakage and tampering, thereby improving the credibility and stability of enterprise document data.

[0105] In one embodiment of the present invention, if the data synchronization request has business permissions, it means that the request has the authority to extract the business document dataset from the business database; if the data synchronization request does not have business permissions, it means that the request task does not have the authority to obtain the business document dataset, so as to prevent data leakage and tampering.

[0106] Furthermore, in this embodiment of the invention, a webhook HTTP request can be created using an open-source framework (such as Spring Boot), and the internal API can be called through this request to extract the business document dataset from the business database. This eliminates the need for a large amount of manpower to extract data and improves the efficiency of subsequent data synchronization.

[0107] In this embodiment of the invention, due to the performance limitations of the business database, the database capacity of the business database cannot be too large. Therefore, it is necessary to perform database sharding to obtain multiple data shards. For example, if the business database is A, then the multiple data shards can be A1, A2, A3, and A4, etc. The purpose of sharding the business database is to prevent the data tables from storing too many records, avoid abnormal interruptions in data synchronization, and ensure that if abnormal conditions occur in the data of one shard, it will not affect the synchronization of data in other shards.

[0108] S250. Perform data preprocessing on the segmented document dataset to obtain a processed document dataset.

[0109] In this embodiment of the invention, the processed document dataset refers to a data set after data processing, including operations such as extraction, transformation, loading, updating, and insertion.

[0110] In this embodiment of the invention, by performing preprocessing on the segmented document dataset, a processed document dataset is obtained, which enables automated data processing. This eliminates the need for manual retrieval of document data from the business database, improves the efficiency of data synchronization, and reduces the possibility of data corruption due to human error during operation, thereby improving the accuracy of data synchronization.

[0111] As an embodiment of the present invention, the step of performing data preprocessing on the segmented document dataset to obtain a processed document dataset includes:

[0112] Construct the data table fields and primary key of the pre-stored data of the segmented document dataset, and generate the index of the data table;

[0113] Create a split document data table based on the data table fields, the primary key, and the index;

[0114] The segmented document data table is loaded, inserted, updated, and transformed using preset rules to obtain the processed document dataset.

[0115] The purpose of creating a split document data table is to enable the data table to be directly displayed to the user terminal based on actual business needs after data synchronization.

[0116] In this embodiment of the invention, the data table fields are the data fields converted from the split document dataset into a data table; the primary key is a data field that describes key information of the data table; and the index is used to query the data table based on the index.

[0117] In this embodiment of the invention, the preset rules include, but are not limited to, filling in missing data values, setting default values, merging data, converting data formats, merging attributes, and splitting attributes. Users can customize these rules based on actual business scenarios. Through these preset rules, business document datasets can be automatically converted into the data required by the actual business scenario, which can effectively avoid data corruption caused by errors during manual operation and facilitate the improvement of the accuracy of subsequent data synchronization.

[0118] For example, if a business document dataset consists of Excel documents containing information about new products, and one of the attributes in this document is "Product A: Specific Information", then this attribute value can be broken down into fields such as product name, product usage method, and product price.

[0119] In one optional embodiment of the present invention, after data conversion, incremental loading of data can be performed. By loading data at a defined loading cycle (such as one hour), manual data operation can be eliminated, thereby improving data processing efficiency.

[0120] S260. The processed document dataset is parsed to obtain a parsed document dataset, and the parsed document dataset is synchronized to the data synchronization target library.

[0121] In this embodiment of the invention, the parsed document dataset refers to a data set that can be directly identified. For example, an Excel document contains the characters value(3+7*8), which, through parsing, can be determined to be the calculation result of the formula 59.

[0122] In this embodiment of the invention, by parsing the processed document dataset to obtain a parsed document dataset, and synchronizing the parsed document dataset to the data synchronization target library, it is possible to adapt to document data of different formats, thereby further improving the accuracy of data synchronization.

[0123] As an embodiment of the present invention, parsing the processed document dataset to obtain a parsed document dataset includes:

[0124] Obtain the parsing object of the processed document dataset, and convert the processed document dataset into a string based on the parsing object;

[0125] Identify the string format of the string, and convert the string into a parsed document dataset of the corresponding format according to the string format.

[0126] The parsing object refers to the data to be parsed in the processing document dataset, including but not limited to characters, numbers, formulas and symbols. The parsing object can be obtained by creating a Workbook object and passing in the processing document dataset.

[0127] In one embodiment of the present invention, the string refers to the data meaning describing the processed document dataset. The string format may include time, text, and date formats.

[0128] In this embodiment of the invention, the parsing of document datasets can be achieved through the third-party library POI to obtain parsed document datasets, enabling adaptive parsing for different formats (such as characters, numbers, formulas, and symbols), which not only improves data analysis capabilities but also ensures the accuracy of processing document datasets.

[0129] In an optional embodiment of the present invention, after the parsing is completed, the program will automatically perform cache cleanup to ensure that the data is not retained during synchronization, thereby further improving the security and confidentiality of the data.

[0130] Furthermore, in this embodiment of the invention, the parsed document dataset can be synchronized to the data synchronization target library using the data synchronization tool cloudcanal.

[0131] Compared with the prior art, the embodiments of this application have the following main advantages:

[0132] In this embodiment of the invention, after obtaining the document dataset to be synchronized and the data synchronization target library, the document dataset is first modeled to obtain a standardized document dataset. This standardized document dataset is then transmitted to a preset business database according to a preset format. This solves the problem of inconsistencies between the entered data and the data in the business database, standardizes the data structure, and improves the accuracy of subsequent data synchronization. Secondly, upon receiving a data synchronization request for the standardized document dataset, the business document dataset is segmented according to the data synchronization request to obtain a segmented document dataset. This ensures the integrity of the synchronized data files even if data synchronization is interrupted, facilitating subsequent improvements in data synchronization accuracy. Furthermore, by performing pre-processing on the segmented document dataset, automated data processing is achieved, eliminating the need for manual retrieval of document data from the business database. This improves data synchronization efficiency and reduces the possibility of data corruption due to human error, thereby increasing the accuracy of data synchronization. Finally, the processed document dataset is parsed to obtain a parsed document dataset, which is then synchronized to the data synchronization target library. This enables adaptation to document data of different formats, further improving the accuracy of data synchronization. Therefore, the data synchronization method proposed in this embodiment of the invention can improve the efficiency and accuracy of data synchronization.

[0133] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0134] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0135] Further reference Figure 3 As a response to the above Figure 2To implement the method shown, this application provides an embodiment of a data synchronization device 310, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0136] This invention provides a data synchronization system, the data synchronization system comprising:

[0137] Modeling module 311 is used to obtain the document dataset to be synchronized and the data synchronization target library, and to perform modeling operations on the document dataset to obtain a standardized document dataset.

[0138] Transmission module 312 is used to transmit the standardized document dataset to a preset business database according to a preset format to obtain a business document dataset;

[0139] The receiving module 313 is used to receive a data synchronization request for the business document dataset;

[0140] The segmentation module 314 is used to perform a segmentation operation on the business document dataset according to the data synchronization request to obtain a segmented document dataset;

[0141] Processing module 315 is used to perform preprocessing on the segmented document dataset to obtain a processed document dataset; and

[0142] The synchronization module 316 is used to parse the processed document dataset to obtain a parsed document dataset, and synchronize the parsed document dataset to the data synchronization target library.

[0143] Compared with the prior art, the embodiments of this application have the following main advantages:

[0144] In this embodiment of the invention, after obtaining the document dataset to be synchronized and the data synchronization target library, the document dataset is first modeled to obtain a standardized document dataset. This standardized document dataset is then transmitted to a preset business database according to a preset format. This solves the problem of inconsistencies between the entered data and the data in the business database, standardizes the data structure, and improves the accuracy of subsequent data synchronization. Secondly, when receiving a data synchronization request for the standardized document dataset, the data synchronization request performs a segmentation operation on the business document dataset to obtain a segmented document dataset. This ensures the integrity of the synchronized data files even if data synchronization is interrupted, facilitating subsequent improvements in data synchronization accuracy. Furthermore, by performing pre-processing on the segmented document dataset, automated data processing is achieved, eliminating the need for manual retrieval of document data from the business database. This improves data synchronization efficiency and reduces the possibility of data corruption due to human error, thereby increasing the accuracy of data synchronization. Finally, by parsing the processed document dataset to obtain a parsed document dataset, and synchronizing the parsed document dataset to the data synchronization target library, adaptation to different document data formats is achieved, further improving the accuracy of data synchronization. Therefore, the data synchronization device proposed in this embodiment of the invention can improve the efficiency and accuracy of data synchronization.

[0145] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0146] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0147] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0148] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as program code for data synchronization methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0149] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run program code stored in the memory 41 or process data, for example, to run the program code for the data synchronization method.

[0150] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0151] This application also provides another embodiment, namely, providing a computer-readable storage medium storing the data synchronization method program, which can be executed by at least one processor to cause the at least one processor to perform the steps of the data synchronization method as described above.

[0152] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware online platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0153] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0154] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A data synchronization method, characterized in that, Includes the following steps: Obtain the document dataset to be synchronized and the data synchronization target library, perform modeling operations on the document dataset to obtain a standardized document dataset; The standardized document dataset is transmitted to a preset business database according to a preset format to obtain a business document dataset. Upon receiving a data synchronization request for the business document dataset; The method further includes, after receiving the data synchronization request and before performing the data synchronization request on the business document dataset to obtain the segmented document dataset, the segmentation operation on the business document dataset to obtain the segmented document dataset. The data synchronization request is parsed to obtain the original data synchronization task. The original data synchronization task is split into multiple data synchronization tasks. The multiple data synchronization tasks are recombined according to the preset task concurrency to obtain a data synchronization request including multiple recombined tasks. The step of performing a segmentation operation on the business document dataset according to the data synchronization request to obtain a segmented document dataset includes the following steps: performing a segmentation operation on the business document dataset according to the data synchronization request including multiple reorganization tasks to obtain a segmented document dataset; The segmented document dataset is preprocessed to obtain a processed document dataset; The processed document dataset is parsed to obtain a parsed document dataset, and the parsed document dataset is synchronized to the data synchronization target library.

2. The data synchronization method according to claim 1, characterized in that, The step of performing data preprocessing on the segmented document dataset according to the data synchronization request to obtain a processed document dataset includes: Construct the data table fields and primary key of the pre-stored data of the segmented document dataset, and generate the index of the data table; Create a split document data table based on the data table fields, the primary key, and the index; The segmented document data table is loaded, inserted, updated, and transformed using preset rules to obtain the processed document dataset.

3. The data synchronization method according to claim 1, characterized in that, The step of parsing the processed document dataset to obtain a parsed document dataset includes: Obtain the parsing object of the processed document dataset, and convert the processed document dataset into a string based on the parsing object; Identify the string format of the string, and convert the string into a parsed document dataset of the corresponding format according to the string format.

4. The data synchronization method according to claim 1, characterized in that, The step of modeling the document dataset to obtain a standardized document dataset includes: Obtain the business requirements and data types of the document dataset; Identify the data structure of the document dataset based on the data type described; Based on the business requirements and the data structure, the document dataset is mapped to a preset document template to obtain the standardized document dataset.

5. The data synchronization method according to claim 1, characterized in that, The step of transmitting the standardized document dataset to a preset business database according to a preset format to obtain a business document dataset includes: Extract field information from the specified document dataset; The field information is marked according to the preset format to obtain the marked field; The tag fields are mapped to the data tables in the business database to obtain the business document dataset.

6. The data synchronization method according to claim 1, characterized in that, The step of segmenting the business document dataset according to the data synchronization request to obtain a segmented document dataset includes: Determine whether the data synchronization request has the necessary business permissions; If the business permission exists for the data synchronization request, the business document dataset is extracted from the business database using a preset interface according to the data synchronization request. The business database is sharded to obtain multiple data shards; The table partitioning operation is performed in multiple data shards to obtain multiple data shards; The business document dataset is segmented based on multiple data partition tables to obtain the segmented document dataset.

7. A data synchronization device, characterized in that, include: The modeling module is used to acquire the document dataset to be synchronized and the data synchronization target library, and to perform modeling operations on the document dataset to obtain a standardized document dataset. The transmission module is used to transmit the standardized document dataset to a preset business database according to a preset format to obtain a business document dataset; The receiving module is used to receive a data synchronization request for the business document dataset; A segmentation module is used to segment the business document dataset according to the data synchronization request to obtain a segmented document dataset. The module further includes the following steps after receiving the data synchronization request for the business document dataset and before performing the segmentation operation on the business document dataset according to the data synchronization request to obtain the segmented document dataset: The data synchronization request is parsed to obtain the original data synchronization task. The original data synchronization task is split into multiple data synchronization tasks. The multiple data synchronization tasks are recombined according to the preset task concurrency to obtain a data synchronization request including multiple recombined tasks. The step of performing a segmentation operation on the business document dataset according to the data synchronization request to obtain a segmented document dataset includes: performing a segmentation operation on the business document dataset according to the data synchronization request including multiple reorganization tasks to obtain a segmented document dataset; The processing module is used to perform preprocessing on the segmented document dataset to obtain a processed document dataset; and The synchronization module is used to parse the processed document dataset to obtain a parsed document dataset, and synchronize the parsed document dataset to the data synchronization target library.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the data synchronization method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data synchronization method as described in any one of claims 1 to 6.

Images