Data Incremental Synchronization Method and Apparatus

Abstract

This application provides a data incremental synchronization method and apparatus. The method involves receiving synchronization information and the amount of data to be synchronized in each batch; determining the data to be synchronized in the current batch from the data source based on the amount of data to be synchronized and the synchronization information, and determining field mapping rules; if the target file corresponding to the target index name exists, comparing the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized; if the version numbers are different, handling the conflict based on a preset conflict strategy; determining a batch insert command for the search server to be synchronized based on the data to be synchronized, causing the search server to respond to the batch insert command and write the data to be synchronized according to the field mapping rules; repeating the above steps until no remaining data is available or the synchronization ends. This method effectively solves the shortcomings of poor data consistency and significantly improves data synchronization consistency, efficiency, and robustness.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data incremental synchronization method and apparatus. Background Technology

[0002] Efficiently and accurately synchronizing data from a data source to a search server is a common requirement. In existing technologies, synchronization can usually be performed by using fixed field mapping rules for full or incremental synchronization at regular intervals. However, the above methods are rigid when dealing with dynamic changes in data structure, requiring frequent manual intervention to adjust the fixed field mapping rules, resulting in high maintenance costs.

[0003] Meanwhile, if version conflicts exist during data synchronization, the lack of a flexible and effective automated processing mechanism can lead to inconsistent or interrupted data synchronization. Furthermore, improper settings for the amount of data to be synchronized in each batch or exceeding the data limit can easily cause synchronization failures, making it difficult to accurately locate the recovery point and affecting synchronization efficiency and reliability. In particular, when the amount of data to be synchronized is large, the adaptability and fault tolerance are poor, making it difficult to meet the needs of efficient and stable data synchronization. Summary of the Invention

[0004] To address the problems in the prior art, this application provides a data incremental synchronization method and apparatus, which can effectively solve the shortcomings of traditional technologies in the process of data synchronization, such as low efficiency, high maintenance costs, difficulty in ensuring data consistency and poor robustness, and significantly improve the efficiency and robustness of data synchronization.

[0005] To solve at least one of the above problems, this application provides the following technical solution:

[0006] Firstly, this application provides a data incremental synchronization method, including:

[0007] It receives the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and the synchronization information, and receives the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position.

[0008] Based on the amount of data to be synchronized and the synchronization information, the data to be synchronized in the current batch is determined in the data source to be synchronized. Based on the data to be synchronized in the current batch, the field mapping rules of the search server to be synchronized are dynamically generated. The field mapping includes field type inference and nested structure processing.

[0009] If the target file corresponding to the target index name exists, compare the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized. If the version numbers are different, process them according to the preset conflict strategy to make the version numbers the same.

[0010] Based on the data to be synchronized, determine the batch insert command for the search server to be synchronized, and make the search server to be synchronized respond to the batch insert command and write the data to be synchronized according to the field mapping rules.

[0011] Repeat the steps of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source or the synchronization ends.

[0012] Furthermore, it also includes: obtaining a synchronization progress file, which is used to store the synchronized records;

[0013] If there is no synchronization start position in the synchronization information and the previous interruption position is stored in the synchronization progress file, the previous interruption position is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position.

[0014] If there is no synchronization start position in the synchronization information and the previous interruption position is not stored in the synchronization progress file, the default start position of the data source to be synchronized is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position.

[0015] If there is no synchronization start position in the synchronization information, the data to be synchronized at the end of the current data source to be synchronized is determined as the synchronization end position.

[0016] Furthermore, it also includes: determining the data record corresponding to the starting offset from the data source to be synchronized based on the synchronization start position;

[0017] Read data records sequentially according to the number of records or data size indicated by the amount of data to be synchronized;

[0018] When the synchronization end position is reached, stop reading and return the data to be synchronized for the current batch.

[0019] Furthermore, it also includes: extracting the field names and field values ​​of the data to be synchronized in the current batch, mapping field values ​​that conform to the date format to date type, mapping field values ​​that conform to numeric type to integer type, and mapping JSON structures that include nested types in the field values ​​to object type;

[0020] When a field name is detected to contain a preset keyword, the field name is processed through a nested structure, and nested structures with a nesting level exceeding the preset nesting level are flattened into single-level key name combinations.

[0021] Furthermore, after repeatedly executing the step of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source or the synchronization end position is reached, the process also includes:

[0022] Once the data to be synchronized is successfully written, update the synchronization progress file so that the synchronization progress file records the currently synchronized records;

[0023] When incremental synchronization mode is enabled, the update log of the synchronization data source is monitored in real time, the incremental data is determined based on the updated content in the update log, and the incremental data is synchronized to the search server to be synchronized.

[0024] Furthermore, it also includes: preset conflict strategies, including comparison strategies, merging strategies, intervention strategies, and rollback strategies, wherein:

[0025] The comparison strategy prioritizes retaining data corresponding to higher version numbers. When version numbers are the same, it compares the timestamps corresponding to each version number and retains data whose timestamps are closer to the current time.

[0026] The merge strategy is used to merge non-conflicting fields in the data corresponding to each version number, and to overwrite conflicting fields with the data to be synchronized.

[0027] Intervention strategies are used to generate conflict reports and pause the synchronization process when an unresolvable conflict is detected.

[0028] The rollback strategy is used to roll back all operations in the current batch when a single record conflict causes the synchronization result of the current batch to fail.

[0029] Furthermore, it also includes: converting the data to be synchronized in the current batch into a batch protocol format supported by the server to be synchronized;

[0030] Add a metadata header to each piece of data to be synchronized. The metadata header includes the index name, document name, and operation type.

[0031] If it is detected that there is data exceeding the data size threshold that the server to be synchronized can accommodate, the data to be synchronized in the current batch will be split into multiple sub-batches of data to be synchronized.

[0032] Secondly, this application provides a data incremental synchronization device, comprising:

[0033] The receiving module is used to receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and the synchronization information, and to receive the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position.

[0034] The processing module is used to determine the data to be synchronized in the current batch in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information. Based on the data to be synchronized in the current batch, it dynamically generates field mapping rules for the search server to be synchronized. The field mapping includes field type inference and nested structure processing.

[0035] The comparison module is used to compare the version number of the data to be synchronized in the data source with the version number of the data stored in the search server when the target file corresponding to the target index name exists. If the version numbers are different, the module will handle the conflict based on a preset conflict strategy to make the version numbers the same.

[0036] The write module is used to determine the batch insert command for the search server to be synchronized based on the data to be synchronized, so that the search server to be synchronized responds to the batch insert command and writes the data to be synchronized according to the field mapping rules.

[0037] The synchronization module is used to repeatedly execute the steps of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source or the synchronization ends.

[0038] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described.

[0039] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data incremental synchronization method described above.

[0040] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the data incremental synchronization method described above.

[0041] As can be seen from the above technical solution, this application provides a data incremental synchronization method and apparatus. It innovatively receives the data source name, the target index name of the search server to be synchronized, and synchronization information, while simultaneously receiving the amount of data to be synchronized in each batch. It extracts the data to be synchronized from the data source and dynamically generates field mapping rules. These field mapping rules include field type inference and nested structure processing. If the target file corresponding to the target index name exists, it compares the version number of the data to be synchronized in the data source with the version number of the data stored in the search server. If the version numbers are different, a preset conflict strategy is used for handling. If the version numbers are the same, a batch insertion instruction is obtained from the data to be synchronized, causing the search server to write the data to be synchronized according to the field mapping rules. This process of batch extraction and synchronization of the data to be synchronized is repeated until there is no remaining data in the data source or the synchronization ends. This method effectively solves the shortcomings of traditional technologies, such as low efficiency, high maintenance costs, difficulty in ensuring data consistency, and poor robustness during data synchronization, significantly improving the efficiency and robustness of data synchronization. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the data incremental synchronization method in an embodiment of this application;

[0044] Figure 2 This is a structural diagram of the data incremental synchronization device in the embodiments of this application;

[0045] Figure 3 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0046] Figure label:

[0047] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0050] To effectively address the shortcomings of traditional technologies in data synchronization, such as low efficiency, high maintenance costs, difficulty in guaranteeing data consistency, and poor robustness, and to significantly improve the efficiency and robustness of data synchronization, this application provides an embodiment of a data incremental synchronization method, see [link to embodiment]. Figure 1 The incremental data synchronization method specifically includes the following:

[0051] Step S101: Receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and the synchronization information, and receive the amount of data to be synchronized in each batch.

[0052] The synchronization information includes the synchronization start position and the synchronization end position.

[0053] Optionally, this embodiment receives the data source name of the data source to be synchronized, so as to clarify the source of the data to be synchronized. At the same time, it receives the target index name of the search server to be synchronized, wherein the target index name is equivalent to specifying a storage location and retrieval identifier for the data to be synchronized in the search server.

[0054] In addition, synchronization information is also received, which includes the synchronization start position and synchronization end position, and clarifies the range of the current synchronization in the data source to be synchronized.

[0055] In addition, the amount of data to be synchronized in each batch can be received, and the data source to be synchronized can be processed in reasonable batches according to the amount of data to be synchronized, so as to avoid resource overload or excessively long synchronization process due to processing too much data to be synchronized at once.

[0056] This embodiment enables the accurate and targeted formulation of synchronization methods based on the data source name, target index name, synchronization information, and the amount of data to be synchronized in each batch. It also allows for the reasonable allocation of resources, ensuring that the synchronization operation of the data to be synchronized can be carried out in the expected manner, and avoiding synchronization errors or failures due to incomplete or incorrect information.

[0057] Step S102: Based on the amount of data to be synchronized and the synchronization information, determine the data to be synchronized in the current batch in the data source to be synchronized, and dynamically generate the field mapping rules of the search server to be synchronized based on the data to be synchronized in the current batch.

[0058] Field mapping includes field type inference and nested structure processing.

[0059] Optionally, in this embodiment, based on the amount of data to be synchronized and the synchronization information, the data to be synchronized in the current batch is accurately located in the data source to be synchronized. The data to be synchronized is used to indicate the start and end positions of the data to be synchronized in the current batch in the data source to be synchronized.

[0060] Furthermore, the start and end positions of the data to be synchronized in each batch can be determined to clarify the data to be synchronized in each batch.

[0061] In addition, based on the data to be synchronized in the current batch, field mapping rules for the search server to be synchronized are dynamically generated. This involves field type inference, that is, by analyzing the values, characteristics, etc. of fields in the data source to be synchronized, the corresponding data types are determined. For example, strings containing date formats are determined to be log types, and pure numbers are determined to be numeric types.

[0062] At the same time, it is necessary to handle nested structures. For example, if there are fields including subfields forming a nested relationship in the current data source to be synchronized, the field mapping rules can correctly map the nested structure to a format that the search server to be synchronized can recognize and store. For example, converting JSON-formatted nested objects into nested field structures supported by the search server to be synchronized.

[0063] This embodiment enables the determination of the data to be synchronized in the current batch based on synchronization information, which can avoid processing irrelevant data and improve synchronization efficiency. The dynamically generated field mapping rules, through field type inference and nested structure processing, enable the data to be synchronized in the data source to be correctly converted into a format that the search server can recognize and store, so that the data to be synchronized can be seamlessly connected and accurately stored, avoiding data loss, storage errors or inability to retrieve due to field mismatch.

[0064] Step S103: If the target file corresponding to the target index name exists, compare the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized. If the version numbers are different, process them according to the preset conflict strategy to make the version numbers the same.

[0065] Optionally, in this embodiment, if the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source is compared with the version number of the corresponding data stored in the search server to be synchronized. The version number can be an incrementing number or a unique number generated by combining timestamps, data change identifiers, etc., to distinguish different state versions of the data.

[0066] When different version numbers are detected, the conflict is handled according to a preset conflict strategy. The preset conflict strategy may include, but is not limited to, overwriting the old version, retaining the new version, and merging data differences, in order to eliminate version differences and bring the version numbers to a consistent state, thereby ensuring the accuracy and consistency of the data.

[0067] This embodiment enables the determination of the version number of the data to be synchronized in the data source and the version number of the data stored in the search server to be synchronized by comparing the version numbers, so as to determine whether the data has changed. In the case of different version numbers, the conflict is handled according to the preset conflict strategy to effectively solve problems such as data update order and duplicate synchronization, so that the data stored in the search server to be synchronized is accurate and kept in a synchronized state with the data source.

[0068] Step S104: Determine the batch insert command for the search server to be synchronized based on the data to be synchronized, and make the search server respond to the batch insert command and write the data to be synchronized according to the field mapping rules.

[0069] Optionally, in this embodiment, based on the analysis of the data structure and content of the data to be synchronized, a batch insert command is generated for the search server to be synchronized. The batch insert command encapsulates the data to be synchronized, and includes the data to be synchronized and its corresponding insert batch. The batch insert command may include the data to be synchronized in the current batch, or it may include multiple data items to be synchronized in the current batch.

[0070] In addition, a batch insert command is sent to the search server to be synchronized, so that the search server responds and parses the batch insert command, converts the data to be synchronized into the target format according to the field mapping rules, and then performs a batch write operation to write the data to be synchronized to the search server, so that the data to be synchronized is accurately stored in the corresponding target index position, thus completing the synchronization of the data to be synchronized.

[0071] This embodiment enables the writing of data to be synchronized into the search server through batch insert commands, which can reduce the number of interactions with the search server and network overhead, improve the overall synchronization efficiency of the data to be synchronized, and meet the needs of business scenarios with high real-time requirements.

[0072] Step S105: Repeat the step of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source or the synchronization ends.

[0073] Optionally, this embodiment repeats the above steps of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and the synchronization information, until there is no remaining data to be synchronized in the data source or the synchronization end position in the synchronization information is reached, ensuring that the data to be synchronized in the data source can be completely, accurately and timely synchronized to the search server to be synchronized.

[0074] This embodiment implements a repeated synchronization process, proceeding step by step according to the amount of data to be synchronized in each batch, avoiding the pressure and potential risks of processing all data at once, while checking the remaining data and the end position of synchronization, so that the search server to be synchronized always has the latest and complete dataset synchronized with the data source.

[0075] Furthermore, the synchronization process can be completed using a shell script. A shell script is a program file that contains various commands in advance for easy execution at once, thus improving the portability of the synchronization.

[0076] This embodiment realizes the complete process from receiving synchronization information to cyclic synchronization, maintaining the data accuracy and consistency between the data source and the search server to be synchronized. It improves the efficiency and performance of data synchronization by reasonably using batch insert commands, reduces resource consumption and time costs during the synchronization process, and enables large-scale data synchronization tasks to be completed efficiently under limited resources and time conditions. It improves operating efficiency and response speed, and can also realize incremental synchronization, increasing the flexibility of synchronization.

[0077] In some embodiments, based on the amount of data to be synchronized and synchronization information, the data to be synchronized in the current batch is determined from the data source to be synchronized, including:

[0078] Get the synchronization progress file, which is used to store the synchronization records that have been synchronized;

[0079] If there is no synchronization start position in the synchronization information and the previous interruption position is stored in the synchronization progress file, the previous interruption position is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position.

[0080] If there is no synchronization start position in the synchronization information and the previous interruption position is not stored in the synchronization progress file, the default start position of the data source to be synchronized is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position.

[0081] If there is no synchronization start position in the synchronization information, the data to be synchronized at the end of the current data source to be synchronized is determined as the synchronization end position.

[0082] Optionally, this embodiment obtains a synchronization progress file, wherein the synchronization progress file contains key information of each synchronization task, especially the synchronization records that have been synchronized. The synchronization records include, but are not limited to, the synchronization start position, synchronization end position, and the amount of data synchronized for each synchronization.

[0083] Once the data to be synchronized in the current batch is determined, obtain the synchronization progress file to understand the progress and results of previous work and determine the starting position for the next synchronization.

[0084] Determining the synchronization start and end positions under different circumstances:

[0085] Scenario 1: If there is a previous interruption point (including the previous end point, i.e., the point where synchronization stopped), and the synchronization information does not explicitly specify the synchronization start point, but the synchronization progress file records the previous interruption point, then the previous interruption point is used as the synchronization start point for the current synchronization, avoiding unnecessary duplicate synchronization and data omissions. Based on the synchronization start point, the data to be synchronized in the current batch is searched and determined in the data source to be synchronized, ensuring that the synchronization process proceeds from where it was not completed last time.

[0086] Scenario 2: In the absence of a previous interruption position, if the synchronization information also lacks a synchronization start position, and the synchronization progress file does not store the previous interruption position, the default start position of the data source to be synchronized is used as the starting point for synchronization. This default start position is typically a preset, stable, and reliable initial position (i.e., querying the table according to its default order and setting the synchronization start position to 1). It marks the starting boundary of synchronizable data in the data source, ensuring that the synchronization task can proceed in an orderly manner from the beginning, guaranteeing that all data to be synchronized is synchronized.

[0087] In addition, the synchronization end position is determined as follows: if there is no synchronization start position in the synchronization information, the data to be synchronized at the end of the current data source to be synchronized is determined as the synchronization end position. This ensures that the current synchronization task can cover all the data in the data source that has not yet been synchronized, and avoids the problem of data not being fully synchronized due to an unreasonable synchronization end position setting.

[0088] Furthermore, to more accurately determine the location of data within the data source to be synchronized, data source location and tracking technologies can be introduced. For example, data block location algorithms in distributed file systems can be used to quickly determine the specific location of data in a large-scale distributed storage environment. For database data sources, transaction log-based tracking technology can be used to monitor data changes in real time and record the update location of changed data. This allows for timely acquisition of accurate synchronization start and end point information, enabling incremental synchronization and improving the timeliness and accuracy of synchronization.

[0089] Furthermore, in terms of managing synchronization progress files, encrypted storage and multiple backup technologies can be used. Encrypted storage ensures the security of sensitive information recorded in the synchronization progress files (such as data source location and synchronization records), preventing unauthorized access and tampering. Multiple backups improve the reliability and availability of the synchronization progress files; even if one copy is damaged or lost, data can be quickly recovered from other copies, avoiding situations where synchronization tasks cannot proceed normally due to the loss of synchronization progress files. Simultaneously, regular cleaning and compression of synchronization progress files optimizes file storage space usage and improves file read and write efficiency.

[0090] This embodiment achieves complete coverage of all data to be synchronized in the data source by acquiring the synchronization progress file and determining the synchronization start and end positions according to different situations, thus avoiding data omissions and duplicate synchronization. At the same time, it reduces unnecessary data scanning and processing work, saves time and resources, and improves data synchronization efficiency.

[0091] In some embodiments, based on the amount of data to be synchronized and synchronization information, the data to be synchronized in the current batch is determined from the data source to be synchronized, including:

[0092] Based on the synchronization start position, determine the data record corresponding to the starting offset from the data source to be synchronized;

[0093] Read data records sequentially according to the number of records or data size indicated by the amount of data to be synchronized;

[0094] When the synchronization end position is reached, stop reading and return the data to be synchronized for the current batch.

[0095] Optionally, in this embodiment, the data record corresponding to the starting offset is located in the data source to be synchronized according to the synchronization start position. The synchronization start position is similar to a guide to determine the position to start searching for the data to be synchronized, that is, the position of the previous synchronization interruption or the specified synchronization start position. The data record corresponding to the starting offset marks the start of this batch of synchronization.

[0096] In addition, after determining the starting data record, the data records are read sequentially according to the number of records or the data size indicated by the amount of data to be synchronized. That is, the data records are read one by one according to the predetermined quantity requirements. The reading process is continuous and orderly to ensure the integrity and consistency of the data. For example, if the amount of data to be synchronized is a specified number of 100 records, the next 100 data records are read starting from the synchronization start position. Or if the amount of data to be synchronized is 1 megabyte, it can be considered that the next 1 megabyte is read starting from the synchronization start position.

[0097] In addition, during the data reading process, the system continuously monitors whether the synchronization end position has been reached. When the synchronization end position is reached, the reading operation is stopped, and the read data record is identified as the data to be synchronized in the current batch and returned.

[0098] Furthermore, data record corresponding to the synchronization start position can be determined through data source location and tracking technology. For example, for message queue-based data sources, offset management can accurately record and track the position of messages in the queue, thereby quickly locating the data record corresponding to the synchronization start position. In distributed files, the file block where the starting offset is located can be found through file block indexing and metadata management, improving the efficiency of data location and reducing the time overhead of data record search.

[0099] Furthermore, during the sequential reading of data records, the data read cache can be expanded. Through appropriate caching strategies, such as pre-reading a certain number of data records into the cache and loading data that may be needed later for synchronization, the frequency of disk input / output operations can be reduced, improving data read speed. Simultaneously, based on the data distribution characteristics and access patterns, the data read order can be optimized. For example, sequential reading of disk files can reduce latency caused by mechanical operations such as head movement, further enhancing data read performance.

[0100] This embodiment achieves accurate and rapid completion of the data transmission process by accurately determining the starting data record, efficiently reading data sequentially, and determining the synchronization end position. When the data volume is large, it can reduce the synchronization time, improve the availability and timeliness of the data, and increase the synchronization flexibility of incremental data.

[0101] In some embodiments, based on the data to be synchronized in the current batch, field mapping rules for the search servers to be synchronized are dynamically generated, including:

[0102] Extract the field names and field values ​​of the data to be synchronized in the current batch, map field values ​​that conform to the date format to date type, field values ​​that conform to numeric type to integer type, and field values ​​that include nested JSON structures to object type;

[0103] When a field name is detected to contain a preset keyword, the field name is processed through a nested structure, and nested structures with a nesting level exceeding the preset nesting level are flattened into single-level key name combinations.

[0104] Optionally, this embodiment extracts the field names and field values ​​of the data to be synchronized in the current batch to identify each field name and its included values.

[0105] The system detects date-formatted portions of field values ​​and maps them to date types. Common date formats include, but are not limited to, "YYYY-MM-DD" (year-month-day) and "YYYY / MM / DD HH:MM:SS" (year-month-day hour:minute:second). By identifying specific date format patterns, date data that might otherwise be stored as a string can be accurately converted into a date type that the target search server can recognize and process. This facilitates date-based queries, sorting, and time range analysis on the search server being synchronized.

[0106] It can also identify data in field values ​​that conform to numeric characteristics and map them to integer types. For example, if a field value contains only numbers and no decimal points, commas, or other symbols, it is determined to be numeric and converted to an integer type. This helps improve the storage efficiency and query performance of data in the search server to be synchronized, especially for scenarios involving numerical range queries and mathematical calculations.

[0107] For JSON structures with nested types in their field values, they are mapped to object types. JSON structures are common in data interaction and storage and can represent complex data relationships and hierarchical structures. By mapping them to object types, the search server to be synchronized can better understand and process nested data, supporting more flexible query and data retrieval methods, such as querying specific field values ​​in nested objects.

[0108] Additionally, after extracting the field names, the system monitors whether the field names contain preset keywords. If the field names contain preset keywords, it indicates that there may be a nested structure. The nesting structure processing mechanism can then be used to parse the field names and identify the nesting level. If the nesting level exceeds a preset nesting level threshold, the nested structure exceeding the preset nesting level threshold will be flattened into a single-level key name combination.

[0109] For example, if a field name is "address.city.name" and the default nesting level is 2, the above field name can be flattened to "address_city_name". This can simplify the data structure, improve the storage and query efficiency of data in the search server to be synchronized, and avoid the data processing complexity and performance problems caused by excessive nesting.

[0110] Furthermore, numerical values ​​including decimal parts can be mapped to floating-point types, and reasonable precision control can be performed according to the precision requirements of the data to be synchronized, such as retaining a specified number of decimal places, to more accurately represent numerical characteristics, meet the needs of scenarios with high data precision, and improve versatility.

[0111] Furthermore, for nested structures, intelligent nested structure analysis can be used to deeply analyze the hierarchical relationships, data types, and semantic meanings of nested structures through static code analysis and data structure parsing algorithms. Simultaneously, users can customize flattening rules according to actual business needs, such as setting specific separators, field name prefixes, or suffix rules to generate flattened field names that better align with business logic and query habits. This improves the flexibility and adaptability of nested structure processing and ensures that the flattened data is easier to understand and use in the search server.

[0112] This embodiment implements dynamically generated field mapping rules to accurately map different types of data fields to the types and structures required by the search server to be synchronized. These can include, but are not limited to, date types, numeric types, object types, and flattened field names. This enables the search server to better understand and process the data to be synchronized, improving the adaptability and query efficiency of the data to be synchronized, meeting various query needs, ensuring the accuracy, integrity, and consistency of the data, improving the quality of the data synchronized to the search server, and avoiding problems such as data type errors, data loss, or chaotic data structures.

[0113] In some embodiments, after repeatedly performing the step of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and synchronization information, until there is no remaining data in the data source or the synchronization end position is reached, the method further includes:

[0114] Once the data to be synchronized is successfully written, update the synchronization progress file so that the synchronization progress file records the currently synchronized records;

[0115] When incremental synchronization mode is enabled, the update log of the synchronization data source is monitored in real time, the incremental data is determined based on the updated content in the update log, and the incremental data is synchronized to the search server to be synchronized.

[0116] Optionally, after the data to be synchronized is successfully written to the search server to be synchronized, this embodiment updates the synchronization progress file in a timely manner to record the currently synchronized data. The synchronization record includes, but is not limited to, information such as the start position, end position, and amount of synchronized data for this synchronization, so as to provide a clear reference and location for possible subsequent synchronization tasks, ensuring that the next synchronization can continue from the correct synchronization start position and avoiding duplicate or missed synchronization of data.

[0117] In addition, when incremental synchronization mode is enabled, the update log of the synchronization data source is monitored in real time. The update log records all data change operations that occur in the data source after full synchronization is completed, such as data changes corresponding to add, modify, and delete operations.

[0118] By parsing the information in the update log, incremental data can be accurately identified, that is, the data portion that has changed after full synchronization. The identified incremental data is then synchronized to the search server to be synchronized, enabling the search server to reflect the latest status of the data source in a timely manner, maintaining the timeliness and consistency of the data. This ensures the continuous maintenance of data accuracy without affecting the normal operation of the business, while reducing resource consumption.

[0119] This embodiment improves the reliability and consistency of data synchronization by updating the synchronization progress file and starting the incremental synchronization mode. The update of the synchronization progress file provides an accurate basis for the management and recovery of synchronization tasks. The incremental mode can continuously maintain the consistency and integrity of data on the basis of full synchronization, avoid data inconsistency caused by synchronization task interruption or data source update, and improve the quality, reliability and scalability of data synchronization.

[0120] In some embodiments, the preset conflict strategy includes a comparison strategy, a merging strategy, an intervention strategy, and a rollback strategy, wherein:

[0121] The comparison strategy prioritizes retaining data corresponding to higher version numbers. When version numbers are the same, it compares the timestamps corresponding to each version number and retains data whose timestamps are closer to the current time.

[0122] The merge strategy is used to merge non-conflicting fields in the data corresponding to each version number, and to overwrite conflicting fields with the data to be synchronized.

[0123] Intervention strategies are used to generate conflict reports and pause the synchronization process when an unresolvable conflict is detected.

[0124] The rollback strategy is used to roll back all operations in the current batch when a single record conflict causes the synchronization result of the current batch to fail.

[0125] Optionally, the preset conflict strategy in this embodiment includes a comparison strategy, a merging strategy, an intervention strategy, and a rollback strategy, wherein:

[0126] The comparison strategy is used when synchronized data conflicts with existing data on the target search server. It prioritizes comparing the version numbers of the two data sets. Data with higher version numbers typically indicates more up-to-date and complete content, and therefore, data with higher version numbers is retained. If the version numbers are the same, the timestamps corresponding to each version number are further compared. Data with timestamps closer to the current time indicates that its generation or update time is closer to the present, potentially containing the latest information changes. Data with timestamps closer to the current time is retained to ensure that, in the event of a conflict, relatively newer data is prioritized, guaranteeing the timeliness and accuracy of the data.

[0127] The merging strategy is used to identify non-conflicting fields in the data corresponding to each version number when conflicts occur, and then merge these non-conflicting fields. The criteria for determining non-conflicting fields typically include their semantics, data type, and business logic. Conflicting fields, i.e., fields with different values ​​in different versions, can be handled by overwriting the conflicting fields with the data to be synchronized. This preserves the non-conflicting parts of the data while ensuring that the synchronized data updates the conflicting parts, allowing the data in the search server to be synchronized to integrate useful information from multiple versions and reflect the latest data status.

[0128] The intervention strategy is used when an unresolvable conflict is detected, such as when complex business logic, data dependencies, or data integrity constraints prevent automatic conflict identification and handling. A detailed conflict report is generated, recording the specific location of the conflict, the conflicting data content, and the version numbers involved. Simultaneously, the synchronization process is paused, awaiting manual intervention. This prevents data inconsistencies or business logic disruptions caused by erroneous data overwriting or merging, ensuring that data accuracy and integrity are manually reviewed and corrected in complex conflict situations.

[0129] The rollback strategy is used in the process of synchronization. If a single record conflict causes the synchronization result of the current batch to fail, the rollback strategy will take effect immediately and roll back all operations of the current batch. That is, the data that has been written to the target search server will be restored to the state before synchronization, and all incomplete change operations will be undone. This ensures that the data consistency in the search server to be synchronized is not compromised, avoids data chaos and unavailability caused by partial successful synchronization and partial failed synchronization, and guarantees the integrity and reliability of the data.

[0130] This embodiment achieves the timeliness and accuracy of data through a preset conflict strategy, while also ensuring the quality of data after synchronization and the stable operation of the search server to be synchronized, thus enhancing adaptability and reliability in the face of complex data environments.

[0131] In some embodiments, determining the batch insert command for the search server to be synchronized based on the data to be synchronized includes:

[0132] Convert the data to be synchronized in the current batch into the batch protocol format supported by the server to be synchronized.

[0133] Add a metadata header to each piece of data to be synchronized. The metadata header includes the index name, document name, and operation type.

[0134] If it is detected that there is data exceeding the data size threshold that the server to be synchronized can accommodate, the data to be synchronized in the current batch will be split into multiple sub-batches of data to be synchronized.

[0135] Optionally, this embodiment converts the data to be synchronized corresponding to the current batch into a batch protocol format supported by the search server to be synchronized. Different search servers to be synchronized may have different data transmission protocols and data format requirements. For example, some servers may require the data to be organized in JSON format, including but not limited to specified fields and structures.

[0136] By converting data into a batch protocol format that the server to be synchronized can recognize and process, the compatibility and efficiency of data during transmission and writing can be improved.

[0137] In addition, a metadata header is added to each piece of data to be synchronized. The metadata header includes, but is not limited to, the index name, the document name, and the operation type. The index name is used to specify the storage location and retrieval scope of the data in the search server to be synchronized. The document name is used to uniquely identify a data record to facilitate querying, updating, and management operations. The operation type is used to indicate the specific operation to be performed on this piece of data to be synchronized, such as insert, update, or delete.

[0138] The metadata header information provides the search server to be synchronized with the context and instructions for data processing, enabling it to correctly perform the corresponding operations on the data to be synchronized.

[0139] In addition, after completing the data format conversion and adding metadata headers, the total data size of the data to be synchronized in the current batch is detected. If the data size exceeds the data size threshold that the server to be synchronized can accommodate, in order to ensure that the data can be successfully written and not affect the performance and stability of the search server to be synchronized, the data to be synchronized in the current batch is split into multiple sub-batches of data to be synchronized, and the data size of each sub-batch is controlled within the acceptable range of the server, thereby ensuring the execution efficiency and reliability of the batch insert command.

[0140] This embodiment achieves efficient and reliable writing of data to be synchronized to the search server by means of data format conversion, adding metadata headers, data size monitoring and batch splitting. The data format adaptation and metadata usage improve the efficiency and accuracy of data processing, while the batch splitting strategy ensures the stability and continuity of the data synchronization process, avoids synchronization failure or server failure caused by data problems, and improves the overall quality and performance of data synchronization.

[0141] To effectively address the shortcomings of traditional technologies in data synchronization, such as low efficiency, high maintenance costs, difficulty in guaranteeing data consistency, and poor robustness, and to significantly improve the efficiency and robustness of data synchronization, this application provides an embodiment of a data incremental synchronization device for implementing all or part of the aforementioned data incremental synchronization. See [link to embodiment]. Figure 2 The data incremental synchronization device specifically includes the following components:

[0142] The receiving module 10 is used to receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and the synchronization information, and to receive the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position.

[0143] The processing module 20 is used to determine the data to be synchronized in the current batch in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information, and dynamically generate field mapping rules for the search server to be synchronized based on the data to be synchronized in the current batch. The field mapping includes field type inference and nested structure processing.

[0144] The comparison module 30 is used to compare the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized when the target file corresponding to the target index name exists. If the version numbers are different, the module will process the conflict based on a preset conflict strategy to make the version numbers the same.

[0145] The writing module 40 is used to determine the batch insert command for the search server to be synchronized based on the data to be synchronized, so that the search server to be synchronized responds to the batch insert command and writes the data to be synchronized according to the field mapping rules.

[0146] The synchronization module 50 is used to repeatedly execute the steps of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source or the synchronization ends.

[0147] As described above, the incremental data synchronization device provided in this application can innovatively receive the data source name, the target index name of the search server to be synchronized, and synchronization information, while simultaneously receiving the amount of data to be synchronized in each batch. It extracts the data to be synchronized from the data source and dynamically generates field mapping rules. These field mapping rules include field type inference and nested structure processing. If the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source to be synchronized is compared with the version number of the data stored in the search server to be synchronized. If the version numbers are different, a preset conflict strategy is used for processing. If the version numbers are the same, a batch insertion instruction is obtained from the data to be synchronized, causing the search server to write the data to be synchronized according to the field mapping rules. This process of batch extraction and synchronization of the data to be synchronized is repeated until there is no remaining data in the data source to be synchronized or the synchronization ends. This method effectively solves the shortcomings of traditional technologies, such as low efficiency, high maintenance costs, difficulty in ensuring data consistency, and poor robustness during data synchronization, significantly improving the efficiency and robustness of data synchronization.

[0148] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in data synchronization, such as low efficiency, high maintenance costs, difficulty in guaranteeing data consistency, and poor robustness, and to significantly improve the efficiency and robustness of data synchronization, this application provides an embodiment of an electronic device for implementing all or part of the aforementioned incremental data synchronization method. The electronic device specifically includes the following components:

[0149] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the incremental data synchronization device and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the incremental data synchronization method and the embodiments of the incremental data synchronization device in the embodiments, the contents of which are incorporated herein, and repeated details will not be described again.

[0150] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0151] In practical applications, the incremental data synchronization method can be partially executed on the electronic device side as described above, or all operations can be completed on the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed on the client device, the client device may further include a processor.

[0152] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0153] Figure 3 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 3 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0154] In one embodiment, the data incremental synchronization method functionality can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0155] Step S101: Receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and the synchronization information, and receive the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position.

[0156] Step S102: Based on the amount of data to be synchronized and the synchronization information, determine the data to be synchronized in the current batch in the data source to be synchronized. Based on the data to be synchronized in the current batch, dynamically generate the field mapping rules for the search server to be synchronized. The field mapping includes field type inference and nested structure processing.

[0157] Step S103: If the target file corresponding to the target index name exists, compare the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized. If the version numbers are different, process them according to the preset conflict strategy so that the version numbers are the same.

[0158] Step S104: Determine the batch insert command for the search server to be synchronized based on the data to be synchronized, so that the search server to be synchronized responds to the batch insert command and writes the data to be synchronized according to the field mapping rules;

[0159] Step S105: Repeat the step of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source or the synchronization ends.

[0160] As described above, the electronic device provided in this application innovatively receives the data source name, the target index name of the search server to be synchronized, and synchronization information. It simultaneously receives the amount of data to be synchronized in each batch, extracts the data to be synchronized from the data source, and dynamically generates field mapping rules. These field mapping rules include field type inference and nested structure processing. If the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source to be synchronized is compared with the version number of the data stored in the search server to be synchronized. If the version numbers are different, a preset conflict strategy is used for processing. If the version numbers are the same, a batch insertion instruction is obtained from the data to be synchronized, causing the search server to write the data to be synchronized according to the field mapping rules. This process of batch extraction and synchronization of the data to be synchronized is repeated until there is no remaining data in the data source to be synchronized or the synchronization ends. This method effectively solves the shortcomings of traditional technologies, such as low efficiency, high maintenance costs, difficulty in ensuring data consistency, and poor robustness during data synchronization, significantly improving the efficiency and robustness of data synchronization.

[0161] In another embodiment, the data incremental synchronization device can be configured separately from the central processing unit 9100. For example, the data incremental synchronization device can be configured as a chip connected to the central processing unit 9100, and the data incremental synchronization method function can be implemented through the control of the central processing unit.

[0162] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 3 All components shown; in addition, the electronic device 9600 may also include Figure 3For components not shown, please refer to existing technologies.

[0163] like Figure 3 As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0164] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0165] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0166] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0167] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0168] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0169] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0170] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the data incremental synchronization method with the execution subject being a server or client in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the data incremental synchronization method with the execution subject being a server or client in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0171] Step S101: Receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and the synchronization information, and receive the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position.

[0172] Step S102: Based on the amount of data to be synchronized and the synchronization information, determine the data to be synchronized in the current batch in the data source to be synchronized. Based on the data to be synchronized in the current batch, dynamically generate the field mapping rules for the search server to be synchronized. The field mapping includes field type inference and nested structure processing.

[0173] Step S103: If the target file corresponding to the target index name exists, compare the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized. If the version numbers are different, process them according to the preset conflict strategy so that the version numbers are the same.

[0174] Step S104: Determine the batch insert command for the search server to be synchronized based on the data to be synchronized, so that the search server to be synchronized responds to the batch insert command and writes the data to be synchronized according to the field mapping rules;

[0175] Step S105: Repeat the step of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source or the synchronization ends.

[0176] As described above, the computer-readable storage medium provided in this application innovatively receives the data source name, the target index name of the search server to be synchronized, and synchronization information, while simultaneously receiving the amount of data to be synchronized in each batch. It extracts the data to be synchronized from the data source and dynamically generates field mapping rules. These field mapping rules include field type inference and nested structure processing. If the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source to be synchronized is compared with the version number of the data stored in the search server to be synchronized. If the version numbers are different, a preset conflict strategy is used for processing. If the version numbers are the same, a batch insertion instruction is obtained from the data to be synchronized, causing the search server to write the data to be synchronized according to the field mapping rules. This process of batch extraction and synchronization of the data to be synchronized is repeated until there is no remaining data in the data source to be synchronized or the synchronization ends. This method effectively solves the shortcomings of traditional technologies, such as low efficiency, high maintenance costs, difficulty in ensuring data consistency, and poor robustness during data synchronization, significantly improving the efficiency and robustness of data synchronization.

[0177] Embodiments of this application also provide a computer program product capable of implementing all steps of the data incremental synchronization method with the execution subject being a server or client in the above embodiments. When this computer program / instruction is executed by a processor, it implements the steps of the data incremental synchronization method. For example, the computer program / instruction implements the following steps:

[0178] Step S101: Receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and the synchronization information, and receive the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position.

[0179] Step S102: Based on the amount of data to be synchronized and the synchronization information, determine the data to be synchronized in the current batch in the data source to be synchronized. Based on the data to be synchronized in the current batch, dynamically generate the field mapping rules for the search server to be synchronized. The field mapping includes field type inference and nested structure processing.

[0180] Step S103: If the target file corresponding to the target index name exists, compare the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized. If the version numbers are different, process them according to the preset conflict strategy so that the version numbers are the same.

[0181] Step S104: Determine the batch insert command for the search server to be synchronized based on the data to be synchronized, so that the search server to be synchronized responds to the batch insert command and writes the data to be synchronized according to the field mapping rules;

[0182] Step S105: Repeat the step of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source or the synchronization ends.

[0183] As described above, the computer program product provided in this application innovatively receives the data source name, the target index name of the search server to be synchronized, and synchronization information. It simultaneously receives the amount of data to be synchronized in each batch, extracts the data to be synchronized from the data source, and dynamically generates field mapping rules. These field mapping rules include field type inference and nested structure processing. If the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source to be synchronized is compared with the version number of the data stored in the search server to be synchronized. If the version numbers are different, a preset conflict strategy is used for processing. If the version numbers are the same, a batch insertion instruction is obtained from the data to be synchronized, causing the search server to write the data to be synchronized according to the field mapping rules. This process of batch extraction and synchronization of the data to be synchronized is repeated until there is no remaining data in the data source to be synchronized or the synchronization ends. This method effectively solves the shortcomings of traditional technologies, such as low efficiency, high maintenance costs, difficulty in ensuring data consistency, and poor robustness during data synchronization, significantly improving the efficiency and robustness of data synchronization.

[0184] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0186] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0188] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A data incremental synchronization method, characterized in that, The method includes: The system receives the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and synchronization information, and receives the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position. Based on the amount of data to be synchronized and the synchronization information, the data to be synchronized in the current batch is determined in the data source to be synchronized. Based on the data to be synchronized in the current batch, the field mapping rules of the search server to be synchronized are dynamically generated. The field mapping includes field type inference and nested structure processing. If the target file corresponding to the target index name exists, compare the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized. If the version numbers are different, process them according to the preset conflict strategy to make the version numbers the same. Based on the data to be synchronized, a batch insert command is determined for the search server to be synchronized, and the search server to be synchronized responds to the batch insert command by writing the data to be synchronized according to the field mapping rules; Repeat the step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source to be synchronized or the synchronization end position is reached; Based on the amount of data to be synchronized and the synchronization information, the data to be synchronized in the current batch is determined from the data source to be synchronized, including: Obtain the synchronization progress file, which is used to store the synchronized records; If the synchronization start position is not present in the synchronization information and the previous interruption position is stored in the synchronization progress file, the previous interruption position is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position. When the synchronization start position is not present in the synchronization information and the previous interruption position is not stored in the synchronization progress file, the default start position of the data source to be synchronized is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position. If the synchronization start position is not present in the synchronization information, the data to be synchronized at the end of the current data source to be synchronized is determined as the synchronization end position. Alternatively, based on the amount of data to be synchronized and the synchronization information, determine the current batch of data to be synchronized from the data source to be synchronized, including: Based on the synchronization start position, determine the data record corresponding to the starting offset from the data source to be synchronized; Read the data records sequentially according to the number of records or the data size indicated by the amount of data to be synchronized; When the synchronization end position is reached, reading stops and the data to be synchronized in the current batch is returned.

2. The method according to claim 1, characterized in that, Based on the data to be synchronized in the current batch, dynamically generate field mapping rules for the search servers to be synchronized, including: Extract the field names and field values ​​of the data to be synchronized in the current batch, map the field values ​​that conform to the date format to the date type, map the field values ​​that conform to the numeric type to the integer type, and map the JSON structure that includes nested types in the field values ​​to the object type; When a preset keyword is detected in the field name, the field name is processed through a nested structure, and the nested structure with a nesting level exceeding the preset nesting level is flattened into a single-level key name combination.

3. The method according to claim 1, characterized in that, After repeatedly executing the step of determining the current batch of data to be synchronized in the data source based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source or the synchronization end position is reached, the method further includes: Once the data to be synchronized is successfully written, update the synchronization progress file so that the synchronization progress file records the currently synchronized records; When incremental synchronization mode is enabled, the update log of the synchronization data source is monitored in real time, incremental data is determined based on the updated content in the update log, and the incremental data is synchronized to the search server to be synchronized.

4. The method according to claim 1, characterized in that, The preset conflict strategies include comparison strategies, merging strategies, intervention strategies, and rollback strategies, wherein: The comparison strategy is used to prioritize retaining data corresponding to the version number with the higher version number. When the version numbers are the same, the timestamps corresponding to each version number are compared, and data whose timestamps are closer to the current time are retained. The merging strategy is used to merge non-conflicting fields in the data corresponding to each version number, and to cover conflicting fields with the data to be synchronized. The intervention strategy is used to generate a conflict report and pause the synchronization process when an unresolvable conflict is detected. The rollback strategy is used to roll back all operations of the current batch when a single record conflict causes the synchronization result of the current batch to fail.

5. The method according to claim 1, characterized in that, Determining the batch insert command for the search server to be synchronized based on the data to be synchronized includes: Convert the data to be synchronized corresponding to the current batch into the batch protocol format supported by the server to be synchronized; Add a metadata header to each piece of data to be synchronized, the metadata header including index name, document name, and operation type; If it is detected that there is data exceeding the data size threshold that the server to be synchronized can accommodate, the data to be synchronized in the current batch will be split into multiple sub-batches of data to be synchronized.

6. A data incremental synchronization device, characterized in that, The device includes: The receiving module is used to receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and the synchronization information, and to receive the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position. The processing module is used to determine the data to be synchronized in the current batch in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information, and to dynamically generate field mapping rules for the search server to be synchronized based on the data to be synchronized in the current batch. The field mapping includes field type inference and nested structure processing. Based on the amount of data to be synchronized and the synchronization information, the data to be synchronized in the current batch is determined from the data source to be synchronized, including: Obtain the synchronization progress file, which is used to store the synchronized records; If the synchronization start position is not present in the synchronization information and the previous interruption position is stored in the synchronization progress file, the previous interruption position is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position. When the synchronization start position is not present in the synchronization information and the previous interruption position is not stored in the synchronization progress file, the default start position of the data source to be synchronized is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position. If the synchronization start position is not present in the synchronization information, the data to be synchronized at the end of the current data source to be synchronized is determined as the synchronization end position. Alternatively, based on the amount of data to be synchronized and the synchronization information, determine the current batch of data to be synchronized from the data source to be synchronized, including: Based on the synchronization start position, determine the data record corresponding to the starting offset from the data source to be synchronized; Read the data records sequentially according to the number of records or the data size indicated by the amount of data to be synchronized; When the synchronization end position is reached, stop reading and return the data to be synchronized in the current batch; The comparison module is used to compare the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized when the target file corresponding to the target index name exists. If the version numbers are different, the module will process the conflict based on a preset conflict strategy to make the version numbers the same. The writing module is used to determine a batch insert command for the search server to be synchronized based on the data to be synchronized, so that the search server to be synchronized responds to the batch insert command and writes the data to be synchronized according to the field mapping rules; The synchronization module is used to repeatedly execute the step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information, until there is no remaining data in the data source to be synchronized or the synchronization end position is reached.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the data incremental synchronization method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the data incremental synchronization method according to any one of claims 1 to 5.

Images