Data increment synchronization method and device

Abstract

The embodiment of the invention provides a data increment synchronization method and device, and the method comprises the steps: receiving synchronization information, and receiving a to-be-synchronized data volume synchronized in each batch; based on the to-be-synchronized data volume and the synchronization information, determining the to-be-synchronized data of the current batch in the to-be-synchronized data source, and determining a field mapping rule; when a target file corresponding to the target index name exists, comparing a version number of to-be-synchronized data in the data source with a version number of data stored in the to-be-synchronized search server, and if the version numbers are different, processing based on a preset conflict strategy; determining a batch insertion command for the to-be-synchronized search server based on the to-be-synchronized data, and enabling the to-be-synchronized search server to write the to-be-synchronized data according to a field mapping rule in response to the batch insertion command; and repeatedly executing the steps until no residual data exists or the synchronization end position is reached. According to the method, the defects of poor data consistency and the like are effectively overcome, and the data synchronization consistency, efficiency and robustness are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for data incremental synchronization. Background Art

[0002] It is a common requirement to efficiently and accurately synchronize data from the data source to the search server. In the existing technology, synchronization can usually be performed in full or timed increments using fixed field mapping rules. However, the above method is rigid when dealing with dynamic changes in data structure and requires frequent manual intervention to adjust the fixed field mapping rules, resulting in high maintenance costs.

[0003] At the same time, if there is a version conflict during the data synchronization process, there is a lack of a flexible and effective automated processing mechanism, which will lead to inconsistent synchronized data or synchronization terminals. In addition, improper setting of the amount of data to be synchronized in each batch or data exceeding the limit can easily lead to synchronization failure, making it difficult to accurately locate the recovery point, affecting synchronization efficiency and reliability. Especially when the amount of data to be synchronized is large, the adaptability and fault tolerance are poor, making it difficult to meet the needs of efficient and stable data synchronization. Summary of the Invention

[0004] In response to the problems in the existing technology, the present application provides a data incremental synchronization method and device, which can effectively solve the shortcomings of traditional technologies in the process of synchronizing data, such as low efficiency, high maintenance cost, difficulty in ensuring data consistency and poor robustness, and significantly improve the efficiency and robustness of data synchronization.

[0005] In order to solve at least one of the above problems, the present application provides the following technical solutions: In a first aspect, the present application provides a method for incremental data synchronization, comprising: Receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and synchronization information, and receive the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position; Based on the amount of data to be synchronized and the synchronization information, the current batch of data to be synchronized is determined in the data source to be synchronized. Based on the current batch of data to be synchronized, the field mapping rules for the search server to be synchronized are dynamically generated. The field mapping includes field type inference and nested structure processing; If the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source is compared with the version number of the data stored in the search server to be synchronized. If the version numbers are different, the conflict is handled based on the preset conflict strategy to make the version numbers the same; Determining a batch insert command for a search server to be synchronized based on the data to be synchronized, and causing the search server to be synchronized to respond to the batch insert command and write the data to be synchronized according to the field mapping rule; The step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information is repeatedly performed until there is no remaining data in the data source to be synchronized or the synchronization end position is reached.

[0006] Furthermore, the method further includes: obtaining a synchronization progress file, the synchronization progress file being used to store synchronized synchronization records; When the synchronization start position does not exist in the synchronization information and the previous interruption position is stored in the synchronization progress file, the previous interruption position is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position; When there is no synchronization start position in the synchronization information and the previous interruption position is not stored in the synchronization progress file, the default start position of the data source to be synchronized is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position; When the synchronization start position does not exist in the synchronization information, the to-be-synchronized data corresponding to the end of the current to-be-synchronized data source is determined as the synchronization end position.

[0007] Furthermore, the method further includes: determining a data record corresponding to a start offset from a data source to be synchronized based on the synchronization start position; Read data records sequentially according to the number of records or data size indicated by the amount of data to be synchronized; When the synchronization end position is reached, reading stops and the data to be synchronized in the current batch is returned.

[0008] Furthermore, the process also includes: extracting the field names and field values of the current batch of data to be synchronized, mapping the field values that conform to the date format into the date type, mapping the field values that conform to the numeric type into the integer type, and mapping the JSON structure including nested types in the field values into the object type; When it is detected that a field name includes preset keywords, the field name is processed through a nested structure, and the nested structure with a nesting level exceeding the preset nesting level is flattened into a single-layer key name combination.

[0009] Furthermore, after repeatedly performing the step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information until there is no remaining data in the data source to be synchronized or the synchronization end position is reached, the method further includes: When the data to be synchronized is written successfully, the synchronization progress file is updated so that the synchronization progress file records the synchronization records that have been synchronized currently; When the incremental synchronization mode is enabled, the update log of the synchronization data source is monitored in real time, the incremental data is determined based on the updated content in the update log, and the incremental data is synchronized to the search server to be synchronized.

[0010] Furthermore, the method further includes: preset conflict strategies including comparison strategy, merge strategy, intervention strategy and rollback strategy, wherein: The comparison strategy is used to prioritize data corresponding to higher version numbers. When the version numbers are the same, the timestamps corresponding to each version number are compared, and data with a timestamp close to the current time is retained. The merge strategy is used to merge the non-conflicting fields in the data corresponding to each version number, overwriting the conflicting fields with the data to be synchronized; The intervention strategy is used to generate a conflict report and suspend the synchronization process when an unresolvable conflict is detected; The rollback policy is used to roll back all operations in the current batch when a single record conflict causes the synchronization result of the current batch to fail.

[0011] Furthermore, the method further includes: converting the to-be-synchronized data corresponding to the current batch into a batch protocol format supported by the to-be-synchronized server; Add a metadata header for each piece of data to be synchronized. The metadata header includes the index name, document name, and operation type. When it is detected that the data size exceeds the threshold that the server to be synchronized can accommodate, the data to be synchronized corresponding to the current batch is split into data to be synchronized corresponding to multiple sub-batches.

[0012] In a second aspect, the present application provides a data incremental synchronization device, comprising: The receiving module is used to receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized and synchronization information, and receive the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position; A processing module is used to determine the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information, and dynamically generate field mapping rules for the search server to be synchronized based on the current batch of data to be synchronized. The field mapping includes field type inference and nested structure processing; A comparison module is used to compare the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized when the target file corresponding to the target index name exists. If the version numbers are different, a conflict is handled based on a preset conflict strategy to make the version numbers the same; A writing module, configured to determine a batch insert command for a search server to be synchronized based on the data to be synchronized, so that the search server to be synchronized responds to the batch insert command and writes the data to be synchronized according to the field mapping rule; The synchronization module is used to repeatedly execute the step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information until there is no remaining data in the data source to be synchronized or the synchronization end position is reached.

[0013] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the described method when executing the program.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the data incremental synchronization method when executed by a processor.

[0015] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which implements the steps of the data incremental synchronization method when executed by a processor.

[0016] It can be seen from the above technical solution that the present application provides a data incremental synchronization method and device, which innovatively receives the data source name, the target index name of the search server to be synchronized, and the synchronization information, and at the same time receives the amount of synchronized data in each batch, extracts the data to be synchronized from the data source, and dynamically generates field mapping rules, wherein the field mapping rules include field type inference and nested structure processing. When the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source to be synchronized is compared with the version number of the data stored in the search server to be synchronized. When the version numbers are different, they are processed according to the preset conflict strategy. When the version numbers are the same, a batch insert instruction is obtained through the data to be synchronized, so that the search server to be synchronized writes the data to be synchronized according to the field mapping rule, and the process of batch extraction and synchronization of the data to be synchronized is repeated until there is no remaining data in the data source to be synchronized or the synchronization end position is reached. This method effectively solves the shortcomings of traditional technologies in the process of synchronizing data, such as low efficiency, high maintenance cost, difficulty in ensuring data consistency, and poor robustness, and significantly improves the efficiency and robustness of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 Schematic diagram of the process of the incremental data synchronization method in the embodiment of the present application; Figure 2 This is a structural diagram of the data incremental synchronization device in an embodiment of the present application; Figure 3 Schematic diagram of the structure of the electronic device in the embodiment of the present application.

[0019] Reference numerals: Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. DETAILED DESCRIPTION

[0020] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0021] The acquisition, storage, use, and processing of data in this application's technical solution comply with relevant national laws and regulations.

[0022] In order to effectively solve the shortcomings of traditional technologies in the process of synchronizing data, such as low efficiency, high maintenance cost, difficulty in ensuring data consistency and poor robustness, and significantly improve the efficiency and robustness of data synchronization, this application provides an embodiment of a data incremental synchronization method, see Figure 1 The data incremental synchronization method specifically includes the following contents: Step S101: receiving the data source name of the data source to be synchronized, the target index name of the search server to be synchronized and synchronization information, and receiving the amount of data to be synchronized in each batch.

[0023] The synchronization information includes a synchronization start position and a synchronization end position.

[0024] Optionally, this embodiment receives the data source name of the data source to be synchronized to clarify the source of the data to be synchronized, and at the same time, receives the target index name of the search server to be synchronized, where the target index name is equivalent to specifying a storage location and retrieval identifier for the data to be synchronized in the search server to be synchronized.

[0025] In addition, synchronization information is also received, wherein the synchronization information includes a synchronization start position and a synchronization end position, and the synchronization information clarifies the range interval of this synchronization in the data source to be synchronized.

[0026] In addition, the amount of data to be synchronized received in each batch can be reasonably batched according to the amount of data to be synchronized, avoiding processing too much data to be synchronized at one time, which may lead to resource overload or an overly lengthy synchronization process.

[0027] This embodiment implements the ability to accurately and specifically formulate synchronization methods and reasonably allocate resources based on the data source name, target index name, synchronization information, and the amount of data to be synchronized in each batch, ensuring that the synchronization operation of the data to be synchronized can be carried out in the expected synchronization method, avoiding synchronization execution errors or failures due to incomplete or erroneous information.

[0028] Step S102: Based on the amount of data to be synchronized and the synchronization information, determine the current batch of data to be synchronized in the data source to be synchronized, and dynamically generate field mapping rules for the search server to be synchronized based on the current batch of data to be synchronized.

[0029] Field mapping includes field type inference and nested structure processing.

[0030] Optionally, this embodiment accurately locates the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information. The data to be synchronized is used to indicate the start and end positions of the current batch of data to be synchronized that needs to be synchronized in the data source to be synchronized.

[0031] Furthermore, the starting position and the ending position of each batch of data to be synchronized that needs to be synchronized can be determined to clarify each batch of data to be synchronized that needs to be synchronized.

[0032] In addition, based on the current batch of data to be synchronized, the field mapping rules for the search server to be synchronized are dynamically generated, which involves field type inference, that is, by analyzing the values, Gess and other characteristics of the fields in the data source to be synchronized, the corresponding data type is determined. For example, a string including a date format is determined as a log type, and a pure number is determined as a numeric type.

[0033] At the same time, nested structures need to be processed. For example, when there are fields including sub-fields in the current data source to be synchronized to form a nested relationship, the field mapping rules can correctly map the nested structure to a format that can be recognized and stored by the search server to be synchronized, such as converting nested objects in JSON format into a nested field structure supported by the search server to be synchronized.

[0034] This embodiment realizes the determination of the current batch of data to be synchronized based on the synchronization information, which can avoid the processing of irrelevant data and improve the synchronization efficiency. The dynamically generated field mapping rules are processed through field type inference and nested structure, so that the data to be synchronized in the data source to be synchronized can be correctly converted into a format that can be recognized and stored by the search server to be synchronized, so that the data to be synchronized can be seamlessly connected and accurately stored, avoiding data loss, storage errors or inability to retrieval due to field mismatch.

[0035] Step S103: When the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source is compared with the version number of the data stored in the search server to be synchronized. If the version numbers are different, processing is performed based on the preset conflict strategy to make the version numbers the same.

[0036] Optionally, this embodiment compares the version number of the data to be synchronized in the data source with the version number of the corresponding data stored in the search server to be synchronized, when the target file corresponding to the target index name exists. The version number can be an increasing number or a unique number generated by combining a timestamp, a data change identifier, etc., to distinguish different status versions of the data.

[0037] When a difference in version numbers is detected, it is processed according to a preset conflict strategy, where the preset conflict strategy may include but is not limited to overwriting the old version, retaining the new version, and merging data differences to eliminate version differences and make the version numbers consistent, thereby ensuring data accuracy and consistency.

[0038] This embodiment realizes that by comparing version numbers, the version number of the data to be synchronized in the data source and the version number of the data stored in the search server to be synchronized can be determined to judge whether the data has changed. In the case of different version numbers, they are processed according to the preset conflict strategy to effectively solve problems such as data update order and repeated synchronization, so that the data stored in the search server to be synchronized is accurate and synchronized with the data source.

[0039] Step S104: determining a batch insert command for the search server to be synchronized based on the data to be synchronized, so that the search server to be synchronized responds to the batch insert command and writes the data to be synchronized according to the field mapping rule.

[0040] Optionally, this embodiment generates a batch insert command for the search server to be synchronized based on the analysis of the data structure and content of the data to be synchronized, wherein the batch insert command encapsulates the data to be synchronized, and the batch insert command includes the data to be synchronized and its corresponding insertion batch. The batch insert command may include the data to be synchronized of the current batch, and may also include multiple pieces of data to be synchronized of the current batch.

[0041] In addition, a batch insert command is sent to the search server to be synchronized, so that the search server to be synchronized responds to and parses the batch insert command, and converts the data to be synchronized into the target format according to the field mapping rules, and then performs a batch write operation to write the data to be synchronized into the search server to be synchronized, so that the data to be synchronized is accurately stored in the corresponding target index location, completing the synchronization of the data to be synchronized.

[0042] This embodiment implements writing the data to be synchronized in the search server to be synchronized through batch insert commands, which can reduce the number of interactions with the search server to be synchronized and network overhead, and improve the overall synchronization efficiency of the data to be synchronized to meet the needs of business scenarios with higher real-time requirements.

[0043] Step S105: Repeat the step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information until there is no remaining data in the data source to be synchronized or the synchronization end position is reached.

[0044] Optionally, this embodiment repeats the above steps of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information, until there is no remaining data to be synchronized in the data source to be synchronized or the synchronization end position in the synchronization information is reached, ensuring that the data to be synchronized in the data source can be synchronized to the search server to be synchronized completely, accurately and timely.

[0045] This embodiment implements the repeated execution of the synchronization process, gradually advancing according to the amount of synchronized data in each batch, avoiding the pressure and potential risks brought by processing all the data at one time, and checking the remaining data and the end position of synchronization at the same time, so that the search server to be synchronized always has the latest and complete data set synchronized with the synchronization data source.

[0046] Furthermore, the synchronization process can be completed through a Shell script, wherein the Shell script is a program file for pre-placing various commands into a file for one-time execution, thereby improving the portability of the synchronization.

[0047] This embodiment implements a complete process from receiving synchronization information to cyclic synchronization, maintains data accuracy and consistency between the data source and the search server to be synchronized, reasonably improves the efficiency and performance of data synchronization through batch insert commands, reduces resource consumption and time cost during the synchronization process, enables large-scale data synchronization tasks to be completed efficiently under limited resources and time conditions, improves operating efficiency and response speed, and can also achieve incremental synchronization, increasing the flexibility of synchronization.

[0048] In some embodiments, based on the amount of data to be synchronized and the synchronization information, determining the current batch of data to be synchronized in the data source to be synchronized includes: Get the synchronization progress file, which is used to store the synchronized synchronization records; When the synchronization start position does not exist in the synchronization information and the previous interruption position is stored in the synchronization progress file, the previous interruption position is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position; When there is no synchronization start position in the synchronization information and the previous interruption position is not stored in the synchronization progress file, the default start position of the data source to be synchronized is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position; When the synchronization start position does not exist in the synchronization information, the to-be-synchronized data corresponding to the end of the current to-be-synchronized data source is determined as the synchronization end position.

[0049] Optionally, this embodiment obtains a synchronization progress file, wherein the synchronization progress file contains key information of all previous synchronization tasks, especially synchronized records, including but not limited to the synchronization start position, synchronization end position, amount of synchronized data, and other data of each synchronization.

[0050] When preparing to determine the data to be synchronized in the current batch, obtain the synchronization progress file to understand the progress and results of previous work and determine the starting position of the next synchronization.

[0051] Determination of synchronization start and end positions in different situations: Scenario 1: If there is a previous interruption location (the previous interruption location includes the previous end location, that is, the location where synchronization was previously stopped), and the synchronization information does not explicitly specify the synchronization start location, but the synchronization progress file records the previous interruption location, the previous interruption location is used as the synchronization start location for the current synchronization, avoiding unnecessary repeated synchronization and data omissions. Based on the synchronization start location, the data to be synchronized in the current batch is searched and determined in the data source to be synchronized, ensuring that the synchronization process continues from the point where it was unfinished last time.

[0052] Scenario 2: When there is no previous interruption location, the synchronization start position is not included in the synchronization information, and the previous interruption location is not stored in the synchronization progress file, the default start position of the data source to be synchronized is used as the synchronization starting point. The default start position is usually a preset, stable and reliable initial position (that is, according to the default table order query, the synchronization start position is set to 1). It marks the starting boundary of the syncable data in the data source, ensuring that the synchronization task can proceed in an orderly manner from the beginning and that all data to be synchronized is synchronized.

[0053] In addition, the synchronization end position is determined: when there is no synchronization start position in the synchronization information, the data to be synchronized corresponding to the end of the current data source to be synchronized is determined as the synchronization end position, ensuring that this synchronization task can cover all data in the data source that has not been synchronized, and avoiding the problem of incomplete data synchronization due to unreasonable synchronization end position setting.

[0054] Furthermore, to more accurately determine the location of data within the data source to be synchronized, data source location and tracking technologies can be introduced. For example, using the data block location algorithm in a distributed file system can quickly determine the specific location of data in a large-scale distributed storage environment. For database data sources, transaction log-based tracking technology can be used to monitor data changes in real time and record the update location of changed data. This allows for timely acquisition of accurate synchronization start and end location information, enabling incremental synchronization and improving the timeliness and accuracy of synchronization.

[0055] Furthermore, in terms of synchronization progress file management, encrypted storage and multiple-copy backup technologies can be used. Encrypted storage ensures the security of sensitive information recorded in synchronization progress files (such as data source location and synchronization history), preventing unauthorized access and tampering. Multiple-copy backup improves the reliability and availability of synchronization progress files. Even if a copy is damaged or lost, data can be quickly restored from other copies, avoiding synchronization tasks that cannot be completed normally due to synchronization progress file loss. At the same time, synchronization progress files are regularly cleaned and compressed to optimize file storage space usage and improve file reading and writing efficiency.

[0056] This embodiment achieves the goal of completely covering all the data to be synchronized in the data source to be synchronized by obtaining the synchronization progress file and determining the synchronization start position and synchronization end position according to different situations, avoiding the problems of data omission and repeated synchronization, and at the same time reducing unnecessary data scanning and processing work, saving time and resources, improving data synchronization efficiency, and improving synchronization efficiency.

[0057] In some embodiments, based on the amount of data to be synchronized and the synchronization information, determining the current batch of data to be synchronized in the data source to be synchronized includes: Based on the synchronization start position, determine the data record corresponding to the start offset from the data source to be synchronized; Read data records sequentially according to the number of records or data size indicated by the amount of data to be synchronized; When the synchronization end position is reached, reading stops and the data to be synchronized in the current batch is returned.

[0058] Optionally, this embodiment locates the data record corresponding to the starting offset in the data source to be synchronized based on the synchronization starting position, wherein the synchronization starting position is similar to a guide to determine the position to start looking for the data to be synchronized, that is, the previous synchronization interruption position or the specified synchronization starting position, and the data record corresponding to the starting offset marks the beginning of this batch synchronization.

[0059] In addition, after determining the starting data record, the data records are read in sequence according to the number of records or data size indicated by the amount of data to be synchronized, that is, the data records are read one by one according to the established quantity requirements. The reading process is continuous and orderly to ensure the integrity and consistency of the data. For example, if the amount of data to be synchronized is a specified number of 100 records, the next 100 data records are read starting from the synchronization starting position, or if the amount of data to be synchronized is 1 megabyte, it can be considered that the next 1 megabyte of characters is read starting from the synchronization starting position.

[0060] In addition, during the data reading process, it is continuously monitored whether the synchronization end position is read. When the synchronization end position is read, the reading operation is stopped, and the read data record is determined as the data to be synchronized in the current batch and returned.

[0061] Furthermore, the data record corresponding to the synchronization start position can be determined through data source positioning and tracking technology. For example, for a data source based on a message queue, the position of the message in the queue can be accurately recorded and tracked through offset management, so as to quickly locate the data record corresponding to the synchronization start position. In a distributed file, the file block where the starting offset is located can be found through file block indexing and metadata management, thereby improving the efficiency of data positioning and reducing the time overhead of data record search.

[0062] Furthermore, during the sequential reading of data records, the data read cache itself can be expanded. By implementing appropriate caching strategies, such as pre-reading a certain number of data records into the cache and pre-loading the synchronized data that may be needed later, the frequency of disk I / O operations can be reduced, thereby increasing data read speed. Furthermore, by optimizing the data read sequence based on the data distribution characteristics and access patterns, such as using sequential reads for disk files, delays caused by mechanical operations such as head movement can be reduced, further improving data read performance.

[0063] This embodiment achieves the goal of accurately and quickly completing the data passing process by accurately determining the starting data record, efficiently reading data sequentially, and determining the synchronization end position. When the data volume is large, it can reduce the synchronization time, improve the availability and timeliness of the data, and increase the synchronization flexibility of incremental data.

[0064] In some embodiments, based on the current batch of data to be synchronized, dynamically generating field mapping rules for the search server to be synchronized includes: Extract the field names and field values of the current batch of data to be synchronized, map the field values that conform to the date format to the date type, map the field values that conform to the numeric type to the integer type, and map the JSON structure including nested types in the field values to the object type; When it is detected that a field name includes preset keywords, the field name is processed through a nested structure, and the nested structure with a nesting level exceeding the preset nesting level is flattened into a single-layer key name combination.

[0065] Optionally, this embodiment extracts the field names and field values of the current batch of data to be synchronized to identify each field name and the value included therein.

[0066] Detects date-formatted portions of field values and maps them to date types. For example, common date formats include, but are not limited to, "YYYY-MM-DD" (year-month-day) and "YYYY / MM / DD HH:MM:SS" (year / month / day hour:minute:second). By recognizing specific date format patterns, date data, which might originally be stored as strings, can be accurately converted to a date type that the target search server can recognize and process, facilitating date-based queries, sorting, and time range analysis on the target search server.

[0067] It can also identify data in field values that match numeric characteristics and map them to integer types. For example, if a field value contains only numbers without decimal points, commas, or other symbols, it is determined to be numeric and converted to an integer type. This improves storage efficiency and query performance on the search server to be synchronized, especially for scenarios involving numeric range queries and mathematical calculations.

[0068] For JSON structures that include nested types in field values, they are mapped to object types. JSON structure is a common structure in data interaction and storage that can represent complex data relationships and hierarchical structures. By mapping it to an object type, the search server to be synchronized can better understand and process nested data, supporting more flexible query and data retrieval methods. For example, specific field values in nested objects can be queried.

[0069] In addition, after extracting the field name, the field name is monitored to see if it contains preset keywords. If the field name contains preset keywords, it indicates that there may be a nested structure. The nested structure processing mechanism can be used to parse the field name and identify the nesting level. If the nesting level exceeds the preset nesting level threshold, the nested structure that exceeds the preset nesting level threshold will be flattened into a single-level key name combination.

[0070] For example, if a field name is "address.city.name" and the preset nesting level is 2, then the above field name will be flattened to "address_city_name". This can simplify the data structure, improve the storage and query efficiency of data in the search server to be synchronized, and avoid the data processing complexity and performance issues caused by too deep nesting levels.

[0071] Furthermore, numerical values including decimal parts can be mapped to floating-point types, and reasonable precision control can be performed according to the accuracy requirements of the data to be synchronized, such as retaining a specified number of decimal places to more accurately represent numerical features, meet the needs of scenarios with higher data precision, and improve versatility.

[0072] Furthermore, nested structures can be processed through intelligent nested structure analysis, using static code analysis and data structure parsing algorithms to deeply analyze the hierarchical relationships, data types, and semantic meanings of nested structures. At the same time, users can customize flattening rules based on actual business needs. For example, specific delimiters, field name prefixes, or suffixes can be set to generate flattened field names that better align with business logic and query habits. This improves the flexibility and adaptability of nested structure processing and ensures that the flattened data is easier to understand and use in the search server.

[0073] This embodiment implements dynamically generated field mapping rules to accurately map different types of data fields to the types and structures required by the search server to be synchronized, which may include but are not limited to date types, numeric types, object types and flattened field names, so that the data to be synchronized can be better understood and processed by the search server to be synchronized, thereby improving the adaptability and query efficiency of the data to be synchronized, meeting various query requirements, ensuring the accuracy, completeness and consistency of the data, improving the quality of data synchronized to the search server to be synchronized, and avoiding problems such as data type errors, data loss or chaotic data structure.

[0074] In some embodiments, after repeatedly performing the step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information until no data remains in the data source to be synchronized or the synchronization end position is reached, the method further includes: When the data to be synchronized is written successfully, the synchronization progress file is updated so that the synchronization progress file records the synchronization records that have been synchronized currently; When the incremental synchronization mode is enabled, the update log of the synchronization data source is monitored in real time, the incremental data is determined based on the updated content in the update log, and the incremental data is synchronized to the search server to be synchronized.

[0075] Optionally, after the data to be synchronized is successfully written into the search server to be synchronized, this embodiment promptly updates the synchronization progress file to record the currently synchronized synchronization records, wherein the synchronization records include but are not limited to the synchronization start position, synchronization end position, amount of synchronized data and other information of this synchronization, so as to provide a clear reference and positioning for subsequent possible synchronization tasks, ensure that the next synchronization can continue from the correct synchronization start position, and avoid duplicate data synchronization or missed synchronization.

[0076] In addition, when incremental synchronization mode is enabled, the update log of the synchronization data source is monitored in real time. The update log records all data change operations that occur in the data source after the full synchronization is completed, such as data changes corresponding to addition, modification, and deletion operations.

[0077] By parsing the information in the update log, we can accurately determine the incremental data, that is, the data part that has changed after the full synchronization, and synchronize the determined incremental data to the search server to be synchronized, so that the search server to be synchronized can promptly reflect the latest status of the data source, maintain the timeliness and consistency of the data, and continuously maintain the accuracy of the data without affecting the normal operation of the business, thereby reducing resource consumption.

[0078] This embodiment improves the reliability and consistency of data synchronization by updating the synchronization progress file and starting the incremental synchronization mode. The update of the synchronization progress file provides an accurate basis for the management and recovery of synchronization tasks. The incremental mode can continuously maintain the consistency and integrity of data on the basis of full synchronization, avoiding data inconsistency caused by synchronization task interruption or data source update, thereby improving the quality, reliability and scalability of data synchronization.

[0079] In some embodiments, the preset conflict strategy includes a comparison strategy, a merge strategy, an intervention strategy, and a rollback strategy, wherein: The comparison strategy is used to prioritize data corresponding to higher version numbers. When the version numbers are the same, the timestamps corresponding to each version number are compared, and data with a timestamp close to the current time is retained. The merge strategy is used to merge the non-conflicting fields in the data corresponding to each version number, overwriting the conflicting fields with the data to be synchronized; The intervention strategy is used to generate a conflict report and suspend the synchronization process when an unresolvable conflict is detected; The rollback policy is used to roll back all operations in the current batch when a single record conflict causes the synchronization result of the current batch to fail.

[0080] Optionally, the preset conflict strategies of this embodiment include a comparison strategy, a merge strategy, an intervention strategy, and a rollback strategy, wherein: When a conflict occurs between synchronized data and existing data on the target search server, the comparison strategy prioritizes comparing the version numbers of the two. Data with a higher version number generally indicates more up-to-date and complete content, so the data corresponding to the higher version number is retained first. If the version numbers are the same, the timestamps corresponding to each version number are further compared. Data with a timestamp closer to the current time indicates that it was generated or updated more recently and may contain the latest information changes. Data with a timestamp closer to the current time is retained to ensure that the more recent data is retained in the event of a conflict, ensuring data timeliness and accuracy.

[0081] In the event of a conflict, the merge strategy identifies non-conflicting fields within the data corresponding to each version number and merges these non-conflicting fields. Non-conflicting fields are typically determined based on their semantics, data type, and business logic. Conflicting fields—fields with different values in different versions—can be handled by having the to-be-synchronized data overwrite the conflicting fields. This preserves the non-conflicting portions of the data while ensuring that the synchronized data can update the conflicting portions. This allows the data on the to-be-synchronized search server to incorporate useful information from multiple versions and reflect the latest data status.

[0082] Intervention strategies are used when unresolvable conflicts are detected, such as those involving complex business logic, data dependencies, or data integrity constraints, making automatic conflict identification and resolution impossible. A detailed conflict report is generated, documenting the specific location of the conflict, the conflicting data content, the version numbers involved, and other information. The synchronization process is also paused pending manual intervention to avoid data inconsistencies or business logic confusion caused by erroneous data overwriting or merging, ensuring that data accuracy and integrity are manually reviewed and repaired in complex conflict situations.

[0083] The rollback strategy is used during the synchronization process. If a single record conflict causes the synchronization result of the current batch to fail, the rollback strategy will take effect immediately and roll back all operations of the current batch, that is, restoring the data written to the target search server to the state before synchronization, and undoing all unfinished change operations to ensure that the data consistency in the search server to be synchronized is not destroyed, avoid data confusion and unavailability caused by partial successful synchronization and partial failed synchronization, and ensure data integrity and reliability.

[0084] This embodiment ensures the timeliness and accuracy of data through a preset conflict strategy, while ensuring the quality of data after synchronization and the stable operation of the search server to be synchronized through the preset conflict strategy, thereby enhancing adaptability and reliability in the face of complex data environments.

[0085] In some embodiments, determining a batch insert command for a search server to be synchronized based on the data to be synchronized includes: Convert the data to be synchronized corresponding to the current batch into the batch protocol format supported by the synchronization server; Add a metadata header for each piece of data to be synchronized. The metadata header includes the index name, document name, and operation type. When it is detected that the data size exceeds the threshold that the server to be synchronized can accommodate, the data to be synchronized corresponding to the current batch is split into data to be synchronized corresponding to multiple sub-batches.

[0086] Optionally, this embodiment converts the data to be synchronized corresponding to the current batch into a batch protocol format supported by the search server to be synchronized, where different search servers to be synchronized may have different data transmission protocols and data format requirements. For example, some servers may require data to be organized in JSON format, including but not limited to specified fields and structures.

[0087] By converting data into a batch protocol format that can be recognized and processed by the server to be synchronized, the compatibility and efficiency of data during transmission and writing can be improved.

[0088] In addition, a metadata header is added to each piece of data to be synchronized, where the metadata header includes but is not limited to the index name, document name and operation type. The index name is used to specify the storage location and retrieval range of the data in the search server to be synchronized, the document name is used to uniquely identify a data record to facilitate query, update and management operations, and the operation type is used to indicate the specific operation performed on the data to be synchronized, such as insert, update or delete.

[0089] The metadata header information provides the search server to be synchronized with the context and instructions for data processing, so that it can correctly perform corresponding operations on the synchronized data.

[0090] In addition, after completing the data format conversion and metadata header addition, the total data size of the data to be synchronized corresponding to the current batch is detected. If it is detected that the data size exceeds the data size threshold that can be accommodated by the server to be synchronized, in order to ensure that the data can be written successfully and does not affect the performance and stability of the search server to be synchronized, the data to be synchronized corresponding to the current batch is split into multiple sub-batches of data to be synchronized, and the data size of each sub-batch is controlled within the acceptable range of the server, thereby ensuring the execution efficiency and reliability of the batch insert command.

[0091] This embodiment implements data format conversion, metadata header addition, data size monitoring and batch splitting to ensure that the data to be synchronized can be written to the search server to be synchronized in an efficient and reliable manner. The adaptation of the data format and the use of metadata improve the efficiency and accuracy of data processing. The batch splitting strategy ensures the stability and continuity of the data synchronization process, avoids synchronization failures or server failures due to data problems, and improves the overall quality and performance of data synchronization.

[0092] In order to effectively solve the shortcomings of traditional technologies in the process of synchronizing data, such as low efficiency, high maintenance cost, difficulty in ensuring data consistency and poor robustness, and significantly improve the efficiency and robustness of data synchronization, this application provides an embodiment of a data incremental synchronization device for realizing all or part of the content of the data incremental synchronization, see Figure 2 The data incremental synchronization device specifically includes the following contents: Receiving module 10, used to receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized and synchronization information, and receive the amount of data to be synchronized in each batch, the synchronization information includes the synchronization start position and the synchronization end position; Processing module 20, for determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information, and dynamically generating field mapping rules for the search server to be synchronized based on the current batch of data to be synchronized, wherein the field mapping includes field type inference and nested structure processing; A comparison module 30 is configured to compare, when a target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized, and to process, based on a preset conflict strategy, if the version numbers are different, to make the version numbers the same; A writing module 40 is configured to determine a batch insert command for a search server to be synchronized based on the data to be synchronized, so that the search server to be synchronized responds to the batch insert command and writes the data to be synchronized according to the field mapping rule; The synchronization module 50 is configured to repeatedly determine the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information until there is no remaining data in the data source to be synchronized or the synchronization end position is reached.

[0093] From the above description, it can be seen that the data incremental synchronization device provided by the embodiment of the present application can innovatively receive the data source name, the target index name of the search server to be synchronized, and the synchronization information, while receiving the amount of data for each batch of synchronization, extract the data to be synchronized from the data source, and dynamically generate field mapping rules, wherein the field mapping rules include field type inference and nested structure processing. When the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source to be synchronized is compared with the version number of the data stored in the search server to be synchronized. When the version numbers are different, they are processed according to the preset conflict strategy. When the version numbers are the same, a batch insert instruction is obtained through the data to be synchronized, so that the search server to be synchronized writes the data to be synchronized according to the field mapping rule, and the process of batch extraction and synchronization of the data to be synchronized is repeated until there is no remaining data in the data source to be synchronized or the synchronization end position is reached. This method effectively solves the shortcomings of traditional technologies in the process of synchronizing data, such as low efficiency, high maintenance cost, difficulty in ensuring data consistency, and poor robustness, and significantly improves the efficiency and robustness of data synchronization.

[0094] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in the process of synchronizing data, such as low efficiency, high maintenance costs, difficulty in ensuring data consistency, and poor robustness, and to significantly improve the efficiency and robustness of data synchronization, the present application provides an embodiment of an electronic device for implementing all or part of the content of the incremental data synchronization method, and the electronic device specifically includes the following content: A processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to implement information transmission between the incremental data synchronization device and related devices such as the core business system, user terminals, and related databases; the logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., but this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the incremental data synchronization method and the incremental data synchronization device in the embodiments, and their contents are incorporated herein, and repeated parts are not repeated.

[0095] It is understandable that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0096] In actual applications, part of the data incremental synchronization method can be executed on the electronic device side as described above, or all operations can be completed on the client device. The specific selection can be based on the processing capabilities of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed on the client device, the client device may also include a processor.

[0097] The aforementioned client device may include a communication module (i.e., a communication unit) capable of establishing a communication connection with a remote server to facilitate data transmission with the server. The server may include a server at the task scheduling center or, in other implementation scenarios, a server on an intermediate platform, such as a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may comprise a single computer device, a server cluster consisting of multiple servers, or a distributed server configuration.

[0098] Figure 3 Schematic block diagram of the system structure of the electronic device 9600 according to an embodiment of the present application. Figure 3 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that the Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0099] In one embodiment, the data incremental synchronization method function may be integrated into the central processing unit 9100. The central processing unit 9100 may be configured to perform the following control: Step S101: receiving the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and synchronization information, and receiving the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position; Step S102: Based on the amount of data to be synchronized and the synchronization information, determine the current batch of data to be synchronized in the data source to be synchronized, and dynamically generate field mapping rules for the search server to be synchronized based on the current batch of data to be synchronized. The field mapping includes field type inference and nested structure processing; Step S103: If the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source is compared with the version number of the data stored in the search server to be synchronized. If the version numbers are different, a conflict resolution is performed based on a preset conflict strategy to make the version numbers the same. Step S104: determining a batch insert command for the search server to be synchronized based on the data to be synchronized, and causing the search server to be synchronized to respond to the batch insert command and write the data to be synchronized according to the field mapping rule; Step S105: Repeat the step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information until there is no remaining data in the data source to be synchronized or the synchronization end position is reached.

[0100] From the above description, it can be seen that the electronic device provided by the embodiment of the present application, through innovatively receiving the data source name, the target index name of the search server to be synchronized and the synchronization information, simultaneously receives the amount of synchronized data in each batch, extracts the data to be synchronized from the data source, and dynamically generates field mapping rules, wherein the field mapping rules include field type inference and nested structure processing, and when the target file corresponding to the target index name exists, compares the version number of the data to be synchronized in the data source to be synchronized with the version number of the data stored in the search server to be synchronized. When the version numbers are different, they are processed according to the preset conflict strategy. When the version numbers are the same, a batch insert instruction is obtained through the data to be synchronized, so that the search server to be synchronized writes the data to be synchronized according to the field mapping rule, and repeats the process of batch extraction and synchronization of the data to be synchronized until there is no remaining data in the data source to be synchronized or the synchronization end position is reached. This method effectively solves the shortcomings of traditional technologies in the process of synchronizing data, such as low efficiency, high maintenance cost, difficulty in ensuring data consistency and poor robustness, and significantly improves the efficiency and robustness of data synchronization.

[0101] In another embodiment, the data incremental synchronization device can be configured separately from the central processing unit 9100. For example, the data incremental synchronization device can be configured as a chip connected to the central processing unit 9100, and the data incremental synchronization method function can be implemented under the control of the central processing unit.

[0102] like Figure 3 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily have to include Figure 3 In addition, the electronic device 9600 may also include all components shown in Figure 3 For components not shown, reference may be made to the prior art.

[0103] like Figure 3 As shown, the central processing unit 9100 is sometimes also referred to as a controller or operation control, and may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0104] Memory 9140 can be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It can store the aforementioned failure-related information and also store programs that execute the relevant information. The CPU 9100 can execute the programs stored in memory 9140 to implement information storage or processing.

[0105] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 may be, for example, a keypad or touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used to display objects such as images and text. The display may be, for example, an LCD display, but is not limited thereto.

[0106] The memory 9140 may be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), or SIM card. Alternatively, it may be a memory that retains information even when power is off, can be selectively erased, and is capable of storing additional data. Examples of such memory are sometimes referred to as EPROMs. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs, or processes used by the central processing unit 9100 to execute operations of the electronic device 9600.

[0107] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, images, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various driver programs for communication functions of the electronic device and / or for executing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0108] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal.

[0109] Based on different communication technologies, multiple communication modules 9110 may be provided in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless local area network modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130, providing audio output via the speaker 9131 and receiving audio input from the microphone 9132, thereby implementing common telecommunication functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Furthermore, the audio processor 9130 is coupled to the central processing unit 9100, enabling local recording via the microphone 9132 and playback of stored audio via the speaker 9131.

[0110] Embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the incremental data synchronization method in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the computer program implements all steps of the incremental data synchronization method in the above-mentioned embodiment, where the execution subject is a server or a client. For example, when the processor executes the computer program, the following steps are implemented: Step S101: receiving the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and synchronization information, and receiving the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position; Step S102: Based on the amount of data to be synchronized and the synchronization information, determine the current batch of data to be synchronized in the data source to be synchronized, and dynamically generate field mapping rules for the search server to be synchronized based on the current batch of data to be synchronized. The field mapping includes field type inference and nested structure processing; Step S103: If the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source is compared with the version number of the data stored in the search server to be synchronized. If the version numbers are different, a conflict resolution is performed based on a preset conflict strategy to make the version numbers the same. Step S104: determining a batch insert command for the search server to be synchronized based on the data to be synchronized, and causing the search server to be synchronized to respond to the batch insert command and write the data to be synchronized according to the field mapping rule; Step S105: Repeat the step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information until there is no remaining data in the data source to be synchronized or the synchronization end position is reached.

[0111] From the above description, it can be seen that the computer-readable storage medium provided by the embodiment of the present application, by innovatively receiving the data source name, the target index name of the search server to be synchronized and the synchronization information, and at the same time receiving the amount of data synchronized in each batch, extracts the data to be synchronized from the data source, and dynamically generates field mapping rules, wherein the field mapping rules include field type inference and nested structure processing, and when the target file corresponding to the target index name exists, compares the version number of the data to be synchronized in the data source to be synchronized with the version number of the data stored in the search server to be synchronized. When the version numbers are different, they are processed according to the preset conflict strategy. When the version numbers are the same, a batch insert instruction is obtained through the data to be synchronized, so that the search server to be synchronized writes the data to be synchronized according to the field mapping rule, and repeats the process of batch extraction and synchronization of the data to be synchronized until there is no remaining data in the data source to be synchronized or the synchronization end position is reached. This method effectively solves the shortcomings of traditional technologies in the process of synchronizing data, such as low efficiency, high maintenance cost, difficulty in ensuring data consistency and poor robustness, and significantly improves the efficiency and robustness of data synchronization.

[0112] Embodiments of the present application also provide a computer program product capable of implementing all steps of the incremental data synchronization method in the above-mentioned embodiment, where the execution subject is a server or a client. When the computer program / instructions are executed by a processor, the steps of the incremental data synchronization method are implemented. For example, the computer program / instructions implement the following steps: Step S101: receiving the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and synchronization information, and receiving the amount of data to be synchronized in each batch. The synchronization information includes the synchronization start position and the synchronization end position; Step S102: Based on the amount of data to be synchronized and the synchronization information, determine the current batch of data to be synchronized in the data source to be synchronized, and dynamically generate field mapping rules for the search server to be synchronized based on the current batch of data to be synchronized. The field mapping includes field type inference and nested structure processing; Step S103: If the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source is compared with the version number of the data stored in the search server to be synchronized. If the version numbers are different, a conflict resolution is performed based on a preset conflict strategy to make the version numbers the same. Step S104: determining a batch insert command for the search server to be synchronized based on the data to be synchronized, and causing the search server to be synchronized to respond to the batch insert command and write the data to be synchronized according to the field mapping rule; Step S105: Repeat the step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information until there is no remaining data in the data source to be synchronized or the synchronization end position is reached.

[0113] From the above description, it can be seen that the computer program product provided by the embodiment of the present application, by innovatively receiving the data source name, the target index name of the search server to be synchronized and the synchronization information, and at the same time receiving the amount of data synchronized in each batch, extracts the data to be synchronized from the data source, and dynamically generates field mapping rules, wherein the field mapping rules include field type inference and nested structure processing. When the target file corresponding to the target index name exists, the version number of the data to be synchronized in the data source to be synchronized is compared with the version number of the data stored in the search server to be synchronized. When the version numbers are different, they are processed according to the preset conflict strategy. When the version numbers are the same, a batch insert instruction is obtained through the data to be synchronized, so that the search server to be synchronized writes the data to be synchronized according to the field mapping rule, and the process of batch extraction and synchronization of the data to be synchronized is repeated until there is no remaining data in the data source to be synchronized or the synchronization end position is reached. This method effectively solves the shortcomings of traditional technologies in the process of synchronizing data, such as low efficiency, high maintenance cost, difficulty in ensuring data consistency and poor robustness, and significantly improves the efficiency and robustness of data synchronization.

[0114] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatuses, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0115] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0118] Specific embodiments are used in the present invention to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A data incremental synchronization method, characterized in that: The method comprises: Receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized, and synchronization information, and receive the amount of data to be synchronized in each batch, wherein the synchronization information includes the synchronization start position and the synchronization end position; Based on the amount of data to be synchronized and the synchronization information, determining a current batch of data to be synchronized in the data source to be synchronized, and dynamically generating field mapping rules for the search server to be synchronized based on the current batch of data to be synchronized, wherein the field mapping includes field type inference and nested structure processing; When the target file corresponding to the target index name exists, comparing the version number of the data to be synchronized in the data source with the version number of the data stored in the search server to be synchronized, and if the version numbers are different, processing them based on a preset conflict strategy to make the version numbers the same; determining a batch insert command for the search server to be synchronized based on the data to be synchronized, and causing the search server to be synchronized to respond to the batch insert command and write the data to be synchronized according to the field mapping rule; The step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information is repeatedly performed until there is no remaining data in the data source to be synchronized or the synchronization end position is reached.

2. The method according to claim 1, characterized in that Determining a current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information includes: Acquire a synchronization progress file, wherein the synchronization progress file is used to store synchronized synchronization records; When the synchronization start position does not exist in the synchronization information and the previous interruption position is stored in the synchronization progress file, the previous interruption position is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position; When the synchronization start position does not exist in the synchronization information and the previous interruption position is not stored in the synchronization progress file, the default start position of the data source to be synchronized is determined as the synchronization start position of the current synchronization, and the data to be synchronized in the current batch is determined according to the synchronization start position; When the synchronization start position does not exist in the synchronization information, the to-be-synchronized data corresponding to the end of the current to-be-synchronized data source is determined as the synchronization end position.

3. The method according to claim 1, characterized in that Determining a current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information includes: Based on the synchronization starting position, determining a data record corresponding to a starting offset from the data source to be synchronized; Reading data records sequentially according to the number of records or data size indicated by the amount of data to be synchronized; When the synchronization end position is read, the reading is stopped and the data to be synchronized in the current batch is returned.

4. The method according to claim 1, wherein Based on the current batch of data to be synchronized, dynamically generate field mapping rules for the search server to be synchronized, including: Extract the field names and field values of the current batch of data to be synchronized, map the field values that conform to the date format to the date type, map the field values that conform to the numeric type to the integer type, and map the JSON structure including nested types in the field values to the object type; When it is detected that the field name includes a preset keyword, the field name is processed through a nested structure, and the nested structure with a nesting level exceeding the preset nesting level is flattened into a single-layer key name combination.

5. The method according to claim 1, wherein After repeatedly performing the step of determining a current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information until no remaining data exists in the data source to be synchronized or the synchronization end position is reached, the method further includes: When the data to be synchronized is written successfully, the synchronization progress file is updated so that the synchronization progress file records the synchronization records that are currently synchronized; When the incremental synchronization mode is started, the update log of the synchronization data source is monitored in real time, incremental data is determined based on the updated content in the update log, and the incremental data is synchronized to the search server to be synchronized.

6. The method according to claim 1, characterized in that The preset conflict strategies include comparison strategy, merge strategy, intervention strategy and rollback strategy, among which: The comparison strategy is used to preferentially retain data corresponding to a version number with a higher version number. When the version numbers are the same, the timestamps corresponding to the version numbers are compared, and data with a timestamp that is closer to the current time is retained. The merging strategy is used to merge the non-conflicting fields in the data corresponding to each version number, and overwrite the conflicting fields with the data to be synchronized; The intervention strategy is used to generate a conflict report and suspend the synchronization process when an unresolvable conflict is detected; The rollback strategy is used to roll back all operations of the current batch when a single record conflict causes the synchronization result of the current batch to be a synchronization failure.

7. The method according to claim 1, characterized in that Determining a batch insert command for the search server to be synchronized based on the data to be synchronized includes: Convert the data to be synchronized corresponding to the current batch into a batch protocol format supported by the server to be synchronized; Add a metadata header to each piece of data to be synchronized, wherein the metadata header includes an index name, a document name, and an operation type; When it is detected that the data size exceeds a threshold that can be accommodated by the to-be-synchronized server, the to-be-synchronized data corresponding to the current batch is split into to-be-synchronized data corresponding to multiple sub-batches.

8. A data incremental synchronization device, characterized in that: The device comprises: A receiving module is used to receive the data source name of the data source to be synchronized, the target index name of the search server to be synchronized and synchronization information, and receive the amount of data to be synchronized in each batch, wherein the synchronization information includes the synchronization start position and the synchronization end position; a processing module, configured to determine a current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information, and dynamically generate field mapping rules for the search server to be synchronized based on the current batch of data to be synchronized, wherein the field mapping includes field type inference and nested structure processing; a comparison module configured to compare, when a target file corresponding to the target index name exists, a version number of the data to be synchronized in the data source with a version number of the data stored in the search server to be synchronized, and to process, based on a preset conflict strategy, when the version numbers are different, so as to make the version numbers the same; a writing module, configured to determine a batch insert command for the search server to be synchronized based on the data to be synchronized, so that the search server to be synchronized responds to the batch insert command and writes the data to be synchronized according to the field mapping rule; The synchronization module is used to repeatedly execute the step of determining the current batch of data to be synchronized in the data source to be synchronized based on the amount of data to be synchronized and the synchronization information until there is no remaining data in the data source to be synchronized or the synchronization end position is reached.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the data incremental synchronization method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data incremental synchronization method described in any one of claims 1 to 7 are implemented.

Images