Data dynamic synchronization method and edge user terminal

Abstract

The application discloses a kind of data dynamic synchronization method and edge user terminal, comprising the following steps: comprising the following steps: step S1: any edge user terminal data request information is copied and distributed to remaining edge user terminal;Step S2: compare the time information of each edge user terminal last time data request, judge whether there is data loss;If any of the edge user terminal exists data loss, then its history data request record synchronization is carried out;If no edge user terminal exists data loss, then normally execute data request;Step S3: update history data request record after data request execution success.The way of global backup rather than distributed storage ensures the reliability and integrity of data;The whole data synchronization process only needs to configure the IP address of edge user terminal, maximally simplifies configuration information, and improves easy operability.

Description

Technical Field

[0001] This invention belongs to the field of ship integrated automation, and specifically relates to a data dynamic synchronization method and an edge user terminal. Background Technology

[0002] In integrated ship automation systems, multiple edge user terminals need to maintain constant data consistency. Furthermore, due to the requirements of integrated ship automation—high reliability, ensuring each user terminal operates independently without interference, guaranteeing software availability and data integrity even under extreme conditions where only one user terminal remains operational; flexibility, adaptability to various data sources and data types within the shipbuilding industry; and ease of operation, allowing even non-software professionals like crew members to modify configurations to handle potential emergencies—a low-latency, highly reliable, flexible, easy-to-operate, and data-integrity-guaranteed synchronization method is crucial and meaningful.

[0003] Currently, there are two main data synchronization methods: one is based on database clusters, which virtualizes all terminal databases into a cluster for overall management and data storage; the other is based on master-slave backup, including both database backup and hard disk hardware backup. The database cluster method has several drawbacks. First, the data in the cluster is distributed across various machines according to a strategy; a large number of nodes losing connection or becoming offline could lead to data loss or even the unavailability of the entire cluster. Second, runtime errors can cause the entire cluster to crash, rendering all cluster data unavailable and reducing system reliability. The master-slave backup method has several drawbacks. First, it requires a designated master; if the master loses or corrupts data, the entire dataset will be corrupted. Second, it relies on specific data storage management software, resulting in low flexibility, high configuration and maintenance difficulty, requiring highly skilled personnel, and is not easy to operate. Therefore, neither of these two solutions can simultaneously achieve high reliability, flexibility, ease of operation, and data integrity.

[0004] Therefore, existing technologies have shortcomings and urgently need improvement. Summary of the Invention

[0005] The purpose of this invention is to provide a dynamic data synchronization method and an edge user terminal to solve the problem that existing data synchronization methods cannot simultaneously achieve high reliability, flexibility, ease of operation, and data integrity.

[0006] This invention provides a dynamic data synchronization method, comprising the following steps: Step S1: Copying and distributing data request information from any edge user terminal to the remaining edge user terminals; Step S2: Comparing the time information of the most recent data request from each edge user terminal to determine whether there is any data missing; if any of the edge user terminals has data missing, synchronizing its historical data request records; if no edge user terminal has data missing, executing the data request normally; Step S3: Updating the historical data request records after the data request is successfully executed.

[0007] Optionally, in some embodiments of the present invention, in step S1, the data request information of any edge user terminal is copied and distributed to the remaining edge user terminals through an intermediate proxy server; wherein the intermediate proxy server is configured with the configuration information of all the edge user terminals.

[0008] Optionally, in some embodiments of the present invention, the intermediate proxy server includes an Nginx intermediate proxy server.

[0009] Optionally, in some embodiments of the present invention, in step S2, the most recent data request time of the edge user terminal to be synchronized is queried in its cache, and a request is sent to query the most recent data request time in the caches of the other edge user terminals. Then, the most recent data request time of the edge user terminal to be synchronized is compared with the most recent data request time of the other edge user terminals in chronological order. If the most recent data request time of the edge user terminal to be synchronized is earlier, it is determined that there is a data gap, and its historical data request record is synchronized. If the most recent data request time of the edge user terminal to be synchronized is later, it is determined that there is no data gap, and the data request is executed normally.

[0010] Optionally, in some embodiments of the present invention, when synchronizing historical data request records of edge user terminals to be synchronized, the following steps are included: Step S21: Send a request to obtain all historical data request records of the other edge user terminals after the most recent data request time of the edge user terminal to be synchronized; Step S22: Based on the request time information of the historical data request records, integrate the historical data request records of all edge user terminals into a union that is sorted by time and does not repeat; Step S23: Based on the union of the historical data request records of the edge user terminals, send simulated requests sequentially to the edge user terminals to be synchronized; if the simulated request fails, resend the simulated request after a preset delay time; if the simulated request succeeds, execute the data request normally.

[0011] Optionally, in some embodiments of the present invention, the simulated request uses RestTemplate to send a simulated HTTP request.

[0012] Optionally, in some embodiments of the present invention, in step S3, after the data request is successfully executed, it is added to the batch data processing program and the historical data request record is updated.

[0013] Optionally, in some embodiments of the present invention, the historical data request record includes: request address, request header information, request body, request time, and the number of the edge user terminal.

[0014] Optionally, in some embodiments of the present invention, the historical data request records are stored in a separate MySQL database with the InnoDB engine.

[0015] Accordingly, the present invention also provides an edge user terminal, comprising: an operation interface for sending data request information; an intermediate proxy server for copying and distributing the data request information to other edge user terminals; a historical verification program for integrating historical data request records of all the edge user terminals; a batch data processing program for updating historical data request records; and a method for performing dynamic data synchronization among multiple edge user terminals.

[0016] In summary, this invention provides a dynamic data synchronization method and an edge user terminal. It transforms ordinary business data into incremental record data by separately storing data request records and comparing the latest request time, avoiding global data queries and comparisons, reducing database pressure caused by data synchronization, and completely decoupling from specific business databases, thus improving flexibility. By using a request-triggered synchronization strategy instead of a timed asynchronous inspection strategy, it minimizes operations on business data with low latency, without affecting business data performance. The use of global backup instead of distributed storage ensures data reliability and integrity. The entire synchronization process only requires configuring the IP address of the edge user terminal, simplifying configuration information and improving ease of operation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the historical verification and data synchronization process of the data dynamic synchronization method in this embodiment of the invention;

[0019] Figure 2 This is a schematic diagram of the structural relationship of the edge user terminal during the dynamic data synchronization process in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0022] Please refer to the following: Figure 1 and Figure 2 This invention provides a method for dynamic data synchronization, comprising the following steps:

[0023] Step S1: Copy and distribute the data request information of any edge user terminal to the remaining edge user terminals.

[0024] Specifically, when a new data request is received on any edge user terminal, that edge user terminal becomes the source edge user terminal for the data request. The source edge user terminal copies the data request information through an intermediate proxy server and distributes it to the other edge user terminals. The intermediate proxy server is preferably an Nginx intermediate proxy server. Nginx is a high-performance HTTP and reverse proxy web server with low memory consumption, high concurrency, and traffic mirroring capabilities, capable of collecting all requests and copying and distributing them to other environments. By configuring the IP addresses of all edge user terminals in Nginx, the entire historical data request record synchronization process only requires configuring the IP addresses of the edge user terminals, maximizing the simplification of configuration information and improving ease of operation.

[0025] Step S2: Compare the time information of the most recent data request from each edge user terminal to determine if there is any missing data. If any edge user terminal has missing data, synchronize its historical data request records. If no edge user terminal has missing data, execute the data request normally. It is understood that, typically, not all edge user terminals are source edge user terminals at the same time. In this case, the data request information received by each edge user terminal will be different. This step can identify edge user terminals with missing data request information and synchronize their historical data request records with those without missing data. Edge user terminals without missing data can skip the historical data request record synchronization step and execute the data request directly.

[0026] like Figure 2 As shown, specifically, in step S2, the most recent data request time is queried from the cache of the edge user terminal to be synchronized, and a request is sent to query the most recent data request time in the caches of the other edge user terminals. Then, the most recent data request time of the edge user terminal to be synchronized is compared with the most recent data request times of the other edge user terminals in chronological order. If the most recent data request time of the edge user terminal to be synchronized is earlier, it is determined that there is a data gap, and its historical data request record is synchronized. If the most recent data request time of the edge user terminal to be synchronized is later, it is determined that there is no data gap, and the data request is executed normally. That is, the last data request time t0 is queried from the cache of the edge user terminal to be synchronized, and at the same time, a data request is sent to query the last request time [t1, t2, ..., t] in the caches of other edge user terminals. n Determine if the last request time t0 of this machine is the same as the last request time [t0, t1, t2, ..., t] of all edge devices. n If t0 is the maximum value in the [database], then the data request is processed normally; otherwise, if t0 is not the maximum value (assuming t0 is not the maximum value), then the request is processed normally. n If the value is the maximum value, then historical data request records will be synchronized on the edge user terminal where data synchronization is to be performed.

[0027] Furthermore, when synchronizing historical data request records on edge user terminals that are to be synchronized with data, the following steps are included:

[0028] Step S21: Send a request to obtain all historical data request records t of the remaining edge user terminals after the most recent data request time of the edge user terminal to be synchronized. n ;

[0029] Step S22: Based on the request time information of the historical data request records, integrate the historical data request records of all the edge user terminals into a union [t0, t1, t2, ..., t] that is sorted by time and does not repeat. n ];

[0030] Step S23: Based on the union of the historical data request records of the edge user terminals, sequentially send simulated requests to the edge user terminals to be synchronized. If the simulated request fails, resend the simulated request after a preset delay time. If the simulated request succeeds, execute the data request normally. The simulated request uses RestTemplate to send simulated HTTP requests. Preferably, if the data request fails or times out, it is retried after a 1-second delay, for a total of three retries. If it still fails after three retries, the historical data request record synchronization step is skipped, and the failed data request records are saved for subsequent manual repair and synchronization. By replacing the timed asynchronous inspection strategy with a request-triggered synchronization strategy, the operation on business data is minimized under low latency conditions, without affecting business data performance.

[0031] Step S3: Update the historical data request record after the data request is successfully executed.

[0032] Specifically, after a data request is successfully executed, each edge user terminal adds it to the batch data processing program and updates the historical data request records through the batch data processing program to synchronize the data of each edge user terminal.

[0033] Preferably, the historical data request record includes: request address, request header information, request body, request time, and the number of the edge user terminal, but is not limited to these.

[0034] Preferably, the historical data request records described in this invention are stored in a separate MySQL database using the InnoDB engine. This means that the historical data request records are stored in a different location than the main business database. By storing the request records separately and comparing the latest request time, ordinary business inventory data is transformed into incremental record data. This avoids global data queries and comparisons, reduces database pressure caused by data synchronization, and completely decouples it from specific business databases, improving flexibility. Using global backup instead of distributed storage ensures data reliability and integrity.

[0035] The present invention also provides an edge user terminal for implementing the aforementioned dynamic data synchronization method, comprising: an operation interface for sending data request information; an intermediate proxy server for copying and distributing the data request information to other edge user terminals; a historical verification program for integrating historical data request records of all the edge user terminals; a batch data processing program for updating historical data request records; and a method for performing the dynamic data synchronization among multiple edge user terminals.

[0036] Preferably, the historical verification procedure is the HandlerInterceptor interceptor in the main program. When verifying related requests, the historical verification procedure uses an RPC remote call framework, bypassing intermediate proxy servers on other edge devices. The call is performed asynchronously using multiple threads, with the main thread waiting for all child threads to complete their calls before executing the judgment operation. If the query call fails or times out, the historical data synchronization process is skipped.

[0037] In summary, this invention provides a dynamic data synchronization method and an edge user terminal. It transforms ordinary business data into incremental record data by separately storing data request records and comparing the latest request time, avoiding global data queries and comparisons, reducing database pressure caused by data synchronization, and completely decoupling from specific business databases, thus improving flexibility. By using a request-triggered synchronization strategy instead of a timed asynchronous inspection strategy, it minimizes operations on business data with low latency, without affecting business data performance. The use of global backup instead of distributed storage ensures data reliability and integrity. The entire synchronization process only requires configuring the IP address of the edge user terminal, simplifying configuration information and improving ease of operation.

[0038] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0039] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0040] The technical solutions provided by the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for dynamic data synchronization, characterized in that, Includes the following steps: Step S1: Copy and distribute the data request information of any edge user terminal to the remaining edge user terminals; Step S2: Compare the time information of the most recent data request of each edge user terminal to determine whether there is any missing data; query the cache of the edge user terminal to be synchronized with data and query the cache of the other edge user terminals with the most recent data request time; then compare the most recent data request time of the edge user terminal to be synchronized with the most recent data request time of the other edge user terminals in time order. If the most recent data request time of the edge user terminal to be synchronized is earlier, it is determined that there is a data gap, and its historical data request record is synchronized. The historical data request record is stored in a different location than the main business database. If the most recent data request time of the edge user terminal to be synchronized is later, it is determined that there is no data gap, and the data request is executed normally. Step S3: Update the historical data request record after the data request is successfully executed.

2. The data dynamic synchronization method as described in claim 1, characterized in that, In step S1, the data request information of any edge user terminal is copied and distributed to the remaining edge user terminals through an intermediate proxy server; wherein The intermediate proxy server contains the configuration information of all the edge user terminals.

3. The data dynamic synchronization method as described in claim 2, characterized in that, The intermediate proxy server includes the Nginx intermediate proxy server.

4. The data dynamic synchronization method as described in claim 1, characterized in that, When synchronizing historical data request records on edge user terminals that are being synchronized, the following steps are included: Step S21: Send a request to obtain all historical data request records of the other edge user terminals after the most recent data request time of the edge user terminal to be synchronized; Step S22: Based on the request time information of the historical data request records, integrate the historical data request records of all the edge user terminals into a union that is sorted by time and does not repeat; Step S23: Based on the union of the historical data request records of the edge user terminal, send simulated requests sequentially to the edge user terminal to be synchronized with data; if the simulated request fails, resend the simulated request after a preset delay time; if the simulated request succeeds, execute the data request normally.

5. The data dynamic synchronization method as described in claim 4, characterized in that, The simulated request uses RestTemplate to send a simulated HTTP request.

6. The data dynamic synchronization method as described in claim 1, characterized in that, In step S3, after the data request is successfully executed, it is added to the batch data processing program, and the historical data request record is updated.

7. The data dynamic synchronization method as described in claim 1, characterized in that, The historical data request record includes: request address, request header information, request body, request time, and the number of the edge user terminal.

8. The data dynamic synchronization method as described in claim 1, characterized in that, The historical data request records are stored in a separate MySQL database using the InnoDB engine.

9. An edge user terminal, characterized in that, include: The user interface is used to send data request information; An intermediate proxy server is used to copy and distribute the data request information to other edge user terminals; A historical verification procedure is used to integrate the historical data request records of all the aforementioned edge user terminals; A batch data processing program used to update historical data request records; The method for dynamic data synchronization as described in any one of claims 1 to 8 is performed between multiple edge user terminals.

Images