A method for cross-service information communication and data synchronization in a container terminal
By embedding listeners and scheduled tasks in each service to synchronize data and events, the method addresses data isolation issues, ensuring timely and accurate communication in container terminal systems, reducing complexity and dependency on external systems.
Patent Information
- Application Number
- CN202311482977.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-08
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2043-11-08
AI Technical Summary
Existing container terminal systems face challenges in real-time data change monitoring and communication between services due to data isolation, leading to reduced system availability, increased complexity, message consistency issues, and dependency on external systems, which are exacerbated by the use of message middleware.
Implementing a listener in each service, a configurable scheduled task, and a unified interface to monitor events, synchronize service operations, redis data, and local cache data, ensuring timely and accurate communication by replacing message middleware.
The solution ensures accurate and timely cross-service information communication, reduces system complexity and dependency, and maintains system stability by eliminating the need for message middleware.
Smart Images
Figure CN117527827B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to container terminal communication technology, and more particularly to a container terminal cross-service information communication and data synchronization method. Background Art
[0002] In the container terminal production control system (TOS) that uses a microservice architecture, there is a problem of data isolation between services. For example, after service A operates on a certain data, service B will not know the change of the data synchronously, which causes service B to fail to respond to the relevant operations in a timely manner. How to monitor data changes in real time and communicate between services is a complex problem. Industrial software systems require stability and reliability, which brings limitations to how to solve the above problems.
[0003] The traditional way to achieve inter-service communication and data synchronization is to use message middleware, but it has the following major disadvantages:
[0004] 1) Reduced system availability: As the number of external dependencies introduced by the system increases, the stability of the system will deteriorate. Once the message middleware goes down, it will have a significant impact on the business.
[0005] 2) Increased system complexity: The introduction of message middleware will greatly increase system complexity. Previously, services could be called synchronously, but after the introduction of message middleware, they will become asynchronous calls. The data link will become more complex and will bring a series of problems.
[0006] 3) Message consistency issues: Because it is an asynchronous call and different services are isolated from each other, it is possible that system A succeeds while system B fails.
[0007] 4) If the network slows down, a buffer that is too small set by the message middleware can cause the client to be idle, but if the network is functioning properly, a buffer that is too large can cause a lot of extra latency.
[0008] Without using message middleware for inter-service communication, data duplication may occur in clustered deployment of services. The traditional way is to use a distributed task scheduling platform for control, which inevitably increases the service's dependence on external systems and creates a certain degree of uncertainty. Summary of the invention
[0009] In view of the above-mentioned defects existing in the prior art, the object of the present invention is to provide a method for cross-service information communication and data synchronization in a container terminal, which can replace the message middleware in each service and ensure the accuracy and timeliness of communication between services.
[0010] To achieve the above object, the present invention adopts the following technical solution:
[0011] A method for cross-service information communication and data synchronization in a container terminal:
[0012] A listener is embedded in each service;
[0013] A configurable scheduled task is embedded in each service;
[0014] A unified interface is set up in each service to monitor the events sent;
[0015] Unify the timing of sending management events for each service;
[0016] Synchronize service operations, redis data, local cache, and local disk data pre-database.
[0017] Preferably, synchronizing the service operation, redis data, local cache, and local disk data pre-database specifically includes the following steps:
[0018] S1. After the front-end system triggers the data change operation, the service processes the changed data uniformly and saves the processed data to the "event log" table;
[0019] S2. The scheduled task is executed once according to the set time to query the data changes in the "Event Log" table. After reading the data, the number in the local cache is first compared with the corresponding object number in the redis data, and the larger one is taken; then it is compared with the number of the data just read. If the number of the data just read is smaller, no operation is performed. If the number of the data just read is larger, the read data is processed;
[0020] S3, convert the format of the read data and save it to the "Event Log" table, modify the local cache first, then modify the redis data cache, and finally send the data out;
[0021] S4. The service receiving data subscribed to the relevant "event object" performs corresponding processing.
[0022] Preferably, step S1 specifically includes:
[0023] When the front-end system is operated, the corresponding service will generate data changes. After the operation is completed, the changed data will be processed and formatted, and the public interface will be called to save the processed data to the "Event Log" table.
[0024] Preferably, the data needs to be processed and screened again before being saved:
[0025] Traverse the data. Each "monitoring object" data must correspond to multiple trigger data in the "event log" table. First, determine whether the trigger data and the "monitoring object" data are empty. If one of them is empty, skip it and enter the next loop; if neither is empty, convert the trigger data format and store it in the "event log" table.
[0026] Preferably, in step S2, the data type polled by the scheduled task of each service is determined according to the "monitoring object" read when the program is started;
[0027] In order to ensure that only one scheduled task can read data at the same time for the same service in a clustered deployment, the redis distributed lock method is adopted. In addition, in order to prevent deadlock or lock failure during data writing, the lua script is also used to lock, unlock and renew the scheduled task; that is,
[0028] If the distributed lock exists, this scheduled task will not be executed; if the distributed lock does not exist, determine whether the set of "monitoring objects" is empty. If it is empty, this scheduled task will not be executed; if it is not empty, query whether the set of "monitoring objects" exists in the database. If none of them exist, this scheduled task will not be executed; if some of them exist, delete the non-existent "monitoring objects" and then execute this scheduled task; if all of them exist, execute this scheduled task, traverse the "monitoring objects", and query all trigger data about the object within the configured hour in the "event log" table.
[0029] Preferably, step S3 specifically includes:
[0030] Traverse the skip list, take out the value in the local cache, and define it as originValue;
[0031] First, update the local cache and update the maximum number in the cacheTrigger data of the previous step to the local cache. If an exception occurs, jump out of the loop directly and enter the next loop. If no exception occurs, update the redis data. If an exception occurs, replace the value of the local cache with originValue and jump out of the current loop. If no exception occurs, send a message by starting the completed listener.
[0032] Preferably, step S4 specifically includes:
[0033] Each service implements an interface for receiving messages, and monitors messages according to the type of "listener object" registered in the listener when the program starts, thereby processing the messages.
[0034] The container terminal cross-service information communication and data synchronization method provided by the present invention can replace the message middleware in each service and ensure the accuracy and timeliness of communication between services. At the same time, it also reduces the coupling dependency between different systems and ensures the timeliness and accuracy of cross-service information communication, thereby replacing the message middleware. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a flow chart of the method for cross-service information communication and data synchronization of a container terminal according to the present invention;
[0036] Figure 2 It is a schematic diagram of the principle of the container terminal cross-service information communication and data synchronization method of the present invention. DETAILED DESCRIPTION
[0037] In order to better understand the above technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0038] A container terminal cross-service information communication and data synchronization method provided by the present invention:
[0039] A listener is embedded in each service;
[0040] A configurable scheduled task is embedded in each service;
[0041] A unified interface is set up in each service to monitor the events sent;
[0042] Unify the timing of sending management events for each service;
[0043] Synchronize service operations, redis data, local cache, and local disk data pre-database.
[0044] Combination Figure 1 and Figure 2 As shown, in the present invention, synchronizing service operations, redis data, local cache, and local disk data pre-database specifically includes the following steps:
[0045] S1. After the front-end system triggers the data change operation, the service processes the changed data uniformly and saves the processed data to the "event log" table;
[0046] S2. The scheduled task is executed once according to the set time to query the data changes in the "Event Log" table. After reading the data, the number in the local cache is first compared with the corresponding object number in the redis data, and the larger one is taken; then it is compared with the number of the data just read. If the number of the data just read is smaller, no operation is performed. If the number of the data just read is larger, the read data is processed;
[0047] S3, convert the format of the read data and save it to the "Event Log" table, modify the local cache first, then modify the redis data cache, and finally send the data out;
[0048] S4. The service receiving data subscribed to the relevant "event object" performs corresponding processing.
[0049] The above step S1 specifically includes:
[0050] When the front-end system is operated, the corresponding service will generate data changes. After the operation is completed, the changed data will be processed and formatted, and the public interface will be called to save the processed data to the "Event Log" table.
[0051] The data needs to be processed and screened again before being saved:
[0052] Traverse the data. Each "monitoring object" data must correspond to multiple trigger data in the "event log" table. First, determine whether the trigger data and the "monitoring object" data are empty. If one of them is empty, skip it and enter the next loop; if neither is empty, convert the trigger data format and store it in the "event log" table.
[0053] In the above step S2, the data type polled by the scheduled task of each service is determined according to the "monitoring object" read when the program is started;
[0054] In order to ensure that only one scheduled task can read data at the same time for the same service in a clustered deployment, the redis distributed lock method is adopted. In addition, in order to prevent deadlock or lock failure during data writing, the lua script is also used to lock, unlock and renew the scheduled task; that is,
[0055] If the distributed lock exists, this scheduled task will not be executed; if the distributed lock does not exist, determine whether the set of "monitoring objects" is empty. If it is empty, this scheduled task will not be executed; if it is not empty, query whether the set of "monitoring objects" exists in the database. If none of them exist, this scheduled task will not be executed; if some of them exist, delete the non-existent "monitoring objects" and then execute this scheduled task; if all of them exist, execute this scheduled task, traverse the "monitoring objects", and query all trigger data about the object within the configured hour in the "event log" table.
[0056] The above step S3 specifically includes:
[0057] Traverse the skip list, take out the value in the local cache, and define it as originValue;
[0058] First, update the local cache and update the maximum number in the cacheTrigger data of the previous step to the local cache. If an exception occurs, jump out of the loop directly and enter the next loop. If no exception occurs, update the redis data. If an exception occurs, replace the value of the local cache with originValue and jump out of the current loop. If no exception occurs, send a message by starting the completed listener.
[0059] The above step S4 specifically includes:
[0060] Each service implements an interface for receiving messages, and monitors messages according to the type of "listener object" registered in the listener when the program starts, thereby processing the messages.
[0061] Example
[0062] Continue to refer Figure 1 and Figure 2 As shown, the container terminal cross-service information communication and data synchronization method of this embodiment specifically includes the following steps:
[0063] S1. When users operate the containers or gates at the terminal through the front-end system (C-end), the corresponding services will generate data changes (see Figure 2A), after the operation is completed, the program will uniformly process and convert the changed data, call the public interface to save the processed data to the "Event Log" table, and the data needs to be processed and screened before storage. Because each operation on the C end may cause multiple data changes, it is necessary to traverse and store multiple data. The specific operation is as follows: traverse the data, each "monitoring object" corresponds to multiple data (trigger) in the "Event Log" table, so first determine whether the trigger data and the "monitoring object" data are empty. If one of the two is empty, skip directly and enter the next loop; if neither is empty, convert the trigger data format and store it in the "Event Log" table (see Figure 2 (middle B).
[0064] S2. Scheduled tasks will poll the "Event Log" table according to the configured time. The data type polled by the scheduled task of each service is determined by the "listening object" read when the program starts. In order to ensure that only one scheduled task can read data at the same time for the same service in the case of clustered deployment, the redis distributed lock method is introduced, and in order to prevent deadlock or lock failure during data writing, the Lua script is used to lock, unlock, and renew the scheduled task. If the distributed lock exists, the scheduled task will not be executed; if the distributed lock does not exist, determine whether the set of "listening objects" is empty. If it is empty, the scheduled task will not be executed; if it is not empty, query whether the set of "listening objects" exists in the database. If none of them exist, the scheduled task will not be executed; if some of them exist, delete the non-existent "listening objects" and execute the scheduled task; if all of them exist, execute the scheduled task, traverse the "listening objects", and query all the data (trigger) about the object within one hour (the time interval is configurable) in the "Event Log" table (see Figure 2 Middle C).
[0065] Traverse the above data (trigger), according to the corresponding "listening object" ID, take out the object number in redis and local cache at the same time, compare the two sizes and take the larger one, define it as cacheKey, distinguish according to the type of "listening object" during the traversal process, take out all trigger data with numbers greater than cacheKey, and define the data as cacheTrigger (see Figure 2 D). Traverse the above data (cacheTrigger), process and convert each piece of data, and insert the processed data into a skip list (to ensure the orderliness of the data, choose to use a skip list).
[0066] S3. After preparing the above skip table data, start updating and sending the data. Traverse the skip table, take out the value in the local cache, and define it as originValue. First update the local cache, update the maximum number in the data cacheTrigger in the previous step to the local cache. If an exception occurs, directly jump out of the loop and enter the next loop to ensure the correctness of the data (see Figure 2 E); if no exception occurs, update the redis data in the same way as above (see Figure 2 F), if an exception occurs, the local cache value is replaced with originValue, and the current loop is jumped out to ensure the correctness of the local cache data; if no exception occurs, the message is sent by starting the completed listener (see Figure 2 Middle G).
[0067] S4. Each service implements the interface for receiving messages, and listens to messages according to the type of "listener object" registered in the listener when the program is started, so as to process the messages (see Figure 2 Middle H).
[0068] In summary, the present invention reduces the coupling dependency between different systems and can ensure the timeliness and accuracy of cross-service information communication, thereby replacing message middleware.
[0069] Those skilled in the art should recognize that the above embodiments are only used to illustrate the present invention, and are not intended to limit the present invention. As long as they are within the spirit of the present invention, any changes or modifications to the above embodiments will fall within the scope of the claims of the present invention.
Claims
1. A method for cross-service information communication and data synchronization in a container terminal, characterized in that: A listener is embedded in each service; A configurable scheduled task is embedded in each service; A unified interface is set up in each service to monitor the events sent; Unify the timing of sending management events for each service; Synchronize service operations, redis data, local cache, and local disk data pre-database. Synchronizing service operations, redis data, local cache, and local disk data pre-database specifically includes the following steps: S1. After the front-end system triggers the data change operation, the service processes the changed data uniformly and saves the processed data to the "event log" table. Step S1 specifically includes: When the front-end system is operated, the corresponding service will generate data changes. After the operation is completed, the changed data is processed and formatted, and the public interface is called to save the processed data to the "event log" table. The data needs to be processed and screened again before being saved: Traverse the data. Each "monitor object" data must correspond to multiple trigger data in the "event log" table. First, determine whether the trigger data and the "monitor object" data are empty. If one of them is empty, skip it and enter the next loop. If neither is empty, convert the trigger data format and store it in the "event log" table. S2. The scheduled task is executed once according to the set time to query the data changes in the "Event Log" table. After reading the data in the "Event Log" table, the number in the local cache is first compared with the corresponding object number in the redis data, and the larger one is taken; then it is compared with the number of the data just read. If the number of the data just read is smaller, no operation is performed. If the number of the data just read is larger, the read data is processed. In step S2, the data type polled by the scheduled task of each service is determined according to the "monitoring object" read when the program is started; In order to ensure that only one scheduled task can read data at the same time for the same service in a clustered deployment, the redis distributed lock method is adopted. In addition, in order to prevent deadlock or lock failure during data writing, the lua script is also used to lock, unlock and renew the scheduled task; that is, If the distributed lock exists, this scheduled task will not be executed; if the distributed lock does not exist, determine whether the set of "listening objects" is empty. If it is empty, this scheduled task will not be executed; if it is not empty, query whether the set of "listening objects" exists in the database. If none of them exist, this scheduled task will not be executed; if some of them exist, delete the non-existent "listening objects" and then execute this scheduled task; if all of them exist, execute this scheduled task, traverse the "listening objects", and query all trigger data about the object within the configured hour in the "event log" table; S3, convert the format of the read data and save it to the "Event Log" table, modify the local cache first, then modify the redis data cache, and finally send the data out; Step S3 specifically includes: Traverse the skip list, take out the value in the local cache, and define it as originValue; First, update the local cache and update the maximum number in the cacheTrigger data of the previous step to the local cache. If an exception occurs, jump out of the loop and enter the next loop. If no exception occurs, update the redis data. If an exception occurs, replace the value of the local cache with originValue and jump out of the current loop. If no exception occurs, send a message by starting the completed listener. S4, subscribing to the service receiving data of the relevant "event object" and performing corresponding processing, step S4 specifically includes: Each service implements an interface for receiving messages, and monitors messages according to the type of "listener object" registered in the listener when the program starts, thereby processing the messages.