Cascade-based multi-platform data synchronization method and system

Abstract

The invention belongs to the field of video monitoring systems, and particularly relates to a cascade-based multi-platform data synchronization method and system, and the method comprises the steps: designing an interface security mechanism of cross-platform data exchange; constructing a data synchronization service architecture, namely deploying corresponding data synchronization services at different levels of platforms to realize resource data synchronization, and supporting data synchronization of multi-level cascade or platform interconnection; performing data initialization synchronization; real-time synchronization of data change is realized; establishing a service heartbeat keep-alive mechanism; service multi-instance deployment is carried out, and efficient data processing and flexible load balancing are achieved. According to the invention, many defects of a data synchronization mechanism of an existing video monitoring system are effectively solved, and the stability, expansibility and data processing efficiency of the system are improved.

Description

Technical Field

[0001] The invention belongs to the field of video monitoring systems, and in particular relates to a cascade-based multi-platform data synchronization method and system. Background Art

[0002] At present, most video surveillance systems on the market exist in the form of independent platforms, and there is a lack of effective data sharing and synchronization mechanisms between these systems. Although the national standard cascade protocol provides certain device connection capabilities, this capability is limited to the basic level and cannot meet the higher requirements for data sharing in complex business scenarios.

[0003] The existing related technologies have at least the following deficiencies: 1. Scalability limitation: The national standard cascade does not support the operation of multiple instances of services. A single service instance must handle all requests from the cascade source. This design limits the scalability of the system. When the number of users grows or the data traffic increases, a single service instance may become a performance bottleneck. It is difficult to improve the overall processing capacity of the system through simple horizontal expansion to meet the needs of large-scale deployment.

[0004] 2. Processing capacity limitations: When faced with a large amount of data from a single cascade source, a single service instance of the national standard cascade protocol may not be able to effectively handle high-concurrency requests, resulting in data processing delays or failures, thus affecting user experience and system reliability.

[0005] 3. System stability risk: Due to the single nature of the service instance, once a problem occurs in the instance, all cascade communications that depend on this instance will be affected. This single point of failure risk increases the risk of system stability, which is a problem that cannot be ignored for video surveillance systems that require high availability.

[0006] 4. Lack of diversified data synchronization: The national standard cascade mainly focuses on the synchronization of basic data such as organizations, equipment, and channels. There is a lack of effective synchronization mechanism for the increasingly diversified data types in video surveillance systems, such as alarms, preset points, project ledgers, tags, etc.

[0007] 5. Difficulty in subdividing functional permissions: Since the national standard cascade permission only has one domain permission, it is difficult to achieve fine-grained permission control. In a complex system, more detailed permission control may be required for different organizations or devices. For example, a certain organization's device may be allowed to view real-time video, but not allowed to perform pan / tilt control. This subdivided permission control requirement is difficult to achieve under the national standard cascade protocol.

[0008] 6. No support for custom coding: The national standard cascade does not provide support for custom coding, which limits the flexibility of organizations in resource management and sharing. When different devices or platforms need to publish resources to the same parent platform, if their channel codes are repeated, resource conflicts will occur, which not only affects the unified management of resources, but also increases the complexity of cross-platform resource sharing. For organizations that need to share resources between multiple parent platforms, this limitation increases the difficulty of adjusting personalized resource codes, limits the organization's ability to allocate and manage resource codes according to specific business logic, and thus constrains the organization's innovation and efficiency in resource sharing and management.

[0009] In order to cope with more diversified and complex business scenarios, technical personnel in this field need a more complete and efficient data sharing mechanism to ensure that data between systems can be exchanged and synchronized smoothly. Summary of the invention

[0010] In view of the shortcomings of the current relevant existing technologies, the present invention is committed to overcoming the defects of the existing video surveillance system data synchronization mechanism and providing a cascade-based multi-platform data synchronization method, aiming to break through the scalability bottleneck and support multi-instance operation of services to adapt to large-scale deployment; enhance data processing efficiency, handle high-concurrency data requests and reduce latency; improve system stability and reduce the risk of single point failures; achieve diversified data synchronization, covering a variety of data types; refine function permission control and accurately limit resource operations; support custom coding and improve resource management flexibility; ensure resource traceability; ensure interface security and data consistency; optimize data consistency maintenance to ensure that the upper-level platform data reflects the status of the lower-level platform in a timely and accurate manner, thereby providing a more complete and efficient data synchronization solution for the video surveillance system to meet the needs of complex business scenarios.

[0011] To achieve the above object, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a multi-platform data synchronization method based on cascade, comprising the following steps: Step S1: Design an interface security mechanism for cross-platform data exchange; Step S2: Building a data synchronization service architecture, including deploying corresponding data synchronization services on different-level platforms to achieve resource data synchronization, and supporting multi-level cascading or platform interconnected data synchronization; Step S3: Perform data initialization synchronization; Step S4: Realize real-time synchronization of data changes; Step S5: Establish a service heartbeat keep-alive mechanism; Step S6: Perform service multi-instance deployment.

[0012] Furthermore, in step S1, the interface security mechanism adopts a unified POST request method combined with the JSON data format for cross-platform data exchange, and integrates interface request header parameters for identity authentication and security verification and corresponding response parameters in the request, wherein the interface request header parameters include one or more of the platform node code, user name, password, and random code; the response parameters include one or more of the response code, response description, and response data.

[0013] Further, in step S2, data synchronization in the data synchronization service architecture is divided into two services including CDS and SDS, wherein CDS is the upper-level data synchronization service and SDS is the lower-level data synchronization service, respectively deploying data synchronization services in the upper-level platform and the lower-level platform to achieve synchronization of organization, equipment, and channel resource data; the data synchronization service architecture supports multi-level cascade data synchronization or two platforms cascaded with each other; The upper-level data synchronization service CDS calls the lower-level data synchronization service SDS to obtain the data information that needs to be shared with the upper-level platform, and then the upper-level platform saves this data information into the local database for storage; The function of the subordinate data synchronization service SDS is to provide an interface for the superior data synchronization service CDS to call and obtain the data information shared to the superior platform; if the subordinate data is changed, the superior data synchronization service CDS interface is also called to notify the superior platform to update the data to ensure the consistency of the superior and subordinate data.

[0014] Furthermore, in step S3, data initialization synchronization includes at least the following steps: S31: After the lower-level data synchronization service SDS is started, it will first call the service online interface of the upper-level data synchronization service CDS. The request header will carry the username and password to perform authentication verification on the upper and lower platforms. If the verification passes, the status of the upper and lower services will be changed to connecting. S32: The service online interface request parameter for requesting the upper data synchronization service CDS will carry the data items that need to be synchronized, including one or more of the organization, device, channel and preset point information that need to be synchronized. After receiving the data, the upper data synchronization service CDS will first save the data items that need to be synchronized; and the synchronization data also has a sequence, that is, the organization information is synchronized first, and then the device and channel information are synchronized according to the organization; S33: After obtaining the organization information that needs to be synchronized, the organization data needs to be processed. The existing organization codes or organization names on the platform are considered to belong to the same organization. These organizations are merged and then saved in the database. The devices, channels, and preset point information under the organization are then synchronized through the organization.

[0015] Furthermore, the organization information is processed according to the organization merger principle and then stored in the database and the relevant data is synchronized. The organization merger principle includes adding a brother node number field to the organization table, setting whether to merge and the merge method options when configuring cascading, and if not merged, it is stored in the database according to the coding prefix rule. If merged, the merge rule is determined based on the mount point and push organization relationship, the organization code and the name length.

[0016] Furthermore, the principles of organizational merger include: a. Add a new field in the organization table, namely the brother node number bro_idx. Nodes whose bro_idx is null can be displayed, while nodes whose bro_idx is not null cannot be displayed. b. When configuring cascading, a configuration item should be included to indicate whether to merge the same organization; c. If not merged, a mount point must be selected and all organizations are stored according to the rules of the encoding prefix; d. If merging, select a mount point. If the mount point is at the same level as the highest level organization pushed, merge it; if the mount point is not at the same level as the highest level organization pushed, do not consider merging, which is equivalent to not merging at all; e. The merging rules are as follows: organizations with a number less than six digits are merged by code; organizations with a number greater than six digits are merged by name; if the parent organization does not meet the merging rules, its child organizations will not be merged.

[0017] Furthermore, in step S4, when the user modifies the data on the lower-level platform, a table change notification will be sent to Kafka, and the lower-level data synchronization service SDS will synchronize the consumption to the upper-level platform in real time to ensure the consistency of the upper and lower-level data; The following are the steps to synchronize data changes: S41: The lower-level data synchronization service SDS monitors the Kafka of the lower-level platform. If there is a data change, it consumes the Kafka information to call the data change interface of the upper-level data synchronization service CDS, and carries the changed data type and data uid identifier; S42: The upper-level data synchronization service CDS receives the changed data record change type and data uid identifier, writes it to Redis for caching, and returns a response result to the lower-level data synchronization service SDS; S43: The upper-level data synchronization service CDS consumes and reads the cached data information, and calls the corresponding data acquisition interface by changing the data type and data uid identifier; S44: after acquiring the data, the acquired data is processed and then stored in the database; S45: Clear the changed data cache just consumed.

[0018] Further, in step S5, In order to ensure the real-time status of the upper and lower cascade services, the service status can be updated in real time when the service is abnormal. For this purpose, a service heartbeat keep-alive mechanism is designed. The following are the detailed steps for service heartbeat keep-alive: S51: The lower-level data synchronization service SDS sends a heartbeat keep-alive request to the upper-level data synchronization service CDS at a predetermined interval; S52: After receiving the message, the upper-level data synchronization service CDS caches a heartbeat key in Redis, sets an expiration time, and then returns the renewal result to the lower-level data synchronization service SDS; S53: If the result returned by requesting the heartbeat keep-alive interface of the upper-level data synchronization service CDS is a failure, a retry mechanism is performed. If the renewal fails after the predetermined number of retries, it indicates that the upper-level service is disconnected. The link status of the cascade is changed to running. If the next renewal is successful, the status is changed to connecting. S54: The upper-level data synchronization service CDS will scan the Redis heartbeat key every predetermined time to see if it exists. If not, it is considered that the lower-level service is disconnected, and the link status of the cascade is changed to running. If the heartbeat key exists next time, the status is changed to connecting.

[0019] Furthermore, the working mode of the service multi-instance deployment is: each service corresponds to a lower / upper node, and supports multi-instance deployment. The service name remains consistent and is used to process only the data corresponding to its own cascade; When there are multiple instances of the upper-level data synchronization service CDS, data is obtained from the Redis queue and processed by type in a preemptive manner; When there are multiple instances of the lower-level data synchronization service SDS, only one service will send a service launch request to the corresponding upper-level service. If the upper-level data synchronization service CDS calls the data acquisition interface, it will be randomly distributed to different instances for execution.

[0020] A second aspect of the present invention provides a cascade-based multi-platform data synchronization system, which is used to implement the above method. The multi-platform data synchronization system includes: The interface security module is used to use a unified POST request method combined with the JSON data format for cross-platform data exchange, and integrates public request header parameters for identity authentication and security verification in each request; Data synchronization service module, including: The upper-level data synchronization service CDS is deployed on the upper-level platform and is used to call the lower-level data synchronization service to obtain the data information to be shared with the upper-level platform and save it to the local database; The lower-level data synchronization service SDS is deployed on the lower-level platform to provide an interface for the upper-level data synchronization service to call and obtain data information and notify the upper-level platform to update the data when the lower-level data changes, so as to achieve the synchronization of resource data such as organizations, equipment, and channels, and support multi-level cascade data synchronization or cascading between two platforms; The data initialization synchronization module is used to call the service online interface of the upper-level data synchronization service CDS with the parameters for identity authentication to perform authentication verification on the upper and lower platforms after the lower-level data synchronization service SDS is started. If the verification is passed, the service status of both parties is changed to being linked, and the lower-level data synchronization service SDS carries the data items to be synchronized when requesting the service online interface of the upper-level data synchronization service CDS. The upper-level data synchronization service CDS saves these data items and synchronizes them in the order of organization first, then according to the organization synchronization device and channel information. At the same time, the obtained organization information is processed according to the organization merger principle and then stored in the warehouse, and then the lower-level device, channel and preset point information is synchronized through the organization; The data change real-time synchronization module is used to enable the subordinate data synchronization service SDS to monitor the data change information of the subordinate platform when the data of the subordinate platform changes, and call the data change interface of the superior data synchronization service CDS, carrying the changed data type and data identifier. After the superior data synchronization service CDS receives the changed data, it records the change type and data identifier and caches it, and returns the response result to the subordinate data synchronization service SDS. The superior data synchronization service CDS reads the cached data information, calls the corresponding data acquisition interface according to the changed data type and data identifier, processes the acquired data and puts it into the warehouse, and cleans up the changed data cache; The service heartbeat keep-alive module is used to enable the subordinate data synchronization service SDS to periodically send heartbeat keep-alive requests to the superior data synchronization service CDS. After receiving the request, the superior data synchronization service CDS sets the heartbeat-related key value in the cache, sets the expiration time, and returns the renewal result to the subordinate data synchronization service SDS. If the subordinate data synchronization service SDS fails to renew, it will retry. If the retry fails multiple times, it is considered that the superior service is disconnected and the cascade link status is modified. If the subsequent renewal is successful, it is modified back to the original status. The superior data synchronization service CDS periodically scans the heartbeat-related key value in the cache. If it does not exist, it is considered that the subordinate service is disconnected and the cascade link status is modified. If it exists, it is modified back to the original status. The service multi-instance deployment module is used to make each service correspond to a subordinate / superior node, support multi-instance deployment, keep the service name consistent, obtain data from the cache queue and process it by type when the superior data synchronization service CDS has multiple instances, and only one service sends a service online request to the superior when the subordinate data synchronization service SDS has multiple instances. When the superior data synchronization service CDS calls the data acquisition interface, it is randomly allocated to different instances for execution.

[0021] The present invention adopts the above technical solution, which has at least the following beneficial effects: 1. The use of multiple instances of services overcomes the scalability limitations of the national standard cascade protocol. This design allows the system to dynamically expand service instances based on the demand for increased user volume or data traffic, thereby breaking through the performance bottleneck of a single service instance, effectively improving the overall processing capacity of the system, and meeting the needs of large-scale deployment. In addition, through load balancing of multiple service instances, the stability risk of the system caused by the failure of a single service instance is reduced. This design significantly improves the fault tolerance of the system, ensures the high availability of the video surveillance system, and avoids the impact of single point failures on the entire system.

[0022] 2. The efficient data processing mechanism can process a large amount of data from a single cascade source, avoiding the high-concurrency request processing bottleneck that a single service instance may encounter in the existing technology. This not only reduces the delay of data processing, but also improves the reliability of the system, thereby optimizing the user experience.

[0023] 3. Not only is it limited to synchronizing basic data, it also expands the types of data that can be synchronized, including alarm information, preset points, project ledgers, tags, etc. This improvement meets the growing diversified data needs of video surveillance systems and enhances data integrity and availability.

[0024] 4. Support for custom coding solves the problem of insufficient flexibility in resource sharing and management. Organizations can assign unique codes to resources based on specific business needs and retain these codes when resources are shared across platforms, thereby avoiding conflicts in resource codes between different platforms and improving the efficiency of resource sharing and the convenience of management.

[0025] 5. Implemented service multi-instance deployment and intelligent load balancing strategy, significantly enhancing the data processing efficiency and scalability of the system. This strategy effectively reduces the risk of single point failure by distributing load among different service instances, providing solid technical support for building an efficient and stable cross-platform cascade system.

[0026] 6. Use the Redis queue cache mechanism to manage the data to be processed, so that multiple service instances can handle the data load evenly. This Redis-based load balancing method not only improves the scalability of the service when processing large amounts of data, but also allows the system to dynamically expand the number of service instances based on actual needs. This design enables the system to flexibly adapt to deployment requirements of different scales, and can maintain efficient and stable operation whether it is responding to daily data processing needs or large amounts of data influx during peak periods.

[0027] 7. It provides a resource code traceability function, which ensures a complete record of the resource path by retaining the platform code through which the resource code has circulated. This function plays an important role in ensuring the authenticity, validity and traceability of information, especially in scenarios where cross-platform tracking and management of resources are required, significantly improving the resource management and auditing capabilities.

[0028] 8. Implemented an efficient, standardized and secure cross-platform data exchange protocol, using a unified POST request method and JSON data format, combined with a public request header for identity authentication and security verification, to ensure the security and consistency of data transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is a schematic diagram of the present invention including interface request header parameters and interface response parameters.

[0031] Figure 2 It is a schematic diagram of the platform deployment architecture of the present invention.

[0032] Figure 3 It is a schematic diagram of the mutual cascading of two platforms of the present invention.

[0033] Figure 4 It is a timing diagram of data initialization of the present invention.

[0034] Figure 5 It is a schematic diagram of a specific implementation method of the organization and combination of the present invention.

[0035] Figure 6 It is a data change timing diagram of the present invention.

[0036] Figure 7 This is a timing diagram of the service keep-alive of the present invention. DETAILED DESCRIPTION

[0037] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0038] A first aspect of this embodiment provides a cascade-based multi-platform data synchronization method, comprising the following steps: Step S1: Design an interface security mechanism for cross-platform data exchange; Step S2: Building a data synchronization service architecture, including deploying corresponding data synchronization services on different-level platforms to achieve resource data synchronization, and supporting multi-level cascading or platform interconnected data synchronization; Step S3: Perform data initialization synchronization; Step S4: Realize real-time synchronization of data changes; Step S5: Establish a service heartbeat keep-alive mechanism; Step S6: Deploy multiple service instances.

[0039] As an implementation method, Figure 1 The figure shows the relevant information of the interface request header parameters and the interface response parameters. In step S1, the interface security mechanism uses a unified POST request method combined with the JSON data format to exchange data across platforms, and integrates the interface request header parameters and the corresponding response parameters used for identity authentication and security verification in the request, wherein the interface request header parameters include one or more of the platform node code, user name, password, and random code; the response parameters include one or more of the response code, response description, and response data.

[0040] As an implementation mode, in step S2, if Figure 2 and Figure 3 As shown in the figure, data synchronization in the data synchronization service architecture is divided into two services including CDS and SDS, where CDS is the upper-level data synchronization service and SDS is the lower-level data synchronization service. Data synchronization services are deployed in the upper-level platform and the lower-level platform respectively to achieve synchronization of organization, equipment, and channel resource data; the data synchronization service architecture supports multi-level cascade data synchronization (such as the direct interaction between the cascade structure on the left and the interconnection architecture on the right) or cascading between two platforms (such as the interaction between platform A, platform B, and platform C included in the cascade architecture on the left); Figure 3 The left side shows the cascade architecture, and the right side shows the interconnect architecture.

[0041] The data synchronization service is described as follows: The upper-level data synchronization service CDS calls the lower-level data synchronization service SDS to obtain the data information that needs to be shared with the upper-level platform, and then the upper-level platform saves this data information into the local database for storage; The function of the subordinate data synchronization service SDS is to provide an interface for the superior data synchronization service CDS to call and obtain the data information shared to the superior platform; if the subordinate data is changed, the superior data synchronization service CDS interface is also called to notify the superior platform to update the data to ensure the consistency of the superior and subordinate data.

[0042] like Figure 4 As shown, this is the data initialization timing diagram of this embodiment, the overall process: Includes the following modules: The upper-level data synchronization service is started (CDS goes online): The entire data initialization process starts with the upper-level data synchronization service (CDS) going online.

[0043] Authentication verification: The superior data synchronization service (CDS) sends an authentication verification request to its corresponding database (CDS-DB). This step is to ensure that CDS has legal authority to obtain and process data.

[0044] CDS-DB performs authentication verification on CDS and returns the result to CDS.

[0045] Data acquisition decision: CDS decides whether to acquire full data from the lower-level data synchronization service (SDS) based on the authentication verification results. This step ensures that data acquisition is performed only when it is legal and necessary.

[0046] Data acquisition and query: If it is decided to acquire data, CDS sends a data acquisition request to SDS.

[0047] After receiving the request, SDS will query its corresponding database (SDS-DB) to determine which data needs to be shared with the superior. This step involves SDS-DB returning data information to SDS.

[0048] Data response and processing: SDS responds to the queried data to CDS. This is a key step for the subordinate to pass data to the superior.

[0049] After receiving the data from SDS, CDS processes the data. The processing operations may include data format conversion, data cleaning, etc. to ensure that the data meets the storage and use requirements of the superior.

[0050] Data storage: The processed data is stored in the database of the upper-level data synchronization service (CDS-DB). This step marks the successful synchronization of data from the lower level to the upper level and completes the data initialization process.

[0051] The functions of each module are: Database of the superior data synchronization service (CDS-DB): The main function is to store data related to the superior data synchronization service (CDS). In the process, it is responsible for authenticating and verifying the CDS, ensuring the legitimacy of data access, and receiving and storing processed data.

[0052] Superior Data Synchronization Service (CDS): As the initiator of data synchronization, it is responsible for authentication verification after going online, deciding whether to obtain data from the subordinate based on the verification result, requesting data from SDS, receiving and processing data from SDS, and finally storing the processed data in CDS-DB.

[0053] Subordinate Data Synchronization Service Database (SDS-DB): stores data related to the subordinate data synchronization service (SDS). In the process, according to the query request of SDS, it returns the data information that needs to be shared with the superior, and assists SDS to complete the data sharing operation.

[0054] Subordinate Data Synchronization Service (SDS): Receives data acquisition requests from the superior data synchronization service (CDS), queries its own database (SDS-DB) to determine the data that needs to be shared, responds to the CDS with these data, and assists in completing the synchronization process of data from the subordinate to the superior. This data initialization sequence diagram ensures that data can be accurately and legally synchronized from the subordinate to the superior through the orderly interaction between these roles, and is processed and stored accordingly.

[0055] See also Figure 4 As shown, in step S3 of this embodiment, data initialization synchronization includes at least the following steps: S31: After the lower-level data synchronization service SDS is started, it will first call the service online interface of the upper-level data synchronization service CDS. The request header will carry the username and password to perform authentication verification on the upper and lower platforms. If the verification passes, the status of the upper and lower services will be changed to connecting. S32: The service online interface request parameter for requesting the upper data synchronization service CDS will carry the data items that need to be synchronized, including one or more of the organization, device, channel and preset point information that need to be synchronized. After receiving the data, the upper data synchronization service CDS will first save the data items that need to be synchronized; and the synchronization data also has a sequence, that is, the organization information is synchronized first, and then the device and channel information are synchronized according to the organization; S33: After obtaining the organization information that needs to be synchronized, the organization data needs to be processed. The existing organization codes or organization names on the platform are considered to belong to the same organization. These organizations are merged and then saved in the database. The devices, channels, and preset point information under the organization are then synchronized through the organization.

[0056] In this embodiment, the organizational information is processed according to the organizational merging principle, stored in the database after processing, and then relevant data is synchronized. The organizational merging principle includes adding a sibling node number field to the organization table, setting options for whether to merge and the merging method when configuring cascading. If not merged, it is stored in the database according to the coding prefix rule. If merged, the merging rule is determined according to the relationship between the mount point and the pushed organization, as well as the length of the organization code and name.

[0057] As an implementation method, the organizational merging principle in this embodiment includes: a. Add a field in the organization table, that is, the number bro_idx of the sibling node. Nodes with bro_idx being null can be displayed, while nodes with bro_idx not being null cannot be displayed; b. When configuring cascading, a configuration item should be included to indicate whether to merge the same organization; c. If not merged, a mount point must be selected, and all organizations are stored in the database according to the coding prefix rule; d. If merged, select a mount point. If the mount point is at the same level as the highest-level organization being pushed, merge; if the mount point is not at the same level as the highest-level organization being pushed, do not consider merging, which is equivalent to not merging all; e. The merging rule is: organizations with less than six digits are merged according to the code; organizations with more than six digits are merged according to the name; if the parent organization does not meet the merging rule, its child organizations are not merged.

[0058] As Figure 5 shown, it is a schematic diagram of the root organization root. In this embodiment, an example of organizational merging is described as follows: a. Store according to the original superior-subordinate relationship of the subordinates. If a certain subordinate organization @A needs to be merged with a certain organization A at the本级 level, mark the sibling node of @A as A, and @A is marked as not displayable, such as Figure 5 @Zhejiang, @Hangzhou, @Binjiang, @Huaye, etc. in the figure.

[0059] b. If there is no corresponding organization at the本级 level for a certain subordinate organization @A, mark the sibling node of @A as null, and @A is marked as displayable, such as all the organizations under @Xiaoshan, @Beijing in the above figure.

[0060] c. For an organization like @Xiaoshan, if there is no such organization at the本级 level at first and it is added by the subordinates first, then when the本级 level needs to add Xiaoshan later, at the same time, modify @Xiaoshan to be not displayable, and the sibling organization is Xiaoshan.

[0061] d. Similarly for c, if there is no Xiaoshan at the本级 level at this time, and two or more subordinates have pushed @Xiaoshan, although the prefixes of these @Xiaoshan are different, they should also be merged.

[0062] As an implementation method, Figure 6 The following is a data change sequence diagram of this embodiment, which describes how to ensure data synchronization between the upper and lower platforms through a series of steps after the user operates the lower-level platform to modify the data. The following is a detailed explanation of the logic in the diagram: 1. Overall process When a user operates the lower-level platform to modify data, a table change notification will be sent to Kafka. This is the starting point of the entire data change synchronization process.

[0063] The lower-level data synchronization service SDS monitors the Kafka of the lower-level platform. Once a data change (Kafka information) is detected, SDS will call the data change interface of the upper-level data synchronization service CDS and carry the changed data type and data uid identifier. This step corresponds to S41 in the implementation step.

[0064] After receiving the changed data, the upper-level data synchronization service CDS records the changed type and data uid identifier, and writes this information to Redis for caching. Then, CDS returns the response result to the lower-level data synchronization service SDS. This step corresponds to S42 in the implementation step.

[0065] The upper-level data synchronization service CDS consumes and reads the cached data information from Redis. Then, according to the changed data type and data uid identifier, the corresponding data acquisition interface is called. This step corresponds to S43 in the implementation step.

[0066] After CDS obtains the data, it processes the data and stores the processed data in the database of the upper-level data synchronization service (CDS-DB). This step corresponds to S44 in the implementation step.

[0067] Finally, CDS cleans up the changed data cache that has just been consumed to ensure the consistency and accuracy of the cached data. This step corresponds to S45 in the implementation steps.

[0068] Second, the functions of each module: CDS-DB: stores data related to CDS. In the process, it serves as the final data storage location and receives processed and synchronized data.

[0069] Upper-level Data Synchronization Service (CDS): Receives data change requests from Lower-level Data Synchronization Service (SDS), records change information to Redis cache, reads cached data and obtains corresponding data, processes data and stores it in CDS-DB, and finally clears the cache.

[0070] Subordinate Data Synchronization Service (SDS): monitors the Kafka messages of the subordinate platform. Once data changes are found, it calls the data change interface of the superior data synchronization service (CDS), passes the changed data type and uid identifier, and receives the response from CDS.

[0071] The Kafka (SDS-Kafka) that monitors the lower-level platform is used as the message source for data changes. When users operate the lower-level platform to modify data, table change notifications are generated for the lower-level data synchronization service (SDS) to monitor and obtain. Through these steps and the interaction between roles, when the data on the lower-level platform changes, the changed data is synchronized to the upper-level platform in real time to ensure the consistency of the upper and lower-level data.

[0072] In step S4, when the user modifies the data on the lower-level platform, a table change notification will be sent to Kafka, and the lower-level data synchronization service SDS will synchronize the consumption to the upper-level platform in real time to ensure the consistency of the upper and lower-level data; The following are the steps to synchronize data changes: S41: The lower-level data synchronization service SDS monitors the Kafka of the lower-level platform. If there is a data change, it consumes the Kafka information to call the data change interface of the upper-level data synchronization service CDS, and carries the changed data type and data uid identifier; S42: The upper-level data synchronization service CDS receives the changed data record change type and data uid identifier, writes it to Redis for caching, and returns a response result to the lower-level data synchronization service SDS; S43: The upper-level data synchronization service CDS consumes and reads the cached data information, and calls the corresponding data acquisition interface by changing the data type and data uid identifier; S44: after acquiring the data, the acquired data is processed and then stored in the database; S45: Clear the changed data cache just consumed.

[0073] As an implementation method, Figure 7 As shown, it is a timing diagram of data keeping alive in this embodiment. This describes the process of ensuring the real-time status of services through the heartbeat mechanism between upper and lower data synchronization services. The following is a detailed explanation of the logic in the diagram, combined with the contents of the briefing book: 1. Overall process Sending heartbeat keep-alive request (lower-level SDS): The lower-level data synchronization service SDS sends a heartbeat keep-alive request to the upper-level data synchronization service CDS every 10 seconds. This is the starting point of the entire keep-alive mechanism, corresponding to step s51 in the briefing document.

[0074] Heartbeat processing and caching (upper CDS): After receiving the heartbeat keep-alive request, the upper data synchronization service CDS caches a heartbeat key in Redis and sets the expiration time to 40 seconds. Then, CDS returns the renewal result to the lower SDS service. This step corresponds to step s52 in the briefing document.

[0075] Failure retry mechanism (upper CDS): If the heartbeat keepalive interface of the upper CDS service returns a failure result, the lower SDS service will perform a retry mechanism. If the renewal fails after 3 retries, it means that the upper service is disconnected. At this time, the lower SDS service will change the link status of the cascade to running. If the next renewal is successful, the status will be changed to connecting. This corresponds to step s53 in the briefing book.

[0076] Heartbeat scanning and status judgment (upper CDS): The upper data synchronization service CDS will scan the heartbeat key in Redis every 30 seconds to see if it exists. If it does not exist, the lower-level service is considered disconnected and the link status of the cascade is changed to running; if the heartbeat key exists next time, the status is changed to connecting. This corresponds to step s54 in the handover document.

[0077] 2. Functions of each module Database of the superior data synchronization service (CDS-DB): Function: Store data related to the superior data synchronization service (CDS).

[0078] Upper-level Data Synchronization Service (CDS): Function: Receive heartbeat keep-alive requests from lower-level Data Synchronization Service (SDS), cache heartbeat keys in Redis and set expiration time, return renewal results to lower-level services, handle retry mechanisms for heartbeat keep-alive failures, and regularly scan heartbeat keys in Redis to determine the status of lower-level services.

[0079] Subordinate Data Synchronization Service (SDS): Function: Periodically (every 10 seconds) send heartbeat keep-alive requests to the superior data synchronization service (CDS), process the retry mechanism according to the results returned by the superior, and modify its own service status after receiving the judgment of loss of connection or reconnection from the superior.

[0080] Subordinate Data Synchronization Service Database (SDS-DB): Function: Stores data related to the subordinate data synchronization service (SDS). In the keep-alive mechanism, it may store relevant data such as service status, which is used for recording and subsequent processing when the service status changes.

[0081] This service heartbeat keep-alive mechanism ensures that the upper and lower data synchronization services can monitor each other's service status in real time, promptly detect and handle service disconnections, and ensure the stability and reliability of the data synchronization service.

[0082] As a preferred implementation, in step S5 of this embodiment, in order to ensure the real-time performance of the upper and lower cascade service status, the service status can be updated in real time when the service is abnormal. For this purpose, a service heartbeat keep-alive mechanism is designed. The following are the detailed steps of the service heartbeat keep-alive: S51: The lower-level data synchronization service SDS sends a heartbeat keep-alive request to the upper-level data synchronization service CDS at a predetermined interval; S52: After receiving the message, the upper-level data synchronization service CDS caches a heartbeat key in Redis, sets an expiration time, and then returns the renewal result to the lower-level data synchronization service SDS; S53: If the result returned by requesting the heartbeat keep-alive interface of the upper-level data synchronization service CDS is a failure, a retry mechanism is performed. If the renewal fails after the predetermined number of retries, it indicates that the upper-level service is disconnected. The link status of the cascade is changed to running. If the next renewal is successful, the status is changed to connecting. S54: The upper-level data synchronization service CDS will scan the Redis heartbeat key every predetermined time to see if it exists. If not, it is considered that the lower-level service is disconnected, and the link status of the cascade is changed to running. If the heartbeat key exists next time, the status is changed to connecting.

[0083] As a preferred implementation, the working mode of the service multi-instance deployment described in this embodiment is: each service corresponds to a lower / upper node, and supports multi-instance deployment. The service name remains consistent, and is used to process only the data corresponding to its own cascade; When there are multiple instances of the upper-level data synchronization service CDS, data is obtained from the Redis queue and processed by type in a preemptive manner; When there are multiple instances of the lower-level data synchronization service SDS, only one service will send a service launch request to the corresponding upper-level service. If the upper-level data synchronization service CDS calls the data acquisition interface, it will be randomly distributed to different instances for execution.

[0084] A second aspect of this embodiment provides a cascade-based multi-platform data synchronization system, which is used to implement the above method. The multi-platform data synchronization system includes: The interface security module is used to use a unified POST request method combined with the JSON data format for cross-platform data exchange, and integrates public request header parameters for identity authentication and security verification in each request; Data synchronization service module, including: The upper-level data synchronization service CDS is deployed on the upper-level platform and is used to call the lower-level data synchronization service to obtain the data information to be shared with the upper-level platform and save it to the local database; The lower-level data synchronization service SDS is deployed on the lower-level platform to provide an interface for the upper-level data synchronization service to call and obtain data information and notify the upper-level platform to update the data when the lower-level data changes, so as to achieve the synchronization of resource data such as organizations, equipment, and channels, and support multi-level cascade data synchronization or cascading between two platforms; The data initialization synchronization module is used to call the service online interface of the upper-level data synchronization service CDS with the parameters for identity authentication to perform authentication verification on the upper and lower platforms after the lower-level data synchronization service SDS is started. If the verification is passed, the service status of both parties is changed to being linked, and the lower-level data synchronization service SDS carries the data items to be synchronized when requesting the service online interface of the upper-level data synchronization service CDS. The upper-level data synchronization service CDS saves these data items and synchronizes them in the order of organization first, then according to the organization synchronization device and channel information. At the same time, the obtained organization information is processed according to the organization merger principle and then stored in the warehouse, and then the lower-level device, channel and preset point information is synchronized through the organization; The data change real-time synchronization module is used to enable the subordinate data synchronization service SDS to monitor the data change information of the subordinate platform when the data of the subordinate platform changes, and call the data change interface of the superior data synchronization service CDS, carrying the changed data type and data identifier. After the superior data synchronization service CDS receives the changed data, it records the change type and data identifier and caches it, and returns the response result to the subordinate data synchronization service SDS. The superior data synchronization service CDS reads the cached data information, calls the corresponding data acquisition interface according to the changed data type and data identifier, processes the acquired data and puts it into the warehouse, and cleans up the changed data cache; The service heartbeat keep-alive module is used to enable the subordinate data synchronization service SDS to periodically send heartbeat keep-alive requests to the superior data synchronization service CDS. After receiving the request, the superior data synchronization service CDS sets the heartbeat-related key value in the cache, sets the expiration time, and returns the renewal result to the subordinate data synchronization service SDS. If the subordinate data synchronization service SDS fails to renew, it will retry. If the retry fails multiple times, it is considered that the superior service is disconnected and the cascade link status is modified. If the subsequent renewal is successful, it is modified back to the original status. The superior data synchronization service CDS periodically scans the heartbeat-related key value in the cache. If it does not exist, it is considered that the subordinate service is disconnected and the cascade link status is modified. If it exists, it is modified back to the original status. The service multi-instance deployment module is used to make each service correspond to a subordinate / superior node, support multi-instance deployment, keep the service name consistent, obtain data from the cache queue and process it by type when the superior data synchronization service CDS has multiple instances, and only one service sends a service online request to the superior when the subordinate data synchronization service SDS has multiple instances. When the superior data synchronization service CDS calls the data acquisition interface, it is randomly allocated to different instances for execution.

[0085] Although the embodiments of the present invention have been shown and described above, it will be appreciated that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, substitute and vary the above embodiments within the scope of the present invention.

Claims

1. A multi-platform data synchronization method based on cascade, characterized in that: The steps include: Step S1: Design an interface security mechanism for cross-platform data exchange; Step S2: Building a data synchronization service architecture, including deploying corresponding data synchronization services on different-level platforms to achieve resource data synchronization, and supporting multi-level cascading or platform interconnected data synchronization; Step S3: Perform data initialization synchronization; Step S4: Realize real-time synchronization of data changes; Step S5: Establish a service heartbeat keep-alive mechanism; Step S6: Perform service multi-instance deployment.

2. The method according to claim 1, characterized in that: In step S1, the interface security mechanism adopts a unified POST request method combined with the JSON data format for cross-platform data exchange, and integrates interface request header parameters for identity authentication and security verification and corresponding response parameters in the request, wherein the interface request header parameters include one or more of the platform node code, user name, password, and random code; the response parameters include one or more of the response code, response description, and response data.

3. The method according to claim 2, characterized in that: In step S2, data synchronization in the data synchronization service architecture is divided into two services including CDS and SDS, wherein CDS is the upper-level data synchronization service and SDS is the lower-level data synchronization service, respectively deploying data synchronization services in the upper-level platform and the lower-level platform to achieve synchronization of organization, equipment, and channel resource data; the data synchronization service architecture supports multi-level cascade data synchronization or two platforms cascaded with each other; The upper-level data synchronization service CDS calls the lower-level data synchronization service SDS to obtain the data information that needs to be shared with the upper-level platform, and then the upper-level platform saves this data information into the local database for storage; The function of the subordinate data synchronization service SDS is to provide an interface for the superior data synchronization service CDS to call and obtain the data information shared to the superior platform; if the subordinate data is changed, the superior data synchronization service CDS interface is also called to notify the superior platform to update the data to ensure the consistency of the superior and subordinate data.

4. The method according to claim 3, characterized in that: In step S3, data initialization synchronization includes at least the following steps: S31: After the lower-level data synchronization service SDS is started, it will first call the service online interface of the upper-level data synchronization service CDS. The request header will carry the username and password to perform authentication verification on the upper and lower platforms. If the verification passes, the status of the upper and lower services will be changed to connecting. S32: The service online interface request parameter for requesting the upper data synchronization service CDS will carry the data items that need to be synchronized, including one or more of the organization, device, channel and preset point information that need to be synchronized. After receiving the data, the upper data synchronization service CDS will first save the data items that need to be synchronized; and the synchronization data also has a sequence, that is, the organization information is synchronized first, and then the device and channel information are synchronized according to the organization; S33: After obtaining the organization information that needs to be synchronized, the organization data needs to be processed. The existing organization codes or organization names on the platform are considered to belong to the same organization. These organizations are merged and then saved in the database. The devices, channels, and preset point information under the organization are then synchronized through the organization.

5. The method according to claim 4, characterized in that: The organization information is processed according to the organization merger principle, stored in the database, and then synchronized with related data. The organization merger principle includes adding a brother node number field to the organization table, setting whether to merge and the merge method options when configuring cascade, if not merged, it is stored according to the coding prefix rule, and if merged, the merge rule is determined based on the relationship between the mount point and the push organization, the organization code, and the name length.

6. The method according to claim 5, characterized in that: The principles for organizational merger include: a. Add a new field in the organization table, namely the brother node number bro_idx. Nodes whose bro_idx is null can be displayed, while nodes whose bro_idx is not null cannot be displayed. b. When configuring cascading, a configuration item should be included to indicate whether to merge the same organization; c. If not merged, a mount point must be selected and all organizations are stored according to the rules of the encoding prefix; d. If merging, select a mount point. If the mount point is at the same level as the highest level organization pushed, merge it; if the mount point is not at the same level as the highest level organization pushed, do not consider merging, which is equivalent to not merging at all; e. The merging rules are as follows: organizations with a number less than six digits are merged by code; organizations with a number greater than six digits are merged by name; if the parent organization does not meet the merging rules, its child organizations will not be merged.

7. The method according to claim 1, characterized in that: In step S4, when the user modifies the data on the lower-level platform, a table change notification will be sent to Kafka, and the lower-level data synchronization service SDS will synchronize the consumption to the upper-level platform in real time to ensure the consistency of the upper and lower-level data; The following are the steps to synchronize data changes: S41: The lower-level data synchronization service SDS monitors the Kafka of the lower-level platform. If there is a data change, it consumes the Kafka information to call the data change interface of the upper-level data synchronization service CDS, and carries the changed data type and data uid identifier; S42； The upper-level data synchronization service CDS receives the changed data, records the changed type and data uid identifier, writes it into Redis for caching, and returns the response result to the lower-level data synchronization service SDS; S43: The upper-level data synchronization service CDS consumes and reads the cached data information, and calls the corresponding data acquisition interface by changing the data type and data uid identifier; S44: after acquiring the data, the acquired data is processed and then stored in the database; S45: Clear the changed data cache just consumed.

8. The method according to any one of claims 1 to 7, characterized in that: In step S5, In order to ensure the real-time status of the upper and lower cascade services, the service status can be updated in real time when the service is abnormal. For this purpose, a service heartbeat keep-alive mechanism is designed. The following are the detailed steps for service heartbeat keep-alive: S51: The lower-level data synchronization service SDS sends a heartbeat keep-alive request to the upper-level data synchronization service CDS at a predetermined interval; S52: After receiving the message, the upper-level data synchronization service CDS caches a heartbeat key in Redis, sets an expiration time, and then returns the renewal result to the lower-level data synchronization service SDS; S53: If the result returned by requesting the heartbeat keep-alive interface of the upper-level data synchronization service CDS is a failure, a retry mechanism is performed. If the renewal fails after the predetermined number of retries, it indicates that the upper-level service is disconnected. The link status of the cascade is changed to running. If the next renewal is successful, the status is changed to connecting. S54: The upper-level data synchronization service CDS will scan the Redis heartbeat key every predetermined time to see if it exists. If not, it is considered that the lower-level service is disconnected, and the link status of the cascade is changed to running. If the heartbeat key exists next time, the status is changed to connecting.

9. The method according to claim 8, characterized in that: The working mode of the service multi-instance deployment is: each service corresponds to a lower / upper node and supports multi-instance deployment; the name of the service remains consistent and is used to process only the data corresponding to its own cascade; When there are multiple instances of the upper-level data synchronization service CDS, data is obtained from the Redis queue and processed by type in a preemptive manner; When there are multiple instances of the lower-level data synchronization service SDS, only one service will send a service launch request to the corresponding upper-level service. If the upper-level data synchronization service CDS calls the data acquisition interface, it will be randomly distributed to different instances for execution.

10. A multi-platform data synchronization system based on cascade, characterized in that: The system is used to implement the method described in any one of claims 1 to 9 above, and the multi-platform data synchronization system includes: The interface security module is used to use a unified POST request method combined with the JSON data format for cross-platform data exchange, and integrates public request header parameters for identity authentication and security verification in each request; Data synchronization service module, including: The upper-level data synchronization service CDS is deployed on the upper-level platform and is used to call the lower-level data synchronization service to obtain the data information to be shared with the upper-level platform and save it to the local database; The lower-level data synchronization service SDS is deployed on the lower-level platform to provide an interface for the upper-level data synchronization service to call and obtain data information and notify the upper-level platform to update the data when the lower-level data changes, so as to achieve the synchronization of organization, equipment, and channel resource data, and support multi-level cascade data synchronization or two platforms cascading each other; The data initialization synchronization module is used to call the service online interface of the upper-level data synchronization service CDS with the parameters for identity authentication to perform authentication verification on the upper and lower platforms after the lower-level data synchronization service SDS is started. If the verification is passed, the service status of both parties is changed to being linked, and the lower-level data synchronization service SDS carries the data items to be synchronized when requesting the service online interface of the upper-level data synchronization service CDS. The upper-level data synchronization service CDS saves these data items and synchronizes them in the order of organization first, then according to the organization synchronization device and channel information. At the same time, the obtained organization information is processed according to the organization merger principle and then stored in the warehouse, and then the lower-level device, channel and preset point information is synchronized through the organization; The data change real-time synchronization module is used to enable the subordinate data synchronization service SDS to monitor the data change information of the subordinate platform when the data of the subordinate platform changes, and call the data change interface of the superior data synchronization service CDS, carrying the changed data type and data identifier. After the superior data synchronization service CDS receives the changed data, it records the change type and data identifier and caches it, and returns the response result to the subordinate data synchronization service SDS. The superior data synchronization service CDS reads the cached data information, calls the corresponding data acquisition interface according to the changed data type and data identifier, processes the acquired data and puts it into the warehouse, and cleans up the changed data cache; The service heartbeat keep-alive module is used to enable the subordinate data synchronization service SDS to periodically send heartbeat keep-alive requests to the superior data synchronization service CDS. After receiving the request, the superior data synchronization service CDS sets the heartbeat-related key value in the cache, sets the expiration time, and returns the renewal result to the subordinate data synchronization service SDS. If the subordinate data synchronization service SDS fails to renew, it will retry. If the retry fails multiple times, it is considered that the superior service is disconnected and the cascade link status is modified. If the subsequent renewal is successful, it is modified back to the original status. The superior data synchronization service CDS periodically scans the heartbeat-related key value in the cache. If it does not exist, it is considered that the subordinate service is disconnected and the cascade link status is modified. If it exists, it is modified back to the original status. The service multi-instance deployment module is used to make each service correspond to a subordinate / superior node, support multi-instance deployment, keep the service name consistent, obtain data from the cache queue and process it by type when the superior data synchronization service CDS has multiple instances, and only one service sends a service online request to the superior when the subordinate data synchronization service SDS has multiple instances. When the superior data synchronization service CDS calls the data acquisition interface, it is randomly allocated to different instances for execution.

Images