Cross-system data synchronization method and system and storage medium

Abstract

The invention discloses a cross-system data synchronization method and system and a storage medium, and relates to the technical field of computers.The method comprises the steps that at least one set of to-be-synchronized data sent by a first system is obtained, and the to-be-synchronized data is added to the tail of a message queue; acquiring a group of to-be-synchronized data from the head of the message queue; based on a target conversion mode corresponding to the to-be-synchronized data, data format conversion is carried out on the to-be-synchronized data to obtain synchronous conversion data, and the synchronous conversion data is matched with a second system format; and writing the synchronous conversion data into a second system to complete data synchronization of the second system and the first system. By adopting the scheme provided by the embodiment of the invention, data synchronization among different systems is facilitated, and the synchronized data conforms to the data format of the corresponding system.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a cross-system data synchronization method, system and storage medium. Background Art

[0002] Nowadays, there are often digital systems in enterprises. The digital systems used by different companies or even different departments of the same company are not consistent, which leads to the decentralization and fragmentation of enterprise data and the formation of data islands. In related technologies, when enterprises use different digital systems, they usually use manual data processing, export and import, database import, and interface synchronization to achieve data synchronization between different systems.

[0003] However, the manual processing method is inefficient. Although manual import and export reduces the workload of manual creation and update of data, the efficiency of manual editing of intermediate data has not been greatly improved, and it cannot adapt to the scenario of large-scale data update. For the database import method, SQL (Structured Query Language) processing also requires a lot of work and is inefficient. Although the interface synchronization method has a faster processing speed, it needs to be customized according to the synchronization system, and corresponding modifications are required when the business changes, which is costly. Therefore, an automated, low-cost cross-system data synchronization solution is urgently needed to complete efficient and high-quality cross-system data synchronization. Summary of the invention

[0004] The purpose of the present invention is to provide a cross-system data synchronization method, system and storage medium, aiming to solve the problems of low efficiency and lack of flexibility of cross-system data synchronization in related technologies and to achieve automated cross-system data synchronization.

[0005] To achieve the above object, on the one hand, the present application provides a cross-system data synchronization method, the method comprising: obtaining at least one set of to-be-synchronized data sent by a first system, and adding the to-be-synchronized data to the end of a message queue; Get a set of data to be synchronized from the head of the message queue; Based on the target conversion mode corresponding to the data to be synchronized, the data to be synchronized is converted into a data format to obtain synchronization conversion data, where the synchronization conversion data matches the second system format; The synchronous conversion data is written into the second system to complete data synchronization between the second system and the first system.

[0006] As a further improvement of the present application, the data format conversion of the data to be synchronized is performed based on the target conversion mode corresponding to the data to be synchronized to obtain the synchronous conversion data, including: Determining a target conversion mode based on the data type corresponding to the to-be-synchronized data, the target conversion mode being used to indicate a target data format of the synchronously-converted data; Based on the target conversion mode, the data to be synchronized is rendered into the target data format to obtain the synchronous conversion data.

[0007] As a further improvement of the present application, the method further includes: In the case where it is detected that the data to be synchronized belongs to the drop-down box data, real-time dense mapping is adopted to map the data to be synchronized into the synchronous conversion data, and the synchronous conversion data is the drop-down box data of the second system.

[0008] As a further improvement of the present application, the method further includes: Based on the business form of the second system, integrity check is performed on the to-be-synchronized data corresponding to each business, and the integrity check at least includes mandatory field verification, field range verification, and event logic verification.

[0009] As a further improvement of the present application, based on the business form of the second system, the required fields of the data to be synchronized corresponding to each business are verified, including: Traversing the business form to determine the required fields corresponding to each business in the second system; Perform non-empty check on the required fields corresponding to each business; Based on the business form of the second system, the field range check is performed on the data to be synchronized corresponding to each business, including: In the case that a range field corresponding to the first business exists as the required field, determining a range limit of the field corresponding to the first business, wherein the range limit is used to specify an interval for filling in data in the range field; Based on the range restriction, the field range check is performed on the range field corresponding to the first service.

[0010] As a further improvement of the present application, the business form based on the second system performs event logic verification on the data to be synchronized corresponding to each business, including: Based on the trigger event type corresponding to each business, determine the trigger event that triggers the required field verification and field range check, and the trigger event type at least includes creating new business data and updating business data; The event logic check is performed based on the event type to which the service corresponding to the data to be synchronized belongs.

[0011] As a further improvement of the present application, when the integrity check fails, the method further includes: Field data that fails validation is marked as an error.

[0012] As a further improvement of the present application, the method further includes: In the case where writing the synchronous conversion data into the second system fails, a retry mechanism is triggered, where the retry mechanism is used to instruct to reacquire the to-be-synchronized data from the first system.

[0013] On the other hand, the present application provides a cross-system data synchronization system, the system comprising: The message queue module is used to obtain at least one group of data to be synchronized sent by the first system, and add the data to be synchronized to the end of the message queue; obtain a group of data to be synchronized from the head of the message queue; A data conversion module, configured to perform data format conversion on the data to be synchronized based on a target conversion mode corresponding to the data to be synchronized, to obtain synchronization conversion data, wherein the synchronization conversion data matches a second system format; The data writing module is used to write the synchronous conversion data into the second system to complete the data synchronization between the second system and the first system.

[0014] As a further improvement of the present application, the system further includes: A template rendering module, used to determine a target conversion mode based on a data type corresponding to the data to be synchronized, wherein the target conversion mode is used to indicate a target data format of the synchronized conversion data; Based on the target conversion mode, the data to be synchronized is rendered into the target data format to obtain the synchronization conversion data; The integrity verification module is used to perform integrity verification on the to-be-synchronized data corresponding to each business based on the business form of the second system; The exception handling module is used to capture exceptions during the synchronization process and trigger the retry mechanism.

[0015] On the other hand, the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the cross-system data synchronization method as described in any of the above aspects.

[0016] Compared with the related art, the beneficial effects of the present invention are: In the embodiment of the present application, during the process of data synchronization between multiple systems, the data to be synchronized sent by the first system is added to the message queue, and the data to be synchronized is obtained from the message queue header during data processing, and the data format is converted so that the converted data matches the format of the second system, thereby realizing cross-system data synchronization. In this process, by introducing a message queue and temporarily storing the data to be synchronized obtained from the first system, the decoupling of the first system and the second system can be realized, and the synchronization request can be processed asynchronously to avoid mutual blocking and waiting between systems, which significantly improves the data synchronization efficiency in the scenario of concurrent requests to be synchronized and shortens the data processing time. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram showing an implementation environment provided by an exemplary embodiment of the present application is shown; Figure 2 A schematic diagram showing description information of customers in a CRM system and a financial system; Figure 3 A flowchart of a cross-system data synchronization method provided by an exemplary embodiment of the present application is shown; Figure 4 A flowchart of a data format conversion process provided by an exemplary embodiment of the present application is shown; Figure 5 A flowchart of an integrity verification process provided by an exemplary embodiment of the present application is shown; Figure 6 A flowchart of a cross-system data synchronization process provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0019] It should be noted that the term "comprises" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0020] The nouns in this embodiment are explained below: Customer Relationship Management (CRM) refers to the process by which an enterprise uses information technology and Internet technology to coordinate the interactions between the enterprise and customers in sales, marketing and services, thereby conducting systematic management and providing customers with innovative and personalized customer interactions and services.

[0021] Data silos refer to the lack of relevance of data in various systems and incompatibility of database batches. Data silos are usually divided into physical and logical types. Physical data silos refer to the fact that data is stored and maintained independently in different departments and isolated from each other, forming physical silos. Logical data silos refer to the fact that different departments understand and define data from their own perspectives, giving different meanings to the same data.

[0022] Decoupling refers to reducing the dependencies between different components in system design so that each part can be developed, tested, and maintained independently. This design approach improves the flexibility and maintainability of the system because changes in one component will not have a direct impact on other components. By using a practical middle layer or message transmission mechanism, such as a message queue, loose coupling between components can be achieved, thereby enhancing the overall stability of the system.

[0023] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application, including a first system 110, a second system 120 and a cross-system data synchronization system (hereinafter referred to as data synchronization system) 130.

[0024] There is a need to synchronize data between the first system 110 and the second system 120. For example, within an enterprise, the sales department uses a CRM system, while the finance department uses a finance system. Data synchronization is required between the CRM system and the finance system, and the types of data definitions for the same data may be different between the first system 110 and the second system 120.

[0025] Please refer to Figure 2, which shows a schematic diagram of the description information of customers in the CRM system and the financial system. In the financial system, the customer description information includes the customer name (Acct Name, 168 characters), the customer address (Acct_Address, 1024 characters), the customer type (acct_type) and the settlement type (payment_method). Among them, the customer type includes "0: ordinary customer, 1: key customer". The settlement type includes "ordinary customer: payment first and then delivery, key customer: delivery first and then payment". In the CRM system, the fields and their descriptions are "name (Account_Name, 200 characters), address (province, 32 characters, encoding), address (city, 32 characters, encoding), address (address, 512 characters), customer type (type, encoding), customer type normal: ordinary customer, KA: key customer". It can be seen that for the CRM system and the financial system, the description method (that is, data format) corresponding to the same information is not the same.

[0026] Therefore, synchronizing data from one system directly to another may result in conflicts, such as customer name length mismatch, customer address format mismatch, customer type mismatch, missing billing type data, and so on.

[0027] In the process of cross-system data synchronization using the solution provided by the embodiment of the present application, when the first system 110 and the second system 120 need to synchronize data, the second system 120 sends a data synchronization request to the first system 110, and the first system 110 sends the data to be synchronized after confirming the data synchronization request. The data synchronization system 130 first fills the data to be synchronized sent by the first system 110 into the tail of the message queue, and when the processing of the preceding data is completed, extracts the previously filled data to be synchronized from the tail of the message queue, and converts the data format according to the corresponding target conversion method, converts it into synchronization conversion data that matches the format of the second system 120, and then writes it into the second system 120, thereby realizing data synchronization between the two systems.

[0028] It should be noted that in the embodiment of the present application, the cross-system data synchronization method is executed by the cross-system data synchronization system 130. For ease of description, the following embodiment is described with the data synchronization system 130 as the execution subject.

[0029] Please refer to Figure 3 , which shows a flowchart of a cross-system data synchronization method provided by an illustrative embodiment of the present application, the method includes the following steps.

[0030] Step 301: obtain at least one set of data to be synchronized sent by the first system, and add the data to be synchronized to the end of the message queue.

[0031] Among them, Message Queue (MQ) refers to a mechanism for passing messages between applications or services. It allows asynchronous communication in different systems and improves the scalability and reliability of the system. By putting messages into a message queue, the sender and receiver do not have to run at the same time, thus achieving a decoupled and loosely coupled design. Message queues are often used to process large amounts of data, handle high-concurrency requests, and implement task scheduling in distributed systems.

[0032] Optionally, the first system 110 is a source system, and the second system 120 is a target system, assuming that data of the source system needs to be synchronized to the target system.

[0033] In an embodiment of the present application, the introduction of a message queue can decouple the source system and the target system. The source system pushes the data to be synchronized to the queue in the form of messages, and the data synchronization system can independently take out the messages from the message queue for processing, thereby reducing the dependency between systems and improving the flexibility of data synchronization.

[0034] Furthermore, the first system pushes the data to be synchronized to the message queue. Even if the data synchronization system fails or the data synchronization system writes data to the second system slowly, the message queue will temporarily store the unprocessed data to be synchronized, which is helpful to ensure that the data to be synchronized is not lost and improve the reliability of the data synchronization system. Furthermore, by temporarily storing the data to be synchronized through the message queue, it can be used as a buffer in scenarios with large data traffic to avoid system overload.

[0035] Step 302: Obtain a group of data to be synchronized from the head of the message queue.

[0036] The first system pushes the data to be synchronized to the end of the message queue, and the data synchronization system obtains the data to be synchronized from the head of the message queue, processes the data, and writes it into the second system, thus realizing asynchronous processing of the first system and the second system and improving the response speed of the system.

[0037] In an embodiment of the present application, the data synchronization system acts as a consumer of the message queue, continuously monitors the message queue, and obtains the synchronization data to be processed from the head of the message queue. The obtained synchronization data is used for subsequent data format conversion and writing operations.

[0038] Step 303 : Based on the target conversion mode corresponding to the data to be synchronized, the data format is converted to obtain synchronous conversion data.

[0039] The synchronous conversion data matches the second system format.

[0040] Since the first system and the second system may use different data formats, data format conversion is required to ensure that the data can be smoothly written into the second system and correctly identified and used.

[0041] Optionally, the target conversion method is determined based on the data type corresponding to the to-be-synchronized data itself or a preset conversion rule.

[0042] During the data format conversion process, the data synchronization system converts the data format of the first system into a data format recognizable by the second system according to the requirements of the target conversion method, for example, converting json format data into xml format, or converting sql data into json data of a specific structure.

[0043] Step 304, writing the synchronous conversion data into the second system to complete data synchronization between the second system and the first system.

[0044] After the data synchronization system converts the data to be synchronized into synchronous conversion data, the synchronous conversion data is written into the second system in the writing method required by the second system, such as calling an API (Application Programming Interface), executing a SQL statement, or directly writing into a database.

[0045] After the synchronously converted data is written into the second system, the data synchronization from the first system to the second system is completed. Subsequently, the data synchronization system reads the data to be synchronized from the message queue header and performs data format conversion.

[0046] To summarize, in the embodiment of the present application, in the process of data synchronization between multiple systems, the data to be synchronized sent by the first system is added to the message queue, and the data to be synchronized is obtained from the message queue header during data processing, and the data format is converted so that the converted data matches the format of the second system, thereby realizing cross-system data synchronization. In this process, by introducing a message queue and temporarily storing the data to be synchronized obtained from the first system, the first system and the second system can be decoupled, and the synchronization request can be processed asynchronously to avoid mutual blocking and waiting between systems, significantly improving the data synchronization efficiency in the scenario of concurrent requests to be synchronized, and shortening the data processing time.

[0047] Please refer to Figure 4 , which shows a flowchart of a data format conversion process provided by an exemplary embodiment of the present application, and the process includes the following steps.

[0048] Step 401, determining a target conversion method based on the data type corresponding to the data to be synchronized.

[0049] The target conversion mode is used to indicate the target data format of the synchronous conversion data.

[0050] Data types refer to the structured attributes of the data to be synchronized, such as customer data, order data, and product data. Different data types may require different conversion methods to adapt to the requirements of the target system.

[0051] After determining the data type of the data to be synchronized, the data synchronization system needs to first determine the original data type of the data type in the first system and the corresponding target data format in the second system. After determining the original data format and the target data format to be converted to, the data synchronization system can determine the target conversion method from the original data format to the target data format.

[0052] The target data format refers to the format standard that needs to be achieved after data conversion. In the embodiment of the present application, the target data format includes one or more of the data formats such as json format, xml format, sql format, js format and yaml format.

[0053] Among them, json (JavaScript Object Notation) format is a lightweight data exchange format that is easy for people to read and write, and easy for machines to parse and generate. It stores data in the form of key-value pairs, supports arrays and nested structures, and is widely used in network data transmission and configuration files. It has good cross-platform and cross-language characteristics.

[0054] The xml (eXtensible Markup Language) format focuses on the content and structure description of data, has good scalability and self-description, and is often used for complex documents and data exchange.

[0055] The SQL format is a standard language for managing relational databases. You can directly operate the database through SQL statements to perform operations such as querying, inserting, updating, and deleting data.

[0056] The js format is a source code format of JavaScript. Through the syntax of JavaScript, data can be dynamically manipulated, including creation, modification, deletion, etc.

[0057] The yaml (YAML Ain't Markup Language) format expresses data structures in a concise and easy-to-read way, uses indentation to represent the hierarchical relationship of data, and supports various data types, such as strings, numbers, Boolean values, and objects. In addition, it also allows adding comments to the data, which has strong readability and maintainability.

[0058] In addition to the above formats, in order to ensure the versatility and wide applicability of the data synchronization system, the solution provided in the embodiment of the present application supports multiple target data formats to meet the docking requirements of various types of systems, and is not limited to the above formats. For example, in some specific industry applications, there may be proprietary data formats for storing and transmitting key business data. In this case, multiple optional target data formats are used to match the proprietary data formats, which is conducive to improving the success rate of cross-system data synchronization and meeting the cross-system data synchronization requirements in different application scenarios.

[0059] Step 402: based on the target conversion mode, render the data to be synchronized into the target data format to obtain synchronization conversion data.

[0060] Rendering refers to structuring and formatting the data to be synchronized according to the requirements of the target data format to generate data that meets the specifications of the target data format.

[0061] For example, if the target data format is json, the data to be synchronized is converted into a json object; if the target data format is xml, the data to be synchronized is converted into an xml document, and so on.

[0062] Step 403: When it is detected that the data to be synchronized belongs to the drop-down box data, real-time dense mapping is adopted to map the data to be synchronized into synchronous conversion data.

[0063] The synchronous conversion data is the drop-down box data of the second system.

[0064] In one possible implementation, the data to be synchronized may include drop-down box data, which is usually used to represent some enumeration values ​​or option lists in the system, such as customer type (ordinary customer, key customer), order status (pending payment, shipped, completed), etc.

[0065] Different systems may have different encoding and text descriptions of drop-down box data. For example, the customer type values ​​in the CRM system are "normal" (ordinary customer) and "KA" (key customer), while the customer type values ​​in the financial system are "0" (ordinary customer) and "1" (key customer).

[0066] In order to solve the problem of value domain mismatch when synchronizing drop-down box data across systems, in an embodiment of the present application, when the data synchronization system detects that the data to be synchronized belongs to the drop-down box data, it uses real-time dense mapping to convert the drop-down box data in the first system into the drop-down box data corresponding to the second system (synchronous conversion data).

[0067] Real-time dense mapping refers to establishing a pre-defined mapping relationship for the drop-down box data. During the data synchronization process, the system can search and apply these mapping relationships in real time and densely to convert the drop-down box data values ​​in the first system into the corresponding values ​​in the second system.

[0068] Optionally, CodeMapping is used to implement real-time dense mapping of the drop-down box data. Please refer to Table 1. The process of dense mapping using CodeMapping is described below by way of example.

[0069] Among them, CodeMapping defines the business type (ACCOUNT_TYPE represents the customer type), the first system field (type), the second system field (acct_type), and the specific mapping relationship. For example, in the original code of the first system, "normal" represents ordinary users and "KA" represents key users, and in the target code of the second system, "0" represents ordinary customers and "1" represents key customers. When performing dense mapping, CodeMapping maps "normal" (source code) "ordinary customer" (source text value) in the first system to "0" (target code) "ordinary customer" (target text value) in the target system, and maps "KA" (source code) "key customer" (source text value) in the source system to "1" (target code) "key customer" (target text value) in the target system.

[0070]

[0071] Table 1 In the embodiment of the present application, the data conversion method is dynamically selected according to the data type, and multiple data conversion formats are supported, thereby further improving the flexibility and adaptability of data format conversion, so that the solution provided by the present application can be better compatible with various different systems, solve problems such as customer name length mismatch, customer address format mismatch, and no settlement type in CRM, and improve the efficiency and accuracy of cross-system data synchronization. In addition, by adopting a real-time dense mapping method, it is ensured that the drop-down box data can be correctly identified in the target system.

[0072] In the process of synchronizing the first system and the second system, in order to further ensure the quality and reliability of data synchronization and avoid writing incomplete or erroneous data into the second system, the integrity of the converted synchronous conversion data can be checked and then written into the second system after the check is completed.

[0073] Optionally, based on the business form of the second system, integrity check is performed on the data to be synchronized corresponding to each business, and the integrity check at least includes mandatory field verification, field range verification and event logic verification.

[0074] The business form refers to the form of each business in the second system, including the data of each business, which contains information such as the fields that each business should include, the data type of the field, the constraint conditions of the field, and the business processing logic.

[0075] If the data synchronization system needs to perform integrity verification, the data synchronization system needs to first obtain the business form of the second system, determine the verification rules, and ensure that the data synchronization results match the data standards of the target system.

[0076] Please refer to Figure 5 , which shows a flow chart of an integrity verification process provided by an exemplary embodiment of the present application. The process includes: Step 501: Based on traversing the business form, determine the required fields corresponding to each business in the second system.

[0077] The required fields refer to the fields that are defined as fields that must be filled in the business form of the second system. If the required fields are missing, the data is considered incomplete and cannot be written into the second system normally.

[0078] Mandatory field validation can check whether the data to be synchronized contains all the mandatory fields specified in the business form of the second system. In the second system, the pre-defined business form specifies the fields that need to be validated for mandatory fields in different events for each business type. For example, the business form of the second system stipulates that for the customer data type, when it is executed at the creation time, the "name, address" field is a mandatory field. In the process of creating a customer data type for a certain business in the form, the "name, address" field needs to be validated for mandatory fields.

[0079] Step 502: Check whether the required fields corresponding to each business are not empty.

[0080] When performing mandatory field verification, the system will traverse the business forms of the second system, determine the mandatory fields corresponding to each business in the second system, and then perform a non-empty check on the mandatory fields corresponding to each business to check whether these mandatory fields exist in the data to be synchronized and are not empty.

[0081] When the non-empty check fails, it indicates that the data is incomplete and the required field verification fails.

[0082] Step 503: If the range field corresponding to the first service is a required field, determine the range limit of the field corresponding to the first service.

[0083] The range restriction is used to specify the interval of data to be filled in the range field.

[0084] Range restriction refers to the restriction on the data value range of certain fields in the business form of the second system, for example, the data type of the field, the length limit of the field, the value range of the field, etc.

[0085] Field range check can check whether the field value in the data to be synchronized meets the range limit defined in the second business form. Specifically, the second system will pre-define the range limit of the fields of each business type. For example, for the "User" business type, the age field is limited to the range of "18~199". And the range limit is written into the range field of the business form.

[0086] Step 504: Based on the range restriction, perform a field range check on the range field corresponding to the first business.

[0087] When performing field range check, if the range field corresponding to the first business is a required field, the system will determine the range limit of the field corresponding to the first business. The range limit is used to specify the interval of data filled in the range field. For example, the range limit of the age field under the "User" business type is determined to be 18~199. Then, based on the range limit, the field range check is performed on the range field corresponding to the first business to check whether the value of the corresponding field in the data to be synchronized falls within the predefined range. If the value of the field in the data to be synchronized exceeds the range limit, the field range check fails, indicating that the data does not meet the specifications.

[0088] Step 505: Based on the trigger event types corresponding to the various services, determine the trigger events that trigger the required field verification and field range check.

[0089] The trigger event type refers to a specific business event that can trigger a data validation operation, which indicates a change in data status or the advancement of a business process.

[0090] Optionally, the trigger event type includes at least creating new business data and updating business data. Creating new business data refers to creating a new business data record in the second system, such as adding a new customer, creating an order, etc. Updating business data refers to modifying an existing business data record in the second system, such as modifying customer information, updating order status, etc.

[0091] Optionally, the trigger event type may also include other types of business events, such as data import events, data approval events, data submission events, etc. The present invention does not limit the types of trigger event types and can be flexibly expanded according to actual business scenarios.

[0092] By performing time logic verification, corresponding verification rules can be dynamically selected and executed according to the business event type corresponding to the data to be synchronized. In the second system, there are pre-set trigger event types corresponding to various business types.

[0093] Step 506: Perform event logic verification based on the event type to which the service corresponding to the data to be synchronized belongs.

[0094] The data synchronization system selectively performs corresponding mandatory field verification or field range verification based on the type of business event currently occurring in the data to be synchronized.

[0095] When the business event type to which the data to be synchronized belongs does not match the predefined trigger event type, or when the event type matches but the data still does not comply with the corresponding validation rules, the event logic check fails, indicating that there are logical errors in the data under a specific business event.

[0096] Optionally, if the event type of the data to be synchronized does not belong to the predefined trigger event type, or although it belongs to the trigger event type, according to the event triggering rules, no verification operation is required under the current event type, the system can skip the corresponding verification step.

[0097] In an embodiment of the present application, by performing event logic verification based on the event type to which the business corresponding to the data to be synchronized belongs, dynamic verification based on event logic can be implemented, making data integrity verification more intelligent and refined, and triggering corresponding verification rules only under necessary business events, thereby avoiding indiscriminate full verification of all data, reducing unnecessary resource consumption, improving system performance, and also ensuring the effectiveness and accuracy of data verification.

[0098] Step 507: If the integrity check fails, an error mark is performed on the field data that fails the check.

[0099] That is, after performing verifications such as required field verification, field range verification, and event logic verification, if it is found that the data to be synchronized does not comply with the predefined business rules, and there are problems such as missing data, data errors, or non-standard data formats, the data that fails the integrity check needs to be marked as an error, which is conducive to error tracing and error resolution.

[0100] Optionally, error marking can be implemented by adding an error identification field, adding an error prompt flag, or adding an error information log file, etc., which is not limited in this embodiment. Through error marking, the data synchronization system or user can quickly identify erroneous data and locate the erroneous field, which is convenient for subsequent error troubleshooting, data correction and problem analysis.

[0101] Please refer to Table 2, which shows a business form provided by an exemplary embodiment of the present application.

[0102]

[0103] Table 2 Among them, the business type "Account" means "data type customer", and "User" means "user". Field means the name of the field corresponding to the database or interface, etc., which can be multiple values. Multiple values ​​are separated by ",". When the value is not unique, it means that the following rules are effective for all fields.

[0104] Action type is the event type, where "C" stands for Create, representing the action of creating new data, such as creating a new account and a new user. In the first item of the business form, for the Account data type, when creating (Action type: C), the name field and the addr field are required fields (the data in the "required" field is 1); in the second item of the business form, for the Account data type when updating (Action type: U), the name field and the addr field are required fields; in the third item of the business form, for the newly created User data, the name field and the addr field are required fields; in the third item of the business form, for the newly created User data, the value range of the age field is between 18 and 199.

[0105] In the embodiment of the present application, the comprehensive application of multiple integrity verification methods such as required field verification, field range verification, and event logic verification is conducive to more comprehensive and effective protection of the quality of synchronized data, ensuring that the data written to the second system is complete and accurate, thereby improving the integrity and accuracy of cross-system data synchronization and reducing the risk of data anomalies in the second system.

[0106] In a possible implementation, when the synchronous conversion data fails to be written into the second system, the data synchronization system triggers a retry mechanism, which is used to instruct to reacquire the data to be synchronized from the first system. In the actual data synchronization process, various abnormal situations may occur, such as network failure of the second system, busy service, database connection abnormality, etc., resulting in failure to write the synchronous conversion data into the second system. In order to increase the robustness and fault tolerance of data synchronization, a retry mechanism is introduced in this implementation.

[0107] The retry mechanism is a mechanism for the system to automatically retry the write operation when a data write operation fails. The retry mechanism is used to instruct to reacquire the data to be synchronized from the first system, which indicates that the retry mechanism in this embodiment does not simply rewrite the previously converted data, but more intelligently reacquires the data to be synchronized from the first system, then obtains the data to be synchronized from the message queue header, and re-executes the data format conversion and subsequent data write operations.

[0108] In the embodiment of the present application, based on the retry mechanism, the data to be synchronized is re-acquired from the first system when writing fails, which can ensure that the retried writing operation uses the latest and most original data, thereby maximizing the success rate of data synchronization.

[0109] Please refer to Figure 6 , which shows a flowchart of a cross-system data synchronization process provided by an illustrative embodiment of the present application.

[0110] First, when there is a need for data synchronization, the first system fills the data to be synchronized and the data synchronization request into the tail of the message queue, and then starts synchronization. The data synchronization system pulls the data to be synchronized from the head of the message queue, and then translates the fields (i.e., converts the data format) to obtain the synchronization conversion data. In the case of drop-down box data, the CodeMapping method is used to convert the drop-down box data. After obtaining the synchronization conversion data, the integrity of the data is checked (integrity check). After the integrity check passes, the synchronization conversion data is rendered based on the data template, and then the API is called to write the rendering result to the second system. In addition, the data synchronization system needs to detect whether the data synchronization is successful. If the synchronization fails, the retry mechanism is triggered. If the synchronization is successful, the data synchronization is completed.

[0111] In a possible implementation, the present application also relates to a cross-system data synchronization system, the system comprising: Message queue module, data conversion module and data writing module.

[0112] The message queue module is used to obtain at least one group of data to be synchronized sent by the first system, and add the data to be synchronized to the end of the message queue; and obtain a group of data to be synchronized from the head of the message queue.

[0113] Optionally, the message queue module receives a data synchronization request from the first system, and stores and manages the synchronization data in the form of messages.

[0114] The data conversion module is used to convert the data format of the data to be synchronized based on the target conversion mode corresponding to the data to be synchronized, so as to obtain synchronous conversion data, and the synchronous conversion data matches the second system format.

[0115] When the message queue module obtains the data to be synchronized from the message queue header, it will pass the data to the data conversion module. After receiving the data to be synchronized, the data conversion module first determines the specific strategy and method of data conversion based on the target conversion method corresponding to the data to be synchronized. After determining the target conversion method, the data conversion module converts the data format of the data to be synchronized according to the requirements of the target conversion method, and converts the data format of the first system into a data format that matches the second system.

[0116] The data writing module is used to write the synchronous conversion data into the second system to complete the data synchronization between the second system and the first system. After the data conversion module generates the synchronous conversion data, it will pass the data to the data writing module. After receiving the synchronous conversion data, the data writing module will write the synchronous conversion data into the second system.

[0117] In some embodiments, the cross-system data synchronization system further includes: a template rendering module, an integrity verification module, and an exception handling module.

[0118] Among them, the template rendering module is used to determine the target conversion method based on the data type corresponding to the data to be synchronized, and the target conversion method is used to indicate the target data format of the synchronous conversion data; based on the target conversion method, the data to be synchronized is rendered into the target data format to obtain the synchronous conversion data.

[0119] When the data conversion module needs to perform data format conversion, the template rendering module is called and informed of the data type of the data to be synchronized and the target conversion method to be adopted. The template rendering module determines the target conversion method based on the data type corresponding to the data to be synchronized, and selects a suitable target data format template according to the target conversion method. Then, the template rendering module renders the data to be synchronized into the target data format based on the target conversion method to obtain synchronized conversion data.

[0120] The integrity verification module is used to perform integrity verification on the data to be synchronized corresponding to each business based on the business form of the second system. The integrity verification module is responsible for executing the various operations of the data integrity verification described in the aforementioned implementation mode. After the data conversion module generates the synchronous conversion data, before the data is actually written into the second system, the integrity verification module needs to perform a comprehensive integrity verification on it to ensure that the quality of the data meets the requirements of the target system.

[0121] An exception handling module is used to capture exceptions during the synchronization process and trigger a retry mechanism. During the data synchronization process, various exceptions may occur. For example, network connection interruptions, target system services being unavailable, data write timeouts, data conversion errors, data verification failures, and so on. The exception handling module is used to capture exceptions during the synchronization process, that is, to detect and identify these exception situations in a timely manner, preventing the entire data synchronization process from being interrupted or failed due to exception situations. When the exception handling module captures an exception, corresponding processing is performed according to the pre-set exception handling strategy. For example, exception logs are recorded, alarm notifications are sent, or a retry mechanism is triggered. When the data writing module fails to write the synchronized converted data to the second system, the exception handling module immediately triggers the retry mechanism, instructing the system to re-obtain the data to be synchronized from the message queue and restart the data synchronization process, attempting to perform data synchronization again.

[0122] In summary, the cross-system data synchronization system can implement the steps in the above-mentioned cross-system data synchronization method. The system has relatively complete functions, a high degree of intelligence and automation, and can meet the complex and changeable cross-system data synchronization requirements of enterprises, providing reliable and efficient technical support for enterprise data integration and applications.

[0123] In an embodiment of the present invention, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to cause a computer to execute the above-mentioned cross-system data synchronization method.

[0124] The cross-system data synchronization method proposed in this application can be implemented by means of computer program instructions. These program instructions can be stored on a non-transitory computer-readable storage medium. When a computer device (such as a server, PC, mobile terminal, etc.) reads and executes the computer instructions stored on this storage medium, the aforementioned cross-system data synchronization method can be implemented to complete cross-system data interaction and sharing.

[0125] A non-transient computer-readable storage medium refers to a storage medium that can store computer instructions and can still be read by a computer after the storage instructions are powered off or removed. The non-transient computer-readable storage medium proposed in the embodiments of the present application includes, but is not limited to, one of: read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disk read-only memory (CD-ROM), digital versatile disk (DVD), magnetic tape, floppy disk, hard disk drive (HDD), solid state drive (SSD), etc. The present application does not limit the specific form of the non-transient computer-readable storage medium, as long as it can store computer instructions and is non-transient.

[0126] The above-mentioned specific embodiments are only used to illustrate the present invention in an exemplary manner. These embodiments are not completely exhaustive and are not intended to limit the scope of the present invention. It is possible to change and modify the disclosed embodiments. Other changes and modifications of the embodiments disclosed by the present invention do not exceed the spirit of the present invention and the scope of protection defined by the claims.

[0127] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-system data synchronization method, characterized in that: The method comprises: Acquire at least one set of data to be synchronized sent by the first system, and add the data to be synchronized to the end of the message queue; Get a set of data to be synchronized from the head of the message queue; Based on the target conversion mode corresponding to the data to be synchronized, the data to be synchronized is converted into a data format to obtain synchronization conversion data, where the synchronization conversion data matches the second system format; The synchronous conversion data is written into the second system to complete data synchronization between the second system and the first system.

2. The method according to claim 1, characterized in that The step of converting the data format of the data to be synchronized based on the target conversion mode corresponding to the data to be synchronized to obtain synchronized converted data includes: Determining a target conversion mode based on the data type corresponding to the to-be-synchronized data, the target conversion mode being used to indicate a target data format of the synchronously-converted data; Based on the target conversion mode, the data to be synchronized is rendered into the target data format to obtain the synchronous conversion data.

3. The method according to claim 2, characterized in that The method further comprises: In the case where it is detected that the data to be synchronized belongs to the drop-down box data, the data to be synchronized is mapped into the synchronous conversion data by using real-time dense mapping, and the synchronous conversion data is the drop-down box data of the second system.

4. The method according to claim 1, characterized in that: The method further comprises: Based on the business form of the second system, integrity check is performed on the to-be-synchronized data corresponding to each business, and the integrity check at least includes mandatory field verification, field range verification, and event logic verification.

5. The method according to claim 4, characterized in that Based on the business form of the second system, the required fields of the data to be synchronized corresponding to each business are verified, including: Traversing the business form to determine the required fields corresponding to each business in the second system; Perform non-empty check on the required fields corresponding to each business; Based on the business form of the second system, the field range check is performed on the data to be synchronized corresponding to each business, including: In the case that a range field corresponding to the first business exists as the required field, determining a range limit of the field corresponding to the first business, wherein the range limit is used to specify an interval for filling in data in the range field; Based on the range restriction, the field range check is performed on the range field corresponding to the first service.

6. The method according to claim 5, characterized in that The business form based on the second system performs event logic verification on the data to be synchronized corresponding to each business, including: Based on the trigger event type corresponding to each business, determine the trigger event that triggers the required field verification and field range check, and the trigger event type at least includes creating new business data and updating business data; The event logic check is performed based on the event type to which the service corresponding to the data to be synchronized belongs.

7. The method according to claim 4, characterized in that In the case where the integrity check fails, the method further comprises: Field data that fails validation is marked as an error.

8. The method according to claim 1, characterized in that The method further comprises: In the case where writing the synchronous conversion data into the second system fails, a retry mechanism is triggered, where the retry mechanism is used to instruct to reacquire the to-be-synchronized data from the first system.

9. A cross-system data synchronization system, characterized in that: The system includes: The message queue module is used to obtain at least one group of data to be synchronized sent by the first system, and add the data to be synchronized to the end of the message queue; obtain a group of data to be synchronized from the head of the message queue; A data conversion module, configured to perform data format conversion on the data to be synchronized based on a target conversion mode corresponding to the data to be synchronized, to obtain synchronization conversion data, wherein the synchronization conversion data matches a second system format; The data writing module is used to write the synchronous conversion data into the second system to complete the data synchronization between the second system and the first system.

10. The system according to claim 9, characterized in that The system further comprises: A template rendering module, used to determine a target conversion mode based on a data type corresponding to the data to be synchronized, wherein the target conversion mode is used to indicate a target data format of the synchronized conversion data; Based on the target conversion mode, the data to be synchronized is rendered into the target data format to obtain the synchronization conversion data; The integrity verification module is used to perform integrity verification on the to-be-synchronized data corresponding to each business based on the business form of the second system; The exception handling module is used to capture exceptions during the synchronization process and trigger the retry mechanism.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the cross-system data synchronization method as described in any one of claims 1 to 8.

Images