Service data transmission system

Abstract

The invention discloses a service data transmission system, which relates to the field of computers, and comprises the following steps: a data production end detects change information of service data, converts an information format of detected reference change information into an information format allowed to be identified by a data application end, and sends the information format to an information interface of the data application end; and the data application end matches a target production end identifier and a target synchronization script corresponding to the target service type from the production end identifier, the service type and the synchronization script which have the corresponding relationship, and calls the target synchronization script to synchronize the changed service data to a second data set corresponding to the target production end identifier, the technical problem that the service data transmission efficiency is low due to the fact that a data application end needs to develop a plurality of data receiving interfaces in a customized mode for a plurality of pieces of service data is solved, and the technical effect that the coupling degree of a system is reduced, so that the service data transmission efficiency is improved is achieved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computers, and more particularly, to a business data transmission system. Background Art

[0002] In the digital transformation of modern enterprises, the cloud platform needs to have the ability to efficiently connect to various third-party systems to achieve unified management and processing of business data from third-party systems.

[0003] In the related art, when it is necessary to transmit the business data of a third-party system to the cloud platform, it is usually necessary to develop a dedicated data receiving interface on the cloud platform to receive the business data of a specific third-party system and process its data format and synchronization operations. Then, the third-party system periodically calls the data receiving interface to push the business data to the cloud platform. However, this approach has various problems: on the one hand, relying on the method of periodically calling the data receiving interface cannot guarantee the real-time nature of the data; on the other hand, since the business data structures and synchronization operations of different third-party systems are different, the cloud platform needs to develop different data receiving interfaces when receiving business data from different third-party systems, which leads to a large development workload. In addition, customized development increases the system coupling. When the business data of the third-party system changes, the data format and synchronization operations defined by the data receiving interface also need to be adjusted accordingly, further increasing the maintenance cost.

[0004] In view of the technical problems in the related art that the data application end needs to customize and develop multiple data receiving interfaces for multiple business data, increasing the system coupling and resulting in low business data transmission efficiency, no effective solution has been proposed yet. Summary of the Invention

[0005] The embodiments of the present application provide a business data transmission system to at least solve the technical problems in the related art that the data application end needs to customize and develop multiple data receiving interfaces for multiple business data, increasing the system coupling and resulting in low business data transmission efficiency.

[0006] According to an embodiment of the embodiments of the present application, a business data transmission system is provided, including: a data application end and multiple data production ends, where the multiple data production ends are all connected to an information interface on the data application end, and each data production end stores the business data required for the data application end to run the corresponding business.

[0007] The target data producer among multiple data producers is used to detect the change information of the first data set storing the target business data. The change information is used to record the business data that has changed in the target business data, the change type of the business data that has changed, and the business type of the business data that has changed. When the reference change information is detected, the information format of the reference change information is converted into the target information format recognizable by the data application end to obtain the target change information. The target change information carrying the target producer identifier is sent to the information interface.

[0008] The data application end is used to receive the target change information carrying the target producer identifier through the information interface; match the target producer identifier and the target synchronization script corresponding to the target business type recorded in the target change information from the production end identifier, business type, and synchronization script with a corresponding relationship. The target synchronization script is used to indicate the synchronization operation required to be performed on the changed business data recorded in the target change information; call the target synchronization script to synchronize the changed business data to the second data set corresponding to the target producer identifier; run the business according to the business data in the second data set.

[0009] Optionally, the target data producer includes: an information collector, a format converter, an information pusher, and a target interface. The information collector is connected to the information pusher through the format converter, and the information pusher is connected to the information interface through the target interface. Among them,

[0010] The information collector is used to detect the change information of the first data set storing the target business data; when the reference change information is detected, send a format conversion request carrying the reference change information to the format converter. The format conversion request is used to request converting the information format of the reference change information into the information format recognizable by the data application end.

[0011] The format converter is used to respond to the format conversion request, convert the information format of the reference change information into the target information format to obtain the target change information, and transmit the target change information to the information pusher.

[0012] The information pusher is used to add the target producer identifier to the target change information and call the target interface to send the target change information carrying the target producer identifier to the information interface.

[0013] Optionally, an information collection tool and a collection configuration tool are run on the information collector. Among them,

[0014] The collection configuration tool is used to configure the collection parameters currently used by the information collection tool. The collection parameters are used to indicate the change types allowed to be collected by the information collection tool.

[0015] An information collection tool is used to detect change information of a first data set according to target collection parameters configured by a collection configuration tool; and in the case of detecting reference change information, send a format conversion request carrying the reference change information to a format converter, where the reference change information records the reference business data in the target business data that has changed, the reference change type of the reference business data, and the reference business type of the reference business data, and the reference change type is the change type indicated by the target collection parameters.

[0016] Optionally, an information collection tool is used to detect change information of a first data set according to target collection parameters configured by a collection configuration tool through the following steps:

[0017] Read the change record file corresponding to the first data set, where the change record file records the change information in the first data set;

[0018] Detect the change information in the change record file according to the target collection parameters.

[0019] Optionally, a first script invocation tool runs on the format converter, and the first script invocation tool is connected to the information collector, where

[0020] The first script invocation tool is used to detect the reference change type recorded in the reference change information carried by the format conversion request sent by the information collector;

[0021] When the reference change type is the new type, respond to the format conversion request, and call the new conversion script to convert the information format of the reference change information into the target information format that the data application side allows to recognize, where the new conversion script allows converting the information format recording the change information of the new type into the target information format that the data application side allows to recognize;

[0022] When the reference change type is the modification type, respond to the format conversion request, and call the modification conversion script to convert the information format of the reference change information into the target information format that the data application side allows to recognize, where the modification conversion script allows converting the information format recording the change information of the modification type into the target information format that the data application side allows to recognize;

[0023] When the reference change type is the deletion type, respond to the format conversion request, and call the deletion conversion script to convert the information format of the reference change information into the target information format that the data application side allows to recognize, where the deletion conversion script allows converting the information format recording the change information of the deletion type into the target information format that the data application side allows to recognize.

[0024] Optionally, an information pusher also runs an identifier reading tool and an information splicing tool, where

[0025] An identification reading tool for reading the target production - end identification of the target data production end;

[0026] An information splicing tool for receiving target change information from a format converter; splicing the target production - end identification read by the identification reading tool to the target change information; and calling a target interface to send the target change information spliced with the target production - end identification to an information interface.

[0027] Optionally, the data application end further includes: a data synchronizer, and the data synchronizer is connected to the information interface, where,

[0028] The data synchronizer is used to screen out a second data set corresponding to the target production - end identification from a production - end identification and data set with a corresponding relationship; and synchronize the changed service data recorded in the target change information to the second data set according to the target production - end identification and the target service type recorded in the target change information.

[0029] Optionally, the data synchronizer runs a set matching tool and a second script calling tool. The set matching tool and the second script calling tool are connected, and the second script calling tool is connected to the information interface, where,

[0030] The set matching tool is used to screen out a second data set corresponding to the target production - end identification from a production - end identification and data set with a corresponding relationship;

[0031] The second script calling tool is used to match a target synchronization script corresponding to the target production - end identification and the target service type from a production - end identification, service type, and synchronization script with a corresponding relationship, where the target synchronization script is used to indicate the synchronization operation required for the changed service data recorded in the target change information; and call the target synchronization script to synchronize the changed service data to the second data set.

[0032] Optionally, the second script calling tool is further used for:

[0033] In the case where the target synchronization script corresponding to the target production - end identification and the target service type cannot be matched from the production - end identification, service type, and synchronization script with a corresponding relationship, prohibit synchronizing the changed service data recorded in the target change information to the second data set, and generate a script addition request carrying the target production - end identification and the target service type, where the script addition request is used to request the target account to create a synchronization script for the changed service data recorded in the target change information.

[0034] Optionally, a request sending interface is also deployed in the data application end. The request sending interface is connected to the second script calling tool, where,

[0035] A request sending interface is used to send a script addition request generated by a second script calling tool to a target account; receive a target production end identifier, a target service type, and a reference synchronization script created by the target account in response to the script addition request; and add the target production end identifier, the target service type, and the reference synchronization script with corresponding relationships to the production end identifier, service type, and synchronization script with corresponding relationships.

[0036] Through this application, the data production end in the business data transmission system detects the change information of business data in real time. When it detects that the change information belongs to the reference change information, it converts the information format of the change information into the information format recognizable by the data application end and then sends it to the information interface of the data application end. After receiving the target production end identifier and the target change information, the data application end matches the target synchronization script corresponding to the target service type recorded in the target production end identifier and the target change information from the production end identifier, service type, and synchronization script with corresponding relationships, and calls the target synchronization script to synchronize the changed business data recorded in the target change information to the second data set corresponding to the target production end identifier, and operates the business according to the business data in the second data set. That is, by completing the format conversion of the change information at the data production end and combining the establishment of the corresponding relationship between the production end identifier, service type, and synchronization script at the data application end, the data application end only needs a unified information interface to receive the business data of multiple data production ends. At the same time, the decoupling of the business data synchronization operation is realized. Therefore, it can solve the technical problems in the related art that the data application end needs to customize and develop multiple data receiving interfaces for multiple business data, increasing the coupling degree of the system and resulting in low business data transmission efficiency. The technical effect of reducing the coupling degree of the system and thus improving the business data transmission efficiency is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] To more clearly illustrate the embodiments of the present application, the accompanying drawings required for use in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0038] Figure 1 is a structural block diagram of a business data transmission system in the related art;

[0039] Figure 2 is a schematic diagram of a business data transmission system according to an embodiment of the present application;

[0040] Figure 3 is a schematic diagram of a business data transmission system according to an embodiment of the present application for business data transmission to a target third-party system;

[0041] Figure 4 It is a schematic diagram of a change information detection of an information collection tool according to an embodiment of the present application;

[0042] Figure 5 It is a schematic diagram of a conversion script according to an embodiment of the present application;

[0043] Figure 6 It is a schematic diagram of a conversion script call according to an embodiment of the present application;

[0044] Figure 7 It is a schematic diagram of a synchronization script according to an embodiment of the present application;

[0045] Figure 8 It is a schematic diagram of a business data transmission task process executed by a business data transmission system according to an embodiment of the present application;

[0046] Figure 9 It is a schematic diagram of a business data collection task process executed by a business data collection tool according to an embodiment of the present application;

[0047] Figure 10 It is a schematic diagram of a format conversion task process executed by a format conversion script according to an embodiment of the present application;

[0048] Figure 11 It is a schematic diagram of a business data processing task process executed by a business data receiving interface according to an embodiment of the present application. Detailed implementation manners

[0049] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0050] It should be noted that in the description of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusions, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0051] Explanations of the nouns involved in the embodiments of the present application are as follows:

[0052] URL (Uniform Resource Locator): Uniform Resource Locator, used to identify the addresses of resources (such as web pages, files, etc.) on the Internet;

[0053] API (Application Programming Interface): Application Programming Interface, a set of predefined functions, protocols, and tools that define the interaction methods between software components;

[0054] Groovy: An agile development language based on the JVM (Java Virtual Machine), which can be combined with Java code, allowing developers to directly call Java classes and methods in Groovy scripts to extend existing Java projects.

[0055] Binlog (Binary Log): Binary Log, a log file in the MySQL database, used to record database change operations, such as INSERT (new), UPDATE (modify), DELETE (delete).

[0056] This application proposes a business data transmission system. Before describing the optional embodiments of this application, in order to better understand the inventive concept of this application and the creativity of this solution, the related technologies are first described: Figure 1 It is a structural block diagram of a business data transmission system in the related technology, as Figure 1 shown. For one data application end and two data production ends, namely data production end A and data production end B, when it is necessary to transmit the business data A in data production end A to the data application end, in the related technology, a data receiving interface A is developed at the data application end to receive business data A, process the data format and synchronization operation of business data A, and write code A for processing business data A in the core code of the data application end. Correspondingly, when it is necessary to transmit the business data B in data production end B to the data application end, it is necessary to develop a data receiving interface B at the data application end to receive business data B, process the data format and synchronization operation of business data B, and write code B for processing business data B in the core code of the data application end.

[0057] The business data transmission system in the related technology has the following technical problems:

[0058] 1) When docking with a new data production end, it is necessary to customize and develop a new data receiving interface at the data application end to process the different data formats of the business data of different data production ends. That is, the data receiving interfaces developed at the data application end are not universal, and the workload of customized development is large;

[0059] 2) When docking with a new data production end, it is necessary to modify the core code of the code data application end to adapt to the synchronization operations of the business data of different data production ends, resulting in a relatively high system coupling degree and an increased maintenance cost.

[0060] To solve the above problems, in this embodiment, a business data transmission system is provided, including: a data application end and multiple data production ends. The multiple data production ends are all connected to an information interface on the data application end. Each data production end stores the business data required for the data application end to run the corresponding business; a target data production end among the multiple data production ends is used to detect the change information of the first data set storing the target business data, where the change information is used to record the business data that has changed in the target business data, the change type of the business data that has changed, and the business type of the business data that has changed; in the case of detecting the reference change information, converting the information format of the reference change information into a target information format that can be recognized by the data application end to obtain the target change information; sending the target change information carrying the target production end identifier to the information interface; the data application end is used to receive the target change information carrying the target production end identifier through the information interface; matching the target production end identifier and the target synchronization script corresponding to the target business type recorded in the target change information from the production end identifier, business type, and synchronization script with a corresponding relationship, where the target synchronization script is used to indicate the synchronization operation required for the changed business data recorded in the target change information; calling the target synchronization script to synchronize the changed business data to the second data set corresponding to the target production end identifier; and running the business according to the business data in the second data set.

[0061] Optionally, in this embodiment, the data application end may be, but is not limited to, a system with data processing and business logic execution capabilities, including: cloud platforms, data warehouses, data middle platforms, and distributed database systems, etc. The data production end may be, but is not limited to, a source system that can provide business data, including: third-party systems, Internet of Things devices, and mobile applications, etc. The information interface may be, but is not limited to, an HTTP / HTTPS API interface.

[0062] Optionally, in this embodiment, the business data may include, but is not limited to, various data generated, collected, and processed by an enterprise during its daily operation and management. The business data may cover multiple business types, and the business data of different business types corresponds to different synchronization operations. For example, for customer data, its synchronization operations may include data verification to ensure the integrity and accuracy of customer information, such as verifying the email format and the validity of the mobile phone number; data cleaning to remove duplicate records and standardize the address format; data association to associate customer transaction records with customer basic information to generate a comprehensive customer profile; data encryption to encrypt sensitive information such as ID numbers and bank card numbers; and real-time update to synchronize the updated customer information to the customer information system in real time. For sales data, its synchronization operations focus on data aggregation, aggregating sales data by time periods such as daily, weekly, and monthly to generate sales reports; data verification to check whether the order amount is greater than zero and whether the order status is valid; data association to associate sales orders with customer information and product information to analyze sales trends; anomaly detection to identify abnormal sales data, such as an abnormally high order amount or an abnormally low sales volume; and real-time synchronization to synchronize sales data to the financial system and the inventory management system in real time to ensure the efficiency of enterprise operation and data consistency.

[0063] Optionally, in this embodiment, the change types of the business data may include, but are not limited to, the types of changes that occur to the business data, such as: addition, modification, and deletion, etc.

[0064] Optionally, in this embodiment, Figure 2 is a schematic diagram of a business data transmission system according to an embodiment of the present application. As Figure 2 shown, the business data transmission system includes: a cloud platform and multiple third-party systems (for example, taking N third-party systems as an example: third-party systems 1 to N). Multiple third-party systems are all connected to an information interface of the cloud platform. Each third-party system stores its own business data (for example, business data 1 to N), and the cloud platform can utilize business data 1 to N to run corresponding services.

[0065] Optionally, in this embodiment, Figure 3 is a schematic diagram of a business data transmission system according to an embodiment of the present application for transmitting business data to a target third-party system. As Figure 3As shown in the figure, when the target third-party system (for example, the third-party system 1) among multiple third-party systems 1 to N conducts business data transmission, the third-party system 1 first detects the change information in the first data set of the target business data (for example, business data 1) it stores. When the reference change information is detected, the third-party system 1 converts the data format of the reference change information into a target information format that the cloud platform can recognize, generates the target change information, and sends the target change information carrying the target production end identifier to the information interface of the cloud platform.

[0066] Optionally, in this embodiment, the third-party system 1 can obtain the change information in the first data set of the business data 1 in multiple ways, such as reading binlog, querying the change log table, or parsing the application log. Among them, the change information records the business data (changed data content) that has changed in the first data set, the change type (such as addition, modification, deletion), and the business type (such as customer data, sales data, etc.).

[0067] Optionally, in this embodiment, the cloud platform stores the corresponding relationship between the production end identifier, the business type, and the synchronization script. The corresponding relationship can be stored in a configuration table, for example. Among them, the production end identifier can be, for example, the system ID (Identifier) of the third-party system, the name, or other unique identifiers; the synchronization script can indicate the synchronization operations required for the change information originating from the corresponding production end identifier and recording the corresponding business type. The synchronization operations can include, for example: according to the change type recorded in the change information, updating the changed business data to the data set corresponding to the corresponding production end identifier, or associating the changed business data recorded in the change information with other data in the data set, and then updating the changed business data to the data set corresponding to the corresponding production end identifier according to the change type recorded in the change information.

[0068] Optionally, in this embodiment, as Figure 3 shown, the cloud platform finds and invokes the corresponding target synchronization script from the configuration table according to the received target production end identifier and the target business type recorded in the target change information, synchronizes the changed business data to the second data set corresponding to the target production end identifier, and then runs the business corresponding to the third-party system 1 based on the business data in the second data set.

[0069] As an alternative solution, the target data production end includes: an information collector, a format converter, an information pusher, and a target interface. The information collector is connected to the information pusher through the format converter, and the information pusher is connected to the information interface through the target interface. Among them, the information collector is used to detect the change information of the first data set storing the target business data; in the case of detecting the reference change information, it sends a format conversion request carrying the reference change information to the format converter, where the format conversion request is used to request to convert the information format of the reference change information into the information format recognizable by the data application end; the format converter is used to respond to the format conversion request, convert the information format of the reference change information into the target information format to obtain the target change information, and transmit the target change information to the information pusher; the information pusher is used to add the target production end identifier to the target change information, and call the target interface to send the target change information carrying the target production end identifier to the information interface.

[0070] Optionally, in this embodiment, the reference change information may but is not limited to the change information allowed by the target third-party system to be received by the cloud platform. For example, Figure 3 As shown, the third-party system 1 includes: an information collector, a format converter, an information pusher, and a target interface. The information collector runs independently in the third-party system 1 and detects the change information of the business data in the first data set in real time.

[0071] Optionally, in this embodiment, the information format of the change information of the business data of the third-party system may be different from the information format recognizable by the cloud platform (such as the JSON format), and the information formats of the change information of the business data from different third-party systems may be different. For example, Figure 3 As shown, when the information collector detects the change information allowed by the third-party system 1 to be received by the cloud platform (i.e., the reference change information), it sends a format conversion request carrying the reference change information to the format converter. After receiving the format conversion request, the format converter responds to the format conversion request, converts the information format of the reference change information into the target information format to obtain the target change information, and transmits the target change information to the information pusher. After receiving the target change information transmitted by the format converter, the information pusher adds the target production end identifier to the target change information, and calls the target interface to send the target change information carrying the target production end identifier to the information interface of the cloud platform.

[0072] As an alternative solution, an information collector runs an information collection tool and a collection configuration tool. The collection configuration tool is used to configure the collection parameters currently used by the information collection tool. The collection parameters are used to indicate the change types allowed to be collected by the information collection tool. The information collection tool is used to detect the change information of the first data set according to the target collection parameters configured by the collection configuration tool. And in the case of detecting reference change information, a format conversion request carrying the reference change information is sent to the format converter. The reference change information records the reference business data in the target business data that has changed, the reference change type of the reference business data, and the reference business type of the reference business data. The reference change type is the change type indicated by the target collection parameters.

[0073] Optionally, in this embodiment, an information collection tool and a collection configuration tool are run on the information collector. The collection configuration tool can be used but is not limited to configuring the collection parameters of the information collection tool. The collection parameters can indicate the data set that the information collection tool needs to collect, the change types allowed to be collected for this data set, and the system identifier. For example, the configuration corresponding to the data set that the information collection tool needs to collect in the collection parameters can be:

[0074] host=127.0.0.1

[0075] port=3306

[0076] username=root

[0077] password=xxx

[0078] Among them, "host", "port", "username", and "password" are the specific parameters of the data set that the information collection tool needs to collect, corresponding to the host address, port number, username, and password. The information collection tool can connect to the corresponding data set through these specific parameters to realize the collection of changes in business data in the data set.

[0079] The configuration corresponding to the change types allowed to be collected for the data set in the collection parameters can be:

[0080] ibase.user_info=UPDATE,DELETE,WRITE

[0081] ibase.department_info=UPDATE,DELETE

[0082] Among them, "ibase.user_info = UPDATE,DELETE,WRITE" can indicate that the change types allowed to be collected for the data set named "user_info" are "UPDATE" (modification), "DELETE" (deletion), and "WRITE" (insertion); "ibase.department_info = UPDATE,DELETE" can indicate that the change types allowed to be collected for the data set named "department_info" are "UPDATE" (modification) and "DELETE" (deletion). An information collection tool with collection parameters configured as "insertion" and "modification" only allows the collection of change information whose recorded change types are "insertion" and "modification".

[0083] Optionally, in this embodiment, Figure 4 is a schematic diagram of an information collection tool in an embodiment of the present application for detecting change information. As Figure 4 shown, after the information collection tool is started, it connects to the first data set according to the configuration of the corresponding data set in the collection parameters and monitors the corresponding binlog file. After detecting a binlog update, it will determine whether the updated change information X meets the collection conditions according to the collection parameters set by the collection configuration tool. If it meets the conditions (for example, when the collection parameter includes "insertion", the change type recorded by the change information X is "insertion"), the information collection tool will encapsulate the change information X into a format conversion request and send it to the format converter; if it does not meet the conditions (for example, when the collection parameter only includes "insertion" and the change type recorded by the change information X is "deletion"), the information collection tool will filter the change information X and continue to monitor the binlog to detect other change information.

[0084] As an optional solution, an information collection tool is used to detect the change information of the first data set according to the target collection parameters configured by the collection configuration tool through the following steps: reading the change record file corresponding to the first data set, where the change record file records the change information in the first data set; detecting the change information in the change record file according to the target collection parameters.

[0085] Optionally, in this embodiment, the change record file can be but is not limited to a file that can record the change information in the first data set, including binlog files, change log tables, or application logs. As Figure 4 shown, the method for the information collection tool to detect the change information of the first data set can be but is not limited to: reading the binlog to implement the monitoring of the change information, and after detecting a binlog update, detecting whether the change information meets the collection conditions according to the target collection parameters set by the collection configuration tool.

[0086] As an alternative solution, a first script calling tool is run on the format converter. The first script calling tool is connected to the information collector. The first script calling tool is used to detect the reference change type recorded in the reference change information carried in the format conversion request sent by the information collector. When the reference change type is the addition type, in response to the format conversion request, the addition conversion script is called to convert the information format of the reference change information into the target information format recognizable by the data application end, where the addition conversion script allows the information format recording the change information of the addition type to be converted into the target information format recognizable by the data application end. When the reference change type is the modification type, in response to the format conversion request, the modification conversion script is called to convert the information format of the reference change information into the target information format recognizable by the data application end, where the modification conversion script allows the information format recording the change information of the modification type to be converted into the target information format recognizable by the data application end. When the reference change type is the deletion type, in response to the format conversion request, the deletion conversion script is called to convert the information format of the reference change information into the target information format recognizable by the data application end, where the deletion conversion script allows the information format recording the change information of the deletion type to be converted into the target information format recognizable by the data application end.

[0087] Optionally, in this embodiment, the first script calling tool may be a service program running on the format converter, which receives the change information through the API interface and then calls the corresponding conversion script according to the preset rules. The manner in which the format converter responds to the format conversion request may include, but is not limited to: the first script calling tool selects the conversion script of the corresponding change type according to the reference change type (such as addition, modification, deletion) recorded in the reference change information carried in the format conversion request, and calls the conversion script of the corresponding change type to convert the information format of the reference change information into the target information format recognizable by the data application end, and then receives the output of the conversion script of the corresponding change type and uses it as the output of the format converter.

[0088] Optionally, in this embodiment, Figure 5 is a schematic diagram of a conversion script according to an embodiment of the present application. As Figure 5 shown, the conversion scripts (including the addition conversion script, the modification conversion script, and the deletion conversion script) may include, but are not limited to, groovy scripts developed according to a unified specification. The unified input parameters are: the table where the business data is located (corresponding to the first data set), the content of the business data before the operation (this field is empty when adding business data), and the content of the business data after the operation (this field is empty when deleting business data). The unified return value is the information format recognizable by the cloud platform (data application end), and the return value includes: the target business type, the target change type, and the changed business data, that is, the target change information.

[0089] Optionally, in this embodiment, multiple conversion scripts are loaded into the third-party system memory for caching, and the first script invocation tool can directly obtain the conversion scripts from the cache when invoking the conversion scripts.

[0090] Optionally, in this embodiment, Figure 6 is a schematic diagram of invoking a conversion script according to an embodiment of the present application. As Figure 6 shown, the first script invocation tool can only select and invoke the corresponding conversion script according to the reference change type recorded in the reference change information. For example, when the reference change type recorded in the reference change information is "new addition", only the new addition conversion script needs to be selected and invoked.

[0091] As an optional solution, the information pusher also runs an identifier reading tool and an information splicing tool. Among them, the identifier reading tool is used to read the target production end identifier of the target data production end; the information splicing tool is used to receive the target change information from the format converter; and splice the target production end identifier read by the identifier reading tool to the target change information; and call the target interface to send the target change information spliced with the target production end identifier to the information interface.

[0092] Optionally, in this embodiment, the production end identifier may include, but is not limited to, a system ID, a system name, or other unique identifier that can indicate the source of business data. The identifier reading tool can obtain the production end identifier from the configuration file, database, or other storage media of the third-party system, and then send the production end identifier to the format converter. The format converter calls the identifier format conversion script to convert the data format of the production end identifier into a data format recognizable by the cloud platform. For example, the identifier reading tool reads the system ID, name, and IP information from the collection parameters in the collection configuration tool, and then calls the identifier format conversion script for data format conversion to obtain the production end identifier as: {name = test system; id = test; ip = 1.1.1.1}.

[0093] Optionally, in this embodiment, after receiving the target change information, the information splicing tool can splice the production end identifier (data format recognizable by the cloud platform) generated by the identifier reading tool into the target change information, so that the target change information can be correctly recognized and processed after being transmitted to the cloud platform.

[0094] Optionally, in this embodiment, after the identification reading tool obtains the production end identification from the configuration file, database, or other storage media of the third-party system, the information splicing tool may first splice the production end identification (data format allowed to be recognized by the third-party system) and the reference change information, and then send the spliced data to the format converter for format conversion. The format converter outputs the target change information carrying the target production end identification with the converted data format, and transmits the target change information carrying the target production end identification with the converted data format to the information pusher. The information pusher directly calls the target interface to send the target change information carrying the target production end identification to the information interface. In the above manner, the format converter only needs to call the conversion script once to output the target change information carrying the target production end identification with the converted data format, improving the efficiency of data format conversion during the business data transmission process.

[0095] Optionally, in this embodiment, the manner in which the information pusher calls the target interface to send data to the information interface of the cloud platform may be, but is not limited to: through an HTTP POST request, sending the target change information spliced with the target production end identification to the cloud platform. The HTTP POST request may include the path of the information interface, the request body, and the data format of the request body. For example, the HTTP POST request may be:

[0096] POST / cloud / third / business / datas

[0097] Content-Type:application / json

[0098] {businessType: target business type

[0099] operateType: target change type

[0100] datas: changed business data

[0101] systemInfo: third-party system information, including system ID, name, IP information}

[0102] Among them, " / cloud / third / business / datas" is the path of the information interface of the cloud platform. "Content-Type:application / json" indicates that the data format in the request body is JSON format. The request body is the data carried in the POST request, that is, the target change information carrying the target production end identification, including the target production end identification (third-party system information), target business type, target change type, and changed business data.

[0103] As an alternative solution, the data application end further includes: a data synchronizer, which is connected to the information interface. Among them, the data synchronizer is used to screen out the second data set corresponding to the target production end identifier from the production end identifiers and data sets with corresponding relationships; according to the target production end identifier and the target business type recorded in the target change information, synchronize the changed business data recorded in the target change information to the second data set.

[0104] Optionally, in this embodiment, the cloud platform further includes a data synchronizer. Assume that a third-party system sends the following target change information with a production end identifier to the cloud platform through the information interface:

[0105]

[0106]

[0107] After receiving the target change information from the information interface, the data synchronizer screens out the second data set corresponding to the target production end identifier from the cloud platform database according to the target production end identifier "systemId", and then selects the corresponding synchronization operation according to the target production end identifier and the target business type recorded in the target change information to synchronize the changed business data recorded in the target change information to the second data set.

[0108] As an alternative solution, the data synchronizer runs a set matching tool and a second script calling tool. The set matching tool is connected to the second script calling tool, and the second script calling tool is connected to the information interface. Among them, the set matching tool is used to screen out the second data set corresponding to the target production end identifier from the production end identifiers and data sets with corresponding relationships; the second script calling tool is used to match the target synchronization script corresponding to the target production end identifier and the target business type from the corresponding production end identifier, business type and synchronization script. Among them, the target synchronization script is used to indicate the synchronization operation required for the changed business data recorded in the target change information; and call the target synchronization script to synchronize the changed business data recorded in the target change information to the second data set.

[0109] Optionally, in this embodiment, the data synchronizer runs a set matching tool. After receiving the target change information carrying the target production end identifier, the set matching tool filters out the second data set corresponding to the target production end identifier from the cloud platform database according to the target production end identifier "systemId". For example, the data sets stored in the cloud platform database are: CRM001_customer_data, Order001_order_data, and CRM002_customer_data. The set matching tool matches the corresponding second data set "CRM001_customer_data" from the data sets according to the target production end identifier "CRM001".

[0110] Optionally, in this embodiment, the data synchronizer also runs a second script calling tool. A configuration table of the corresponding relationship between the production end identifier, business type, and synchronization script is stored in the cloud platform. Assume that the following corresponding relationships of the synchronization script are configured in the configuration table:

[0111] systemId:CRM001

[0112] businessType:customer_data

[0113] scriptPath:scripts / sync_customer_data.py

[0114] The second script calling tool can match the corresponding target synchronization script "scripts / sync_customer_data.py" from the configuration table according to the target production end identifier CRM001 and the target business type "customer_data", and call "scripts / sync_customer_data.py" to perform a synchronization operation on the changed business data in the target change information.

[0115] Optionally, in this embodiment, the synchronization script can be but is not limited to a groovy script developed according to a unified specification. Figure 7 It is a schematic diagram of a synchronization script in an embodiment of the present application. As Figure 7 shown, the unified input parameters of the groovy script are: target change type, target production end identifier (third-party system information), and changed business data, and the unified return value: true / false (synchronization success / synchronization failure), indicating whether the synchronization operation is successful.

[0116] As an optional solution, the second script calling tool is also used for:

[0117] In the case where the target synchronization script corresponding to the target production end identifier and the target service type cannot be matched from the production end identifier, service type, and synchronization script with a corresponding relationship, it is prohibited to synchronize the changed service data recorded in the target change information to the second data set, and a script addition request carrying the target production end identifier and the target service type is generated, where the script addition request is used to request the target account to create a synchronization script for the changed service data recorded in the target change information.

[0118] Optionally, in this embodiment, when the second script calling tool fails to match the target synchronization script corresponding to the target production end identifier and the target service type from the configuration table, the cloud platform will be prohibited from synchronizing the changed service data to the second data set, and a script addition request will be generated. The script addition request carries the target production end identifier and the target service type, and is used to request the target account to create a synchronization script for the changed service data.

[0119] As an alternative solution, a request sending interface is also deployed in the data application end, and the request sending interface is connected to the second script calling tool, where

[0120] The request sending interface is used to send the script addition request generated by the second script calling tool to the target account; receive and store the reference synchronization script returned by the target account in response to the script addition request; create the corresponding relationship between the target production end identifier, the target service type, and the reference synchronization script; and add the target production end identifier, the target service type, and the reference synchronization script with the corresponding relationship to the production end identifier, service type, and synchronization script with the corresponding relationship.

[0121] Optionally, in this embodiment, the request sending interface first sends the script addition request generated by the second script calling tool to the target account; then, receives and saves the reference synchronization script input by the user on the cloud platform data processing script management page, where the cloud platform data processing script management page visually displays the corresponding relationship between the production end identifier, service type, and synchronization script stored in the current cloud platform; the synchronization operation indicated by the reference synchronization script is consistent with the synchronization operation required for the changed information originating from the target production end identifier and recording the target service type; then establish the corresponding relationship between the target production end identifier, the target service type, and the reference synchronization script; and finally save the corresponding relationship in the configuration table storing the corresponding relationship between the production end identifier, service type, and synchronization script.

[0122] Optionally, in this embodiment, the request sending interface also allows receiving the script information actively added or modified by the target account, including: the user inputs the corresponding production end identifier, business type, and reference synchronization script on the cloud platform data processing script management page, and the cloud platform data processing script management page returns the input production end identifier, business type, and reference synchronization script to the request sending interface. For example, the user inputs on the cloud platform data processing script management page:

[0123] {"systemId":"CRM003",

[0124] "businessType":"new_customer_data",

[0125] "scriptPath":"scripts / sync_new_customer_data.py"}

[0126] It means that the user has actively added the change information corresponding to the production end identifier "CRM003" and business type "new_customer_data" that needs to execute the synchronization operation, that is, the synchronization operation indicated by "sync_new_customer_data.py". After receiving the above information, the request sending interface stores the reference synchronization script, creates the target production end identifier, target business type, and reference synchronization script with corresponding relationships, and adds these information to the configuration table. The updated configuration table adds the following corresponding relationships:

[0127] systemId:CRM003

[0128] businessType:new_customer_data

[0129] scriptPath:scripts / sync_new_customer_data.py

[0130] Optionally, in this embodiment, in order to better understand the process of the above business data transmission system executing the business data transmission task, the following further describes the above process of executing the business data transmission task in combination with optional embodiments, but it is not used to limit the technical solutions of the embodiments of the present application.

[0131] In this embodiment, a business data transmission system is provided, Figure 8 which is a schematic diagram of the process of a business data transmission system executing a business data transmission task according to an embodiment of the present application, as Figure 8 shown, mainly including the following parts:

[0132] 1) The data production end collects business data. A third-party system (data production end) develops a business data collection tool (information collector). Figure 9 It is a schematic diagram of the process of a business data collection tool executing a business data collection task according to an embodiment of the present application. As Figure 9 shown, the business data collection tool determines the business data tables (first data set) to be collected and the allowed business types (such as new, modified, and deleted) according to the collection parameters in the collection configuration tool, and then dynamically monitors the binlog file of the business database according to the collection parameters. The binlog file records the changes in business data caused by the operations of third-party system users (including new, modified, and deleted, etc.).

[0133] 2) Format conversion of the changed business data at the data production end. Figure 10 It is a schematic diagram of the process of a format conversion script executing a format conversion task according to an embodiment of the present application. As Figure 10 shown, after the information collector collects the changed business data (reference change information) that meets the collection parameters through binlog time monitoring and binlog event filtering, the first script call tool in the business data conversion script (format conversion script) selects and calls the corresponding conversion script (new conversion script, modified conversion script, or deleted conversion script) according to the change type of the changed business data, converts the data structure (information format) of the changed business data into the data structure required by the cloud platform (data application end), obtains the target changed business data, and then uses the business data pusher (information pusher) to splice the target changed business data with the third-party system information (target production end identifier) and then call the target interface to send an HTTP POST request to call the business data receiving interface (information interface) of the cloud platform to push the target changed business data to the cloud platform.

[0134] 3) The data application end receives and processes the target changed business data. A unified business data receiving interface is developed in the cloud platform, which can receive the changed business data from multiple different third-party systems. Figure 11 It is a schematic diagram of the process of a business data receiving interface executing a business data processing task according to an embodiment of the present application. As Figure 11As shown in the figure, for the target changed business data (including target business type, target change type, target changed business data, and target third-party system information (target production end identifier)) pushed by a third-party system received, the business data receiving interface queries, according to the corresponding relationship between the third-party system information, business type, and business data processing script (synchronization script) stored in the business data processing configuration table, for the target business data processing script that matches the target third-party system information and target changed business data. After it is found that the target business data processing script exists, the script is called to perform synchronization operations on the target changed business data, such as data assembly, association with cloud platform resources, etc., to achieve dynamic processing and storage (synchronization operation) of different business data by calling different business data processing scripts.

[0135] 4) Visual configuration of cloud platform business data processing scripts. The cloud platform development business data processing script management interface (request sending interface), as Figure 11 shown in the figure, when the business data receiving interface fails to query from the business data processing configuration table for the business data processing script that matches the target production end identifier and target changed business data, the business data processing script management interface can send a script addition request to the business data processing script management page, requesting the target account to create a business data processing script for the changed business information that comes from the target production end identifier and belongs to the target changed business data. After the target account returns the reference business data processing script (reference synchronization script), the reference business data processing script is stored, the corresponding relationship among the target production end identifier, target changed business data, and reference business data processing script is created, and this corresponding relationship is stored in the business data configuration table. Also, the configuration table and the reference business data processing script are synchronously cached in the business data processing cache script of the cloud platform. The business data receiving interface can directly obtain from the business data processing cache script the corresponding relationship among the target production end identifier, target changed business data, and reference business data processing script, and call the reference business data processing script. Before storing the reference business data processing script, the business data processing script management interface also provides: a function to trial-run the reference business data processing script, where the user can click the "Trial Run" button to test whether the function of the reference business data processing script meets the expectations; a function to detect dangerous instructions in the script, such as using text analysis and sandbox technology to verify the reference business data processing script; and a function to encrypt the script to prevent the reference business data processing script from being tampered with or leaked.

[0136] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method.

[0137] Based on such understanding, the technical solution of this application, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.

[0138] Specific examples in this embodiment can refer to the examples described in the above embodiments and exemplary embodiments, and will not be repeated here.

[0139] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0140] The above has introduced in detail a service data transmission system provided by this application. Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A business data transmission system, characterized in that, Including: A data application end and multiple data production ends. Multiple said data production ends are all connected to an information interface on the data application end. Each said data production end stores the service data required for the data application end to run the corresponding service. A target data production end among multiple said data production ends is used to detect the change information of a first data set storing target service data. Wherein, the change information is used to record the service data that has changed in the target service data, the change type of the service data that has changed, and the service type of the service data that has changed. When detecting the reference change information, convert the information format of the reference change information into a target information format recognizable by the data application end to obtain target change information. Send the target change information carrying the target production end identifier to the information interface. The data application end is used to receive the target change information carrying the target production end identifier through the information interface. Match the target production end identifier and the target synchronization script corresponding to the target service type recorded in the target change information from the production end identifier, service type, and synchronization script with a corresponding relationship. Wherein, the target synchronization script is used to indicate the synchronization operation required to execute the changed service data in the target change information. Invoke the target synchronization script to synchronize the changed service data to a second data set corresponding to the target production end identifier. Run the service according to the service data in the second data set.

2. The system according to claim 1, wherein The target data production end includes: an information collector, a format converter, an information pusher, and a target interface. The information collector is connected to the information pusher through the format converter, and the information pusher is connected to the information interface through the target interface. Wherein, The information collector is used to detect the change information of a first data set storing target service data. When detecting the reference change information, send a format conversion request carrying the reference change information to the format converter. Wherein, the format conversion request is used to request converting the information format of the reference change information into an information format recognizable by the data application end. The format converter is used to respond to the format conversion request, convert the information format of the reference change information into the target information format to obtain the target change information, and transmit the target change information to the information pusher. The information pusher is used to add the target production end identifier to the target change information, and invoke the target interface to send the target change information carrying the target production end identifier to the information interface.

3. The system according to claim 2, wherein An information collection tool and a collection configuration tool are run on the information collector. Wherein, The collection configuration tool is used to configure the collection parameters currently used by the information collection tool. Wherein, the collection parameters are used to indicate the change types allowed to be collected by the information collection tool. The information collection tool is used to detect the change information of the first data set according to the target collection parameters configured by the collection configuration tool; and in the case of detecting the reference change information, send a format conversion request carrying the reference change information to the format converter, where the reference change information records the reference service data that has changed in the target service data, the reference change type of the reference service data, and the reference service type of the reference service data, and the reference change type is the change type indicated by the target collection parameters.

4. The system according to claim 3, wherein The information collection tool is used to detect the change information of the first data set according to the target collection parameters configured by the collection configuration tool through the following steps: Read the change record file corresponding to the first data set, where the change record file records the change information in the first data set; Detect the change information in the change record file according to the target collection parameters.

5. The system according to claim 2, wherein A first script call tool is run on the format converter, and the first script call tool is connected to the information collector, where The first script call tool is used to detect the reference change type recorded in the reference change information carried by the format conversion request sent by the information collector; When the reference change type is the new type, respond to the format conversion request, and call the new conversion script to convert the information format of the reference change information into the target information format recognizable by the data application end, where the new conversion script allows converting the information format recording the change information of the new type into the target information format recognizable by the data application end; When the reference change type is the modification type, respond to the format conversion request, and call the modification conversion script to convert the information format of the reference change information into the target information format recognizable by the data application end, where the modification conversion script allows converting the information format recording the change information of the modification type into the target information format recognizable by the data application end; When the reference change type is the deletion type, respond to the format conversion request, and call the deletion conversion script to convert the information format of the reference change information into the target information format recognizable by the data application end, where the deletion conversion script allows converting the information format recording the change information of the deletion type into the target information format recognizable by the data application end.

6. The system according to claim 2, wherein The information pusher also runs an identifier reading tool and an information splicing tool, where The identifier reading tool is used to read the target production end identifier of the target data production end; The information splicing tool is used to receive the target change information from the format converter; splice the target production end identifier read by the identifier reading tool to the target change information; and call the target interface to send the target change information spliced with the target production end identifier to the information interface.

7. The system according to claim 1, wherein the data application end further includes: a data synchronizer, the data synchronizer is connected to the information interface, wherein the data synchronizer is used to screen out the second data set corresponding to the target production end identifier from the production end identifier and the data set with a corresponding relationship; synchronize the change service data recorded in the target change information to the second data set according to the target production end identifier and the target service type recorded in the target change information.

8. The system according to claim 7, wherein the data synchronizer runs a set matching tool and a second script calling tool, the set matching tool and the second script calling tool are connected, and the second script calling tool is connected to the information interface, wherein the set matching tool is used to screen out the second data set corresponding to the target production end identifier from the production end identifier and the data set with a corresponding relationship; the second script calling tool is used to match the target synchronization script corresponding to the target production end identifier and the target service type from the production end identifier, service type and synchronization script with a corresponding relationship, wherein the target synchronization script is used to indicate the synchronization operation required to be performed on the change service data recorded in the target change information; and call the target synchronization script to synchronize the change service data to the second data set.

9. The system according to claim 8, wherein the second script calling tool is further used for: in the case where the target synchronization script corresponding to the target production end identifier and the target service type cannot be matched from the production end identifier, service type and synchronization script with a corresponding relationship, prohibit synchronizing the change service data recorded in the target change information to the second data set, and generate a script addition request carrying the target production end identifier and the target service type, wherein the script addition request is used to request the target account to create a synchronization script for the change service data recorded in the target change information.

10. The system according to claim 9, wherein a request sending interface is further deployed in the data application end, and the request sending interface is connected to the second script calling tool, wherein the request sending interface is used to send the script addition request generated by the second script calling tool to the target account; receive the target production end identifier, the target service type and the reference synchronization script with a corresponding relationship created by the target account in response to the script addition request; and add the target production end identifier, the target service type and the reference synchronization script with a corresponding relationship to the production end identifier, service type and synchronization script with a corresponding relationship.

Images