Data processing method and system

Abstract

The invention discloses a data processing method and a system comprising a data distribution node and a plurality of data processing nodes, and the data processing nodes perform data processing on first data processed by an upstream data processing node based on corresponding data processing logic to obtain second data. And the data distribution node determines a target data processing node and corresponding target data according to the data distribution strategy, and distributes the target data to the target subscriber. Thus, the multiple data processing nodes adopt an upstream and downstream chain type data processing mode, subsequent processing can be carried out based on the data processed by the upstream data processing nodes, when data processing logic configuration is carried out on the data processing nodes, only processing logic for the upstream data of the nodes needs to be configured, the configuration workload is reduced, and the configuration efficiency is improved. And the probability of data processing exception is reduced. Moreover, the data distribution node can distribute the data processed by any data processing node based on the data distribution strategy, so that the data distribution is more flexible and efficient.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method and system. Background Art

[0002] In the data subscription technology based on distributed architecture, the data subscription system includes a data processing module and a subscriber. The data processing module obtains the source data, processes the source data based on the subscription requirements of the subscriber, and distributes the processed data to the corresponding subscriber. Since the source of the source data is different, the data format will be different, and the data type or data format subscribed by different subscribers may be different, so it is necessary to perform data processing such as format conversion on the data before distributing the processed data to the subscriber.

[0003] Currently, for data processing, at least one data processing module is deployed, and the data processing logic of each data processing module is configured so that each data processing module obtains source data, processes the source data based on the configured data processing logic, and then sends the processed target data to the corresponding subscriber. Assume that there are two data processing modules, for example, data processing module A converts the source data into data A as target data and sends the target data to subscriber A, and data processing module B converts the source data into data A, and then converts data A into data B as target data and sends the target data to subscriber B. Therefore, as long as a new method of data processing is required, or data needs to be distributed to new subscribers, a new data processing module needs to be added, and the entire data processing logic of converting from source data to target data needs to be configured for the new data processing module, that is, the entire data processing logic of converting the source data into data A, then converting data A into data B, and then converting data B into other formats to obtain the target data needs to be configured. Therefore, in the prior art, in order to configure the data processing logic for a newly added data processing module, it is necessary to configure the entire data processing logic that converts source data into target data. On the one hand, this increases the workload for users, and on the other hand, it is easy for data processing logic configuration errors to occur, resulting in data processing abnormalities, which is not conducive to data processing. Summary of the invention

[0004] The present application provides a data processing method and system for solving the problem in the prior art that the data processing logic configuration for a newly added data processing module requires the configuration of the entire data processing logic for converting source data into target data, which on the one hand increases the workload for users and on the other hand is prone to data processing logic configuration errors, resulting in data processing abnormalities and being unfavorable for data processing.

[0005] Among them, a chain data processing method is adopted, so that the data processing node performs subsequent processing based on the data processed by the upstream data processing node, so that when configuring the data processing logic for the data processing node, it is only necessary to configure the processing logic that converts the first data processed by the upstream processing node into the second data, and there is no need to configure the entire data processing logic from the source data to the second data, thereby reducing the workload of data processing logic configuration, thereby reducing the configuration error rate of data processing logic, and reducing the probability of data processing anomalies. In addition, by configuring the data distribution node, it is possible to obtain the data processed by any data processing node and distribute the data to any subscriber, making data processing and data distribution more flexible and efficient.

[0006] To solve the above technical problems, on the first aspect, an implementation method of the present application discloses a data processing method, which is applied to a data processing system, wherein the data processing system includes a data distribution node and multiple data processing nodes, and the method includes: the data processing node processes first data based on corresponding data processing logic to obtain second data, wherein the first data is obtained by the data processing node from an upstream data processing node based on the corresponding data processing logic, and is data processed by the upstream data processing node based on the corresponding data processing logic; the data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, and distributes the target data to the target subscriber, wherein the data distribution strategy is obtained based on the subscription requirements of the subscriber.

[0007] The data processing method provided by the implementation of the present application configures a data distribution node and multiple data processing nodes, so that the data processing node can obtain the first data processed by the upstream data processing node based on the corresponding data processing logic based on the corresponding data processing logic, and then performs data processing based on its own data processing logic to obtain the processed second data. In this way, a chain data processing method is adopted, so that the data processing node performs subsequent processing based on the data processed by the upstream data processing node, so that when the data processing logic is configured for the data processing node, it is only necessary to configure the processing logic that converts the first data processed by the upstream data processing node into the second data, and there is no need to configure the entire data processing logic from the source data to the second data, thereby reducing the workload of data processing logic configuration, thereby reducing the configuration error rate of the data processing logic, and reducing the probability of data processing anomalies. In addition, by configuring the data distribution node, it is possible to obtain the data processed by any data processing node and distribute the data to any subscriber, making data processing and data distribution more flexible and efficient.

[0008] Furthermore, the data distribution node can distribute the data (first data or second data) processed by any data processing node to the target subscriber according to the data distribution strategy. In this way, by configuring multiple data processing nodes and corresponding data processing strategies, different data processing nodes can be used to perform different data processing on different data. Furthermore, by configuring the data distribution node, data acquisition after processing by any data processing node and data distribution to any subscriber can be achieved, making data processing and data distribution more flexible and efficient.

[0009] According to another specific implementation of the present application, an implementation of the present application discloses a data processing method, wherein the data processing system includes a management center and a configuration center, the configuration center stores a configuration file, and the configuration file is generated based on the user's data processing needs. The method also includes: the management center creates a data processing node according to the configuration file stored in the configuration center, and enables the data processing node to obtain corresponding data processing logic based on the configuration file.

[0010] By adopting the above technical solution, a new data processing node is created based on the configuration file, and the data processing logic of the data processing node is configured, so that the configuration of the data processing node is simpler and more efficient.

[0011] According to another specific implementation of the present application, the data processing method disclosed in the implementation of the present application, the configuration file includes a data query script and a data verification script, and the data processing node processes the first data based on the corresponding data processing logic to obtain the second data, including: the data processing node obtains the first data from the upstream data processing node according to the data query script; the data processing node verifies the legitimacy of the first data according to the data verification script, and when it is determined that the first data is legal data, processes the first data to obtain the second data.

[0012] By adopting the above technical solution, the data query script can be configured to enable the data processing node to obtain the first data in the upstream data processing node based on the user's data processing needs, so that the data acquisition method can meet the user's needs. And after the data is verified for legitimacy by configuring the data verification script, the data is processed to ensure the security of data processing.

[0013] According to another specific implementation of the present application, the data processing method disclosed in the implementation of the present application, the configuration file also includes a data batch update script, and the method also includes: the data processing node updates the second data to the storage medium corresponding to the data processing node according to the data batch update script, so that the downstream data processing node of the data processing node obtains the second data from the storage medium.

[0014] By adopting the above technical solution, the second data can be stored in the storage medium by configuring the data batch update script, which facilitates the tracing of the data processing process.

[0015] According to another specific implementation of the present application, the data processing method disclosed in the implementation of the present application, the data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, and distributes the target data to the target subscriber, including: the data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, and determines the target message middleware according to the data distribution strategy, and distributes the target data to the target subscriber based on the target message middleware.

[0016] By adopting the above technical solution, the target data is distributed to the target subscriber based on the target message middleware, which can make data distribution more efficient.

[0017] According to another specific implementation of the present application, the data processing method disclosed in the implementation of the present application, the data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, including: the data distribution node determines the target data processing node from the data processing nodes according to the data distribution strategy, and takes the first data or the second data corresponding to the target data processing node as the target data; or the data distribution node determines the target data processing node from the data processing nodes according to the data distribution strategy, and processes the first data or the second data corresponding to the target data processing node based on the corresponding data processing logic to obtain the target data.

[0018] By adopting the above technical solution, the data distribution node distributes the first data or the second data as the target data to the target subscriber based on the data distribution strategy, or processes the first data or the second data based on the corresponding data processing logic to obtain the target data, and distributes the target data to the target subscriber, so that data distribution can be more flexible based on the user's data processing requirements and distribution requirements, and better meet user needs.

[0019] According to another specific implementation of the present application, the data processing method disclosed in the implementation of the present application also includes: the management center deletes the data processing nodes that do not have corresponding downstream data processing nodes according to the user's data processing requirements.

[0020] By adopting the above technical solution, the management center can delete the data processing nodes according to the user's data processing needs, so that the data processing link can be configured on demand and better meet the user's needs.

[0021] According to another specific implementation of the present application, in the data processing method disclosed in the implementation of the present application, the first data and the second data are structured data obtained based on the database change data capture technology, and the data processing is data format conversion processing.

[0022] According to another specific implementation of the present application, the data processing method disclosed in the implementation of the present application, a data processing model is deployed in a data processing node, and the data processing node processes the first data based on the corresponding data processing logic to obtain the second data, including: the data processing node calls the data processing model to process the first data based on the corresponding data processing logic to obtain the second data.

[0023] By adopting the above technical solution, the data processing node calls the data processing model to process the first data based on the corresponding data processing logic to obtain the second data, making the data processing method based on the data processing model safer and more efficient.

[0024] On the second aspect, the implementation method of the present application also discloses a data processing system, characterized in that the data processing system includes a data distribution node and multiple data processing nodes, wherein the data processing node is used to process first data based on corresponding data processing logic to obtain second data, the first data is obtained by the data processing node from an upstream data processing node based on the corresponding data processing logic, and is data processed by the upstream data processing node based on the corresponding data processing logic; the data distribution node is used to determine the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, and distribute the target data to the target subscriber, wherein the data distribution strategy is obtained based on the subscription requirements of the subscriber.

[0025] In a third aspect, an implementation method of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement a data processing method provided in any one of the implementation methods of the first aspect above.

[0026] In a fourth aspect, an implementation of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements a data processing method as provided in any one of the implementations of the first aspect above.

[0027] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can also be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a structural diagram of a data processing system provided in an embodiment of the present application;

[0029] Figure 2 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0030] Figure 3 It is a schematic diagram of the principle of a data processing method provided in an embodiment of the present application;

[0031] Figure 4 It is a logical schematic diagram of a data processing method provided in an embodiment of the present application;

[0032] Figure 5 It is a schematic diagram of the principle of a data processing system executing a data processing method provided in an embodiment of the present application;

[0033] Figure 6 It is a timing diagram of the data processing method provided in the embodiment of the present application;

[0034] Figure 7 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] At present, the data subscription and distribution technology based on distributed architecture mainly collects data, converts the data format, and distributes data to corresponding subscribers for customized development of downstream businesses. If the distributed data needs to be supplemented or monitored, customized embedding points are required for data collection, which will cause great intrusion to the entire data distribution business.

[0036] As mentioned above, in the data subscription technology based on the distributed architecture, the data subscription system includes a data processing module and a subscriber. The data processing module processes the data based on the subscription requirements of the subscriber and distributes the processed data to the corresponding subscriber. However, different subscribers may subscribe to different data types or data formats, so the data needs to be processed such as format conversion before the processed data is distributed to the subscriber.

[0037] In the prior art, the data processing module is written based on code so that the data processing module implements the corresponding data processing logic. However, for downstream subscribers, if subscription data needs to be added, it is necessary to modify the configuration code of the data processing module, thereby modifying the data processing logic of the data processing module, which increases the risk of service exception scenarios (such as data processing exceptions).

[0038] In addition, due to the different sources of data, the formats of the collected data may be diverse. The data processing logic configuration for converting source data into target data is performed for each data processing module, which brings great complexity to the orderly processing of data in the distributed system.

[0039] Furthermore, currently, for data processing, at least one data processing module is deployed, and the data processing logic of each data processing module is configured so that each data processing module obtains the source data, processes the source data based on the configured data processing logic, and then sends the processed target data to the corresponding subscriber. Assume that there are two data processing modules, for example, data processing module A converts the source data into data A and uses it as target data, and sends the target data to subscriber A, and data processing module B converts the source data into data A, and then converts data A into data B and uses it as target data, and sends the target data to subscriber B. That is, the current data processing and data push distribution are all customized one-to-one. That is, the prior art usually has each data processing module in the cloud server perform separate logical processing and distribution on the source data, or distributes and filters the processed data separately before sending it out. However, if a new method of data processing is required, or if there is a new subscriber who needs to subscribe to a data format that is different from the data format subscribed by other subscribers, it is necessary to configure a new data processing module and configure the data processing logic of the data processing module so that the data processing module performs data processing of the corresponding data processing logic on the source data based on the corresponding data processing logic, and then distributes the processed target data to the new subscriber. In other words, as long as a new method of data processing is required, or data needs to be distributed to new subscribers, a new data processing module needs to be added, and the entire data processing logic for converting the source data into the target data needs to be configured, that is, the entire data processing logic for converting the source data into data A, then converting data A into data B, and then converting data B into other formats to obtain the target data needs to be configured.

[0040] Therefore, in the prior art, in order to configure the data processing logic for a newly added data processing module, it is necessary to configure the entire data processing logic that converts source data into target data. On the one hand, this increases the workload for users, and on the other hand, it is easy for data processing logic configuration errors to occur, resulting in data processing abnormalities, which is not conducive to data processing.

[0041] Based on this, the present application proposes a data processing method and system, which adopts a chain data processing method, so that data processing and processed data are always circulated in the chain data flow. When configuring the data processing logic for the data processing node, it is only necessary to configure the processing logic for converting the first data processed by the upstream data processing node into the second data, and there is no need to configure the entire data processing logic from the source data to the second data, thereby reducing the workload of data processing logic configuration, thereby reducing the configuration error rate of the data processing logic, and reducing the probability of data processing anomalies. And by configuring the data distribution module based on the subscription needs of the downstream subscriber, it is convenient to send the subscription data, and it is also convenient to select the data processing node for subscription.

[0042] Next, the data processing method and system of the present application are described in detail.

[0043] See also Figure 1 The data processing system provided by the implementation method of the present application includes a data distribution node and multiple data processing nodes.

[0044] Among them, the data processing node is used to process the first data based on the corresponding data processing logic to obtain the second data, and the first data is obtained by the data processing node from the upstream data processing node based on the corresponding data processing logic, and is the data processed by the upstream data processing node based on the corresponding data processing logic.

[0045] The data distribution node is used to determine the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, and distribute the target data to the target subscriber, wherein the data distribution strategy is obtained based on the subscription requirements of the subscriber.

[0046] Furthermore, in the implementation of the present application, the data processing system includes a management center and a configuration center. The configuration center is used to store configuration files, and the configuration files are generated based on the user's data processing needs.

[0047] Exemplarily, configuration files generated by users based on data processing requirements are stored in the configuration center of the data processing system.

[0048] The management center is used to create data processing nodes according to the configuration files stored in the configuration center, and enable the data processing nodes to obtain corresponding data processing logic based on the configuration files.

[0049] Exemplarily, the management center is used to manage all data processing nodes, and to create new data processing nodes based on configuration files, so that the data processing nodes obtain corresponding data processing logic based on the configuration files.

[0050] Therefore, in the implementation of the present application, the management center is also used to modify the data processing logic of the corresponding data processing node according to the configuration file. In this way, the data processing node is configured based on the configuration file without the need for code writing, making the creation of the data processing node and the corresponding data processing logic configuration more concise and efficient.

[0051] Further, see Figure 2 The data processing method provided by the implementation of this application is applied to Figure 1 The data processing system shown can be deployed in a cloud server, so the data processing system provided by the implementation method of the present application can be applied to a cloud server, and specifically includes the following steps.

[0052] S100, the data processing node processes the first data based on the corresponding data processing logic to obtain the second data, where the first data is obtained by the data processing node from the upstream data processing node based on the corresponding data processing logic and is the data processed by the upstream data processing node based on the corresponding data processing logic.

[0053] S200, the data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the distribution strategy, and distributes the target data to the target subscriber, wherein the distribution strategy is obtained based on the configuration operation performed by the user on the data distribution node according to the subscription requirements of the subscriber.

[0054] It should be noted that there is no restriction on the execution order of step S100 and step S200. Step S100 may be executed first and then step S200, or step S200 may be executed first and then step S100, or step S100 and step S200 may be executed simultaneously.

[0055] In an implementation of the present application, the first data and the second data are structured data obtained based on a database change data capture (CDC) technology.

[0056] For example, Figure 3 As shown, data processing node A serves as the upstream data processing node of data processing node B and data processing node D, data processing node B serves as the downstream data processing node of data processing node A, and data processing node D serves as the downstream data processing node of data processing node A.

[0057] Data processing node A obtains a driving data source (i.e., source data), processes the source data based on the corresponding data processing logic 1, and obtains data in format A (as an example of second data); data processing node B obtains format A data (as an example of first data) from data processing node A based on the corresponding data processing logic 2, processes the format A data based on the corresponding data processing logic 2, and obtains format B data (as another example of second data); data processing node D obtains format A data (as an example of first data) from data processing node A based on the corresponding data processing logic 3, processes the format A data based on the corresponding data processing logic 3, and obtains format D data (as another example of second data).

[0058] In the implementation of the present application, the data processing node is mainly used to perform corresponding data format conversion processing according to its own data processing logic configuration to generate driving data (i.e., first data) for the downstream data processing node. Therefore, each data processing node only needs to be responsible for its own data output and does not need to pay attention to the data flow in the entire data link.

[0059] It should be noted that the driving data source obtained by the data processing node A can be specifically the CDC log data generated based on the CDC technology obtained from the database when there is data saved in the database, that is, a structured data, which serves as the driving data source of the data processing node. Therefore, there is a node in the database that obtains the driving data source, which serves as the upstream data processing node of the data processing node A.

[0060] Furthermore, in the implementation of the present application, the data processing node obtains the first data from the upstream data processing node based on the corresponding data processing logic. Specifically, it may be obtaining all the first data processed by the upstream data processing node, or it may be obtaining part of the first data processed by the upstream data processing node. The specific details can be determined according to the data processing logic configured based on the user's data processing requirements.

[0061] In one implementation of the present application, data processing is data format conversion processing. That is, source data, format A data, format B data, and format C data can specifically be data of different formats. Of course, data processing can also be other, such as data type conversion processing (that is, source data, format A data, format B data, and format C data are specifically data of different data types), or data screening processing (that is, source data includes format A data, format A data includes format B data, and format B data includes format C data), etc.

[0062] In this way, the entire data flow continuously generates and updates data due to the input of the driving data source, and the data will generate data in different formats when passing through each data processing node (because the data processing logic configuration of each data processing node is different, the output data format and data structure will also be different).

[0063] Furthermore, in the implementation of the present application, users can write configuration files based on data processing requirements and store the configuration files in a configuration center.

[0064] When the management center detects that there is a new configuration file in the configuration center, it creates a data processing node according to the new configuration file and enables the data processing node to obtain the corresponding data processing logic based on the configuration file.

[0065] For example, if data processing in other formats is required, a new data processing node needs to be created, such as Figure 3As shown, the user writes a configuration file based on data processing requirements and stores the configuration file to the configuration center. The management center creates a data processing node C according to the configuration file. Data processing node C is the downstream data processing node of data processing node D (that is, data processing node D is the upstream data processing node of data processing node C). Data processing node C obtains format D data from data processing node D based on the corresponding data processing logic 4, and then processes the format D data based on the corresponding data processing logic 4 to obtain format C data.

[0066] Furthermore, similar to the logic of adding a new data processing node, the user can edit and modify the configuration file of each data processing node, and save the modified configuration file in the configuration center after modification. When the management center detects that there is a modified configuration file in the configuration center, it modifies the data processing logic of the corresponding data processing node according to the modified configuration file, so that the corresponding data processing node performs corresponding data processing based on the modified data processing logic. For example, if the user modifies the configuration file of data processing node A based on data processing requirements, the management center will update the data processing logic 1 of data processing node A according to the modified configuration file, so that data processing node A performs data processing based on the modified data processing logic 1.

[0067] It should be noted that if the data processing node to be modified has a downstream data processing node (that is, a non-leaf node), then when the corresponding configuration file is modified, only new data processing logic can be added. For example, the data processing node can only add the data field that needs to be obtained (that is, the first data) for data processing, and the processed data field (that is, the second data), but the original data field cannot be deleted, that is, the original data processing logic must be retained.

[0068] Furthermore, in the implementation of the present application, the configuration file includes a data query script and a data verification script.

[0069] In one implementation of the present application, a data processing node processes first data based on corresponding data processing logic to obtain second data, including: the data processing node obtains the first data of an upstream data processing node according to a data query script.

[0070] Exemplarily, based on data processing requirements, the user determines that corresponding data processing needs to be performed based on the first data output by the upstream data processing node, and then writes a data query script, which indicates that the data output by the upstream data processing node can be obtained as driving data to obtain business data from the structured database (that is, to obtain the previous node driving data), and executes the driving data based on the data query script to obtain a query result set, that is, to obtain part of the data or all of the data based on the data query script (as different examples of the first data).

[0071] Furthermore, a data verification script is written, and the data processing node verifies the legitimacy of the first data according to the data verification script. When it is determined that the first data is legal data, the first data is processed to obtain the second data.

[0072] Exemplarily, a data verification script is written so that the data processing node performs data legitimacy verification on the obtained result set data (i.e., the first data) based on the data verification script, so that when the first data is determined to be legal data, the first data is processed according to the data verification script to obtain the target structured data (i.e., the second data). If the business party (i.e., the subscriber) subscribes to the interface of this data processing node according to the customized configuration of the business party, the distribution module obtains the data distribution strategy according to the customized configuration of the business party, determines the data processing node subscribed by the business party according to the data distribution strategy (as an example of the target data processing node), and after screening or reconstructing the structured data (i.e., the second data) corresponding to the target data processing node (i.e., determining the target data corresponding to the target data processing node), the target data is sent to the business party.

[0073] In one implementation of the present application, the configuration file also includes a data batch update script, and the data processing node updates the second data to the storage medium of the data processing node according to the data batch update script, so that the downstream data processing node of the data processing node obtains the second data from the storage medium.

[0074] Exemplarily, the data processing node persists the target structured data (i.e., the second data) according to the data batch update script to store the target structured data in the storage medium of the database, so that the target structured data is a CDC data stream (i.e., log data) as the driving data of the next data processing node (i.e., the downstream data processing node).

[0075] Therefore, in the implementation of this application, Figure 4As shown, for each data processing node, a single configuration of a data query script (also known as a query script), a data verification script, and a data batch update script (also known as a batch update script) can be performed to hot deploy the generated configuration file script to the data processing system. The processing logic of the data processing node engine is to load the configuration file script, read the driving data of the previous data processing node (also known as the data processed by the upstream data processing node), execute the driving data based on the data query script to obtain the query result set (that is, the first data is obtained based on the corresponding data processing logic), verify the legitimacy of the result data set data based on the data verification script, and generate the target structured data (that is, the second data) after the corresponding data processing (for example, the target structured data is obtained based on the code {"id":"test","is_valid":"1"}). The target structured data is a list of fields that can be subscribed to by the subscriber. If the business party (that is, the subscriber) customizes the data according to the business party's Configure the interface for subscribing to this data processing node (for example, based on the code api: "(interface name)", the data subscription of the corresponding data processing node is performed), and the distribution module obtains the data distribution strategy according to the customized configuration of the business party, so as to determine the data processing node subscribed by the business party according to the data distribution strategy (as an example of the target data processing node), and after screening or reconstructing the structured data corresponding to the target data processing node (that is, the second data) (that is, determining the target data corresponding to the target data processing node), the target data is sent to the business party (for example, based on the code {"api": "(interface name)", "data": [{"id": "test", "is_valid": "1"}]}). Further, the target structured data (that is, the second data) is persisted according to the data batch update script to store the target structured data in the storage medium of the database, so that the target structured data is a CDC data stream (that is, log data) as the driving data of the next data processing node (that is, the downstream data processing node).

[0076] Therefore, in the implementation of this application, the data processing node can generate a list of fields that can be subscribed by the business party by analyzing the data validation script (the list is automatically generated as the single configuration is hot-deployed in the data processing system), and the system portal can show the business party the data fields that the data processing node can subscribe to, so that the subscriber can subscribe to the data of the corresponding data structure. For example, a list of fields that the business subscriber needs to subscribe to can be generated based on the following code:

[0077]

[0078]

[0079] Push configuration includes data query scripts and data batch update scripts. By configuring these scripts, decoupling from business logic can be achieved.

[0080] Furthermore, in the implementation of the present application, data processing nodes can be created according to data processing requirements.

[0081] Exemplarily, taking the creation of data processing node C as an example, the user determines based on data processing requirements that corresponding data processing needs to be performed based on format D data (as an example of first data) output by data processing node D, and then writes a data query script, which indicates that format D data can be used as driving data to obtain business data from a structured database.

[0082] Furthermore, a data verification script and a data batch update script are written to generate a configuration file script. The completed configuration file script is stored in the public configuration center of the data processing system. The management center of the data processing system will automatically identify and load the configuration file script, and create a data processing node C, so that the data processing node C performs corresponding data processing on the format D data (as another example of the first data) based on the corresponding data verification script to obtain the format C data (as another example of the second data).

[0083] Furthermore, if subscriber user C wants to subscribe to format C data, he can see the data fields that data processing node C can subscribe to (including field attributes and field definitions of data processing node C), and thus can choose to subscribe to data processing node C.

[0084] Then the user writes a subscription configuration, which declares which data fields user C needs, and declares which message middleware platform (that is, including the message middleware address) user C needs the data processing system to send data to. User C stores the subscription configuration in the configuration center of the data distribution module, so that the data distribution module loads the subscription configuration to obtain a data distribution strategy, connects to data processing node C based on the data distribution strategy, distributes format C data corresponding to data processing node C to the target message middleware, and sends format C data to user C based on the target message middleware.

[0085] Furthermore, in the implementation of the present application, if a data processing node does not have a corresponding downstream data processing node (that is, the data processing node is a leaf node), the management center deletes the data processing node according to the user's data processing requirements.

[0086] That is, if the data processing node to be deleted has downstream data processing nodes (i.e. non-leaf nodes), deletion is not allowed. If the data processing node to be deleted does not have downstream data processing nodes (i.e. leaf nodes), the configuration file can be deleted. When the management center cannot read the configuration file of the data processing node, the processing thread of the data processing node will be revoked to delete the data processing node.

[0087] Furthermore, in the implementation method of the present application, a data processing model is deployed in the data processing node, and the data processing node processes the first data based on the corresponding data processing logic to obtain the second data, including: the data processing node calls the data processing model to process the first data based on the corresponding data processing logic to obtain the second data.

[0088] Exemplarily, corresponding data processing logic is executed based on the data processing model, first data is obtained from an upstream data processing node, and data processing is performed on the first data to obtain second data.

[0089] Furthermore, in an implementation of the present application, since the subscriber can configure the address of the message middleware in the subscription configuration, in an implementation of the present application, the data distribution node can obtain the data distribution strategy for the subscriber based on the subscription configuration.

[0090] That is, in the implementation method of the present application, the data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, and distributes the target data to the target subscriber, including: the data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, and determines the target message middleware according to the data distribution strategy, and distributes the target data to the target subscriber based on the target message middleware.

[0091] Exemplarily, the data distribution strategy includes subscription configurations for multiple subscribers. Thus, based on the subscription configurations of each subscriber, the target message middleware is determined to obtain the data required by the subscriber (i.e., target data) from the target data processing node, and the target data is distributed to the target subscriber based on the target message middleware.

[0092] Furthermore, in one implementation of the present application, the data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, including: the data distribution node determines the target data processing node from the data processing nodes according to the data distribution strategy, and takes the first data or second data corresponding to the target data processing node as the target data.

[0093] Exemplarily, the data distribution node subscribes to the data of the data processing node based on the data distribution strategy to determine the target data processing node according to the data distribution strategy, takes the first data or the second data (partial data or all data) corresponding to the target data processing node as the target data, and distributes the target data to the target subscriber based on the target message middleware.

[0094] In another implementation of the present application, the data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, including: the data distribution node determines the target data processing node from the data processing nodes according to the data distribution strategy, and processes the first data or the second data corresponding to the target data processing node based on the corresponding data processing logic to obtain the target data.

[0095] Exemplarily, the data distribution node can also obtain the characteristic configuration of the data to be distributed based on the corresponding data processing logic, and judge the data to be distributed. If it is data that meets the push rule, it will bridge the message middleware or the message middleware such as webSocket according to the corresponding data distribution strategy, and send the target data to the downstream destination (that is, the subscriber). Of course, it is also possible to restructure the first data or the second data and send it according to the special definition of the data format by the target subscriber, that is, before the data is sent, the data structure is converted based on the data processing rules to obtain the target data, and the target data is distributed to the target subscriber based on the target message middleware.

[0096] It should be noted that if the target data processing node is an upstream data processing node, the target data is the first data, and if the target data processing node is a downstream data processing node, the target data is the second data. According to the present application, the first data of the upstream data processing node, that is, the second data obtained after data processing of the acquired data, is also the first data acquired by the downstream data processing node.

[0097] The data processing method and system provided by the implementation of the present application are actually a data distribution method based on the CDC (Change Data Capture) technology of structured data storage in a database, and a cloud computing system for realizing dynamic data distribution. By obtaining database structured data as a driving data stream, a cluster system of chain data processing and distribution composed of multiple processing models / programs (i.e., data processing nodes) is adopted. First, a data stream is generated as a driving data stream according to the structured data CDC, and then each data processing model engine in the data processing system (as an example of a data processing node) reads the corresponding data processing logic configuration, and after running, the structured data is processed based on the CDC of the database to obtain the processed data. Among them, the processing logic of each data processing model engine may involve operations such as querying the database and converting data types. In this way, the data processing model engine loaded with the data processing logic configuration obtains the target structured data (i.e., the second data) after processing the CDC data stream. The data processing node will add and update the target structured data to the storage medium according to the data batch update script in the configuration file script. Therefore, in the process of triggering persistence, the change of the storage medium will be triggered, and the target structured data in the storage medium will flow to the next data processing node (that is, the downstream data processing node) as the data source, and serve as the data source of the next data processing node (that is, the first data), so that the downstream data processing node can perform data format conversion processing based on the corresponding data processing logic configuration to obtain data in a new format (that is, the second data). By connecting multiple data processing nodes in series in this way, a structured data configuration distribution system can be formed, and the downstream business party can subscribe to the data of any data processing node, so that the data distribution node obtains the target data of the target data processing node and distributes it to the corresponding business party, so that the business party can subscribe to the desired target structured data.

[0098] In this way, the data-driven processing method is a chain structure. In the process of data format conversion and push, the processing model / program of each data processing node only needs to care about the data format conversion at this step. Through the assembly of processing models / programs of multiple data processing nodes, multiple data processing flow networks are finally formed. While ensuring data consistency among all computing power clusters, it achieves decoupling from the business system that generates the driving data source, and supports functions such as high availability, dynamic configuration, and hot deployment of data processing nodes, thereby improving the efficiency of internal data processing in the enterprise.

[0099] In another implementation of the present application, Figure 5As shown, the data processing system includes a data flow system (that is, a data cluster) and a distribution module. The data flow system includes multiple data processing modules. The data processing module 1 obtains the driving data source as the CDC data source (as an example of the first data) based on the corresponding configuration file 1, and obtains the second data after performing corresponding data processing, and determines whether user subscription distribution is required. If user subscription distribution is required, the second data is sent to the distribution module. If user subscription distribution is not required, the second data is stored in the corresponding storage medium so that the second data serves as the CDC data flow of the data processing module 2. The data processing module 1 obtains the driving data source as the CDC data source (as an example of the first data) based on the corresponding configuration file 2, and obtains the second data after performing corresponding data processing, and determines whether user subscription distribution is required. If user subscription distribution is required, the second data is sent to the distribution module. If user subscription distribution is not required, the second data is stored in the corresponding storage medium as the CDC data flow of the next data processing module.

[0100] The distribution module can subscribe to the second data of any one of the data processing modules to distribute the second data to the corresponding message middleware (such as MQ) or to the corresponding interface API.

[0101] It should be noted that the specific data processing process of the data processing module is similar to the data processing process of the aforementioned data processing node, and the specific distribution process of the distribution module is similar to the specific distribution process of the aforementioned distribution node.

[0102] Thus, in the entire chain data flow system, the downstream business party can freely subscribe to the data of one or more data processing modules, so that the distribution module distributes the data of the corresponding data processing module to the corresponding subscriber.

[0103] The implementation method of this application builds a universal data distribution system executed on a cloud server. It provides a distribution system design method for complex downstream data subscription requirements and subscription changes. This method can enable each data processing node to adopt different data processing logic configurations and assemble them into a meshed data processing system, which can push and distribute customized data to downstream businesses without the need for service restart and deployment.

[0104] Therefore, especially for the cloud services of the Internet of Vehicles, the written data often needs the support of other data (configuration data, basic data, etc.) for subsequent business development. In various cloud business services, the demand scenarios and data formats for data are different, so there are also different requirements for the format, real-time performance, transmission method, reconciliation function, abnormal retransmission function, etc. of data push. Therefore, it is necessary to process the data multiple times with different logics. By configuring multiple data processing nodes, the data can flow between multiple data processing nodes and can output the data processed by any data processing node, which is convenient for data processing and data acquisition.

[0105] The data processing method and system provided by the implementation of the present application are based on a distributed architecture, which increases the fault tolerance and scalability of data processing services and data distribution services; they can be decoupled from the business system that generates the driving data source, without affecting the business processing of the driving data source end, and the work at the data push level does not need to be considered during the development of the business system; they are not limited to the format of the driving data, and can be any form of data, which will be chain-triggered after being transmitted to the data processing system as a driving data source, thereby generating the next batch of driving data; they can push and distribute target data in near real time, so that even when the database is updated with instantaneous large batches of data, the maximum processing speed of the service load can be stably maintained and processed at a uniform speed, It can achieve the effect of shaving peaks and filling valleys. For abnormal data reconciliation during data distribution, since the pushed target data must exist in the database target table (that is, the storage medium), reconciliation is convenient. For retransmission of abnormal data, it is only necessary to re-trigger a single field in the target data table to resend the data, which is simple to implement. Each data processing node program does not need to be developed separately, and the function of each data processing node can be realized according to the configuration file. The flexibility, configurability, real-time nature, scalability, message reconciliation, and message reissue of data distribution can all be taken into account. By customizing the configuration of data flow and distributing data with the help of message middleware, the upstream and downstream system businesses are truly decoupled.

[0106] That is, the data processing method and system provided by the implementation of the present application configure the broadcast stream data source of the data (that is, configure the distribution data source) through a customized stream processing job; by decoupling from the business system of the configuration-driven data source, the business system only needs to ensure that the configuration data is available; the entire data stream is a chained data stream, which can monitor the data health and flow of each data processing node, so that the data processing results are visualized; if there is an abnormal loss of status data of the stream processing job, only the data update in the result table (that is, the storage medium) is required to supplement the lost configuration data, which is convenient for data supplementation; modifications to the configuration data can be updated to the broadcast stream in quasi-real time to ensure the real-time nature of data processing.

[0107] Furthermore, the data processing system provided by the implementation method of the present application, as a data middle platform, provides support for relational data distribution to downstream businesses; supports the development of various message middleware, http, and websocket to achieve diversified data docking; different downstream businesses can ensure data tenant isolation, and can also ensure field permission isolation to achieve data format customization; to dock with new data processing nodes, there is no need to publish the service, only to add a new configuration file to complete the distribution of downstream data, so that data processing nodes can be hot deployed; for business data with high real-time requirements, the system can achieve timely changes and timely push to ensure the real-time distribution of data.

[0108] Furthermore, in the implementation of the present application, for other CDC data stream driven data distribution methods based on database logs, for example, for non-chained data stream architectures, after being driven by database logs, a chained data processing method is adopted to implement the data format conversion and data distribution permission logic in the service code logic, and finally the target data is obtained for distribution.

[0109] Furthermore, in the implementation of this application, if Figure 6 As shown, when a new subscriber needs to be connected or a new data processing node needs to be added, the specific steps are as follows.

[0110] S310, the user / administrator queries the wide table fields that need to be subscribed.

[0111] That is, first, the user / administrator determines the wide table fields to be queried (ie, target data), and sends the wide table fields to be queried to the management center of the data processing system (ie, the distribution system Portal).

[0112] If a data field corresponding to a data processing node that meets the conditions can be found, step S320 is executed; if a data field corresponding to a data processing node that meets the conditions cannot be found, step S360 is executed.

[0113] S320, the distribution system Portal returns the matching node fields.

[0114] That is, the data field of the data processing node that can output the wide table field is returned (that is, the target data is returned).

[0115] Next, add a subscription to distribute the data stream.

[0116] S330, the user / administrator writes the subscription fields and formats and receives the Topic configuration upload.

[0117] That is, write the subscription fields and formats and receive subscription configurations such as Topics and upload them to the distribution module configuration center.

[0118] S340, the distribution module configuration is loaded into the service.

[0119] That is, the distribution module configuration center loads the subscription configuration into the distribution service.

[0120] S350, the distribution service distributes the corresponding data to the corresponding Topic.

[0121] That is, based on the distribution service, the distribution node obtains the data distribution strategy according to the subscription configuration (that is, the subscription requirements of the subscriber), obtains the target data from the target data processing node, and distributes the corresponding target data to the response Topic (that is, distributes it to the corresponding subscriber).

[0122] S360, the distribution system Portal sends a message to the user / administrator that no matching node is found.

[0123] That is, the management center of the data processing system returns that no matching node is found (that is, returns that there is no data processing node that can output the wide table field).

[0124] S370, the user / administrator edits the data engine configuration script (ie, writes the configuration file script).

[0125] S380, the user / administrator uploads the configuration file script to the data configuration center (ie, the configuration center).

[0126] S390, the data configuration center loads the configuration into the service.

[0127] That is, the data configuration center loads the configuration file script into the data processing service and generates a new data processing node, so that the data processing node performs data processing based on the data processing logic of the configuration file script to obtain data that meets the conditions.

[0128] S410, the data processing service forwards the node data to the distribution service.

[0129] That is, the data processing service forwards the data generated by the data processing node to the distribution service, so as to distribute the target data of the target data processing node to the corresponding Topic (that is, to the corresponding subscriber) based on the distribution service.

[0130] S420: The data configuration center displays the data field of the node on the distribution system portal.

[0131] That is, the data configuration center displays the data field format processed by the data processing node on the distribution system portal.

[0132] S430, the user / administrator queries the wide table fields that need to be subscribed.

[0133] That is, the user / administrator queries the wide table fields that need to be subscribed to perform data subscription of the wide table fields of the corresponding data processing node.

[0134] That is, in the implementation of the present application, when a user adds a new subscription, there is no need to start or stop the service, and the expansion and addition of the data processing node service is imperceptible to the user's use of data. When using the data processing system, the user does not need to write code, which can reduce development costs. Through the distribution system portal, you can clearly see in which data processing node the data you need can be subscribed, which can reduce the cost of use. Therefore, the data processing system provided by the implementation of the present application also includes a display interface, which can display the data fields corresponding to each data processing node, so that the user can know the data format of the second data generated after each data processing node performs the corresponding data processing, so that the subscriber can know which data processing node's data field needs to be subscribed to in order to perform subscription configuration, thereby enabling the distribution node to subscribe to the corresponding data processing node based on the subscription configuration to distribute the data processed by the data processing node to the corresponding subscriber.

[0135] The data processing method provided by the implementation method of the present application is not only applicable to cloud servers, but also to electronic devices, which are deployed with the data processing system provided by the implementation method of the present application.

[0136] The electronic equipment may specifically be vehicles, mobile phones, computers, computer clusters, cabinets and other electronic equipment. Figure 7 , Figure 7 The figure is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Figure 7 As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .

[0137] The processor 122 executes the computer execution instructions stored in the memory, so that the processor 122 executes part of the technical solutions of the data processing method in the above embodiment. The processor 122 can be a general-purpose processor, including a central processing unit CPU, a network processor (NP), etc.; it can also be a digital data processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0138] The memory 123 is connected to the processor 122 via a system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0139] By way of example and not limitation, the memory 123 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 123 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 123 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 123 is a non-volatile solid-state memory. In a particular embodiment, the memory 123 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0140] The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0141] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0142] An embodiment of the present application also provides a chip for executing instructions, which is used to execute the technical solution of the data processing method in the above embodiment.

[0143] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a processor of an electronic device, the processor of the electronic device executes the technical solution of the data processing method of the above embodiment.

[0144] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes program code. When the program product runs on a processor of an electronic device, the program code is used to enable the processor of the electronic device to execute the steps of the method according to various exemplary implementations of the present application described above in this specification. For example, the electronic device may execute the data processing method recorded in the embodiments of the present application.

[0145] The program product may employ any combination of one or more readable media. The readable medium may be a readable data medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0146] The implementation method of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the technical solution of the data processing method in the above embodiment.

[0147] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable information processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable information processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0148] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0149] These computer program instructions can also be loaded onto a computer or other programmable information processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0150] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0151] It should be noted that, in addition to the implementation of the present application described in the above-mentioned specific embodiments, those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application is introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to the implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may extend based on the claims of the present application. In order to provide an in-depth understanding of the present application, the above description contains many specific details, and the present application may also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0152] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0153] The terms “first”, “second”, etc. are only used for distinguishing descriptions and should not be understood as indicating or implying relative importance.

[0154] Although the present application has been illustrated and described by referring to certain preferred implementations of the present application, it should be understood by those skilled in the art that the above contents are further detailed descriptions of the present application in combination with specific implementations, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. Those skilled in the art may make various changes in form and details, including making several simple deductions or substitutions, without departing from the spirit and scope of the present application.

Claims

1. A data processing method, characterized in that: Applied to a data processing system, the data processing system includes a data distribution node and a plurality of data processing nodes, the method includes: The data processing node processes the first data based on the corresponding data processing logic to obtain second data, where the first data is data obtained by the data processing node from an upstream data processing node based on the corresponding data processing logic and processed by the upstream data processing node based on the corresponding data processing logic; The data distribution node determines a target data processing node and target data corresponding to the target data processing node according to a data distribution strategy, and distributes the target data to a target subscriber, wherein the data distribution strategy is obtained based on a subscription requirement of the subscriber.

2. The data processing method according to claim 1, characterized in that: The data processing system further includes a management center and a configuration center, wherein the configuration center stores a configuration file, and the configuration file is generated based on the data processing requirements of the user. The method further includes: The management center creates the data processing node according to the configuration file stored in the configuration center, and enables the data processing node to obtain corresponding data processing logic based on the configuration file.

3. The data processing method according to claim 2, characterized in that: The configuration file includes a data query script and a data verification script, and the data processing node processes the first data based on the corresponding data processing logic to obtain the second data, including: The data processing node obtains the first data from the upstream data processing node according to the data query script; The data processing node verifies the legitimacy of the first data according to the data verification script, and when determining that the first data is legal data, performs data processing on the first data to obtain second data.

4. The data processing method according to claim 3, characterized in that: The configuration file also includes a data batch update script, and the method also includes: The data processing node updates the second data to a storage medium corresponding to the data processing node according to the data batch update script, so that a downstream data processing node of the data processing node obtains the second data from the storage medium.

5. The data processing method according to any one of claims 1 to 4, characterized in that: The data distribution node determines a target data processing node and target data corresponding to the target data processing node according to a data distribution strategy, and distributes the target data to a target subscriber, including: The data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, determines the target message middleware according to the data distribution strategy, and distributes the target data to the target subscriber based on the target message middleware.

6. The data processing method according to any one of claims 1 to 5, characterized in that: The data distribution node determines the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, including: The data distribution node determines a target data processing node from the data processing nodes according to the data distribution strategy, and uses the first data or the second data corresponding to the target data processing node as the target data; or The data distribution node determines a target data processing node from the data processing nodes according to the data distribution strategy, and processes the first data or the second data corresponding to the target data processing node based on corresponding data processing logic to obtain the target data.

7. The data processing method according to any one of claims 2 to 6, characterized in that: The method further comprises: The management center deletes the data processing node for which there is no corresponding downstream data processing node according to the data processing requirements of the user.

8. The data processing method according to any one of claims 1 to 7, characterized in that: The first data and the second data are structured data obtained based on a database change data capture technology, and the data processing is a data format conversion process.

9. The data processing method according to any one of claims 1 to 8, characterized in that: The data processing node has a data processing model deployed therein, and the data processing node processes the first data based on the corresponding data processing logic to obtain the second data, including: The data processing node calls the data processing model to perform data processing on the first data based on corresponding data processing logic to obtain the second data.

10. A data processing system, characterized in that: The data processing system includes a data distribution node and a plurality of data processing nodes, wherein: The data processing node is used to perform data processing on first data based on corresponding data processing logic to obtain second data, wherein the first data is data obtained by the data processing node from an upstream data processing node based on the corresponding data processing logic and processed by the upstream data processing node based on the corresponding data processing logic; The data distribution node is used to determine the target data processing node and the target data corresponding to the target data processing node according to the data distribution strategy, and distribute the target data to the target subscriber, wherein the data distribution strategy is obtained according to the subscription requirements of the subscriber.

Images