Data processing systems, methods, equipment and media suitable for industrial big data

Abstract

The present invention provides a data processing system suitable for industrial big data, including a registration center module that records the startup status of services of various modules in the system, the service type corresponding to the service, and the service identifier; a language translation module that converts the interface file corresponding to the interface description language into interface codes of various programming languages; and a system configuration module that configures the rendering of the screen and the various functions of the data source and data item. The data fusion module collects data item data, matches the data item list according to the trigger conditions, and caches the execution interface for other modules to query. The data processing flow configuration module is used for querying and configuring the data processing flow, switching the historical version of the data processing flow, and modifying the status of the data processing flow. The quality management module monitors the execution of the specific data processing flow, and calculates the quality score of the data processing flow according to the indicator factors in the execution of the data processing flow. The present invention improves the bottleneck of the system processing capacity and integrates the data source.

Description

Technical Field

[0001] The present invention relates to the field of computer information technology, and in particular to a data processing system and method, equipment and medium suitable for industrial big data. Background Art

[0002] At present, the country is vigorously developing smart manufacturing, which is essentially the product of the continuous integration and development of information technology and manufacturing technology. The introduction of information technology is accompanied by the demand for industrial big data processing methods. The characteristics of industrial big data are real-time and closed-loop. Every minute and every second, a large number of sensors are generating data, and several systems are statistically analyzing and analyzing these data to create new data. Such a huge amount of data is often difficult for the original single-machine architecture system to handle.

[0003] In industrial scenarios, data sources often go through many systems during the entire product life cycle. In the actual generation process, each system in the factory has its own responsible staff, and the relevant computer skills used by staff in different areas are generally different, making it difficult to have a unified data processing method.

[0004] After searching, the patent document CN107632842B discloses a rule configuration and publishing method, system, device and storage medium, including a rule engine providing a rule configuration interface and obtaining the configuration data of the rule; the rule engine provides a plug-in upload interface and obtains the diversion strategy plug-in and the execution strategy plug-in; the rule engine provides a rule set configuration interface and obtains the rule set data configured by the user in the preset scenario; the rule engine generates a file package to be published; the rule engine compiles the file package to be published into an executable file and publishes it to the target object. This prior art is a rule engine system based on Drools, which focuses on publishing the java class files generated based on Drools to each server and applying them. It relies on the open source rule engine Drools to configure its related business logic, has a high learning cost, and has relatively weak data processing capabilities. It is not suitable for processing data processing logic with complex business, complex calculations, and frequent iterative updates in industrial scenarios in smart manufacturing.

[0005] Patent document CN105467953A discloses a knowledge representation and automated application method for industrial big data, by 1) creating a corresponding category of intelligent objects for each type of knowledge resource in an industrial enterprise to obtain an intelligent object library; 2) defining various corresponding judgment conditions and calculation models for each set business problem, and establishing the knowledge automation rules of the business problem; 3) creating the business process of the business problem according to the knowledge automation rules and business logic of the business problem; 4) determining the event data to be processed from the real-time information data of the industrial enterprise according to the configuration of the start node of the business process, and then generating an event statement according to the event data and sending it to the event engine; 5) the event engine searches for matching nodes or intelligent objects from the business process according to the received event statement for processing, and saves the processing results. The business process configuration of the knowledge automation rules and business logic involved in the prior art is mainly for the configuration of the process, in which the key data processing method depends on the data processing method provided by the system itself, and the relevant capabilities of data processing are limited by the system.

[0006] Patent document CN106199421A discloses an early warning method and system based on industrial big data, including: step S1, feature point detection of real-time signals, selecting the signal with a total duration of n that crosses the highest value of the preset alarm threshold in the real-time signal as the sampling signal; step S2, similarity calculation of the sampling signal with N historical alarm signals, and obtaining N similarity calculation results; step S3, normalizing the N similarity calculation results, and obtaining the normalized similarity calculation results; step S4, predicting the future change trend of the sampling signal according to the normalized similarity calculation results. The early warning method involved in the prior art is a relatively detailed but single data processing method, and its disadvantage is that it is a relatively single system function.

[0007] Therefore, there is an urgent need to develop and design a data processing system, method, equipment, and medium suitable for industrial big data that can integrate data sources and improve the processing capabilities of the data processing system. Summary of the invention

[0008] In view of the defects in the prior art, the purpose of the present invention is to provide a data processing system and method, equipment, and medium suitable for industrial big data, so as to overcome the shortcomings of the above-mentioned general data processing system in processing industrial big data, such as limited processing capacity of a single-machine architecture, a single data source, and poor scalability of business logic and system coupling, so as to improve the bottleneck of system processing capacity, integrate data sources, support distributed expansion and development cooperation in multiple programming languages, and optimize and consolidate knowledge assets in the business field.

[0009] A data processing system suitable for industrial big data provided by the present invention includes:

[0010] Registration center module: provides registration services for services in other modules, so that services in other modules can obtain service proxies based on service types, and can push the start, stop and change of each service to services that have subscribed to service information changes;

[0011] System configuration module: configure and query the various data sources and data items required by the system. Each data item can have corresponding data source configuration, data item name, data item description, data item acquisition trigger condition, data item structure type and data item value type;

[0012] Data fusion module: acquires and caches data according to the data source configured by the system configuration module and the different triggering modes of the corresponding trigger configuration;

[0013] Data processing flow configuration module: used to configure and query data processing flows. Each data processing flow has a corresponding data processing service type configuration. For data processing flows that use different types of data processing services, there are custom configurations for different types of data processing services.

[0014] Data processing scheduling module: monitors the start and stop of data processing services in the data processing module through the registration center module, and horizontally expands the processing capacity of the data processing system suitable for industrial big data;

[0015] Data processing module: It is divided into different types of data processing services. Each data processing service is deployed in any computer under the same network. Its types are distinguished according to the interface it implements;

[0016] Quality management module: monitors the execution of specific data processing processes and calculates the quality score of the data processing process based on the indicator factors in the execution of the data processing process.

[0017] Preferably, it also includes:

[0018] Language translation module: converts the data processing service interface described by the pre-agreed interface interactive data language into various supported programming languages;

[0019] Result display module: used to display the historical query results of the data fusion module and the input parameters and output parameters of the historical execution process of the data processing flow scheduled by the data processing scheduling module.

[0020] Preferably, the system configuration module includes querying data sources, adding data sources, querying data items, adding data items, modifying data items, and deleting data items.

[0021] Preferably, the triggering mode of the trigger configuration in the data fusion module includes timed triggering and event triggering. The timed triggering is to actively obtain corresponding data according to the configured timed triggering conditions; the event triggering is to monitor the corresponding events and obtain data after receiving the event information.

[0022] Preferably, the data processing flow configuration module includes querying data processing flow, adding data processing flow, modifying data processing flow, data processing flow version configuration, data processing flow activation / invalidation configuration and deleting data processing flow.

[0023] Preferably, when the trigger conditions of the data processing flow configuration are met, the data processing scheduling module collects the corresponding data from the data fusion module, and calls the corresponding data processing flow execution service according to the corresponding load balancing strategy. When the execution result is obtained, the execution result is stored and pushed out so that other systems can perform corresponding operations based on the processing result.

[0024] Preferably, the data processing module includes multiple data processing services, which are distributed on any machine in the same network and are divided into different data processing services according to different implementation methods of the data processing service interface, including java data processing services and python data processing services.

[0025] According to a data processing method applicable to industrial big data provided by the present invention, data processing is performed using the above-mentioned data processing system applicable to industrial big data, comprising the following steps:

[0026] Configuration process steps: In the system configuration module and data processing process configuration module, configure the data source, data item and data processing process in sequence;

[0027] Data fusion step: when the trigger condition of a data item is met, a list of data items that meet the trigger condition is obtained, and data is acquired, and the acquisition result is converted into a corresponding data structure and cached;

[0028] Data processing flow execution steps: when the triggering conditions of the data processing flow are met, obtain the data processing flow list that meets the triggering conditions, integrate the data item list required by the data processing flow in the data processing flow list, and then obtain the data result set of the data list from the data fusion module. Subsequently, according to the load balancing strategy, a specific data processing service is selected for each data processing flow to execute the data processing flow. When the execution result of the data processing flow is returned, the result is stored and pushed to the message channel configured by the data processing flow;

[0029] Quality management steps: Every once in a while, quality management will statistically query the historical execution records of the data processing flow and the content of the data processing flow itself, and then statistically verify related factors to calculate the quality score of the data processing flow. If the score is not qualified, the data processing flow status will be modified to invalid through the data processing flow configuration module.

[0030] According to a computer-readable storage medium storing a computer program provided by the present invention, the steps of the above method are implemented when the computer program is executed by a processor.

[0031] A data processing device provided according to the present invention includes the above-mentioned data processing system suitable for industrial big data or the above-mentioned computer-readable storage medium storing a computer program.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present invention encapsulates multiple data sources and multiple types of data at the industrial site, and can well expand horizontally in data acquisition and data processing methods, thereby improving the load capacity of the entire system.

[0034] 2. The present invention provides execution services corresponding to various programming languages, and can fully utilize the development capabilities of existing personnel.

[0035] 3. The present invention can well access the message queue of production events, obtain data in real time and return results by setting up a system configuration module, a data fusion module, a data processing flow configuration module, a data processing scheduling module and a data processing module.

[0036] 4. The present invention improves the quality management function by setting up a quality management module, thereby being able to better maintain the business logic in the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0038] Figure 1 It is a framework diagram of a data processing system applicable to industrial big data in the present invention;

[0039] Figure 2 The figure is a flowchart of the steps of the data processing method applicable to industrial big data in the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0041] like Figure 1 As shown, the present invention provides a data processing system suitable for industrial big data, comprising:

[0042] Registration center module: provides registration services for services in other modules, so that services in other modules can obtain service proxies based on service types, and can push the start, stop and changes of each service to services that have subscribed to service information changes. The registration center module includes service registration and query functions. The registration center module of the entire system is unique. When services in other modules are started, they will send their own registration information, such as service type and unique service identifier, to the registration center module. The registration center module records this information to provide other services for query. In addition, the registration center module maintains a heartbeat connection with all other services. When a service stops, the registration center can discover and modify the service record accordingly.

[0043] System configuration module: configure and query each data source and each data item required by the system. Each data item can have corresponding data source configuration, data item name, data item description, data item acquisition trigger condition, data item structure type and data item value type; including data source query, data source addition, data item query, new data item, modification of data item and deletion of data item. Data source query is to query and display the data source list; data source addition is to add a data source to the data source list, and it is necessary to configure the data source name, data source description, data source interface implementation class name and data source required parameters; data item query is to query all data items under the group according to the group of data items; adding a data item is to add a data item to the group of data items and configure its name, describe the corresponding data source and data item related parameter configuration, data structure and trigger conditions; modifying a data item is to select a data item in the data item query result for modification, and its data source, data structure and collection conditions can be modified. When the modification is confirmed, it will prompt the specific data processing flow that may be affected by the modification of the data item; deleting a data item is to select a data item in the data query result for deletion. Deleting a data item requires confirmation, and it will prompt the specific data processing flow that may be affected by the deletion of the data item.

[0044] Data fusion module: acquires and caches data according to the different triggering modes of the data source configured by the system configuration module and the corresponding trigger configuration; the triggering modes of the trigger configuration in the data fusion module include timed triggering and event triggering. The timed triggering is to actively acquire the corresponding data according to the configured timed triggering conditions; the event triggering is to monitor the corresponding events and acquire the data after receiving the event information. The data fusion module can be allocated by consistent hashing strategy to obtain the data items it is responsible for, so as to realize the distributed deployment of the data fusion module.

[0045] Specifically, the data fusion module is mainly responsible for acquiring data according to the data items configured by the system configuration module. Data acquisition is based on trigger conditions, dependencies, and data sources. When the trigger conditions are met, the data items that meet the trigger conditions are collected and grouped according to the dependencies. The subsequent groups are classified according to the data source. Data items from the same data source will be acquired in one thread and the results will be cached according to the data items. Therefore, one data acquisition may involve multiple data items, which can effectively reduce repeated acquisition and reduce the system burden. There may be dependencies between data items. When the data fusion module is triggered, the dependencies of each data item are automatically calculated to ensure that the dependent data items and the dependent data items are correctly divided in the acquisition order. And not all data items will be acquired. Only data items referenced by the effective data processing flow or dependent by other data items will enter the data acquisition list when the task is triggered.

[0046] Data processing flow configuration module: used to configure and query data processing flows. Each data processing flow has a corresponding data processing service type configuration. For data processing flows that use different types of data processing services, there are custom configurations for various types of data processing services; including querying data processing flows, adding new data processing flows, modifying data processing flows, data processing flow version configuration, data processing flow activation / invalidation configuration, and deleting data processing flows.

[0047] Specifically, the above processes are:

[0048] To query the data processing flow, you can query all the data processing flows under the data processing flow group.

[0049] Adding a new data processing flow means adding a new data processing flow to the data processing flow group and configuring its data processing flow name, data processing flow description, trigger conditions, data processing service type, data items required for the data processing service, and configuration items required for a specific data processing service, as well as the message channel pushed after the data processing flow is executed. Similarly, in a new data processing flow, the configuration items required for a specific data processing service and the type of data processing service can be encapsulated into a template. If the configuration of the data processing flow template is used in the data processing flow, there is no need to configure the additional business logic required for a specific data processing service. You only need to configure the data processing flow name, data processing flow description, trigger conditions, and data items required for the data processing flow, so as to facilitate the rapid deployment and adjustment of the same business logic in the application.

[0050] To modify a data processing flow, you need to select a data processing flow in the data processing flow query to modify it. When modifying a data processing flow, if the content that is modified is not related to the business logic, such as the data processing flow name, data processing flow description, trigger conditions, and the message channel pushed after execution, the data processing flow will be modified directly. However, when the data processing flow modification involves the type of data processing service, the data items required by the data processing service, and the configuration items required for a specific data processing service, the system will match the historical modification results based on these configuration items involving business logic. If there is a corresponding data processing flow version in the history, the staff will be prompted to switch versions instead of modifying the business logic. Otherwise, a new data processing flow version of the data processing flow will be added to facilitate the subsequent switching of business logic.

[0051] Data processing flow version configuration is to select a data processing flow to switch versions in the data processing flow query, and enter the query of the historical versions of the data processing flow. Select one of the historical versions to set it as the application version of the current data processing flow.

[0052] Data processing flow activation / invalidation configuration. In the data processing flow query, select a data processing flow and modify its activation and invalidation status. If the data processing flow is in an activated state, then the data processing flow can be scheduled by the data processing scheduling module. Otherwise, if it is in an invalid state, the data processing scheduling module will ignore the data processing flow and the data processing flow will not be triggered.

[0053] Delete the data processing flow. In the data processing flow query, select a data processing flow to delete. A prompt will pop up and you need to confirm before you can delete it.

[0054] Data processing scheduling module: monitors the start and stop of data processing services in the data processing module through the registration center module, and horizontally expands the processing capacity of the data processing system suitable for industrial big data; when the trigger conditions of the data processing process configuration are met, collect the corresponding data from the data fusion module, and call the corresponding data processing process execution service according to the corresponding load balancing strategy. When the execution result is obtained, the execution result is stored and pushed out so that other systems can perform corresponding operations based on the processing result.

[0055] Specifically, the data processing scheduling module includes monitoring and maintenance of data processing services in the data processing module.

[0056] First, obtain the current information and status of each service and the changes of the subscription service from the registration center module. When a new data processing service appears in the system, the information of the data processing service is maintained, and the link object with the data processing service is obtained. According to the type of data processing service, the required data processing flow is sent to the data processing service, and the data processing service itself maintains the processing logic of the data processing flow; when a data processing service in the system loses response or the data processing service program is deregistered, the corresponding data processing service is removed from the maintenance table.

[0057] The data processing scheduling module also includes the data processing flow scheduling function. When monitoring the events triggered by the data processing flow, the data processing flows that meet the conditions are taken out, the data items they need are integrated, and unified into the data fusion module to obtain data in batches. Subsequently, the corresponding data processing service is selected according to the load balancing strategy according to the type of data processing service of each data processing flow, and the data processing flow execution method of the linked object of the data processing service is called in an asynchronous callback manner. When the data processing service completes the data processing flow, the callback function of the data processing scheduling module will be triggered, which will record the execution status and execution results of the data processing flow, and push the data processing flow to different message channels for consumption by third-party systems according to the configuration of the data processing flow.

[0058] Data processing module: divided into different types of data processing services, each data processing service is deployed in any computer under the same network, and its type is distinguished according to the interface method it implements; including multiple data processing services, data processing services are distributed on any machine in the same network, and are divided into different data processing services according to different implementation methods of the data processing service interface, including java data processing services and python data processing services. This module maintains the specific implementation framework of the data processing flow. When the interface of the data processing module is called, according to the data processing flow identifier and input parameters passed in during the call, the corresponding data processing flow content is executed and the result is returned.

[0059] Specifically, for Java data processing services, the data processing services are implemented using Java language and data processing flow engine technology. The data processing flow parsing syntax adopts Java-like syntax, supports Chinese, has a low usage threshold, high scalability, and is easy to encapsulate.

[0060] For Python data processing services, it is a data processing service implemented in Python language. By dynamically generating Python scripts and importing environment dependency libraries, it is very convenient to use Python's dependency packages required for data analysis.

[0061] Quality management module: monitors the execution of specific data processing processes, and calculates the quality score of data processing processes based on the indicator factors in the execution of data processing processes; the quality management module is mainly responsible for the closed-loop management of specific data processing processes. Every other cycle, the module will count and analyze the operation records and operation status of the data processing process version applied by the currently effective data processing process, and count the quality score of the data processing process based on the success rate of data processing process execution, the system resources occupied by data processing process execution, the deviation between the time consumed by data processing process execution and the expected amount, and the integrity of the data processing process itself. The statistical method is not linear, and each factor has its own threshold. When a factor exceeds the threshold, the overall quality score will be unqualified. When all relevant factors are within the threshold range, the quality score of the data processing process will be within the qualified range according to the weighted average algorithm. For unqualified data processing processes, the quality management module will make the data processing process enter an invalid state and notify the staff.

[0062] Language translation module: converts the data processing service interface described by the pre-agreed interface interactive data language into various supported programming languages, including C++, C-Sharp, Java, JavaScript, Objective-C, PHP, Python, Ruby and Swift. The corresponding language implements the generated interface code specifically, and then deploys it to achieve the expansion of the data processing service corresponding to a programming language. With the support of the language translation module, the types of languages ​​supported by the data processing module can be expanded, breaking away from the limitation of a single programming language, better compatible with the programming skills used by on-site developers, and reducing the cost of learning the language.

[0063] Result display module: used to display the historical query results of the data fusion module and the input parameters and output parameters of the historical execution process of the data processing flow scheduled by the data processing scheduling module. It is mainly responsible for displaying the request results and status of the historical data requests of the data items. It can also view the input and output of each data processing flow in the data processing flow scheduling process, as well as the execution time, execution status, and execution resource consumption. There are different display forms for different data structures. Curve type data is displayed in a line chart, table type data is displayed in a table, and single value type data directly displays the corresponding value.

[0064] like Figure 2 As shown, the present invention provides a data processing method applicable to industrial big data, which uses the above-mentioned data processing system applicable to industrial big data to process data, including the following steps:

[0065] Configuration process steps: In the system configuration module and the data processing process configuration module, configure the data source, data items, and data processing process in sequence. In the configuration process, the configuration of the data source includes the data source name, data source description, data source interface implementation class name, and the parameters required for the data source configuration. The configuration of the data item includes the data item name, data item description, data source configuration, data item data structure, data item parameters, and data item triggering conditions. In the configuration of the data processing process, it is necessary to configure the data processing process name, data processing process description, data processing process triggering conditions, data processing service type, data items required for the data processing service, and the configuration items required for specific data processing services, and the message channel pushed after the data processing process is executed.

[0066] Data fusion step: When the trigger condition of the data item is met, obtain the list of data items that meet the trigger condition, and perform data acquisition, convert the acquisition results into the corresponding data structure, and cache them. In the data fusion process, when the trigger condition is met, extract the list of data items that meet the trigger condition, among which the data items that are not referenced by the data processing process or other data items will be filtered, and then grouped according to the dependency relationship and data source configuration of the data items, and then assign threads in turn according to the grouping results, and perform data acquisition in the threads. According to the acquisition results, the content in the results is assigned to each data item according to the data item parameters, and the content will be converted into the corresponding data item structure according to the data item structure. The existing structure types include single point structure, curve structure, and table structure. The value of the data item will be cached later.

[0067] Data processing flow execution steps: When the trigger condition of the data processing flow is met, obtain the data processing flow list that meets the trigger condition, integrate the data item list required by the data processing flow in the data processing flow list, and then obtain the data result set of the data list from the data fusion module. Subsequently, select a specific data processing service for each data processing flow according to the load balancing strategy to execute the data processing flow. When the data processing flow execution result is returned, the result is stored and pushed to the message channel configured by the data processing flow. In the data processing flow execution process, when the trigger condition is met, the data processing flow that meets the trigger condition is taken out, and the invalid data processing flow will be filtered out, and it will be organized into a data processing flow list. The data items involved in the data processing flow list are collected and obtained from the data fusion module. After obtaining the data, select the respective data processing service for each data processing flow according to the load balancing strategy of the data processing scheduling module, and then pass the name of the data processing flow and the trigger parameter to the corresponding data processing service to execute the data processing flow. When the data processing flow is executed, the data processing scheduling module obtains the data processing flow result, stores the data processing flow result and pushes it to the push message channel configured by the data processing flow.

[0068] Quality management steps: Every cycle, the quality management will statistically query the historical execution records of the data processing process and the content of the data processing process itself, and then statistically check the relevant factors to calculate the quality score of the data processing process. If the score is unqualified, the data processing process status will be modified to invalid through the data processing process configuration module. In the quality management process, every cycle, the process history records of the activated data processing process will be queried, and the quality score will be calculated based on the success rate of the data processing process execution within the cycle time, the system resources occupied by the data processing process execution, the difference between the time consumed by the data processing process execution and the expected time, and the integrity of the data processing process itself. Each factor has its threshold. When the corresponding factor exceeds the threshold, the quality score is unqualified. When all factors are within the threshold range, the corresponding quality score is obtained according to the weighted average method. If the quality score is unqualified, it will automatically enter the configuration process, modify the corresponding data processing process to invalid in the data processing process configuration module, and notify the staff.

[0069] Result display step: After the data fusion module queries the relevant data items, the relevant historical records can be viewed and displayed through the result display module. After the data processing scheduling module and the data processing module complete the data processing flow execution process, the relevant historical records can be viewed and displayed through the result display module. In the result display process, you can query the historical query records of the data items according to the data items, and then view the results of the data query and the results after the results are mapped to the data items through a historical record, or query the query results of the data processing flow according to the data processing flow, and then view the input, output, execution time, execution status, and execution consumption resources of the data processing flow when the data processing flow is executed through a historical record. The display of input, output, and data item results has different display methods according to the data structure type. The single point structure directly displays the data, the curve structure displays it as a line chart according to the data, and the table structure displays it in a table form according to the data.

[0070] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0071] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A data processing system suitable for industrial big data, characterized in that: include: Registration center module: provides registration services for services of other modules, so that services of other modules can obtain service agents according to service types, and can push the start, stop and changes of each service to services that have subscribed to service information changes; the registration center module includes service registration and query functions, and the registration center module of the entire system is unique. When services of different modules are started, registration information will be sent to the registration center module, which records the registration information and provides other services for query; the registration center module maintains a heartbeat connection with all other services. When a service stops, the registration center can discover and modify the service record accordingly; System configuration module: configure and query the various data sources and data items required by the system. Each data item can have corresponding data source configuration, data item name, data item description, data item acquisition trigger condition, data item structure type and data item value type; Data fusion module: acquires and caches data according to the data source configured by the system configuration module and the different triggering modes of the corresponding trigger configuration; Data processing flow configuration module: used to configure and query data processing flows. Each data processing flow has a corresponding data processing service type configuration. For data processing flows that use different types of data processing services, there are custom configurations for different types of data processing services. Data processing scheduling module: monitors the start and stop of data processing services in the data processing module through the registration center module, and horizontally expands the processing capacity of the data processing system suitable for industrial big data; Data processing module: It is divided into different types of data processing services. Each data processing service is deployed in any computer under the same network. Its types are distinguished according to the interface it implements; Quality management module: monitors the execution of specific data processing processes and calculates the quality score of the data processing process based on the indicator factors in the execution of the data processing process; Also includes: Language translation module: converts the data processing service interface described by the pre-agreed interface interactive data language into various supported programming languages; Result display module: used to display the historical query results of the data fusion module and the input parameters and output parameters of the historical execution process of the data processing flow scheduled by the data processing scheduling module; The data processing flow configuration module includes querying data processing flow, adding data processing flow, modifying data processing flow, data processing flow version configuration, data processing flow activation / invalidation configuration and deleting data processing flow; When the trigger conditions configured for the data processing flow are met, the data processing scheduling module collects the corresponding data from the data fusion module and calls the corresponding data processing flow execution service according to the corresponding load balancing strategy. When the execution result is obtained, the execution result is stored and pushed out so that other systems can perform corresponding operations based on the processing result.

2. The data processing system suitable for industrial big data according to claim 1 is characterized in that: The system configuration module includes querying data source, adding data source, querying data item, adding data item, modifying data item and deleting data item.

3. The data processing system suitable for industrial big data according to claim 1 is characterized in that: The triggering modes of the trigger configuration in the data fusion module include timed triggering and event triggering. The timed triggering is to actively obtain the corresponding data according to the configured timed triggering conditions; the event triggering is to monitor the corresponding events and obtain data after receiving the event information.

4. The data processing system suitable for industrial big data according to claim 1, characterized in that: The data processing module includes multiple data processing services, which are distributed on any machine in the same network and are divided into different data processing services according to different implementation methods of the data processing service interface, including java data processing services and python data processing services.

5. A data processing method suitable for industrial big data, characterized in that: The data processing system applicable to industrial big data according to any one of claims 1 to 4 is used to process data, comprising the following steps: Configuration process steps: In the system configuration module and data processing process configuration module, configure the data source, data item and data processing process in sequence; Data fusion step: when the trigger condition of a data item is met, a list of data items that meet the trigger condition is obtained, and data is acquired, and the acquisition result is converted into a corresponding data structure and cached; Data processing flow execution steps: when the triggering conditions of the data processing flow are met, obtain the data processing flow list that meets the triggering conditions, integrate the data item list required by the data processing flow in the data processing flow list, and then obtain the data result set of the data list from the data fusion module. Subsequently, according to the load balancing strategy, a specific data processing service is selected for each data processing flow to execute the data processing flow. When the execution result of the data processing flow is returned, the result is stored and pushed to the message channel configured by the data processing flow; Quality management steps: Every once in a while, quality management will statistically query the historical execution records of the data processing flow and the content of the data processing flow itself, and then statistically verify related factors to calculate the quality score of the data processing flow. If the score is not qualified, the data processing flow status will be modified to invalid through the data processing flow configuration module.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 5 are implemented.

7. A data processing device, characterized in that: A data processing system suitable for industrial big data comprising any one of claims 1 to 4 or a computer-readable storage medium storing a computer program as claimed in claim 6.

Images