Custom data processing method, system and equipment and medium

Abstract

The invention relates to the technical field of Internet of Things, in particular to a user-defined data processing method, system and device and a medium, and the method comprises the steps: obtaining data processing logic information, and carrying out the recognition and division of the data processing logic information, and obtaining logic module information; splitting the logic module information to obtain minimum function component information, and performing flow definition on the minimum function component information; acquiring selection information for selecting a plurality of pieces of minimum function component information, acquiring selection connection information among the minimum function component information, and constructing to obtain processing flow information according to the selection information and the selection connection information, the problems that service scene adaptability is insufficient and non-technical user friendliness is poor in the perception data circulation process are solved, and data processing flexibility, adaptability and operation and maintenance efficiency are improved.

Description

Technical Field

[0001] The present application relates to the technical field of Internet of Things, and in particular to a method, system, device and medium for processing customized data. Background Art

[0002] As an important bridge connecting the physical world and the digital world, the Internet of Things realizes data collection, transmission and analysis through sensors, devices and Internet technology, bringing tremendous changes and opportunities to various industries.

[0003] In the operation of the IoT platform, the degree of customization of the perception data flow process is not high, making it difficult to flexibly respond to the needs of various business scenarios. The platform's adaptability to changing business scenarios is limited, and it cannot effectively respond to rapidly changing market demands. As a result, it is not friendly enough to users with non-technical backgrounds, making it difficult for ordinary users to easily use and manage the IoT system. The existence of these problems directly affects the overall efficiency of the IoT platform and the user experience. Specifically, the low degree of customization and insufficient adaptability lead to inefficient data processing and flow, increase the cost and complexity of system operation and maintenance, and further limit the platform's flexibility and efficiency in responding to complex business needs. The above problems need to be solved. Summary of the invention

[0004] In order to solve the problems of insufficient adaptability to business scenarios and poor friendliness for non-technical users caused by the perception data flow process, and to improve the flexibility, adaptability and operation and maintenance efficiency of data processing, this application provides a custom data processing method, system, device and medium, which adopts the following technical solutions:

[0005] In a first aspect, the present application provides a method for processing user-defined data, comprising:

[0006] Acquire data processing logic information, identify and divide the data processing logic information, and obtain logic module information;

[0007] Split the logic module information to obtain the minimum functional component information, and define the process of the minimum functional component information;

[0008] The selection information for selecting several minimum functional component information is obtained, and the selection connection information between the minimum functional component information is obtained, and the processing flow information is constructed according to the selection information and the selection connection information.

[0009] Preferably, it also includes:

[0010] Obtain task execution information and execution time information, and run processing flow information according to the task execution information and execution time information.

[0011] Preferably, the specific steps of identifying and dividing the data processing logic information to obtain the logic module information are:

[0012] The node model is used to identify the diversity of data sources, define data processing requirements, and control data flow for data processing logic information to obtain logic module information.

[0013] Preferably, the minimum functional component information includes MQTT message subscription component information, MQTT message push component information, Kafka message subscription component information, Kafka message publishing component information, HTTP request component information, HTTP response component information, routing component information, and function component information.

[0014] In a second aspect, the present application provides a custom data processing system, comprising:

[0015] The first information division module is used to obtain data processing logic information, identify and divide the data processing logic information, and obtain logic module information;

[0016] The second information division module is used to split the logic module information to obtain the minimum functional component information and define the process of the minimum functional component information;

[0017] The process construction module is used to obtain selection information for selecting several minimum functional component information and to obtain selection connection information between the minimum functional component information, and to construct processing process information according to the selection information and the selection connection information.

[0018] Preferably, it also includes:

[0019] The time setting module is used to obtain task execution information and execution time information, and run the processing flow information according to the task execution information and execution time information.

[0020] Preferably, the first information division module identifies and divides the data processing logic information, and the specific steps of obtaining the logic module information are:

[0021] The node model is used to identify the diversity of data sources, define data processing requirements, and control data flow for data processing logic information to obtain logic module information.

[0022] Preferably, the minimum functional component information obtained by the second information division module includes MQTT message subscription component information, MQTT message push component information, Kafka message subscription component information, Kafka message publishing component information, HTTP request component information, HTTP response component information, routing component information, and function component information.

[0023] In a third aspect, the present application provides a custom data processing device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the custom data processing method as described above.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the custom data processing method as described above when run.

[0025] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0026] The present application obtains data processing logic information, identifies the data processing logic information, divides different types of data processing logic information, obtains several logic module information, further decomposes each logic module information, obtains minimum functional component information, and defines the process for it. After the setting is completed, selection information for selecting several minimum functional component information is obtained, and selection connection information between the minimum functional component information is obtained. According to the selection information and the selection connection information, the processing flow information is constructed, so that the user can adjust the processing flow at will, solve the problems of insufficient adaptability to business scenarios and poor friendliness for non-technical users brought about by the perception of data flow process, and improve the flexibility, adaptability and operation and maintenance efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a custom data processing method described in an embodiment of the present application.

[0028] Figure 2 It is a module diagram of a custom data processing system described in an embodiment of the present application.

[0029] Description of reference numerals:

[0030] 1. First information division module; 2. Second information division module; 3. Process construction module. DETAILED DESCRIPTION

[0031] The following combination Figure 1-Figure 2 The present application is described in further detail. The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting.

[0032] The current IoT platform faces many challenges in the process of data processing and circulation, which are mainly reflected in: low degree of customization. Existing platforms often preset fixed data processing processes, which are difficult to meet the personalized needs in specific business scenarios. Insufficient adaptability. With the diversified development of business, the platform responds slowly to rapidly changing business needs. Lack of user friendliness. For users with non-technical backgrounds, the threshold for configuring complex business logic and scheduled tasks is high. Low operation and maintenance efficiency and lack of effective tools to support efficient management of data processing processes lead to increased operation and maintenance costs, so a data processing method is provided.

[0033] Reference Figure 1 , a custom data processing method involved in this application specifically includes:

[0034] Acquire data processing logic information, identify and divide the data processing logic information, and obtain logic module information;

[0035] Split the logic module information to obtain the minimum functional component information, and define the process of the minimum functional component information;

[0036] The selection information for selecting several minimum functional component information is obtained, and the selection connection information between the minimum functional component information is obtained, and the processing flow information is constructed according to the selection information and the selection connection information.

[0037] Specifically, the present application obtains data processing logic information, identifies the data processing logic information, divides different types of data processing logic information, obtains several logic module information, further splits each logic module information to obtain minimum functional component information, and defines the process for it. After the setting is completed, selection information for selecting several minimum functional component information is obtained, and selection connection information between the minimum functional component information is obtained. According to the selection information and the selection connection information, processing process information is constructed, so that users can adjust the processing process at will, improve the data processing flexibility, adaptability and operation and maintenance efficiency of the Internet of Things platform, and solve the problems of weak customization ability of perception data flow, insufficient adaptability to business scenarios, poor friendliness for non-technical users, and lack of efficient visual configuration tools in the prior art.

[0038] As one implementation method, the specific steps of identifying and dividing the data processing logic information to obtain the logic module information are as follows:

[0039] The node model is used to identify the diversity of data sources, define data processing requirements, and control data flow for data processing logic information to obtain logic module information.

[0040] Specifically, the embodiments of the present application are based on the idea of ​​componentization, and identify the complex logic in the data flow process according to the diversity of data sources, data processing requirements, and data flow control, and break down the originally integrated complex processing logic into the smallest manageable functional units, which are standardized to form independent and unified components, and different business needs are met based on the combination of components.

[0041] Among them, the component processing logic is to disassemble complex business processes into a series of manageable and reusable components. This process follows the three principles of data source diversity, data processing requirements, and data flow control, and ultimately achieves modularization and standardization of logic. After obtaining the principle information, the embodiment of the present application uses the node model of the LiteFlow framework, namely the Node model, based on the componentization idea to modularize the complex logic in the data flow process. Each node represents a minimum logical processing unit, corresponding to the diversity of data sources, specific data processing requirements, or a link in data flow control.

[0042] Identify according to the diversity of data sources. Combining the characteristics of IoT data and business needs, common data sources of IoT perception data include MQTT, Kafka and HTTP. LiteFlow's node architecture designs dedicated access nodes for multiple data sources such as MQTT, Kafka, HTTP, etc. Each node serves as a data entry, identifies the source of the data, and preliminarily defines the basic processing path of the data, laying the foundation for the positioning of subsequent logical processing units.

[0043] Defined according to data processing requirements. Analyze the characteristics of IoT data and the specific requirements of business scenarios, and use LiteFlow's conditional branching logic to identify the minimum necessary steps for data processing. This includes but is not limited to data cleaning, format conversion, delay processing, etc. Each step is regarded as a logical processing unit, each carrying a specific data processing task, and together constitutes a complex logic.

[0044] And data flow control. Using LiteFlow process design, simulate and map the flow of data on the processing link. Based on business rules and data status, by configuring the dependencies and conditional transfers between nodes, accurately plan the flow path of data between different processing units. This process ensures that data can be effectively directed to the processing link that best suits its characteristics, achieving the optimal configuration of the logical processing unit.

[0045] Based on LiteFlow's logic identification and mapping mechanism, by carefully analyzing the data source, clarifying the specific needs of data processing, and pre-planning the control logic of data flow, we successfully identified and defined the smallest logical processing unit in the complex processing flow, laying the foundation for the subsequent splitting and standardized construction of components.

[0046] As one implementation method, the minimum functional component information includes MQTT message subscription component information, MQTT message push component information, Kafka message subscription component information, Kafka message publishing component information, HTTP request component information, HTTP response component information, routing component information, and function component information.

[0047] Specifically, the embodiment of the present application breaks down the originally integrated complex processing logic into the smallest manageable functional units. After identifying and defining the smallest logical processing unit, the chain call and conditional branch mechanism of LiteFlow are used to further break down these logical units into the smallest manageable functional components. Each component focuses on a single processing task. The smallest manageable functional components include MQTT message subscription, MQTT message push, Kafka message subscription, Kafka message publishing, HTTP request, HTTP response, routing, and function.

[0048] Among them, MQTT message subscription, MQTT message push, Kafka message subscription, Kafka message publishing, HTTP request and HTTP response describe specific data source access points, which are mainly defined by the identification results of the dimension of data source diversity and are direct ways for data to flow in and out.

[0049] According to predefined rules or dynamic conditions, data is directed to different processing paths, and the identification result of the associated data flow control dimension is defined to obtain routing component information.

[0050] It encapsulates specific business logic or algorithm operations, such as data cleaning, format conversion, numerical calculation, etc., to ensure the accuracy and availability of data, and defines the function component information based on the identification results of this dimension defined in association with data processing requirements.

[0051] Through LiteFlow's configuration mechanism, functional components are organized in the form of nodes to form a clear task chain. Disassembly not only simplifies the complexity of a single logical processing unit, but also makes the relationship between components modular and configurable, creating conditions for subsequent management and reuse.

[0052] As one of the implementation methods, a series of abstract base classes and interfaces are designed under the LiteFlow framework so that all components follow consistent development standards.

[0053] Specifically, in order to achieve standardization and unification of components, a set of common interface specifications and configuration modes are defined for functional nodes, so that they are consistent in terms of data input and output formats, error handling mechanisms, and ways of interacting with other components.

[0054] It defines common interfaces. It designs a series of abstract base classes and interfaces, such as IComponent, IDataProcessor, IEventEmitter, etc. The above interfaces specify the basic behaviors and functions that components should have, such as data input and output, event triggering, error handling, etc. Regardless of the specific implementation details of the components, as long as these interfaces are implemented, the consistency and interoperability between components can be guaranteed.

[0055] For example, MQTT and Kafka nodes, as data sources, need to follow the basic behavior of the IComponent interface so that all components can be initialized, executed, and destroyed consistently.

[0056] As one of the implementation methods, based on standardized components, the process definition file is clearly defined, the connection relationship between components, data flow and execution conditions are specified, and the components are assembled in an orderly manner. When facing new business needs or changes, only the components and the connections between them need to be adjusted or increased or decreased without rebuilding the entire system.

[0057] Specifically, the embodiment of the present application adopts a visual rule canvas engine. LiteFlow and LogicFlow technologies are integrated to create a user-friendly visual rule configuration interface. Users can intuitively drag components on the canvas without programming knowledge to obtain selection information and selection connection information for selecting several minimum functional component information, and the system defines data processing rules and forwarding paths to achieve customized processing and forwarding of IoT data or middleware data.

[0058] During the user's use, based on the LogicFlow graphical process design tool, standardized components can be intuitively dragged onto the canvas to obtain selection information. Through wiring operations, data processing processes that meet specific business scenarios can be quickly built to obtain selection connection information. Whether it is simple data forwarding, complex data conversion and filtering, or business logic involving multi-conditional branch judgments, it can be achieved through the flexible combination of components. The component parameter configuration interface allows users to adjust the behavior of components according to real-time business needs, such as modifying data processing rule information, setting trigger conditions for scheduled task information, etc. This highly flexible component design allows users with non-technical backgrounds to quickly build data processing solutions that meet diverse business needs, greatly improving the adaptability of the system and the productivity of users.

[0059] As one implementation method, task execution information and execution time information are obtained, and the processing flow information is run according to the task execution information and execution time information.

[0060] Specifically, the embodiment of the present application enhances the timing scheduling mechanism. It integrates the Quartz scheduling framework to achieve highly flexible timing task configuration. Users can set the precise time, interval period and other parameters of task execution through an intuitive interface, and the system automatically arranges the rule tasks into the task queue according to the time sequence, so that the rule information is accurately executed.

[0061] Among them, the integration of the Quartz scheduling framework includes: integrating the Quartz library and adding Quartz dependencies to the project. Configure the Quartz Scheduler. When the application starts, initialize the Quartz Scheduler instance and configure the database connection and thread pool settings to facilitate the management and execution of tasks. Define Jobs and Triggers. Jobs represent the task logic that needs to be executed. They usually implement the Quartz Job interface or inherit the JobBase class. Triggers define when and how the Job is executed, including one-time execution, fixed-frequency execution, Cron expression execution, etc., that is, task execution information. Register tasks and triggers: Register the defined Jobs and Triggers with the Scheduler so that the tasks can be executed at a predetermined time point or period, that is, execution time information.

[0062] Among them, the scheduled scheduling is extracted as a minimum manageable functional unit and standardized into a scheduled task node to achieve highly flexible scheduled task configuration.

[0063] Users can set the precise time, interval and other parameters of task execution through an intuitive interface. Implement a visual Cron expression generator. Users can operate through a graphical interface, such as checking a specific time point, selecting a repetition period, etc., which will be automatically converted into the corresponding Cron expression string in the background, or directly configure the Cron expression in the front end. According to the execution time or Cron expression set by the user, the Quartz scheduled task framework is used to arrange the task execution plan.

[0064] The system automatically puts rule tasks into the task queue in time sequence. Parse Trigger rules. When the user saves the task configuration, the system will parse the Trigger, such as SimpleTrigger and CronTrigger rules, and calculate the first execution time and subsequent execution plan of the task. Build a scheduling plan. The Scheduler builds an execution plan for a period of time in the future based on the Trigger rules and the current time to ensure that each task is ready to execute at the right time. Task queuing. According to the execution plan, the system adds the task instance JobDetail+Trigger to be executed to the internal task queue in chronological order, waiting for the execution thread pool to allocate resources for execution. Scheduling execution. The Scheduler has one or more threads that continuously monitor the task queue. When the scheduled execution time of the task is reached, the task instance is taken out of the queue and assigned to the thread pool to execute the Job.

[0065] This enables visual configuration of the entire chain, from the collection, transmission, rule definition, matching, execution to data push of IoT sensor data, and the entire process can be customized through a graphical interface, greatly improving the convenience and intuitiveness of the configuration. The data processing capability is enhanced, and through efficient parsing, extraction, filtering and processing of device data, it is ensured that the data meets the expected quality standards before it is transferred to a third-party platform. The flexibility and adaptability of the system are improved. The componentization and visual design enable the system to respond quickly to business changes, lower the technical threshold, and enable non-technical personnel to efficiently build complex data processing logic. Optimize operation and maintenance costs and efficiency: Through automated scheduling and management, manual intervention is reduced, operation and maintenance costs are reduced, and the efficiency of overall data processing and circulation is improved.

[0066] Reference Figure 2 , a custom data processing system is provided for an embodiment of the present application, the system comprising:

[0067] The first information division module 1 is used to obtain data processing logic information, identify and divide the data processing logic information, and obtain logic module information;

[0068] The second information division module 2 is used to split the logic module information to obtain the minimum functional component information and define the process of the minimum functional component information;

[0069] The process construction module 3 is used to obtain selection information for selecting several minimum functional component information and to obtain selection connection information between the minimum functional component information, and to construct the processing process information according to the selection information and the selection connection information.

[0070] As one implementation mode, it also includes:

[0071] The time setting module is used to obtain task execution information and execution time information, and run the processing flow information according to the task execution information and execution time information.

[0072] As one implementation mode, the first information division module identifies and divides the data processing logic information, and the specific steps of obtaining the logic module information are as follows:

[0073] The node model is used to identify the diversity of data sources, define data processing requirements, and control data flow for data processing logic information to obtain logic module information.

[0074] As one of the implementation modes, the minimum functional component information obtained by the second information division module includes MQTT message subscription component information, MQTT message push component information, Kafka message subscription component information, Kafka message publishing component information, HTTP request component information, HTTP response component information, routing component information, and function component information.

[0075] An embodiment of the present application provides a custom data processing device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the custom data processing method as described above.

[0076] An embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned custom data processing method when running.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and product can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0078] In several embodiments provided in this application, it should be understood that the disclosed methods, systems, devices and program products may be implemented in other ways.

[0079] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0080] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A custom data processing method, characterized in that: include: Acquire data processing logic information, identify and divide the data processing logic information, and obtain logic module information; Split the logic module information to obtain the minimum functional component information, and define the process of the minimum functional component information; The selection information for selecting several minimum functional component information is obtained, and the selection connection information between the minimum functional component information is obtained, and the processing flow information is constructed according to the selection information and the selection connection information.

2. The method for processing user-defined data according to claim 1, characterized in that: Also includes: Obtain task execution information and execution time information, and run processing flow information according to the task execution information and execution time information.

3. The method for processing user-defined data according to claim 1, characterized in that: The specific steps of identifying and dividing the data processing logic information to obtain the logic module information are as follows: The node model is used to identify the diversity of data sources, define data processing requirements, and control data flow for data processing logic information to obtain logic module information.

4. The method for processing user-defined data according to claim 3, characterized in that: The minimum functional component information includes MQTT message subscription component information, MQTT message push component information, Kafka message subscription component information, Kafka message publishing component information, HTTP request component information, HTTP response component information, routing component information, and function component information.

5. A custom data processing system, characterized in that: include: The first information division module is used to obtain data processing logic information, identify and divide the data processing logic information, and obtain logic module information; The second information division module is used to split the logic module information to obtain the minimum functional component information and define the process of the minimum functional component information; The process construction module is used to obtain selection information for selecting several minimum functional component information and to obtain selection connection information between the minimum functional component information, and to construct processing process information according to the selection information and the selection connection information.

6. The user-defined data processing system according to claim 5, characterized in that: Also includes: The time setting module is used to obtain task execution information and execution time information, and run the processing flow information according to the task execution information and execution time information.

7. The user-defined data processing system according to claim 5, characterized in that: The first information division module identifies and divides the data processing logic information, and the specific steps of obtaining the logic module information are: The node model is used to identify the diversity of data sources, define data processing requirements, and control data flow for data processing logic information to obtain logic module information.

8. The user-defined data processing system according to claim 7, characterized in that: The minimum functional component information obtained by the second information division module includes MQTT message subscription component information, MQTT message push component information, Kafka message subscription component information, Kafka message publishing component information, HTTP request component information, HTTP response component information, routing component information, and function component information.

9. A custom data processing device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the custom data processing method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the custom data processing method according to any one of claims 1 to 4 when running.

Images