Lightweight editable streaming data processing framework

Abstract

The invention discloses a lightweight editable streaming data processing framework which is characterized by comprising the following steps: A, loading an operator library; b, loading a json text, and creating a computational flow; c, starting node input and processing data; and D, obtaining a result by the terminal node. Compared with the prior art, the system can quickly adapt to various complex data processing requirements through modular design and an automatic process, a convenient and efficient data processing solution is provided for a user, and the system has wide application prospects and commercial value.

Description

Technical Field

[0001] The present invention relates to the technical field of computer software, and particularly to a lightweight and editable streaming data processing framework. Background Art

[0002] With the rapid development of computer vision and image processing technologies, many software tools aiming to simplify the development process have emerged in the market. For example, existing software pre-packages various operators of OpenCV and provides an intuitive drag-and-drop interface to edit the image processing process. This approach indeed provides great convenience for entry-level users or rapid prototyping.

[0003] However, for professional developers, these tools have significant limitations. Firstly, developers hope to be able to customize the data structure to adapt to special application scenarios. Secondly, they need to manually control the data flow and precisely manage the way data is transmitted in the system. In addition, using self-developed image processing algorithms is also the key to improving system performance and achieving differentiation, but existing software often has difficulty supporting this. Finally, being able to conveniently edit algorithm parameters for fine-grained optimization and adjustment is also a basic requirement of professional developers.

[0004] In addition, most other similar streaming processing frameworks serve the Internet industry, and their architectures and design concepts are not entirely applicable to the field of image processing. These frameworks usually rely on third-party components such as Kafka, increasing the complexity and maintenance cost of the system. This is not an ideal choice for developers who need efficient and stable image processing solutions. Summary of the Invention

[0005] In view of this, the present invention discloses a lightweight and editable streaming data processing framework, which meets the in-depth requirements of professional developers for data structure, data flow, algorithm integration, and parameter tuning, avoids excessive dependence on third-party components, and provides a high-performance and scalable solution. The technical solution of the present invention is as follows:

[0006] The present invention discloses a lightweight and editable streaming data processing framework, including the following steps:

[0007] A. Load the operator library;

[0008] B. Load the json text and create a calculation flow;

[0009] C. The start node inputs and processes data;

[0010] D. The end node obtains the result.

[0011] Specifically, step B is to load a JSON text that complies with the framework specification and create the computational flow described therein.

[0012] Specifically, the start node inputs data, and the data automatically executes the processing program of the node and streams the processing result to the next node.

[0013] Specifically, an E step is provided between step B and step C; step E is specifically to perform a normalization check on the input JSON text. If it passes, step B is carried out; if it fails, the program exits.

[0014] Specifically, an F step is provided after step E; step F is specifically to register a callback function for the end node and automatically process the result of the end node.

[0015] The advantages of the present invention are specifically as follows:

[0016] 1. The framework adopts a lightweight architecture, reducing system resource consumption and improving operation efficiency, and is applicable to resource-constrained environments and data processing scenarios requiring high performance.

[0017] 2. By loading a JSON text that complies with the framework specification, users can edit and configure the computational flow by themselves, flexibly customize the data processing process according to actual needs, and improve the adaptability of the system.

[0018] 3. The entire data processing process is clearly divided into steps such as loading the operator library, creating the computational flow, data input processing, and result acquisition. The responsibilities of each module are clear, which is convenient for maintenance and expansion.

[0019] 4. A normalization check on the JSON text is added between creating the computational flow and data processing to ensure that the input data and configuration meet the requirements, avoiding running exceptions caused by incorrect data formats, and enhancing the stability of the system.

[0020] 5. Registering a callback function for the end node can automatically process the result of the end node, facilitating subsequent operations on the processed data, such as storage, analysis, or triggering other processes. It improves the system's ability to handle abnormal situations and ensures the reliability of data processing. Description of the Drawings

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only one embodiment of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1Flow chart of steps A - D of the present invention;

[0023] Figure 2 Schematic diagram of steps A - F of the present invention. Detailed implementation manners

[0024] The following will clearly and completely describe the technical solutions of the present invention in conjunction with the embodiments of the present invention and their accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts fall within the scope of protection of the present invention.

[0025] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs; the terms used in the detailed implementation manners are only for the purpose of describing specific embodiments and are not intended to limit the present invention; the terms "including" and "having" and any variations thereof in the description of the specification and claims of the present invention and the above accompanying drawings are intended to cover non - exclusive inclusion.

[0026] In the description of the detailed implementation manners of the present invention, technical terms such as "first" and "second" are only used to distinguish different objects and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity, specific order or primary - secondary relationship of the indicated technical features. In the description of the embodiments of the present invention, "a plurality of" means two or more unless otherwise specifically defined.

[0027] Referring to "embodiments" in the present invention means that specific features, structures or characteristics described in conjunction with the embodiments can be included in at least one embodiment of the present invention. The phrase appears in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described in the present invention can be combined with other embodiments.

[0028] In the description of the embodiments of the present invention, the term "and / or" is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in the present invention generally represents an "or" relationship between the associated objects before and after.

[0029] It should be noted that for the convenience of description, in the following embodiments, all the same technical features are marked with the same symbols.

[0030] With the rapid development of computer vision and image processing technologies, many software tools aiming to simplify the development process have emerged in the market. For example, existing software pre-packages various operators of OpenCV and provides an intuitive drag-and-drop interface to edit the image processing pipeline. This approach indeed offers great convenience for entry-level users or rapid prototyping.

[0031] However, for professional developers, these tools have significant limitations. Firstly, developers hope to be able to customize data structures to adapt to special application scenarios. Secondly, they need to manually control the data flow and precisely manage the way data is transmitted in the system. In addition, using self-developed image processing algorithms is also the key to improving system performance and achieving differentiation, but existing software often has difficulty supporting this. Finally, being able to conveniently edit algorithm parameters for fine-grained optimization and adjustment is also a basic requirement of professional developers.

[0032] In addition, most other similar streaming processing frameworks serve the Internet industry, and their architectures and design concepts are not fully applicable to the field of image processing. These frameworks usually rely on third-party components such as Kafka, increasing the complexity and maintenance cost of the system. This is not an ideal choice for developers who need efficient and stable image processing solutions.

[0033] In view of this, the present invention discloses a lightweight and editable streaming data processing framework, which meets the in-depth requirements of professional developers for data structures, data flows, algorithm integration, and parameter tuning, avoids excessive reliance on third-party components, and provides a high-performance and scalable solution. The technical solution of the present invention is as follows:

[0034] The present invention discloses a lightweight and editable streaming data processing framework, as Figure 1 - Figure 2 shown, including the following steps:

[0035] A. Load the operator library;

[0036] Among them, the system loads a predefined operator library. The operator library contains the implementations of various data processing operations, such as filtering, aggregation, transformation, cleaning, etc. These operators are organized in a modular manner, supporting users to expand and customize according to needs, conveniently adding new data processing functions, and enhancing the scalability of the system.

[0037] B. Load the json text and create a computation flow;

[0038] Among them, the system loads a JSON format configuration file that complies with the framework specifications. This JSON text details the structure of the computation flow, the configuration of each node, and the data processing process. By parsing the JSON text, the system automatically creates a computation flow network, establishes connections between nodes, and forms a complete data processing link.

[0039] C. The start node inputs and processes data;

[0040] Among them, after the computation flow is successfully created, the start node begins to receive data input. The data source can be real-time stream data, batch file data, database reading, etc. The start node preprocesses the input data, such as data format conversion, preliminary filtering, etc.

[0041] In some specific embodiments, after the data is processed at the start node, according to the configuration of the computation flow, the processing programs of subsequent nodes are automatically executed. Each node performs specific processing operations on the received data according to the set operators and passes the results to the next node, realizing the step-by-step processing and transfer of data.

[0042] D. The end node obtains the result.

[0043] Among them, the end node outputs the processed data result. The user can obtain the result from the end node for subsequent data analysis, visualization display, or business applications. With the mechanism of the callback function, the processing and application of the result become more flexible and efficient.

[0044] In some specific embodiments, step B is specifically to load a json text that complies with the framework specifications and create the computation flow described by it.

[0045] In some specific embodiments, the start node inputs data, and the data automatically executes the processing program of the node and streams the processing result to the next node.

[0046] In some specific embodiments, there is an E step between steps B and C; step E is specifically to perform a normative check on the input json text. If it passes, step B is carried out; if it fails, the program exits.

[0047] Among them, this check includes:

[0048] Syntax check: Verify whether the syntax of the JSON text is correct to avoid parsing failure due to format errors.

[0049] Structure check: Ensure that the configuration items in the JSON text comply with the framework specifications, including necessary fields and correct values.

[0050] Security check: Detect whether there are potential security risks in the JSON text, such as malicious code injection, etc.

[0051] In the above content, the system will only continue to execute the subsequent steps if the check passes; if the check fails, the system will exit the program and prompt the user to make modifications.

[0052] In some specific embodiments, an F step is provided after the E step; specifically, the F step is to register a callback function for the end node and automatically process the result of the end node.

[0053] In some implementable embodiments, the RxCpp library is introduced first, then the data flow of the node is defined, then the node processing logic is implemented, and finally the end callback is registered. Based on this, the subscription of the node and the registration of the callback function are achieved through RxCpp to achieve the effect of asynchronous streaming programming.

[0054] In some implementable embodiments, the present invention includes the header files provided by the framework, uses the templates and macros provided by it, writes the algorithm class according to the specifications and compiles it into a dynamic library, and the framework automatically loads the dynamic library to import the algorithm class.

[0055] In some implementable embodiments, the present invention prepares a json text that conforms to the framework specifications, and calls the corresponding API to generate a calculation flow.

[0056] The advantages of the present invention are as follows:

[0057] 1. The framework adopts a lightweight architecture, reducing system resource consumption and improving operation efficiency, and is suitable for resource-constrained environments and data processing scenarios that require high performance.

[0058] 2. By loading a JSON text that conforms to the framework specifications, users can edit and configure the calculation flow by themselves, flexibly customize the data processing process according to actual needs, and improve the adaptability of the system.

[0059] 3. The entire data processing process is clearly divided into steps such as loading the operator library, creating a calculation flow, data input processing, and result acquisition. The responsibilities of each module are clear, which is convenient for maintenance and expansion.

[0060] 4. A normative check on the JSON text is added between creating the calculation flow and data processing to ensure that the input data and configuration meet the requirements, avoiding operation exceptions caused by incorrect data formats, and enhancing the stability of the system.

[0061] 5. Registering a callback function at the end node can automatically process the result of the end node, facilitating subsequent operations on the processed data, such as storage, analysis, or triggering other processes. It improves the system's ability to handle abnormal situations and ensures the reliability of data processing.

[0062] Based on the above content, this streaming data processing framework is applicable to fields such as real-time data processing, big data analysis, and Internet of Things data collection. Its characteristics of lightweight and editability give it significant advantages in data processing scenarios that require quick response and flexible configuration. It can play an important role in fields such as financial real-time transaction analysis, industrial sensor data monitoring, and network real-time log processing.

Claims

1. A lightweight editable streaming data processing framework, characterized in that It includes the following steps: A. Load the operator library; B. Load the json text and create a calculation flow; C. The start node inputs and processes data; D. The end node obtains the result.

2. The lightweight editable streaming data processing framework according to claim 1, wherein Specifically, in step B, load the json text that conforms to the framework specification and create the calculation flow described by it.

3. A lightweight editable streaming data processing framework according to claim 1, characterized in that Specifically, in step C, the start node inputs data, and the data automatically executes the processing program of the node and streams the processing result to the next node.

4. A lightweight editable streaming data processing framework according to claim 1, characterized in that, There is an E step between steps B and C; Specifically, in step E, perform a normative check on the input json text. If it passes, proceed to step B; if it fails, exit the program.

5. A lightweight editable streaming data processing framework according to claim 4, characterized in that There is an F step after step E; Specifically, in step F, register a callback function for the end node to automatically process the result of the end node.

Images