Stream data processing method and device, computer device and storage medium

Abstract

The embodiment of the application belongs to the field of big data, and relates to a stream data processing method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining device source data from a target device; performing standardization processing on the device source data based on a physical model corresponding to the target device to obtain device stream data; obtaining a stream data processing task chain corresponding to the target device, wherein the stream data processing task chain comprises a plurality of data processing tasks, the data processing tasks exist in a task pool, and the data processing tasks comprise general type tasks and custom type tasks; and performing data processing on the device stream data through the stream data processing task chain. The application improves the stream data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a streaming data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In the field of AIoT (Artificial Intelligence of Things, integrating AI (Artificial Intelligence) and IoT (Internet of Things) technologies), streaming data processing is a crucial means of cleaning, aggregating, and re-enabling device data. Streaming data processing functions are determined based on business needs, and typically involve multiple processing stages, some of which, such as data filtering, may be generic. However, existing streaming data processing technologies often require developers to write code specifically for different business scenarios. This coding process is complex, difficult to learn, and the code development is one-off, lacking dynamic orchestration. Generic data processing stages need to be repeatedly implemented, resulting in low streaming data processing efficiency. Summary of the Invention

[0003] The purpose of this application is to provide a streaming data processing method, apparatus, computer device, and storage medium to solve the problem of low efficiency in streaming data processing.

[0004] To address the aforementioned technical problems, this application provides a streaming data processing method, employing the following technical solution:

[0005] Obtain device source data from the target device;

[0006] Based on the object model corresponding to the target device, the device source data is standardized to obtain device flow data;

[0007] Obtain the streaming data processing task chain corresponding to the target device. The streaming data processing task chain contains multiple data processing tasks. The data processing tasks exist in the task pool and include general type tasks and custom type tasks.

[0008] The device stream data is processed through the stream data processing task chain.

[0009] To address the aforementioned technical problems, this application also provides a streaming data processing apparatus, which employs the following technical solution:

[0010] The data acquisition module is used to acquire device source data from the target device;

[0011] The standard processing module is used to standardize the source data of the device based on the object model corresponding to the target device to obtain device flow data;

[0012] The task chain acquisition module is used to acquire the streaming data processing task chain corresponding to the target device. The streaming data processing task chain contains multiple data processing tasks. The data processing tasks exist in the task pool and include general type tasks and custom type tasks.

[0013] The data processing module is used to process the device stream data through the stream data processing task chain.

[0014] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0015] Obtain device source data from the target device;

[0016] Based on the object model corresponding to the target device, the device source data is standardized to obtain device flow data;

[0017] Obtain the streaming data processing task chain corresponding to the target device. The streaming data processing task chain contains multiple data processing tasks. The data processing tasks exist in the task pool and include general type tasks and custom type tasks.

[0018] The device stream data is processed through the stream data processing task chain.

[0019] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0020] Obtain device source data from the target device;

[0021] Based on the object model corresponding to the target device, the device source data is standardized to obtain device flow data;

[0022] Obtain the streaming data processing task chain corresponding to the target device. The streaming data processing task chain contains multiple data processing tasks. The data processing tasks exist in the task pool and include general type tasks and custom type tasks.

[0023] The device stream data is processed through the stream data processing task chain.

[0024] Compared with existing technologies, the embodiments of this application have the following main advantages: Original device source data is obtained from the target device; based on the object model corresponding to the target device, the device source data is standardized to obtain device stream data, making the data more standardized and easier to understand; a stream data processing task chain corresponding to the target device is obtained, containing multiple data processing tasks, with different data processing tasks corresponding to different process nodes in the stream data processing; data processing tasks exist in a task pool, including general-type tasks and custom-type tasks; general-type tasks implement relatively conventional data processing processes, while custom-type tasks implement data processing processes with more business attributes; the complete data processing process is decoupled, allowing data processing tasks to be selected from the task pool according to requirements, achieving task reuse, reducing repetitive development processes, lowering development difficulty, and improving stream data processing efficiency; the stream data processing task chain enables one-stop processing of device stream data, further improving stream data processing efficiency. Attached Figure Description

[0025] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0027] Figure 2 This is a flowchart of an embodiment of the streaming data processing method according to this application;

[0028] Figure 3 This is a schematic diagram of the structure of one embodiment of the streaming data processing apparatus according to this application;

[0029] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0031] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0033] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a processing device 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and processing device 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0034] Users can use terminal devices 101, 102, and 103 to interact with processing device 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0035] Terminal devices 101, 102, and 103 are Internet of Things (IoT) devices. They can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.; they can also be various electronic and electrical devices without displays and without support for web browsing, such as electricity meters.

[0036] The processing device 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103, and can perform streaming data processing on the source data generated by IoT devices; the processing device 105 can also be a terminal device.

[0037] It should be noted that the streaming data processing method provided in the embodiments of this application is generally executed by a processing device, and correspondingly, the streaming data processing device is generally disposed in the processing device.

[0038] It should be understood that Figure 1 The number of terminal devices, networks, and processing devices shown is merely illustrative. Any number of terminal devices, networks, and processing devices can be included depending on implementation needs.

[0039] Continue to refer to Figure 2 A flowchart of an embodiment of the streaming data processing method according to this application is shown. The streaming data processing method includes the following steps:

[0040] Step S201: Obtain device source data from the target device.

[0041] In this embodiment, the streaming data processing method runs on an electronic device (e.g., Figure 1 The processing device shown can communicate with the terminal device via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future wireless connection methods.

[0042] Specifically, the processing device obtains device source data from the target device, which can be one or more terminal devices. The device source data is the data in its original form from the target device. The processing device can obtain the device source data in various ways, such as by calling an interface.

[0043] Step S202: Based on the object model corresponding to the target device, the device source data is standardized to obtain device flow data.

[0044] Specifically, source data is data on the original form of the target device, which is not conducive to reading and understanding. To facilitate data reading, understanding, calculation, or storage, a physical model corresponding to the target device can be obtained.

[0045] A product model is a digital description of a product, defining its functions. It abstracts and summarizes the functions of products from different brands and categories, forming a "standard product model" that facilitates the description, control, and understanding of product functions using a unified language. A product model consists of several "parameters," which are further categorized into attributes, methods, and events based on the type of function they describe. Attributes generally describe the device's state and support reading and setting; methods refer to the device's capabilities or methods that can be invoked externally, allowing the setting of input and output parameters, and each parameter must be an attribute. Compared to attributes, services can implement more complex business logic through a single command; events describe events reported by the device to the cloud, can contain multiple input parameters, and each parameter must be an attribute.

[0046] Based on the object model, the source equipment data is standardized to obtain the equipment flow data. Compared to the source equipment data, the equipment flow data has a more standardized format, is easier to understand and read, and facilitates subsequent data processing.

[0047] Step S203: Obtain the streaming data processing task chain corresponding to the target device. The streaming data processing task chain contains multiple data processing tasks. The data processing tasks exist in the task pool and include general type tasks and custom type tasks.

[0048] Specifically, this application establishes a streaming data processing task chain, and the target device has at least one streaming data processing task chain. This application divides the processing of streaming data into different process nodes, each process node having at least one data processing task, and the data processing tasks at each process node together constitute the streaming data processing task chain for the target device.

[0049] This application establishes a task pool, which stores multiple different data processing tasks. It supports visual configuration and task reuse, allowing users to select necessary data processing tasks from the pool to build streaming data processing task chains. Data processing tasks can be broadly categorized into general-type tasks and custom-type tasks. Different data processing needs may involve the same or similar data processing procedures, which appear as general-type tasks; while some data processing procedures are strongly related to business needs or requirements, and these appear as custom-type tasks.

[0050] By decoupling the data processing process, some data processing tasks can be implemented directly by selecting data processing tasks from the task pool, or by selecting data processing tasks from the task pool and making adaptive modifications. This achieves task reuse in a simple way, simplifies the work of developers, avoids repetitive work, and improves the efficiency of streaming data processing.

[0051] Step S204: Process the device stream data through the stream data processing task chain.

[0052] Specifically, streaming data is processed through a streaming data processing task chain to obtain the desired results from the user. It can be understood that the streaming data processing task chain can be pre-built, allowing direct processing of the received streaming data based on the task chain, further improving processing efficiency. Alternatively, the streaming data processing task chain can be built after the streaming data is received, reducing the storage and operational burden on the processing devices.

[0053] In this embodiment, raw device source data is obtained from the target device. Based on the object model corresponding to the target device, the device source data is standardized to obtain device stream data, making the data more standardized and easier to understand. A stream data processing task chain corresponding to the target device is obtained. This task chain contains multiple data processing tasks, each corresponding to a different process node in the stream data processing. The data processing tasks exist in a task pool and include general-type tasks and custom-type tasks. General-type tasks implement relatively conventional data processing procedures, while custom-type tasks implement data processing procedures with more business attributes. The complete data processing process is decoupled, allowing the selection of data processing tasks from the task pool as needed. This achieves task reuse, reduces repetitive development processes, lowers development difficulty, and improves stream data processing efficiency. The stream data processing task chain enables one-stop processing of device stream data, further improving stream data processing efficiency.

[0054] Furthermore, before the step of obtaining the streaming data processing task chain corresponding to the target device, the process may include: obtaining each data processing task added on the creation page; obtaining the task connection lines between each data processing task; and encapsulating the tasks based on each data processing task and the task connection lines between each data processing task to obtain the streaming data processing task chain.

[0055] Specifically, this application supports the visual configuration of streaming data processing task chains / data processing tasks. The terminal device can display a creation page, where users can add multiple data processing tasks in the canvas area of ​​the creation page by dragging, dropping, and creating new tasks.

[0056] Data processing tasks are displayed graphically in the canvas area, such as as rectangles. The graph can display the number of process nodes corresponding to the data processing task and the task name. The graph has shape and color, which can be used to distinguish different data processing tasks or different types of data processing tasks.

[0057] Data processing tasks have task configuration information. Adding this configuration information is equivalent to adding specific information to the task template, thereby achieving specific functions. For example, in a data filtering task, if it is necessary to select voltage values ​​greater than a voltage threshold from device stream data, then the voltage threshold and its magnitude relationship constitute the task configuration information. Users can configure the task configuration information for data processing tasks on the creation page.

[0058] The acquisition of equipment source data and the standardization of equipment source data can also be configured as data processing tasks in the manner described above.

[0059] Users can add task connections between various data processing tasks. These task connections are directional, and their direction represents the direction of data flow.

[0060] Tasks are encapsulated based on each data processing task and the task connections between them to obtain a streaming data processing task chain. In this embodiment, the streaming data processing task chain is pre-built, so when device streaming data is received, data processing can be performed directly according to the existing streaming data processing task chain, improving response speed.

[0061] Furthermore, the steps described above for obtaining the data processing tasks added on the creation page may include: displaying a task pool containing multiple data processing tasks; and adding at least one data processing task to the creation page according to a selection instruction for at least one data processing task in the task pool.

[0062] Specifically, the terminal device can display a task pool containing multiple existing data processing tasks. Users can view the details of each data processing task, select the required task, and trigger a selection command. The terminal device then adds the data processing task to the creation page based on the selection command, enabling task reuse.

[0063] After a data processing task is added to the creation page, users can modify the task configuration information to meet their own needs. For example, if a data processing task has a threshold feature with a value of 100, but the user needs a threshold of 200, then this threshold feature in the task configuration information can be modified.

[0064] In this embodiment, a task pool containing multiple data processing tasks is shown. At least one data processing task is added to the creation page according to the user's selection instructions, realizing task reuse. The visual configuration method reduces the development difficulty and improves the efficiency of streaming data processing.

[0065] Furthermore, in another embodiment of this application, the step of obtaining each data processing task added on the creation page may further include: obtaining task configuration information added on the creation page; creating at least one data processing task according to the task configuration information, and adding at least one data processing task to the task pool.

[0066] Specifically, users can select data processing tasks from the task pool or create new tasks directly. The creation page provides task creation templates, allowing users to enter various necessary task configuration information in a WYSIWYG manner, thereby defining the data processing flow. The processing device creates new data processing tasks based on the task configuration information.

[0067] New data processing tasks can also be added to the task pool for reuse in the future.

[0068] In this embodiment, the task configuration information added on the creation page is obtained, and a data processing task is created based on the task configuration information. The WYSIWYG configuration method reduces and improves the generation efficiency of data processing tasks, thereby ensuring the efficiency of streaming data processing. The created data processing task is added to the task pool so that the data processing task can be reused.

[0069] In this embodiment, each data processing task added on the creation page and the task connection lines between each data processing task are obtained. The tasks are encapsulated according to each data processing task and the task connection lines between each data processing task to obtain the streaming data processing task chain. This realizes visual configuration, reduces development difficulty, and improves streaming data processing efficiency.

[0070] Furthermore, the steps described above for obtaining the streaming data processing task chain corresponding to the target device may include: querying the task chain information corresponding to the target device and extracting each task identifier from the task chain information; extracting the data processing tasks corresponding to each task identifier from the task pool; and encapsulating the tasks according to the task chain information and each data processing task to obtain the streaming data processing task chain corresponding to the target device.

[0071] Specifically, in another embodiment of this application, instead of generating the streaming data processing task chain in advance, the configuration of the streaming data processing task chain can be completed in advance to obtain the task chain information.

[0072] Upon receiving device stream data, the task chain information corresponding to the target device is queried. The task chain information contains the task identifiers of each required data processing task. Then, the data processing tasks corresponding to each task identifier are temporarily extracted from the task pool.

[0073] The task chain information can also include data flow information. The function of the data flow information is equivalent to the task connection line mentioned earlier. Based on the data flow information, each data processing task is encapsulated to obtain the streaming data processing task chain corresponding to the target device.

[0074] In this embodiment, streaming data processing task chains can be generated and destroyed in real time, thereby reducing the operating and storage pressure on processing devices without having to store a large number of streaming data processing task chains in memory or on the hard drive. In this embodiment, when configuring streaming data processing task chains, if a data processing task from the task pool is selected and the task configuration information is modified, a new data processing task needs to be created in the task pool to distinguish it from the original data processing task, which acts as a template.

[0075] In this embodiment, the task chain information corresponding to the target device is queried, the task identifiers in the task chain information are extracted, the data processing tasks corresponding to each task identifier are extracted from the task pool, and the tasks are encapsulated according to the task chain information and each data processing task to temporarily generate a streaming data processing task chain, which can reduce the pressure on the operation and storage of the processing device.

[0076] Furthermore, the streaming data processing task chain includes a data selection task, a data filtering task, and a custom task. The steps of processing device streaming data through the streaming data processing task chain can include: extracting first target type data from the device streaming data through the data selection task; filtering the first target type data through the data filtering task to obtain second target type data; and performing business processing on the second target type data through the custom task.

[0077] Specifically, the data processing tasks in the streaming data processing task chain can be categorized by function as data selection tasks, data filtering tasks, and custom tasks.

[0078] Typically, a data selection task selects the data to be processed from the device stream data to obtain the first target type of data. For example, when the target device is an electrical appliance, the device stream data may include voltage data, current data, resistance data, etc. If a user wants to analyze voltage, they can select voltage data through the data selection task to obtain the first target type of data.

[0079] Then, the data of the first target type is filtered through a data filtering task to obtain the data of the second target type. The business orientation of the data filtering task is more obvious than that of the data selection task. For example, if a user wants to generate a report on voltage anomaly monitoring, the data filtering task can be used to filter out voltage points that exceed the voltage threshold.

[0080] Finally, business processing is performed on the second target type of data through custom tasks. Custom tasks have the most obvious business orientation and are usually the end point of the stream data processing task chain.

[0081] In one embodiment, performing business processing on second target type data through a custom task may include: performing business processing on second target type data through a custom task to achieve event push, attribute push, service call, or data storage.

[0082] Specifically, custom tasks perform business processing on the second target type of data, achieving purposes such as event pushing, attribute pushing, service calls, or data storage. Event pushing can push the processing results to the target object in the form of events, such as monitoring voltage and triggering an alert event to notify relevant personnel when an anomaly is detected. Attribute pushing can add the processing results as attribute information to the object to which the attribute belongs, such as calculating the power consumption for a certain day and recording the power consumption in the power consumption statistics of a table. Service calls can call relevant interfaces or services based on the processing results. When configuring data processing tasks, it is necessary to configure the interfaces or services that may be called, or customize the data processing logic according to some software development kits (SDKs). Data storage simply stores the processing results according to a preset method.

[0083] In this embodiment, after the custom task performs business processing on the second target type data, it can perform event push, attribute push, service call or data storage, which meets diverse business scenarios.

[0084] In this embodiment, the device stream data is processed sequentially by the data selection task, the data filtering task, and the custom task, realizing one-stop processing of the device stream data.

[0085] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0086] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0087] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a streaming data processing apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0088] like Figure 3 As shown, the streaming data processing device 300 described in this embodiment includes: a data acquisition module 301, a standard processing module 302, a task chain acquisition module 303, and a data processing module 304, wherein:

[0089] The data acquisition module 301 is used to acquire device source data from the target device.

[0090] The standard processing module 302 is used to standardize the source data of the device based on the object model corresponding to the target device to obtain the device flow data.

[0091] The task chain acquisition module 303 is used to acquire the streaming data processing task chain corresponding to the target device. The streaming data processing task chain contains multiple data processing tasks. The data processing tasks exist in the task pool and include general type tasks and custom type tasks.

[0092] The data processing module 304 is used to process device stream data through a stream data processing task chain.

[0093] In this embodiment, raw device source data is obtained from the target device. Based on the object model corresponding to the target device, the device source data is standardized to obtain device stream data, making the data more standardized and easier to understand. A stream data processing task chain corresponding to the target device is obtained. This task chain contains multiple data processing tasks, each corresponding to a different process node in the stream data processing. The data processing tasks exist in a task pool and include general-type tasks and custom-type tasks. General-type tasks implement relatively conventional data processing procedures, while custom-type tasks implement data processing procedures with more business attributes. The complete data processing process is decoupled, allowing the selection of data processing tasks from the task pool as needed. This achieves task reuse, reduces repetitive development processes, lowers development difficulty, and improves stream data processing efficiency. The stream data processing task chain enables one-stop processing of device stream data, further improving stream data processing efficiency.

[0094] In some optional implementations of this embodiment, the streaming data processing device 300 may further include: a task acquisition module, a connection acquisition module, and a task encapsulation module, wherein:

[0095] The task acquisition module is used to acquire the various data processing tasks added on the creation page.

[0096] The connection acquisition module is used to acquire the task connection lines between various data processing tasks.

[0097] The task encapsulation module is used to encapsulate tasks based on each data processing task and the task connections between them, thus obtaining a streaming data processing task chain.

[0098] In this embodiment, each data processing task added on the creation page and the task connection lines between each data processing task are obtained. The tasks are encapsulated according to each data processing task and the task connection lines between each data processing task to obtain the streaming data processing task chain. This realizes visual configuration, reduces development difficulty, and improves streaming data processing efficiency.

[0099] In some optional implementations of this embodiment, the task acquisition module may include: a task pool display submodule and a task addition submodule, wherein:

[0100] The Task Pool Display submodule is used to display the task pool, which contains multiple data processing tasks.

[0101] The task addition submodule is used to add at least one data processing task to the creation page based on the selection instruction for at least one data processing task in the task pool.

[0102] In this embodiment, a task pool containing multiple data processing tasks is shown. At least one data processing task is added to the creation page according to the user's selection instructions, realizing task reuse. The visual configuration method reduces the development difficulty and improves the efficiency of streaming data processing.

[0103] In some alternative implementations of this embodiment, the task acquisition module may include: a configuration acquisition submodule and a task creation submodule, wherein:

[0104] The configuration retrieval submodule is used to retrieve the task configuration information added on the creation page.

[0105] The task creation submodule is used to create at least one data processing task based on the task configuration information and add at least one data processing task to the task pool.

[0106] In this embodiment, the task configuration information added on the creation page is obtained, and a data processing task is created based on the task configuration information. The WYSIWYG configuration method reduces and improves the generation efficiency of data processing tasks, thereby ensuring the efficiency of streaming data processing. The created data processing task is added to the task pool so that the data processing task can be reused.

[0107] In some optional implementations of this embodiment, the task chain acquisition module 303 may include: an information query submodule, a task extraction submodule, and a task encapsulation submodule, wherein:

[0108] The information query submodule is used to query the task chain information corresponding to the target device and extract the task identifiers from the task chain information.

[0109] The task extraction submodule is used to extract data processing tasks from the task pool that correspond to each task identifier.

[0110] The task encapsulation submodule is used to encapsulate tasks based on task chain information and each data processing task to obtain the stream data processing task chain corresponding to the target device.

[0111] In this embodiment, the task chain information corresponding to the target device is queried, the task identifiers in the task chain information are extracted, the data processing tasks corresponding to each task identifier are extracted from the task pool, and the tasks are encapsulated according to the task chain information and each data processing task to temporarily generate a streaming data processing task chain, which can reduce the pressure on the operation and storage of the processing device.

[0112] In some optional implementations of this embodiment, the streaming data processing task chain includes a data selection task, a data filtering task, and a custom task. The data processing module 304 may include: a data extraction submodule, a data filtering submodule, and a data processing submodule, wherein:

[0113] The data extraction submodule is used to extract the first target type of data from the device stream data through the data selection task.

[0114] The data filtering submodule is used to filter the first target type data through a data filtering task to obtain the second target type data.

[0115] The data processing submodule is used to perform business processing on the second target type data through custom tasks.

[0116] In this embodiment, the device stream data is processed sequentially by the data selection task, the data filtering task, and the custom task, realizing one-stop processing of the device stream data.

[0117] In some optional implementations of this embodiment, the data processing submodule is also used to perform business processing on the second target type data through custom tasks to realize event push, attribute push, service call or data storage.

[0118] In this embodiment, after the custom task performs business processing on the second target type data, it can perform event push, attribute push, service call or data storage, which meets diverse business scenarios.

[0119] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0120] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0121] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0122] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for streaming data processing methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0123] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the streaming data processing method.

[0124] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0125] The computer device provided in this embodiment can execute the above-described streaming data processing method. The streaming data processing method here can be any of the streaming data processing methods described in the various embodiments above.

[0126] In this embodiment, raw device source data is obtained from the target device. Based on the object model corresponding to the target device, the device source data is standardized to obtain device stream data, making the data more standardized and easier to understand. A stream data processing task chain corresponding to the target device is obtained. This task chain contains multiple data processing tasks, each corresponding to a different process node in the stream data processing. The data processing tasks exist in a task pool and include general-type tasks and custom-type tasks. General-type tasks implement relatively conventional data processing procedures, while custom-type tasks implement data processing procedures with more business attributes. The complete data processing process is decoupled, allowing the selection of data processing tasks from the task pool as needed. This achieves task reuse, reduces repetitive development processes, lowers development difficulty, and improves stream data processing efficiency. The stream data processing task chain enables one-stop processing of device stream data, further improving stream data processing efficiency.

[0127] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the streaming data processing method described above.

[0128] In this embodiment, raw device source data is obtained from the target device. Based on the object model corresponding to the target device, the device source data is standardized to obtain device stream data, making the data more standardized and easier to understand. A stream data processing task chain corresponding to the target device is obtained. This task chain contains multiple data processing tasks, each corresponding to a different process node in the stream data processing. The data processing tasks exist in a task pool and include general-type tasks and custom-type tasks. General-type tasks implement relatively conventional data processing procedures, while custom-type tasks implement data processing procedures with more business attributes. The complete data processing process is decoupled, allowing the selection of data processing tasks from the task pool as needed. This achieves task reuse, reduces repetitive development processes, lowers development difficulty, and improves stream data processing efficiency. The stream data processing task chain enables one-stop processing of device stream data, further improving stream data processing efficiency.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0130] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method of processing streaming data, the method comprising: The method comprises the following steps: obtaining device source data from a target device; standardizing the device source data based on a physical model corresponding to the target device to obtain device stream data; obtaining a stream data processing task chain corresponding to the target device, wherein the stream data processing task chain comprises a plurality of data processing tasks, the data processing tasks exist in a task pool, and the data processing tasks comprise general type tasks and custom type tasks; performing data processing on the device stream data through the stream data processing task chain; before the step of obtaining the stream data processing task chain corresponding to the target device, the method further comprises: obtaining each data processing task added in a creation page; obtaining task connection lines between the data processing tasks; performing task packaging based on the data processing tasks and the task connection lines to obtain the stream data processing task chain; the stream data processing task chain comprises a data selection task, a data filtering task, and a custom task, and the step of performing data processing on the device stream data through the stream data processing task chain comprises: extracting first target type data from the device stream data through the data selection task; performing data filtering on the first target type data through the data filtering task to obtain second target type data; performing business processing on the second target type data through the custom task; the step of performing business processing on the second target type data through the custom task comprises: performing business processing on the second target type data through the custom task to realize event pushing, attribute pushing, service calling, or data storage.

2. The stream data processing method of claim 1, wherein, the step of obtaining each data processing task added in the creation page comprises: displaying a task pool, wherein the task pool comprises a plurality of data processing tasks; adding at least one data processing task in the task pool to the creation page according to a selection instruction of the at least one data processing task.

3. The stream data processing method of claim 1, wherein, the step of obtaining each data processing task added in the creation page further comprises: obtaining task configuration information added in the creation page; creating at least one data processing task according to the task configuration information and adding the at least one data processing task to the task pool.

4. The stream data processing method of claim 1, wherein, the step of obtaining the stream data processing task chain corresponding to the target device comprises: querying task chain information corresponding to the target device and extracting each task identifier in the task chain information; extracting data processing tasks corresponding to the task identifiers from the task pool; performing task packaging based on the task chain information and the data processing tasks to obtain the stream data processing task chain corresponding to the target device.

5. A stream data processing apparatus, characterized by comprising: The stream data processing device implements the steps of the stream data processing method according to any one of claims 1 to 4, and comprises: a data acquisition module configured to obtain device source data from a target device; a standard processing module configured to standardize the device source data based on a physical model corresponding to the target device to obtain device stream data; A task chain acquisition module is configured to acquire a stream data processing task chain corresponding to the target device, the stream data processing task chain including a plurality of data processing tasks, the data processing tasks existing in a task pool, and the data processing tasks including general type tasks and customized type tasks. A data processing module is configured to perform data processing on the device stream data through the stream data processing task chain. 6.A computer device, comprising a memory and a processor, wherein the memory stores computer readable instructions, and the processor executes the computer readable instructions to implement the steps of the stream data processing method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the steps of the stream data processing method according to any one of claims 1 to 4.

Images