Embedded device data real-time analysis method and device based on DuckDB

Abstract

The invention relates to a DuckDB-based embedded device data real-time analysis method and device, and the method comprises the steps: receiving to-be-processed data, carrying out the parallel scheduling of execution threads of a plurality of data processing rules according to a preset processing period, and carrying out the concurrent processing of a plurality of paths of data streams; serially executing a series of processing jobs on the execution thread of each data processing rule according to a preset processing assembly line sequence, and forming a data processing assembly line by taking the output of the previous job as the input of the next job; respectively verifying before and after the execution thread of each job is started, and after the verification is passed, acquiring data from the data source according to the configured input mapping and caching the data; and processing by different data processing functions according to the job type, and sending a processing result to a specified target through configured output mapping. According to the invention, the data processing efficiency is optimized, the flexibility and compatibility of the system are enhanced, and efficient and reliable real-time data analysis is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of embedded device data processing and analysis, and in particular relates to a method and device for real-time analysis of embedded device data based on DuckDB. Background Art

[0002] With the rapid development of Internet of Things technology, embedded devices have been widely used in various fields. These devices generate a large amount of data during operation. How to process this data efficiently and in real time has become a key issue in the current technological development. Traditional data processing methods mainly rely on remote servers or large data centers, but this method has obvious limitations in embedded device applications, such as network latency, high data transmission costs, and privacy protection. It can no longer meet the real-time requirements in some scenarios. Edge computing, as an emerging computing model, pushes computing tasks from the cloud to the edge of the network, that is, the device or terminal side, to achieve local processing and rapid response of data. However, embedded devices are usually resource-constrained, such as processing power, storage space, and energy consumption. Therefore, how to achieve efficient real-time data analysis on embedded devices has become a technical problem that needs to be solved urgently.

[0003] At present, there are some data processing and analysis tools for embedded devices on the market, but they usually have the following problems: first, the data processing efficiency is low and it is difficult to meet the needs of real-time analysis; second, the functions are single and it is difficult to support the processing of multiple data sources and data formats; third, the configuration is complex and the usage threshold is high, which is not conducive to rapid deployment and application.

[0004] DuckDB is a lightweight, efficient, powerful, embedded executable column-based database, which is particularly suitable for running on devices with limited resources. Therefore, the present invention proposes a real-time analysis device for embedded device data based on duckdb to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a method and device for real-time analysis of embedded device data based on DuckDB, so as to solve the problems of low efficiency, single function and complex configuration in the prior art.

[0006] To achieve one of the above-mentioned purposes, an embodiment of the present invention provides a real-time analysis method for embedded device data based on DuckDB, which is characterized by comprising: Receive data to be processed, and schedule multiple execution threads of data processing rules in parallel according to the preset processing cycle, and process multiple data streams concurrently; The execution thread of each data processing rule executes a series of processing jobs in series according to the preset processing pipeline sequence, and forms a data processing pipeline by using the output of the previous job as the input of the next job; Verification is performed before and after each job execution thread starts. After verification, data is obtained from the data source and cached according to the configured input mapping; Different data processing functions are used to process the data according to the job type, and the processing results are sent to the specified target through the configured output mapping.

[0007] As a further improvement of an embodiment of the present invention, the method further includes, after the system is started, initializing configuration information for data processing; After the initialization is completed, the configuration information is verified, and if the verification passes, data processing is performed according to the configuration information.

[0008] As a further improvement of an embodiment of the present invention, the method further includes that the configuration information includes a custom Grok expression group, DataMapper data mapping and data processing rules; The DataMapper data mapping includes the connection information of the data source and the target, the mapping of the data source field to the target field and the data type conversion rules, as well as the configuration of the input mapping and the output mapping; The data sources include Redis, MQTT, MySQL, memory queue, internal table persistence, Elasticsearch, Syslog service, HTTP and static files; The targets include Elasticsearch, MySQL (Doris), Syslog, memory queues, and internal tables; The input mapping is used to define the conversion rules from the data source to the internal data format of DuckDB, and the output mapping is used to define the conversion rules from the internal data format of DuckDB to the target format.

[0009] As a further improvement of an embodiment of the present invention, the method further includes that each job is verified before and after the execution of the thread, including: The pre-job execution verification includes input data source verification and job configuration verification; the input data source verification is used to verify the availability of the data source and the correctness of the data format; the job configuration verification is used to verify the correctness and validity of the data processing function; Post-job execution verification includes result set format verification and optional result set content verification; the result set format verification is used to verify whether the format of the result set meets expectations; the result set content verification is used to verify whether the content of the result set meets expectations based on predefined rules or thresholds, including verifying whether the values ​​in the result set are within a preset range or verifying whether the data in the result set meets a specific pattern.

[0010] As a further improvement of an embodiment of the present invention, the method further includes: the cache adopts a memory buffer based on DuckDB to improve data access efficiency and realize data transfer between previous and next jobs.

[0011] As a further improvement of an embodiment of the present invention, the method further includes: a data processing function is used to perform data conversion, filtering and calculation operations; the data processing function includes SQL filtering, Grok filtering and user-defined function processing.

[0012] The SQL filtering is performed by DuckDB, the Grok filtering is performed by the Grok library, and the user-defined function is loaded and executed through a plug-in mechanism.

[0013] As a further improvement of an embodiment of the present invention, the method also includes, after the execution thread is completed, releasing the used DuckDB buffer and recovering related memory resources to prepare for the next scheduling and execution of data processing rules and maintain efficient and stable system operation.

[0014] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention provides a real-time analysis device for embedded device data based on DuckDB, characterized in that it includes a receiving module, a pipeline processing module, a data verification module and an output module; The data receiving and scheduling module is used to receive data to be processed, and according to a preset processing cycle, to schedule execution threads of multiple data processing rules in parallel, and to process multiple data streams concurrently; The pipeline processing module is used to execute a series of processing jobs in series according to a preset processing pipeline sequence for the execution thread of each data processing rule, and form a data processing pipeline by taking the output of a previous job as the input of a subsequent job; The data verification module is used to perform verification before and after the execution thread of each job is started. After the verification is passed, data is obtained from the data source according to the configured input mapping and cached; The output module is used to process the job by different data processing functions according to the job type, and send the processing result to the specified target through the configured output mapping.

[0015] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention further provides an electronic device, comprising a memory and a processor, characterized in that the memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the real-time analysis method of embedded device data based on DuckDB as described above are implemented.

[0016] To achieve one of the above-mentioned purposes of the invention, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores a computer program, and wherein when the computer program is executed by a processor, the steps in the above-mentioned method for real-time analysis of embedded device data based on DuckDB are implemented.

[0017] Compared with the prior art, the present invention provides a method and device for real-time analysis of embedded device data based on DuckDB, which optimizes data processing efficiency by parallel scheduling and serial execution of data processing rules; adopts input-output mapping and multiple data source support to enhance the flexibility and compatibility of the system; ensures the accuracy and stability of data processing through verification mechanism and caching strategy, and realizes efficient and reliable real-time data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is an overall flow chart of the real-time analysis method of embedded device data based on DuckDB described in the present invention.

[0019] Figure 2 It is a specific flow chart of the real-time analysis method of embedded device data based on DuckDB described in the present invention.

[0020] Figure 3 It is a schematic diagram of the architecture of the embedded device data real-time analysis device based on DuckDB described in the present invention.

[0021] Figure 4 It is an overall design diagram of the embedded device data real-time analysis device based on DuckDB described in the present invention. DETAILED DESCRIPTION

[0022] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0023] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0024] In the first embodiment of the present invention, the present invention provides a real-time analysis method of embedded device data based on DuckDB, such as Figure 1 As shown, the method includes, S1: Receive data to be processed, and schedule multiple execution threads of data processing rules in parallel according to a preset processing cycle to concurrently process multiple data streams; S2: The execution thread of each data processing rule executes a series of processing jobs in series according to the preset processing pipeline sequence, and forms a data processing pipeline by using the output of the previous job as the input of the next job; S3: Verification is performed before and after the execution thread of each job. After verification, data is obtained from the data source and cached according to the configured input mapping; S4: Different data processing functions are used to process the job according to the job type, and the processing results are sent to the specified target through the configured output mapping.

[0025] In a specific embodiment of the present invention, after the system is started, configuration information for data processing is initialized; After the initialization is completed, the configuration information is verified, and if the verification passes, data processing is performed according to the configuration information.

[0026] Furthermore, the initialization after system startup includes: Initialization process: After the system is started, the configuration is initialized first. This process includes loading configuration information from storage media (such as the local file system or a remote configuration server) to ensure that each module can operate according to the preset configuration parameters. Configuration parameter loading: The system reads relevant parameters from the configuration information. These parameters define the system's operating environment, data processing logic, input and output stream paths and formats, etc. During the initialization process, the system will also set some default values ​​to prevent certain key parameters from being missing in the configuration information. Component initialization: After loading the configuration parameters, the system will initialize each core component in turn, including the input stream module, the core job module, and the output stream module. The initialization of each component will be specifically set according to the parameters in the configuration information. For example, the input stream module will configure the data source path and data format, the core job module will configure the DuckDB engine and data processing rules, and the output stream module will configure the data target location, etc.

[0027] Furthermore, configuration information verification includes: Verification steps: After initialization is completed, the system will verify the configuration information. The verification process mainly includes the following aspects: Legality verification: Check whether the configuration information format is correct and whether the parameter value is within the legal range. For example, check whether the path exists, whether the format is correct, whether the numerical parameter is within the allowable range, etc. Integrity verification: Ensure that all necessary parameters are included in the configuration information. For missing parameters, the system will record a warning log and prompt the user to supplement them. Consistency verification: Verify whether the logical relationship between each parameter is consistent. For example, the paths of the input stream and the output stream cannot be the same, the processing rules should match the input data format, etc. Verification result processing: After the verification passes, the system will record a successful verification log and enter the data processing stage; if the verification fails, the system will record an error log and prompt the user to make corrections. At this time, the system enters standby mode, waiting for the user to correct the configuration information and restart.

[0028] In a specific embodiment of the present invention, the configuration information includes a custom Grok expression group, a DataMapper data mapping, and a data processing rule; The DataMapper data mapping includes the connection information of the data source and the target, the mapping of the data source field to the target field and the data type conversion rules, as well as the configuration of the input mapping and the output mapping; The data sources include Redis, MQTT, MySQL, memory queue, internal table persistence, Elasticsearch, Syslog service, HTTP and static files; The targets include Elasticsearch, MySQL (Doris), Syslog, memory queues, and internal tables; The input mapping is used to define the conversion rules from the data source to the internal data format of DuckDB, and the output mapping is used to define the conversion rules from the internal data format of DuckDB to the target format.

[0029] It should be noted that Grok expressions are used to parse and extract fields from unstructured text data. Custom Grok expression groups allow users to define regular expressions that conform to a specific format to extract key information from logs or other unstructured data sources. Grok expressions are mainly used to extract required data fields from Syslog services, HTTP logs, and static files for subsequent processing.

[0030] It should be noted that DataMapper data mapping specifically includes: Connection information for data sources and targets: defines how to connect to various data sources and data targets, including connection URLs, ports, authentication information, etc. Mapping of data source fields to target fields: defines how to map fields in the data source to target fields, including the correspondence between field names. Data type conversion rules: defines data type conversion rules from data source fields to target fields. Ensures that data types between different systems can be correctly converted during data transmission. Input mapping: defines conversion rules from data sources to DuckDB internal data formats. Including field name mapping, data type conversion, and necessary data cleaning and preprocessing. Output mapping: defines conversion rules from DuckDB internal data formats to target formats. Ensures that data processed by DuckDB can be correctly transmitted to the target system, and that the format and type meet the requirements of the target system.

[0031] The data sources include: Redis: supports reading data from the Redis cache database, which is often used for real-time data stream processing. MQTT: supports reading data from the MQTT message queue, which is often used for data collection from Internet of Things (IoT) devices. MySQL: supports reading data from the MySQL relational database, which is often used for structured data processing. Memory queue: supports reading data from the system memory queue, which is often used for high-throughput data stream processing. Internal table persistence: supports reading data from the system's internal persistence table to facilitate historical tracing and analysis of data. Elasticsearch: supports reading data from the Elasticsearch search engine, which is often used for processing logs and full-text retrieval data. Syslog service: supports reading log data from the Syslog service, which is often used for the collection and analysis of system and network logs. HTTP: supports obtaining data from the HTTP service interface, which is often used to obtain dynamic data from Web services. Static files: supports reading local or remote static files, which is often used for batch data analysis.

[0032] Data target specific packages: Elasticsearch: Supports writing processed data to Elasticsearch for search and analysis. MySQL (Doris): Supports writing data to MySQL database or Apache Doris data warehouse for structured data storage and analysis. Syslog: Supports writing data to Syslog service for system and network log collection. Memory queue: Supports writing data to system memory queue for real-time data processing. Internal table: Supports writing data to system internal table for persistent data storage and subsequent analysis.

[0033] Input mapping is used to define the conversion rules from the data source to the internal data format of DuckDB, including: Field name mapping: Map the field names in the data source to the field names in the DuckDB table. Data type conversion: Convert the field types in the data source to the corresponding types in DuckDB. Data cleaning and preprocessing: Perform necessary data cleaning and preprocessing during the data conversion process.

[0034] Output mapping is used to define the conversion rules from DuckDB internal data format to the target format, including: Field name mapping: Map the field names in the DuckDB table to the field names in the target system. Data type conversion: Convert the field types in DuckDB to the corresponding types in the target system. Data formatting: Perform necessary formatting on the data according to the requirements of the target system.

[0035] In a specific implementation of the present invention, each job is verified before and after the execution of the thread, specifically, The pre-job execution verification includes input data source verification and job configuration verification; the input data source verification is used to verify the availability of the data source and the correctness of the data format; the job configuration verification is used to verify the correctness and validity of the data processing function; Post-job execution verification includes result set format verification and optional result set content verification; the result set format verification is used to verify whether the format of the result set meets expectations; the result set content verification is used to verify whether the content of the result set meets expectations based on predefined rules or thresholds, including verifying whether the values ​​in the result set are within a preset range or verifying whether the data in the result set meets a specific pattern.

[0036] It should be noted that pre-job verification specifically includes: Input data source verification: Verify the availability of the data source, including checking whether the data source is online and whether the connection information is correct; verify whether the data format of the data source is correct, such as checking the field type, data length, data integrity, etc. Job configuration verification: Verify the correctness of SQL statements, including syntax checking, field reference checking, table name checking, etc., to ensure that SQL statements can be executed correctly; verify the correctness of Grok expressions, to ensure that custom Grok expressions can correctly parse and match input data; verify the correctness and validity of user-defined functions, including syntax checking and logical correctness checking, to ensure that custom functions can be executed correctly and return expected results.

[0037] The post-job execution verification specifically includes: Result set format verification: Verify whether the format of the result set meets expectations, including checking the number of fields, field types, field order, etc., to ensure that the result set can be correctly parsed and used; Result set content verification: Verify the content of the result set according to predefined rules or thresholds to ensure that the data in the result set meets expectations; Verify whether the values ​​in the result set are within the preset range, such as checking whether the values ​​of certain key fields are within a reasonable range; Verify whether the data in the result set conforms to a specific pattern, such as checking the data format, string pattern, etc.

[0038] In a specific embodiment of the present invention, the cache uses a memory buffer based on DuckDB to improve data access efficiency and realize data transfer between previous and next jobs.

[0039] It should be noted that the memory buffer based on DuckDB can provide fast data read and write operations, thereby improving data access efficiency. This method is suitable for scenarios that require fast temporary storage and access to small amounts of data.

[0040] In a specific implementation of the present invention, the data processing function is used to perform data conversion, filtering and calculation operations; the data processing function includes SQL filtering, Grok filtering and user-defined function processing.

[0041] The SQL filtering is performed by DuckDB, the Grok filtering is performed by the Grok library, and the user-defined function is loaded and executed through a plug-in mechanism.

[0042] It should be noted that data processing functions are used to perform various data conversion, filtering and calculation operations during data processing; SQL filtering can filter, aggregate, sort and convert data by writing SQL query statements; SQL filtering is suitable for structured data processing and can use DuckDB's high-performance query engine to efficiently process large-scale data. Grok is a powerful regular expression library that is often used to parse and extract specific fields in log data; by defining Grok expressions, unstructured text data can be parsed and filtered to extract the required information; Grok filtering is suitable for processing unstructured data such as log data and system output. User-defined function processing writes custom functions according to specific needs to implement specific data processing logic; custom functions can be used to implement complex business logic, specific calculation rules or data conversion operations; user-defined functions provide flexible expansion capabilities, allowing the system to adapt to various special needs.

[0043] In a specific embodiment of the present invention, after the execution thread is completed, the used DuckDB buffer is released and the related memory resources are recovered to prepare for the next scheduling and execution of the data processing rules and maintain the efficiency and stability of the system operation.

[0044] It should be noted that in order to ensure the efficiency and stability of the system operation, the resources used must be released in time after the execution thread completes the data processing operation. By releasing the DuckDB buffer and recycling related memory resources, it can prepare for the scheduling and execution of the next data processing rule, avoiding resource leakage and excessive memory usage.

[0045] Specifically, DuckDB buffer release: After the execution thread is completed, the system will call DuckDB's memory management interface to release the allocated memory buffer; releasing the memory buffer can ensure that no longer used memory resources are recycled in time to avoid memory leaks. Memory resource recycling: In addition to releasing DuckDB's memory buffer, the system also needs to recycle other allocated memory resources; through the garbage collection mechanism or manual memory management, release the temporary memory allocated during the data processing process.

[0046] In the second embodiment of the present invention, the present invention provides a real-time analysis device for embedded device data based on DuckDB, such as Figure 3 As shown, the system includes a receiving module 1, a pipeline processing module 2, a data verification module 3 and an output module 4; The receiving module 1 is used to receive data to be processed, and according to a preset processing cycle, to schedule execution threads of multiple data processing rules in parallel, and to process multiple data streams concurrently; The pipeline processing module 2 is used to execute a series of processing jobs in series according to a preset processing pipeline sequence for the execution thread of each data processing rule, and form a data processing pipeline by taking the output of the previous job as the input of the next job; The data verification module 3 is used to perform verification before and after the execution thread of each job is started. After the verification is passed, data is obtained from the data source according to the configured input mapping and cached; The output module 4 is used to process the job by different data processing functions according to the job type, and send the processing result to the specified target through the configured output mapping.

[0047] In embodiment three of the present invention, the present invention provides an electronic device, comprising a memory and a processor, characterized in that the memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the real-time analysis method of embedded device data based on DuckDB as described above are implemented.

[0048] In a fourth embodiment of the present invention, the present invention provides a storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps in the above-mentioned method for real-time analysis of embedded device data based on DuckDB.

[0049] In summary, the method and device for real-time analysis of embedded device data based on DuckDB provided by the present invention optimize the data processing efficiency through parallel scheduling and serial execution of data processing rules; adopt input-output mapping and multiple data source support to enhance the flexibility and compatibility of the system; ensure the accuracy and stability of data processing through verification mechanism and caching strategy, and realize efficient and reliable real-time data analysis.

[0050] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0051] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the modules described above can refer to the corresponding process in the aforementioned method implementation, and will not be repeated here.

[0052] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present implementation scheme.

[0053] In addition, each functional module in each embodiment of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0054] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for a computer system (which can be a personal computer, a server, or a network system, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0055] Finally, it should be noted that the above implementation modes are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned implementation modes, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned implementation modes, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various implementation modes of the present application.

Claims

1. A real-time analysis method for embedded device data based on DuckDB, characterized in that: include, Receive data to be processed, and schedule multiple execution threads of data processing rules in parallel according to the preset processing cycle, and process multiple data streams concurrently; The execution thread of each data processing rule executes a series of processing jobs in series according to the preset processing pipeline sequence, and forms a data processing pipeline by using the output of the previous job as the input of the next job; Verification is performed before and after each job execution thread starts. After verification, data is obtained from the data source and cached according to the configured input mapping; Different data processing functions are used to process the data according to the job type, and the processing results are sent to the specified target through the configured output mapping.

2. The method for real-time analysis of embedded device data based on DuckDB according to claim 1, characterized in that: Also includes, After the system is started, the configuration information for data processing is initialized; After the initialization is completed, the configuration information is verified, and if the verification passes, data processing is performed according to the configuration information.

3. The method for real-time analysis of embedded device data based on DuckDB according to claim 2, characterized in that: The configuration information includes custom Grok expression groups, DataMapper data mapping and data processing rules; The DataMapper data mapping includes the connection information of the data source and the target, the mapping of the data source field to the target field and the data type conversion rules, as well as the configuration of the input mapping and the output mapping; The data sources include Redis, MQTT, MySQL, memory queue, internal table persistence, Elasticsearch, Syslog service, HTTP and static files; The targets include Elasticsearch, MySQL (Doris), Syslog, memory queues, and internal tables; The input mapping is used to define the conversion rules from the data source to the internal data format of DuckDB, and the output mapping is used to define the conversion rules from the internal data format of DuckDB to the target format.

4. The method for real-time analysis of embedded device data based on DuckDB according to claim 1, characterized in that: Each job is verified before and after the execution of the thread, including: The pre-job execution verification includes input data source verification and job configuration verification; the input data source verification is used to verify the availability of the data source and the correctness of the data format; the job configuration verification is used to verify the correctness and validity of the data processing function; Post-job execution verification includes result set format verification and optional result set content verification; the result set format verification is used to verify whether the format of the result set meets expectations; the result set content verification is used to verify whether the content of the result set meets expectations based on predefined rules or thresholds, including verifying whether the values ​​in the result set are within a preset range or verifying whether the data in the result set meets a specific pattern.

5. The method for real-time analysis of embedded device data based on DuckDB according to claim 4, characterized in that: The cache uses a memory buffer based on DuckDB to improve data access efficiency and realize data transfer between previous and next jobs.

6. The method for real-time analysis of embedded device data based on DuckDB according to claim 5, characterized in that: The data processing function is used to perform data conversion, filtering and calculation operations; the data processing function includes SQL filtering, Grok filtering and user-defined function processing; The SQL filtering is performed by DuckDB, the Grok filtering is performed by the Grok library, and the user-defined function is loaded and executed through a plug-in mechanism.

7. The method for real-time analysis of embedded device data based on DuckDB according to claim 1, characterized in that: Also includes, After the execution thread is completed, the used DuckDB buffer is released and the related memory resources are recovered to prepare for the scheduling and execution of the next data processing rule and maintain the efficiency and stability of the system operation.

8. A real-time analysis device for embedded device data based on DuckDB, characterized in that: It includes a receiving module, a pipeline processing module, a data verification module and an output module; The receiving module is used to receive the data to be processed, and according to a preset processing cycle, to schedule the execution threads of multiple data processing rules in parallel, and to process multiple data streams concurrently; The pipeline processing module is used to execute a series of processing jobs in series according to a preset processing pipeline sequence for the execution thread of each data processing rule, and form a data processing pipeline by taking the output of a previous job as the input of a subsequent job; The data verification module is used to perform verification before and after the execution thread of each job is started. After the verification is passed, data is obtained from the data source according to the configured input mapping and cached; The output module is used to process the job by different data processing functions according to the job type, and send the processing result to the specified target through the configured output mapping.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores a computer program that can be run on the processor, and when the program is executed on the processor, the steps in the real-time analysis method of embedded device data based on DuckDB as described in any one of claims 1 to 7 are implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps in the real-time analysis method of embedded device data based on DuckDB as described in any one of claims 1 to 7 are implemented.

Images