Data processing system, device and method for end-side system

Abstract

The application relates to the technical field of end-side time sequence data processing, in particular to a data processing system, equipment and method of an end-side system. The system collects and processes sensor time sequence data through a data collector, stores the data compressed by a data compressor into a memory and sends the data to a data analyzer; the data analyzer generates a fault processing instruction according to the compressed data, a first processor searches the memory data to locate a fault position according to the fault processing instruction, and a second processor further searches corresponding data to complete fault analysis. Therefore, the problems of low processing efficiency of end-side high-frequency time sequence data and limited storage resources are solved.

Description

Technical Field

[0001] This application relates to the field of edge-side timing data processing technology, and in particular to a data processing system, device and method for an edge-side system. Background Technology

[0002] In industrial high-frequency multi-sensor data processing scenarios, the edge device needs to receive, analyze, and store massive amounts of time-series data in real time. Due to the limited storage and computing resources of the edge device, directly processing massive amounts of raw data will significantly consume resources, resulting in low efficiency of high-frequency time-series data processing and limited storage resources, which in turn makes it difficult to achieve efficient data querying and fault analysis. Summary of the Invention

[0003] This application provides a data processing system, device, and method for edge systems to solve problems such as low efficiency and limited storage resources in edge high-frequency time-series data processing.

[0004] The first aspect of this application provides a data processing system for an edge-side system, comprising: a data collector for collecting and processing sensor time-series data; a data compressor for compressing the processed sensor time-series data, storing the compressed data in memory, and sending it to a data analyzer; a data analyzer for generating fault handling instructions based on the analysis results of the compressed data; a first processor for generating a first retrieval request based on the fault handling instructions, invoking a database engine based on the first retrieval request to obtain first target data from memory, and if a fault location is located based on the first target data, sending the fault location to a second processor; and a second processor for generating a second retrieval request based on the fault location, invoking a database engine based on the second retrieval request to obtain second target data from memory, and generating fault analysis results based on the second target data.

[0005] Based on the above technical means, the embodiments of this application collect, process, compress, store, and perform hierarchical retrieval and analysis of sensor time-series data on the edge side. First, the data is processed and compressed and stored in memory space, which greatly reduces the occupation of the limited storage resources on the edge side by massive high-frequency data. Then, the fault location and in-depth analysis are completed sequentially by two-level processors, which reduces redundant data reading and writing and computational overhead, and significantly improves the processing efficiency and query response speed of edge-side time-series data. Thus, under the condition of limited edge-side resources, efficient and low-latency equipment fault diagnosis and analysis can be achieved.

[0006] Optionally, it also includes: a page manager for accelerating the lookup and access speed of the database engine; a transaction processor for processing data in the memory space; and a data decompressor for decompressing the data in the memory space and sending it to the database engine. The database engine receives a first retrieval request from the first processor and a retrieval request from the second processor to retrieve data from the memory space, and decompresses the retrieved data using the data decompressor before sending it to at least one of the first processor and the second processor.

[0007] Based on the above technical means, this application embodiment further optimizes the data retrieval and read / write process on the client side by adding a page manager, transaction processor, and data decompressor to work in conjunction with the database engine. The page manager can effectively improve the search and access efficiency of the database engine, the transaction processor ensures the stability and reliability of memory data processing, and the data decompressor realizes on-demand decompression of retrieved data. With the hierarchical fault analysis logic of the two-level processor, while alleviating the storage pressure on the client side, it further reduces data query latency and improves the processing efficiency and fault diagnosis accuracy of high-frequency time-series data on the client side.

[0008] Optionally, the edge system includes a first computing resource and a second computing resource running synchronously. A data compressor, a data decompressor, a page manager, a transaction processor, and a database engine run on the first computing resource, while a data analyzer, a first processor, and a second processor run on a graphics processor.

[0009] Based on the aforementioned technical means, this embodiment of the application rationally allocates the computing resources of the edge system. Basic data operations such as data compression, decompression, database access, and transaction processing are assigned to the first computing resource, while intelligent processing tasks such as data analysis, fault location, and deep analysis are assigned to the graphics processor. This achieves precise adaptation and efficient utilization of computing resources. This resource allocation method avoids resource competition between different types of tasks, fully leverages the performance advantages of various computing resources, further reduces edge data processing latency, improves processing efficiency, and maximizes the utilization of edge hardware resources. While ensuring data processing and fault diagnosis effectiveness, it effectively saves equipment operating costs and adapts to the resource-constrained needs of industrial high-frequency multi-sensor data processing scenarios.

[0010] Optionally, the page manager's processing flow includes: if a new data slot is detected to be written, a new index structure is created, wherein the index structure includes a timestamp and a data page pointer, each data page stores sensor time-series data, and the size of each data page is smaller than the cache size of the first computing resource; the sorting of the index results in the data table is updated according to the timestamp of the index structure, and the data in memory is paged according to the index structure of the data table.

[0011] Based on the above technical means, the embodiments of this application construct a lightweight index structure based on timestamps and data page pointers through a page manager, and control the size of a single data page within the first computing resource cache. At the same time, the index sorting is dynamically updated according to the timestamp to realize data paging management. This not only greatly improves the database engine's speed in locating and searching time-series data, but also fully adapts to the limited cache resources on the client side, reduces data read and write overhead, and further reduces query latency, allowing the client side to still efficiently and stably complete fault retrieval and analysis in high-frequency time-series data scenarios.

[0012] Optionally, the transaction processor's processing flow includes: establishing an array index and a one-dimensional array for the data slot; calculating the value of the one-dimensional array based on the data write completion status of the data slot; if the array index is a table-level index, mapping the array index of the data slot to the data page pointer of the table-level index; if the array index is a page-level index, maintaining the visibility status of the data slot based on the one-dimensional array.

[0013] Based on the aforementioned technical means, this embodiment establishes an array index and a one-dimensional array for data slots using a transaction processor. It dynamically calculates the one-dimensional array value based on the data write completion status and implements data page pointer mapping and data slot visibility status maintenance for table-level and page-level indexes, respectively. This achieves orderly management and efficient control of on-device memory data. This processing method can accurately track data write status, clearly define data index levels, effectively avoid data read / write conflicts, ensure the stability and accuracy of memory data storage and retrieval, further optimize the accuracy of data retrieval, reduce invalid data access overhead, adapt to the high-concurrency processing requirements of high-frequency time-series data on the device side, and help improve overall data processing efficiency and the reliability of fault diagnosis.

[0014] Optionally, the data collector's processing flow includes: generating corresponding data slots based on PLC-related metadata and system global time; sending the sensor timing data collected by the PLC to the data calibrator; writing the sliced ​​data without timestamps into the data slots through the data calibrator; and writing the timestamps of the first and last data points into the metadata area of ​​the data slots as the start and end timestamps of the data slots; and sending the data to the data filler through the data calibrator.

[0015] Based on the aforementioned technical means, this embodiment of the application generates corresponding data slots by combining PLC-related metadata and system global time through a data collector. After calibration, the sensor timing data collected by the PLC not only completes the slicing and writing of data without timestamps, but also records the timestamps of the first and last data points in the data slot into the metadata area as start and end identifiers. This data is then synchronously pushed to the data filler by the data calibrator. This process achieves standardized collection, precise time calibration, and orderly transmission of sensor timing data, ensuring the time consistency and integrity of the data collected at the edge, avoiding data misalignment problems caused by missing or disordered timestamps. This provides an accurate and reliable data foundation for subsequent data compression, retrieval analysis, and fault location, further guaranteeing the accuracy and efficiency of high-frequency multi-sensor data processing at the edge.

[0016] Optionally, the data compressor's processing flow includes: extracting data points from each data slot in the processed sensor time-series data; compressing the data points according to a one-dimensional vector distribution to obtain a first set of vector values ​​and a second set of vector values; generating a two-dimensional vector based on the first set of vector values ​​and the second set of vector values; and using the two-dimensional vector to cover the data points of each data slot in place.

[0017] Based on the aforementioned technical means, this embodiment of the application extracts sensor time-series data points from each data slot using a data compressor, compresses the data using a one-dimensional vector distribution method, obtains two sets of vector values, generates a two-dimensional vector, and then uses the two-dimensional vector to overwrite the original data points of the data slot in place. This compression method is specifically adapted to the distribution characteristics of time-series data on the edge side. While ensuring data compression effect and significantly reducing the data storage space occupied, the in-place overwrite mode reduces additional memory overhead, effectively alleviating the problem of limited storage resources on the edge side. At the same time, it simplifies the compression process, reduces computational complexity, adapts to the limited computing resources on the edge side, avoids excessive computing power consumption during the compression process leading to processing delays, provides efficient data support for subsequent data retrieval and fault analysis, and further improves the overall data processing efficiency on the edge side.

[0018] Optionally, the data compressor allocates a separate computing core to each data slot, and each computing core performs independent compression processing.

[0019] Based on the aforementioned technical means, this embodiment of the application allocates a separate computing core to each data slot of the data compressor, enabling the compression processing of each data slot to be carried out independently and in parallel. This distributed parallel processing method effectively avoids computing power competition during multi-data-slot compression, significantly improves the overall compression efficiency of massive high-frequency sensor time-series data, and shortens the data compression time. At the same time, each computing core focuses on the compression task of a single data slot, which can reduce the complexity of single-task processing, reduce computing power loss during compression, fully adapt to the limited computing resources on the edge, further alleviate the pressure on edge storage resources, and buy time for subsequent data retrieval, fault analysis, and other stages, ensuring the efficient and stable operation of the edge system in high-frequency multi-sensor data scenarios.

[0020] A second aspect of this application provides an end-side device, including a data processing system of the end-side system described above.

[0021] A third aspect of this application provides a data processing method for an edge-side system. The method is applied to the data processing system of the edge-side system described in the above embodiments. The method includes the following steps: collecting and processing sensor time-series data using a data collector; compressing the processed sensor time-series data using a data compressor, storing the compressed data in memory, and sending it to a data analyzer; generating a fault handling instruction using the data analyzer based on the analysis results of the compressed data; generating a first retrieval request using a first processor based on the fault handling instruction, invoking a database engine based on the first retrieval request to obtain first target data from the memory; if a fault location is located based on the first target data, sending the fault location to a second processor; generating a second retrieval request using the second processor based on the fault location, invoking a database engine based on the second retrieval request to obtain second target data from the memory, and generating a fault analysis result based on the second target data.

[0022] Based on the above technical means, the embodiments of this application realize the whole process of time-series data acquisition, processing compression, storage and forwarding to fault instruction generation, hierarchical retrieval and location and in-depth analysis on the end side. Data processing and compression are brought forward to reduce the storage pressure on the end side. Then, memory data is retrieved through the linkage of two-level processors. In the resource-constrained end-side environment, the efficiency of time-series data processing is significantly improved, the query latency is reduced, and accurate and efficient device fault location and analysis are achieved.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is an architecture diagram of a data processing system for an edge system provided according to an embodiment of this application; Figure 2 This is a data processing system diagram of the end-side system provided in the embodiments of this application; Figure 3 This is a two-level index architecture diagram of the page manager provided in the embodiments of this application; Figure 4 This is a flowchart illustrating the structure and processing flow of the data collector provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an end-side device according to an embodiment of this application; Figure 6 This is a flowchart of a data processing method for an end-side system according to an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0026] Currently, the widespread adoption of the Industrial Internet of Things (IIoT) in fields such as intelligent manufacturing, energy and power, and rail transportation has led to the generation of massive amounts of time-series data from equipment sensors. High-frequency, multi-sensor industrial scenarios require edge systems to simultaneously receive, analyze, and store large volumes of data. Related technologies include centralized computing, distributed computing, cloud computing, and cloud-edge integrated computing. Cloud computing, with its strong storage and elastic computing power, has become an ideal platform for large-scale equipment diagnostics, employing a dual-driven architecture of data and models. High-frequency data processing from multiple sensors has also formed a typical workflow from acquisition and preprocessing to analysis and decision-making. However, these technologies have the following shortcomings: Firstly, edge-side resources are far inferior to those in the cloud. Traditional methods of accessing large models are slow and have high latency, and edge-side devices are mostly single machines, making it impossible to build a distributed platform. Edge-side storage is limited, and both processing and compressing data before or after processing have drawbacks. Secondly, it fails to utilize the hardware architecture of CPU (Central Processing Unit) and NPU (Neural Processing Unit), lacks a single end-device integrated processing solution, and fails to maximize the utilization of resources of a single end-device to save costs.

[0027] In summary, the relevant technologies are insufficient to meet the needs of a single end-side device to efficiently complete the entire data processing process, and cannot adapt to the requirements of low latency, high reliability, and low cost for industrial end-side devices.

[0028] It should be noted that, in this application, spatiotemporal data refers to an ordered set of data recorded in chronological order, containing characteristics such as timestamps, continuity, high frequency, spatial coordinates, and multi-dimensionality; a time-series database refers to a database system specifically used for storing, managing, and analyzing time or spatiotemporal sequence data; data ingestion refers to the process of collecting data from multiple sources and importing it into a database for subsequent storage, processing, and analysis; cloud-edge integrated computing technology is an architecture that integrates cloud computing and edge computing, with the cloud providing large-scale storage and processing capabilities, and the edge processing data locally, optimizing bandwidth and response speed; the multi-sensor high-frequency data processing and analysis process involves end-to-end processing of high-frequency time-series data from multiple types of sensors, used in scenarios such as equipment monitoring and fault diagnosis; the NPU is designed specifically for artificial intelligence computing, with significant advantages in energy efficiency and speed; centralized computing refers to data being centrally stored on a single device, relying on high-performance hardware, with centralized management but high costs; distributed computing refers to data being distributed and stored across multiple nodes, possessing characteristics such as redundancy and independence, and requiring assurance of data consistency; cloud computing refers to the adoption of a data and model dual-driven architecture in the industrial field to achieve cloud-based data governance, model training, and intelligent decision-making.

[0029] The following description, with reference to the accompanying drawings, describes a data processing system, device, and method for an edge-side system according to embodiments of this application. Addressing the issues of low efficiency and limited storage resources in edge-side high-frequency time-series data processing mentioned in the background, this application provides a data processing system for an edge-side system. By designing a novel data framework and data flow mode adapted to the limited resources and localized processing requirements of the edge-side system, it simultaneously utilizes the CPU and NPU for parallel computing, employs a processing flow that analyzes small and large models, a two-stage concurrent data compression and analysis method, a parallel processing multi-sensor time-series data storage format, a two-level index architecture based on timestamps, and a parallel visibility check structure based on bitmaps. This maximizes the utilization of edge-side computing resources, saves memory storage space, and improves data processing speed and throughput. Ultimately, it enables localized model diagnosis and detection in high-frequency, multi-sensor scenarios using a single edge-side device, achieving cost reduction and efficiency improvement.

[0030] Specifically, Figure 1 This application provides a data processing system for an end-side system.

[0031] like Figure 1 As shown, the data processing system 10 of the end-side system includes: a data collector 101, a data compressor 102, a data analyzer 103, a first processor 104, and a second processor 105.

[0032] The data collector 101 is used to collect and process sensor time-series data; the data compressor 102 is used to compress the processed sensor time-series data, store the compressed data in memory, and send it to the data analyzer 103; the data analyzer 103 is used to generate fault handling instructions based on the analysis results of the compressed data; the first processor 104 is used to generate a first retrieval request based on the fault handling instructions, call the database engine based on the first retrieval request to obtain first target data from memory, and if the fault location is located based on the first target data, the fault location is sent to the second processor 105; the second processor 105 is used to generate a second retrieval request based on the fault location, call the database engine based on the second retrieval request to obtain second target data from memory, and generate fault analysis results based on the second target data.

[0033] It is understood that the embodiments of this application collect, process, compress, store, and perform hierarchical retrieval and analysis of sensor time-series data on the edge side. First, the data is processed and compressed and stored in memory space, which greatly reduces the occupation of the limited storage resources on the edge side by massive high-frequency data. Then, the fault location and in-depth analysis are completed sequentially by two-level processors, which reduces redundant data reading and writing and computational overhead, and significantly improves the processing efficiency and query response speed of edge time-series data. Thus, under the condition of limited edge resources, efficient and low-latency equipment fault diagnosis and analysis can be achieved.

[0034] It should be noted that the first processor 104 is used to generate a first retrieval request according to the fault handling instructions, and to locate the fault location based on the first target data obtained from the memory space; the first retrieval request is a retrieval instruction generated by the first processor 104 according to the fault handling instructions, used to retrieve relevant data from the memory space; the first target data is relevant data obtained by the database engine from the memory space according to the first retrieval request, used to realize the fault location. The second processor 105 is used to generate a second retrieval request according to the fault location after the fault location is determined, and to complete the final fault analysis based on the second target data obtained from the memory space; the second retrieval request is a retrieval instruction generated by the second processor 105 according to the located fault location, used to further retrieve relevant data; the second target data is relevant data obtained by the database engine from the memory space according to the second retrieval request, used to realize in-depth fault analysis.

[0035] Specifically, such as Figure 2As shown, the data collector 101 generates data slots based on PLC (Programmable Logic Controller) related metadata and system global time. The size of the data slot is adapted to the CPU / NPU cache line (64Kbytes), and the number of data points is an even multiple. The original data slot structure is shown in Table 1. Table 1

[0036] The ellipsis at the end represents the continuously expandable bits of the original time-series data points, used to flexibly add more time-series sampling data points according to the actual sensor sampling length and sampling duration, with no fixed upper limit. The metadata area stores the number, type, start and end timestamps, and sampling frequency of data points; the data compressor 102 uses a wavelet algorithm to compress the data points in the data slot according to a one-dimensional vector distribution, obtaining... (Approximate coefficient set) (Detail coefficient set) Two sets of vectors are combined into a two-dimensional vector and then cover the original data slots in place. The compressed data slot structure is shown in Table 2: Table 2

[0037] in, The ellipsis following a column indicates a sequence of approximate coefficient units that can be arranged continuously and expanded in batches. The ellipsis following the column indicates a sequence of detail coefficient units that can be arranged continuously and expanded in batches, adapting to the length of the complete feature vector output after wavelet compression, and achieving compact and aligned storage of the full compressed features. Unlike conventional compression schemes that first release old memory and then apply for new memory blocks, this application directly completes data rewriting and feature reorganization within the allocated memory space of the original data slot, without the need to apply for additional memory space, and without memory expansion, data migration, or memory fragmentation overhead; the data analyzer 103 directly analyzes the compressed two-dimensional vector data, filters abnormal data, and generates fault handling instructions; the first processor 104 (small model processor) calls the database engine to obtain historical data to locate the fault location, and the second processor 105 (large model processor) calls the database engine to obtain more data to generate the final fault analysis result. The whole process forms an integrated workflow mode of "acquisition-compression-analysis-hierarchical fault diagnosis", which is suitable for scenarios with limited end-side resources and can achieve high-frequency, multi-sensor localized model diagnosis and detection through a single device.

[0038] In this embodiment of the application, it further includes: a page manager for accelerating the lookup and access speed of the database engine; a transaction processor for processing data in the memory space; and a data decompressor for decompressing the data in the memory space and sending it to the database engine. The database engine receives a first retrieval request from the first processor 104 and a retrieval request from the second processor 105 to retrieve data from the memory space, and decompresses the retrieved data using the data decompressor and sends it to at least one of the first processor 104 and the second processor 105.

[0039] It is understood that the embodiments of this application further optimize the data retrieval and read / write process on the client side by adding a page manager, transaction processor, and data decompressor to work in conjunction with the database engine. The page manager can effectively improve the lookup and access efficiency of the database engine, the transaction processor ensures the stability and reliability of memory data processing, and the data decompressor realizes on-demand decompression of retrieved data. With the hierarchical fault analysis logic of the two-level processor, the data query latency is further reduced while alleviating the storage pressure on the client side, and the processing efficiency and fault diagnosis accuracy of high-frequency time-series data on the client side are improved.

[0040] Specifically, such as Figure 3 As shown, the page manager is responsible for building a two-level index structure to support subsequent data retrieval; the transaction processor maintains the write and visibility status of data slots through a bitmap visibility data structure, and performs parallel visibility checks through AND operations of table-level and page-level bitmaps; the data decompressor uses wavelet algorithm inverse operation to recover the original timestamp of the data point according to the start timestamp, end timestamp, and sampling frequency of the data slot metadata area; the database engine receives retrieval requests from the first processor 104 and the second processor 105, calls the page manager to locate the data address, calls the data decompressor to complete the data decompression, and finally feeds back the decompressed data to the corresponding processor. The four work together to ensure efficient and accurate data retrieval.

[0041] In this embodiment, the end-side system includes a first computing resource and a second computing resource that run synchronously. The data compressor 102, the data decompressor, the page manager, the transaction processor, and the database engine run on the first computing resource, while the data analyzer 103, the first processor 104, and the second processor 105 run on the second computing resource.

[0042] It is understood that this application embodiment rationally allocates the computing resources of the edge system, assigning basic data operations such as data compression, decompression, database access, and transaction processing to the first computing resource, and allocating intelligent processing tasks such as data analysis, fault location, and deep analysis to the graphics processor, thereby achieving precise adaptation and efficient utilization of computing resources. This resource allocation method avoids resource competition between different types of tasks, fully leverages the performance advantages of various computing resources, further reduces edge data processing latency and improves processing efficiency, and maximizes the utilization of edge hardware resources. While ensuring the effectiveness of data processing and fault diagnosis, it effectively saves equipment operating costs and adapts to the resource-constrained needs of industrial high-frequency multi-sensor data processing scenarios.

[0043] It should be noted that the first computing resource is the central processing unit in the edge system, which is mainly used to perform basic data reading, writing and operation operations such as data compression and decompression, memory data paging management, data visibility maintenance and database engine scheduling; the second computing resource is the graphics processing unit in the edge system, specifically the neural network processor, which is mainly used to perform model inference-type intelligent processing tasks such as time series data anomaly analysis, fault location, and deep fault diagnosis. The two types of computing resources are independent of each other, run synchronously and in parallel, and do not wait for each other.

[0044] Specifically, the CPU is responsible for performing basic data operations such as data compression, data decompression, page index management, transaction visibility processing, and database engine scheduling. The data compressor 102 allocates a separate CPU core to each data slot to achieve multi-core parallel compression. The graphics processor is an NPU, which is responsible for performing intelligent inference tasks such as data analysis, fault location by the first processor 104, and deep fault analysis by the second processor 105. The two run synchronously and do not wait for each other, giving full play to the basic computing advantages of the CPU and the parallel inference advantages of the NPU, maximizing the use of edge hardware resources, avoiding computing power competition between different types of tasks, and reducing processing latency.

[0045] In this embodiment, the page manager's processing flow includes: if a new data slot is detected to be written, a new index structure is created, wherein the index structure includes a timestamp and a data page pointer, each data page stores sensor time-series data, and the size of each data page is smaller than the cache size of the first computing resource; the sorting of the index results in the data table is updated according to the timestamp of the index structure, and the data in memory is paged according to the index structure of the data table.

[0046] It is understood that the embodiments of this application construct a lightweight index structure based on timestamps and data page pointers through the page manager, and control the size of a single data page within the first computing resource cache. At the same time, the index sorting is dynamically updated according to the timestamp to realize data pagination management. This not only greatly improves the database engine's speed in locating and searching time-series data, but also fully adapts to the limited cache resources on the client side, reduces data read and write overhead, and further reduces query latency, so that the client side can still efficiently and stably complete fault retrieval and analysis in high-frequency time-series data scenarios.

[0047] Specifically, such as Figure 3 As shown, the page manager's index structure is a standard two-dimensional array KV structure. Each independent PLC's corresponding data table has an independent index, where K is a timestamp (corresponding to the earliest start time of all data pages within that range), sorted in ascending order of time to ensure that the timestamp of the next K is later than the end time of the previous KV; V is a pointer to all data pages belonging to that table within that timestamp range, i.e. The table contains pointers to all data pages belonging to it; each data page stores only the time-series data of one sensor, and the data page size is smaller than the CPU L2 cache size to reduce cache-missing; when a new data slot is detected to be written, the page manager only creates the corresponding index entry without moving the data itself, and updates the sorting of the index results in the data table according to the timestamp of the index structure. The data in memory is paged according to the index structure, and the data page indexes under the same K are required to be time-aligned, equal in size, and have the same number of key-value pairs that can be stored. Each index header also contains transaction information maintained by the transaction processor to further improve data retrieval efficiency.

[0048] In this embodiment of the application, the processing flow of the transaction processor includes: establishing an array index and a one-dimensional array for the data slot; calculating the value of the one-dimensional array based on the data write completion status of the data slot; if the array index is a table-level index, mapping the array index of the data slot to the data page pointer of the table-level index; if the array index is a page-level index, maintaining the visibility status of the data slot based on the one-dimensional array.

[0049] It is understood that this application embodiment establishes an array index and a one-dimensional array for data slots through a transaction processor, dynamically calculates the one-dimensional array value based on the data write completion status, and implements data page pointer mapping and data slot visibility status maintenance for table-level indexes and page-level indexes respectively, achieving orderly management and efficient control of on-device memory data. This processing method can accurately track data write status, clearly divide data index levels, effectively avoid data read / write conflicts, ensure the stability and accuracy of memory data storage and retrieval, further optimize the accuracy of data retrieval, reduce invalid data access overhead, adapt to the high-concurrency processing requirements of high-frequency time-series data on the device side, and help improve overall data processing efficiency and fault diagnosis reliability.

[0050] Specifically, the transaction processor maintains data slots in columnar storage (each data slot contains only one sensor information). It establishes an array index for the data slots and a one-dimensional bitmap array (the one-dimensional array is the bitmap array itself). The value of the bitmap array is calculated based on the data slot's data write completion status. If the array index is a table-level index, the data slot's array index is directly mapped to the data page pointer of the table-level index. The array index corresponds to the position of K in the table index, requiring no additional storage. Each bit in the bitmap array represents the corresponding data page in V, with 1 indicating the page is full and 0 indicating the page is not full. If the array index is a page-level index, then it is based on the bitmap... The array maintains the visibility status of data slots. The length of the bitmap array is the number of key-value pairs contained in the page. Each bit stores the compressed write status of the corresponding data slot (0 indicates no data / not written incomplete, 1 indicates data has been written and compressed incomplete). During visibility checks, the table-level bitmap is checked first. If all bits are 1, the data in that time slice is visible. If any bit is 0, all page-level bitmaps are read and ANDed. Bits with a result of 1 are visible. The bitmap can call the CPU's SIMD instruction set to perform parallel checks. Its bitmap structure is shown in Table 3 (Visibility Data Structure Table), which greatly improves transaction processing efficiency.

[0051] Table 3

[0052] In this embodiment, the processing flow of the data collector 101 includes: generating corresponding data slots based on PLC-related metadata and system global time; sending the sensor timing data collected by the PLC to the data calibrator; writing the sliced ​​data without timestamps into the data slots through the data calibrator; and writing the timestamps of the first and last data points into the metadata area of ​​the data slots as the start and end timestamps of the data slots; and sending the data to the data filler through the data calibrator.

[0053] It is understood that in this embodiment, the data collector 101 combines PLC-related metadata with the system global time to generate corresponding data slots. After calibration, the sensor timing data collected by the PLC not only completes the slicing and writing of data without timestamps, but also records the timestamps of the first and last data points in the data slot into the metadata area as start and end identifiers. Then, the data calibrator synchronously pushes the data to the data filler. This process realizes the standardized collection, accurate time calibration, and orderly transmission of sensor timing data, ensuring that the data collected at the end side has time consistency and integrity, avoiding data misalignment problems caused by missing or disordered timestamps. It provides an accurate and reliable data foundation for subsequent data compression, retrieval analysis, and fault location, further ensuring the accuracy and efficiency of high-frequency multi-sensor data processing at the end side.

[0054] It should be noted that the first data point refers to the earliest sampled frame in time among the multi-frame sensor timing data to be written to the current data slot after being sliced ​​by the data calibrator; the last data point refers to the latest sampled frame in time among the timing data in this slice. These two data points correspond to the start and end points on the timeline of all the sampled data written to the data slot this time. The data calibrator only writes the timestamps corresponding to the start and end points to the metadata area. The timestamps of the intermediate sampling points can be calculated based on the sampling frequency and start and end timestamps recorded in the data slot, eliminating the need for point-by-point storage. This reduces data storage overhead and ensures timing integrity.

[0055] Specifically, such as Figure 4 As shown, the data collector 101 preloads the sensor information of the PLC device (such as sensor frequency, data type, etc.) and collects the data of each PLC in the form of a single table. The data slot generator generates and initializes the corresponding data slots according to the relevant metadata of the PLC and the system global time. The size of the data slot is adapted to the CPU / NPU cache line (64Kbytes), and the number of data points is an even multiple. The metadata area stores the number of data points, data type, start timestamp, end timestamp, and sampling frequency. After the sensor timing data collected by the PLC is sent to the data calibrator, the data calibrator slices the data according to the amount of data that the corresponding data slot can store. The data without timestamps after slicing is written into the data slot. At the same time, the timestamps of the first and last data points are written into the metadata area of ​​the data slot as the start timestamp and end timestamp of the data slot. After the data calibration is completed, the data calibrator sends the data to the data filler, which pushes it to the data compressor 102 for subsequent compression processing to ensure that the collected data is standardized, complete, and time-consistent.

[0056] In this embodiment of the application, the processing flow of the data compressor 102 includes: extracting data points of each data slot in the processed sensor time-series data; compressing the data points according to a one-dimensional vector distribution to obtain a first set of vector values ​​and a second set of vector values; generating a two-dimensional vector based on the first set of vector values ​​and the second set of vector values; and using the two-dimensional vector to cover the data points of each data slot in place.

[0057] It is understood that in this embodiment, the sensor time-series data points of each data slot are extracted by the data compressor 102, and the data is compressed using a one-dimensional vector distribution method to obtain two sets of vector values ​​and generate a two-dimensional vector. The two-dimensional vector is then used to overwrite the data points of the original data slot. This compression method is specifically adapted to the distribution characteristics of the time-series data on the edge side. While ensuring the data compression effect and significantly reducing the data storage space occupied, the in-situ overwrite mode reduces additional memory overhead, effectively alleviating the problem of limited storage resources on the edge side. At the same time, it simplifies the compression process, reduces computational complexity, adapts to the limited computing resources on the edge side, and avoids excessive computing power consumption during the compression process, which may lead to processing delays. This provides efficient data support for subsequent data retrieval and fault analysis, further improving the overall data processing efficiency on the edge side.

[0058] It should be noted that the first set of vector values ​​is an approximation coefficient vector obtained by compressing the one-dimensional time-series data points within the data slot, used to characterize the overall trend and low-frequency components of the sensor time-series data; the second set of vector values ​​is the corresponding detail coefficient vector, used to characterize the fluctuations and high-frequency components of the sensor time-series data. The sum of the lengths of the two sets of vectors is consistent with the length of the original data points, which can completely characterize the feature information of the original time-series data; the two-dimensional vector is composed of the approximation coefficient vector and the detail coefficient vector concatenated in sequence, which can represent the original one-dimensional time-series data in a more compact data form. Using this two-dimensional vector to cover the original data points within the data slot in-situ can achieve data compression without occupying additional memory space, facilitating subsequent storage and analysis, and reducing the storage pressure on the edge system.

[0059] Specifically, the data compressor 102 extracts data points from each data slot sent by the data collector 101, and uses a wavelet compression algorithm to compress the data points according to a one-dimensional vector distribution to obtain the first set of vector values. Approximate coefficient group) and second group of vector values ​​( The two sets of vectors (detail coefficient groups) are both half the length of the original one-dimensional data. After combining the two sets of vectors to generate a two-dimensional vector, the data area of ​​each data slot is covered in place without the need to allocate additional memory, which greatly reduces the memory consumption during compression. Each set of one-dimensional vectors is 64K bytes long and can be loaded into the CPU cache at once. It is compressed in parallel through SIMD (Single Instruction, Multiple Data) instructions to improve the compression speed. At the same time, the compressed two-dimensional vector can be directly sent to the data analyzer 103 for asynchronous analysis without waiting for the compression to complete, which further improves the processing efficiency.

[0060] In this embodiment, the data compressor 102 allocates a separate computing core to each data slot, and each computing core performs independent compression processing.

[0061] It is understood that this embodiment of the application allocates a separate computing core to each data slot of the data compressor 102, enabling the compression processing of each data slot to be carried out independently and in parallel. This distributed parallel processing method effectively avoids computing power competition when compressing multiple data slots, significantly improves the overall compression efficiency of massive high-frequency sensor time-series data, and shortens the data compression time. At the same time, each computing core focuses on the compression task of a single data slot, which can reduce the complexity of single-task processing, reduce computing power loss during the compression process, fully adapt to the limited computing resources on the edge, further alleviate the pressure on edge storage resources, and buy time for subsequent data retrieval, fault analysis, and other stages, ensuring the efficient and stable operation of the edge system in high-frequency multi-sensor data scenarios.

[0062] Specifically, the data compressor 102 allocates a separate CPU computing core to each data slot. Since the data in each data slot is independent after being processed by the data collector 101 and has no relation to each other, it does not need to depend on the data in other data slots. Therefore, each computing core can independently execute the wavelet compression algorithm to compress the corresponding data slot separately, realizing parallel compression of multiple CPU cores. This effectively avoids computing power competition when compressing multiple data slots, significantly improves the overall compression efficiency of massive high-frequency sensor time series data, shortens compression time, and at the same time reduces the processing complexity of a single core, reduces computing power loss, and adapts to the limited computing resources on the edge.

[0063] According to the data processing system of the end-side system proposed in the embodiments of this application, by collecting, processing, compressing, storing and hierarchically retrieving and analyzing sensor time-series data at the end, the data is first processed and compressed and then stored in memory space, which greatly reduces the occupation of the limited storage resources of the end-side by massive high-frequency data; then, the fault location and in-depth analysis are completed sequentially by two-level processors, reducing redundant data reading and writing and computational overhead, significantly improving the efficiency of end-side time-series data processing and query response speed, thereby achieving efficient and low-latency equipment fault diagnosis and analysis under the condition of limited end-side resources.

[0064] Figure 5 This application provides an end-side device, such as... Figure 5 As shown, the data processing system 10 includes the end-side system of the above embodiment.

[0065] The following section will elaborate on the workflow of the data processing method of the edge system through a specific embodiment, such as... Figure 6 As shown, this method is applied to the data processing system of the terminal system in the above embodiment, and the specific steps are as follows: In step S601, sensor timing data is collected by a data collector and processed.

[0066] It is understood that the embodiments of this application use a data collector to collect and process sensor time-series data in a standardized manner. This can encapsulate discrete sensor data into a standard data slot structure and record key time information in the metadata area. This not only ensures the integrity and time consistency of the time-series data and avoids data corruption, but also provides a standardized data foundation for subsequent compression, retrieval and fault diagnosis, thereby improving the reliability and efficiency of edge data processing.

[0067] In step S602, the processed sensor time-series data is compressed by a data compressor, and the compressed data is stored in memory and sent to a data analyzer.

[0068] It is understood that the embodiments of this application use a data compressor to compress the sensor time-series data into a compact form, which greatly reduces the amount of data while retaining the complete data characteristics. Storing the compressed data in memory can effectively reduce the storage pressure on the edge system. At the same time, it is sent to the data analyzer simultaneously, which can realize parallel processing of compression and analysis, reduce the overall time consumption, and improve the system operating efficiency.

[0069] In step S603, a fault handling instruction is generated by the data analyzer based on the analysis results of the compressed data.

[0070] It is understood that the embodiments of this application directly perform anomaly analysis on the compressed data through a data analyzer, without the need to decompress and restore the original data first. This can significantly reduce computational overhead and processing latency while ensuring the accuracy of anomaly identification, and quickly generate corresponding fault handling instructions based on the analysis results, providing clear triggering basis for subsequent graded fault location and diagnosis, and improving system response speed.

[0071] In step S604, the first processor generates a first retrieval request according to the fault handling instruction, calls the database engine according to the first retrieval request to obtain the first target data from the memory space, and if the fault location is located according to the first target data, the fault location is sent to the second processor.

[0072] It is understood that the embodiments of this application initiate a preliminary search through the first processor, which can quickly locate relevant time-series data based on fault handling instructions, efficiently retrieve the first target data in the memory space to complete the coarse location of the fault, realize the hierarchical processing logic of first locating and then deeply analyzing, effectively reduce invalid data retrieval and calculation, and improve the response speed of fault diagnosis and the utilization rate of system resources.

[0073] In step S605, the second processor generates a second retrieval request based on the fault location, calls the database engine based on the second retrieval request to obtain second target data from the memory space, and generates fault analysis results based on the second target data.

[0074] It is understood that the embodiments of this application initiate a refined search based on the determined fault location through the second processor, obtain more targeted second target data and complete in-depth analysis, which can achieve accurate fault diagnosis on the basis of coarse location, improve the accuracy and reliability of analysis results while reducing redundant data processing, and provide a strong basis for subsequent fault handling.

[0075] It should be noted that the explanation of the aforementioned data processing system embodiment of the end-side system also applies to the data processing method of the end-side system in this embodiment, and will not be repeated here.

[0076] According to the data processing method of the end-side system proposed in the embodiments of this application, by realizing the whole process of time-series data acquisition, processing compression, storage and forwarding to fault instruction generation, hierarchical retrieval and location and in-depth analysis on the end side, the data processing and compression are brought forward to reduce the storage pressure on the end side. Then, the memory data is retrieved through the linkage of two-level processors. In the resource-constrained end-side environment, the efficiency of time-series data processing is significantly improved, the query latency is reduced, and accurate and efficient equipment fault location and analysis are achieved.

[0077] In summary, the embodiments of this application have at least the following beneficial effects: (1) A data framework and flow mode for multi-sensor fault diagnosis is proposed, which can be adapted to scenarios with limited storage and computing resources on the end side. It can support the acquisition, processing, storage and analysis of high-frequency and ultra-high-frequency data, adapt to localized multi-sensor diagnosis needs, and achieve cost reduction and efficiency improvement.

[0078] (2) Enables synchronous parallel computing between CPU and NPU without waiting for each other, maximizes the use of edge system resources, ensures that data compression, storage and analysis are carried out synchronously in ultra-high frequency data scenarios, improves resource utilization and reduces diagnostic processing latency.

[0079] (3) The processing flow of analysis, small model and large model is adopted. By taking advantage of the low probability of failure, normal and noisy data are intercepted by analysis and small model, reducing the dependence on large model, greatly improving the data acquisition speed, and reducing processing time and delay.

[0080] (4) A two-stage data compression and processing method and data storage format are proposed to realize the concurrent execution of compression and analysis, and the compressed data can be directly used for analysis. At the same time, it supports multi-sensor time series data multi-dimensional parallel processing, reduces memory occupation, and improves data acquisition and processing efficiency.

[0081] (5) Design a two-level index architecture based on timestamps and a visibility data structure based on bitmaps. This allows for the rapid creation of indexes without data movement, enabling parallel visibility checks, further reducing memory usage and data processing latency.

[0082] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0083] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0084] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0085] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0086] Those skilled in the art will understand that all or part of the steps of the methods implementing the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0087] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A data processing system for an edge-side system, characterized in that, include: A data collector for collecting and processing sensor time-series data; A data compressor is used to compress the processed sensor time-series data, store the compressed data in memory, and send it to a data analyzer. The data analyzer is used to generate fault handling instructions based on the analysis results of the compressed data. The first processor is configured to generate a first retrieval request according to the fault handling instruction, call the database engine according to the first retrieval request to obtain first target data from the memory space, and if the fault location is located according to the first target data, send the fault location to the second processor. The second processor is configured to generate a second retrieval request based on the fault location, invoke the database engine based on the second retrieval request to obtain second target data from the memory space, and generate fault analysis results based on the second target data. Page manager, used to accelerate the lookup and access speed of the database engine; A transaction processor is used to process data in the memory space; A data decompressor is used to decompress the data in the memory space and then send it to the database engine. The database engine receives a first retrieval request from the first processor and a retrieval request from the second processor to retrieve data from the memory space, and then decompresses the retrieved data using the data decompressor and sends it to at least one of the first processor and the second processor. The page manager's processing flow includes: If a new data slot write is detected, a new index structure is created, wherein the index structure includes a timestamp and a data page pointer, each data page stores sensor time-series data, and the size of each data page is smaller than the cache size of the first computing resource; Update the sorting of the index results in the data table based on the timestamp of the index structure. The data in memory is managed by paging based on the index structure of the data table; The processing flow of the transaction processor includes: Establish the array index and one-dimensional array of the data slots; Calculate the value of the one-dimensional array based on the data write completion status of the data slot; If the array index is a table-level index, the array index of the data slot is mapped to the data page pointer of the table-level index. If the array index is a page-level index, the visibility state of the data slot is maintained based on the one-dimensional array.

2. The data processing system of the end-side system according to claim 1, characterized in that, The endpoint system includes a first computing resource and a second computing resource that run synchronously. The data compressor, the data decompressor, the page manager, the transaction processor, and the database engine run on the first computing resource, while the data analyzer, the first processor, and the second processor run on the second computing resource.

3. The data processing system of the end-side system according to claim 1, characterized in that, The data collector's processing flow includes: Generate corresponding data slots based on PLC-related metadata and system global time; The PLC collects the sensor timing data and sends it to the data calibrator. The data calibrator writes the sliced ​​data without timestamps into the data slot and writes the timestamps of the first and last data points into the metadata area of ​​the data slot as the start and end timestamps of the data slot. The data is sent to the data filler via the data calibrator.

4. The data processing system of the end-side system according to claim 1, characterized in that, The data compressor's processing flow includes: Extract the data points of each data slot from the processed sensor time-series data; The data points are compressed according to a one-dimensional vector distribution to obtain a first set of vector values ​​and a second set of vector values. A two-dimensional vector is then generated based on the first set of vector values ​​and the second set of vector values. The two-dimensional vector is used to cover the data points of each data slot in place.

5. The data processing system of the end-side system according to claim 4, characterized in that, The data compressor allocates a separate computing core to each data slot, and each computing core performs independent compression processing.

6. An end-side device, characterized in that, The data processing system includes the end-side system as described in any one of claims 1-5.

7. A data processing method for an end-side system, characterized in that, The method is applied to the data processing system of the end-side system according to any one of claims 1-5, wherein the method includes the following steps: The sensor timing data is collected by a data collector and then processed. The processed sensor time-series data is compressed using a data compressor, and the compressed data is stored in memory and sent to a data analyzer. The data analyzer generates fault handling instructions based on the analysis results of the compressed data. The first processor generates a first retrieval request according to the fault handling instructions, and calls the database engine according to the first retrieval request to obtain first target data from the memory space. If the fault location is located according to the first target data, the fault location is sent to the second processor. The second processor generates a second retrieval request based on the fault location, calls the database engine based on the second retrieval request to obtain second target data from the memory space, and generates fault analysis results based on the second target data.

Images