Heterogeneous hardware accelerated supply chain enterprise carbon emission intelligent terminal
By adopting heterogeneous hardware acceleration technology on supply chain smart terminals, carbon emission data is collected, stored, analyzed and transmitted in real time, the shortcomings of data collection, storage, analysis and transmission in the existing technology are solved, and efficient, accurate and real-time carbon emission data management is achieved, supporting the green and low-carbon transformation of enterprises.
Patent Information
- Application Number
- CN202411502545.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-02-25
AI Technical Summary
The existing technology has shortcomings in the collection, storage, analysis and transmission of carbon emission data, which makes it difficult for the data to meet the needs of green and low-carbon transformation and low-carbon development.
The supply chain smart terminal based on heterogeneous hardware acceleration is adopted to collect data in real time by deploying carbon emission gas sensors, efficient storage and management is used for optimized design of timing database system, and the data is cleaned and preprocessed through the analysis module, and finally the data is safely and quickly transmitted to the cloud through the network module.
It realizes efficient, accurate and real-time monitoring and analysis of carbon emission data, improves the speed and efficiency of data processing, meets the real-time and accuracy needs of carbon emission data, and provides strong data support for the green transformation and low-carbon development of enterprises.
Smart Images

Figure CN119515296A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of heterogeneous devices, and more specifically, to a supply chain intelligent terminal based on heterogeneous hardware acceleration. Background Art
[0002] As the concept of low carbon gradually becomes a global consensus, the global supply chain is rapidly transforming towards green and low carbon, which further intensifies the competition for low carbon management capabilities among international trade suppliers. In order to follow the trend of green and low carbon and take a leading position in the international changes, State Grid, as the leading enterprise of the national supply chain, should actively lead the decarbonization of the supply chain and accelerate the pace of green and low carbon transformation.
[0003] Carbon emissions are a key indicator of green and low-carbon work. The accuracy and real-time nature of their data are crucial for formulating effective emission reduction strategies and tracking the implementation of emission reduction targets.
[0004] In the real power system supply chain network, due to the close cooperation between enterprises and complex business processes, carbon emission data often has the characteristics of high frequency and real-time. Therefore, enterprises need to quickly capture, process and analyze this data in the supply chain in order to adjust production plans and procurement strategies in a timely manner, thereby reducing unnecessary carbon emissions.
[0005] However, there are two major defects in the current carbon emission collection and measurement process. First, the carbon emission calculation methods of domestic power supply chain enterprises and institutions are mostly based on traditional manual accounting methods. However, with the continuous expansion of the scale of the power supply chain and the sharp increase in the amount of carbon emission data, the traditional method of manually filling in or manually importing carbon emission data is no longer applicable. The self-reported carbon emissions of enterprises based on the existing manual accounting method are quite different from the actual emissions. These deviations occur in the statistical deviations of various energy consumption levels and the measurement deviations of carbon emission factors, indicating that enterprises have not deliberately distorted their emissions, but the accuracy of traditional manual accounting methods has certain limitations. Secondly, the existing carbon metering equipment has low collection accuracy and cannot achieve high-frequency carbon emission data collection, so it is impossible to achieve real-time and dynamic carbon emission monitoring. Therefore, the data obtained based on the existing carbon emission online monitoring system is too discrete, and it is impossible to obtain scientific carbon emission data with time continuity and reasonable spatial coverage, which makes it difficult to meet the needs of modeling and evaluation. In addition, traditional data processing and analysis usually rely on software-level computing units, which are limited by the performance bottleneck of general computing units, resulting in low efficiency in processing large-scale time series data and unable to meet the requirements of real-time and accuracy.
[0006] In summary, comprehensive and accurate tracking and measurement of greenhouse gas emissions of enterprises, and real-time and effective data analysis will help to reasonably allocate carbon emission quotas to enterprises in the power supply chain, promote the efficient operation of the carbon trading market, and thus promote the market mechanism of carbon emission rights trading. Enterprises in the power supply chain need to seek a more efficient, flexible and intelligent data management method to better monitor and analyze carbon emission data, and provide strong support for their green transformation and low-carbon development. Summary of the invention
[0007] The purpose of the embodiments of the present disclosure is to provide a supply chain intelligent terminal based on heterogeneous hardware acceleration. The present invention aims to solve the deficiencies of existing carbon emission data management methods in data collection, storage, analysis and transmission, and to achieve efficient, accurate and real-time monitoring and analysis of carbon emission data by introducing heterogeneous hardware acceleration technology.
[0008] In a general aspect, a supply chain intelligent terminal based on heterogeneous hardware acceleration is provided, comprising four parts: a collection module, a storage module, an analysis module, and a network module;
[0009] The collection module collects carbon emission data of each link in the power grid supply chain in real time during the production, storage, maintenance and scrapping processes through carbon emission gas sensors deployed on the end side;
[0010] The storage module adopts an optimized time series database system to construct a programmable network hardware read and write request processing submodule, a query optimization submodule based on hardware incremental materialized views, and a hardware-based time series data compression storage submodule to achieve storage and management of a large amount of carbon emission time series data uploaded by the acquisition module;
[0011] The analysis module ensures the accuracy and validity of carbon emission data by cleaning and preprocessing the data stored in the database;
[0012] The network module is responsible for transmitting the processed data securely and quickly to the cloud or other terminal devices.
[0013] The carbon emission gas sensor is an ULTRAMAT 6 infrared gas analyzer.
[0014] The storage module receives the time series data of the content of harmful substances produced by combustion monitored by the sensor of the acquisition module, and stores and manages it through a time series database system based on programmable network hardware acceleration;
[0015] The time series database system based on programmable network hardware acceleration includes a read and write request processing module based on programmable network hardware, a query optimization module based on hardware incremental materialized view, and a time series data compression storage module based on hardware;
[0016] The read and write request processing module based on programmable network hardware is used to implement the processing of read and write requests of the time series database in the network hardware, and realizes its functions through the writing process of time series data and the process of executing read requests in the network hardware.
[0017] The writing process of the time series data is divided into the following steps:
[0018] 1) Traffic filtering: The traffic filtering module matches the write request of time series data, removes it from the normal network data processing flow, and forwards it to the hardware related to the writing of time series data. Specifically, this module implements the recognition of filter requests based on the UDP custom protocol, where the user data contains magic numbers, dictionary codes, and serialized data points, where the data points contain fixed-length measurement values, timestamps, and variable-length arrays, the latter of which store string codes to identify measurements, metrics, and time series data labels. Data packets are sent repeatedly to simplify parsing and avoid disorder. After matching the data packet, the write parser will parse the data points to be written from the data packet based on the above protocol.
[0019] 2) Write parsing: The write parsing module parses several data points from the write data packet and pushes these data points into a circular buffer through the internal bus. Each element in the buffer queue is a page, and each page has four states:
[0020] Writing state: When a page is in the writing state, the writing parsing module will write the data point to the page. Only one page is in the writing state at the same time.
[0021] Read-only state: When a page in the write state is full, it will be marked as read-only. The data of the page in the read-only state will not be changed. There may be multiple pages in the read-only state at the same time.
[0022] Persistent state: The persistent state means that the data of the interface needs to be written to the SSD. Read-only pages will be marked as persistent in two cases: the first case is that the creation time of the oldest page exceeds the threshold from the current time; the second case is that the number of read-only pages exceeds 10. At the same time, only one page in the system is in persistent state.
[0023] Idle state: The idle state means that the page can be written to. After the current page is full, the write module will randomly select a page from the idle pages as the next write page; in addition, when all the data in the page in the persistent state is written to the SSD, the page will be marked as idle.
[0024] 3) Disk access: The disk access module persists the filled areas in the ring buffer to the SSD through PCIe Peer ToPeer communication. The specific logic of disk writing is as follows: first, the disk access proxy module polls the page status and writes the data in the pages with persistent status to the SSD. When enough pages are filled, the disk access proxy module marks the oldest created page as persistent and generates an interrupt on the PCIe bus. The program running in the CPU captures the interrupt, opens the file in the SSD through a system call, obtains its address information and sends it to the disk access proxy module through the PCIe bus. The disk access proxy module writes the filled pages to the address sent by the host and marks these pages as idle.
[0025] The process of executing a read request in the network hardware specifically includes:
[0026] 1) Intercept read requests: The packet format of a read request is the same as that of a write request, so the system identifies the read request from the packet and removes it from the normal network data flow in the same way.
[0027] 2) Generate query plan: In order to simplify the system design, the client will complete the parsing of user queries and directly transmit the serialized query expression in the read request data packet. The query plan is serialized into a binary tree represented by an array in the read request data packet. This system only considers queries that can be mapped to a combination of the four operators of scanning, filtering, projection, and aggregation. After completing the parsing of the query request, this system can know whether the current request can be fully executed in the network hardware through the data provided by the metadata directory. If the current request can be executed, the specific location of the data waiting to be queried will be determined by scanning the metadata directory and the page directory, and the encoded query destination address will be stored together with the query metadata in a first-in-first-out queue.
[0028] 3) Execute the request: The executor obtains the query plan from the first-in-first-out queue, then reads the time series data required for the query from the write-ahead log cache and the first-level file after it is written to the disk, and inputs the time series data and the query metadata into the executor. The output of the executor is written to a temporary memory area. This memory area is a group of HBM pages organized in the form of a linked list. After completing the writing, the executor writes the query results to the responder, which reads the query results and constructs a return data packet to complete the execution of the query request.
[0029] The query optimization submodule based on hardware incremental materialized view consists of two parts: 1) materialized view creation tool and 2) materialized view runtime environment. The creation tool is responsible for converting user queries that can be incrementally calculated into a format that can be loaded into programmable network hardware. The runtime environment is a set of components running on programmable network hardware, responsible for the computational execution and state management of hardware materialized views. This section will introduce the system design of these two parts respectively.
[0030] 1) Materialized view creation tool
[0031] The input of the materialized view creation tool is the SQL query statement written by the user. First, the creation tool parses the input into a logical execution plan. And determines whether the logical execution plan can be calculated incrementally. Secondly, the creation tool maps the logical execution plan that can be calculated incrementally into a set of source code files that meet the high-level synthesis specification. Finally, the creation tool calls the high-level synthesis tool to synthesize the generated source code file into a binary bit stream file for configuring the programmable logic gate array, and loads it into the materialized view running environment on the programmable network hardware.
[0032] The first step of determining whether a user query can be incrementally calculated is to check whether the user query is bounded, where bounded means that the amount of data in the materialized view is a constant and does not increase with the increase of data in the materialized sequence. The second step of determining whether a user query can be incrementally calculated is to check whether the operator therein can be incrementally calculated. When the output query only contains operators that have been implemented in the materialized view running environment, it will be determined as a query that can be accelerated by incremental materialized views.
[0033] 2) Operating environment of materialized views
[0034] The operating environment of the incremental materialized view consists of three parts: an input module, a state storage module, and an output module.
[0035] · Input module: The input module is responsible for filtering out logic related to materialized view writing from write requests. Specifically, the input module filters out write requests that may update registered materialized views from the set of all write requests. When data is written according to the method of the time series data writing process, all write requests filtered out in the network will be copied to the input module of the materialized view operating environment at the same time. The input module will determine whether the materialized view has been registered on the object being written based on the sequence identifier in the write request and the information in the metadata storage. Requests that may cause the materialized view status to be updated will be passed to the downstream together with the view identifier.
[0036] ·State storage module: The state storage module is used to execute the hardware materialized view calculation logic generated by the user query. Specifically, the hardware materialized view generated by the user query runs on a partially reconfigurable area of the programmable logic gate array, and the logic in this area can be dynamically updated without affecting other areas. The operating environment of the materialized view reserves three interfaces in the reconfigurable area: the first interface is connected to the first-in-first-out queue outside the reconfigurable area, and is used to read the write requests filtered out in the input module; the second interface is connected to the state storage of the materialized view, and is used to read and write the state generated when the materialized view is updated; the third interface is connected to the output module, and is used to store the calculation results of the materialized view in the primary storage.
[0037] Output module: The output module is responsible for writing the output of the materialized view into the memory of the programmable logic gate array to support queries.
[0038] The hardware-based time series data compression storage submodule is used to implement compression-related computing logic for processing time series data in programmable network hardware, and is designed into two parts: the first part is a merge compressor, which is used to merge and compress several uncompressed files on the disk into a more compact format, and merge and compress several secondary files into tertiary files. Specifically, the merge compressor can be divided into four parts: a trigger, a feeder module, a fan-out module, and a serializer.
[0039] 1) The trigger will write a task descriptor into the feeder module after the number of secondary files exceeds 40. The task descriptor includes metadata such as the measurement value type of the current sequence and the location information of the secondary files.
[0040] 2) The feeder module will request the contents of the primary file from the disk access agent according to the task descriptor; these contents will be fed to the merge sort module in units of lines. The merge sort module runs a multi-way merge sort algorithm and outputs row format data that is ordered in time. Each row of data consists of a timestamp and a measurement value, and is 128 bits long.
[0041] 3) The fan-out module has two functions: the first function is to convert row-formatted data into column-formatted data. The fan-out module will split every eight rows of data into two 512-bit data packets and pass them to the compression module, where the timestamp and integer measurement value will be passed to the integer compression module; the floating-point measurement value will be passed to the floating-point compression module. The second function is to count the number of data points, the maximum and minimum values of the measurement value, the average of the measurement value, and other information; after processing a certain number of data points, the fan-out module will generate a special checkpoint data for the downstream module.
[0042] 4) The function of the serializer is to write data into the third-level file. The third-level file is divided into three parts: the first part is the timestamp area, which continuously stores several blocks of compressed timestamps; the second part is the measurement value area, which continuously stores several blocks of measurement value information compressed by the algorithm; the last part is the index area, which stores the timestamp sequence, offset index and pre-calculated results for fast retrieval.
[0043] The second part is the real-time data scanning operator, which is used to execute user queries that require scanning compressed data. Its structure includes two parts: a scheduler and a merging module.
[0044] 1) The scheduler scans the three-level file according to the input execution plan, and completes the scanning of historical data through two stages: reading the index area data and reading the data area. When the scheduler accesses the index area, it first stores the timestamp index in the three-level file index area into the HBM memory of the programmable network hardware; then it runs a binary search algorithm on the ordered timestamp index to determine the range of the data involved in this query in the file. When the scheduler accesses the data area, it reads different areas in the file according to different input plans. When the input query does not contain an aggregation operation, the scheduler will read the timestamp area and the measurement value area of the three-level file at the same time, and write the compressed data read into different decompression modules according to the data type. When the input query operation involves an aggregation operation, the scheduler will process two types of data blocks separately. The data in the first type of data blocks is completely contained in the scanned area, and the scheduler will read the pre-calculated results of these blocks from the three-level file and send them to the downstream. Only part of the data in the second type of data blocks is within the scope of this scan, and the scheduler will send the data of these blocks to the corresponding decompression module.
[0045] 2) The merging module has two different working modes. When there is no aggregation operation, the merging module merges the column format data output from different decompression modules into row format data and then outputs it; when there is an aggregation operation, the merging module first merges the pre-calculated results, then performs real-time aggregation calculations on the data that is not completely included in the scanning range, and finally merges the above two parts of the results and outputs them.
[0046] The analysis module inputs the original time series data from the acquisition module, and the output after cleaning and preprocessing will be used as the input of the data transmission operation of the downstream network module;
[0047] The cleaning and preprocessing method is as follows: first, detecting outliers in time series data by using a K-means clustering algorithm, checking data points in a data set, and grouping similar data points into K clusters, and then distinguishing anomalies by measuring the distance from a data point to its nearest centroid, and if the distance is greater than a certain threshold, marking the data point as an anomaly; secondly, interpolating abnormal and missing data by using an interpolation method; by segmenting the entire interpolation interval and fitting a polynomial function on each segment, the missing data is completed, the accuracy and stability of the prediction are improved, and the processed data is selected as the input of the data transmission operation of the downstream network module.
[0048] The network module inputs real-time data from terminal devices and edge nodes, and transmits it to the cloud center node through the network for further processing and analysis; the output is the processed and analyzed data results, which are used to support carbon reduction decision-making and optimization of enterprises on the chain;
[0049] A power grid supply chain big data transmission architecture based on cloud-edge-end collaboration is adopted to ensure that the data in the time series database can be effectively sent to the cloud, thereby supporting subsequent data analysis and business decisions. The architecture consists of three layers: cloud center server, edge node and smart terminal. The cloud server and edge node are interconnected through a wide area data transmission network. The smart terminal communicates with nearby edge nodes through the local area network. At the same time, a proprietary wide area communication network is also deployed to interact with the cloud center.
[0050] The innovation of the embodiment of the present invention lies in:
[0051] 1. Accurate and efficient collection of carbon emission data: Through high-frequency real-time sensor equipment, real-time collection of carbon emission data can be achieved, reducing manual intervention and errors.
[0052] 2. Efficiently process large-scale carbon emission data: Use a time series database system based on programmable network hardware acceleration to improve the speed and efficiency of data processing.
[0053] 3. Provide accurate data analysis: Through the built-in analysis module, the collected carbon emission data is cleaned, preprocessed and preliminarily analyzed.
[0054] 4. Safely transmit data to the cloud: Through the network module, the cloud-edge-end architecture is used to transmit the data pre-processed by the acquisition module to the cloud device. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.
[0056] Figure 1 is a schematic diagram showing an architecture diagram of a supply chain intelligent terminal based on heterogeneous hardware acceleration according to an embodiment of the present disclosure;
[0057] Figure 2 is a schematic diagram showing the technical specifications of the ULTRAMAT 6 infrared gas analyzer according to an embodiment of the present disclosure;
[0058] Figure 3 is a schematic diagram showing an architecture diagram of a time series database system based on programmable network hardware acceleration according to an embodiment of the present disclosure;
[0059] Figure 4 is a schematic diagram showing a process of performing writing in network hardware according to an embodiment of the present disclosure;
[0060] Figure 5 is a schematic diagram showing a flow chart of network hardware executing a read request according to an embodiment of the present disclosure;
[0061] Figure 6 is a schematic diagram showing a materialized view operation flow chart according to an embodiment of the present disclosure;
[0062] Figure 7 is a schematic diagram showing the structure of a combined compressor component according to an embodiment of the present disclosure;
[0063] Figure 8 is a schematic diagram showing the structure of a historical data scanning operator according to an embodiment of the present disclosure;
[0064] Fig. 9 It is a schematic diagram of a data management architecture diagram of cloud-edge-end collaboration of a power grid supply chain big data center according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0065] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.
[0066] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.
[0067] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.
[0068] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.
[0069] In the specification, when an element (such as a layer, a region or a substrate) is described as being “on”, “connected to” or “coupled to” another element, the element may be directly “on”, “connected to” or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to” or “directly coupled to” another element, there may be no other elements present therebetween.
[0070] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.
[0071] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.
[0072] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.
[0073] Figure 1 It is a schematic diagram showing a supply chain intelligent terminal based on heterogeneous hardware acceleration according to an embodiment of the present disclosure.
[0074] This intelligent terminal is an intelligent hardware device that integrates four modules: collection, storage, analysis, and network. It is based on the Internet of Things and big data technology and aims to solve the needs of power system supply chain enterprises for real-time monitoring, efficient storage, accurate analysis, and secure transmission of carbon emission data. Among them, the collection module collects carbon emission data from various links in the power grid supply chain, such as production, warehousing, maintenance, and scrapping, in real time through the carbon emission gas sensor deployed on the terminal side; the storage module adopts an optimized time series database system to achieve efficient storage and management of a large amount of carbon emission time series data uploaded by the collection module. This module is the core module of this intelligent terminal; the analysis module ensures the accuracy and effectiveness of carbon emission data by cleaning and preprocessing the data stored in the database; the network module is responsible for transmitting the processed data safely and quickly to the cloud or other terminal devices. The modules work together to form a complete set of carbon emission data management intelligent terminals, providing all-round data support and solutions for the decarbonization of the State Grid power supply chain enterprises.
[0075] Acquisition module:
[0076] The acquisition module collects carbon emission data in real time with low latency through sensors deployed in various links of the enterprise in the power grid supply chain, providing raw data support for subsequent data analysis and processing. Specifically, this terminal system uses the ULTRAMAT 6 infrared gas analyzer for high-frequency and real-time carbon emission collection. Due to its low detection limit and airtight isolation, it can measure low concentrations and has a long service life in harsh environments. The analyzer has 4 ranges, can realize internal / external switching, and supports automatic range switching. This flexibility ensures accurate measurement of gases of various concentrations. Its main technical specifications are as follows Figure 2 .
[0077] At the same time, the analyzer can enter a stable working state within 30 minutes and reach maximum accuracy after two hours. At a flow rate of 1 ml / min, the dead time is approximately 0.5 to 5s, and the internal signal processing time is less than 1s, which provides favorable conditions for high-frequency and real-time carbon emissions collection. In addition, the analyzer allows an ambient temperature range of -30°C to +70°C, and an operating temperature range of +5°C to +45°C. The adaptability of the wide temperature range enables it to operate stably under different environmental conditions, reducing the instability of data collection. The above settings can be equipped in various links of production, warehousing, maintenance and scrapping of power grid supply chain enterprises, and can realize real-time, precise and automated carbon emission accounting of the acquisition module. Through real-time monitoring of data, the accuracy and real-time nature of carbon emission accounting data collection can be greatly improved.
[0078] Storage Module:
[0079] The storage module receives time series data such as the carbon dioxide emissions generated by combustion, the content of harmful substances such as sulfur dioxide and nitrogen oxides in chimney exhaust gas monitored by the acquisition module sensor, and stores and manages them through a time series database system based on programmable network hardware acceleration.
[0080] Programmable network hardware (FPGAs) can process multiple data streams simultaneously, usually with low processing latency, and can be flexibly designed and optimized according to demand to achieve high-efficiency, high-frequency time series data processing. The carbon emission data obtained from the acquisition module has the characteristics of large data volume and strong real-time performance, which requires timely and effective monitoring and analysis in order to adjust the production process or take environmental protection measures in a timely manner. Therefore, this module designs a time series database system based on programmable network hardware acceleration. The specific composition of the system is as follows.
[0081] The system mainly includes three core sub-modules: read and write request processing module based on programmable network hardware, query optimization module based on hardware incremental materialized view, and time series data compression storage module based on hardware. The system architecture diagram is shown in the figure below. Figure 3 shown.
[0082] (1) Read and write request processing submodule based on programmable network hardware:
[0083] The read and write request processing submodule based on programmable network hardware is used to implement the read and write requests of the time series database in the network hardware. The writing process of time series data is as follows: Figure 4 As shown, it is divided into the following steps:
[0084] 1) Traffic filtering: The traffic filtering module matches the write request of time series data, removes it from the normal network data processing flow, and forwards it to the hardware related to the writing of time series data. Specifically, this module implements the recognition of filter requests based on the UDP custom protocol, where the user data contains magic numbers, dictionary codes, and serialized data points, where the data points contain fixed-length measurement values, timestamps, and variable-length arrays, the latter of which store string codes to identify measurements, metrics, and time series data labels. Data packets are sent repeatedly to simplify parsing and avoid disorder. After matching the data packet, the write parser will parse the data points to be written from the data packet based on the above protocol.
[0085] 2) Write parsing: The write parsing module parses several data points from the write data packet and pushes these data points into a circular buffer through the internal bus. Each element in the buffer queue is a page, and each page has four states:
[0086] Writing state: When a page is in the writing state, the writing parsing module will write the data point to the page. Only one page is in the writing state at the same time.
[0087] Read-only state: When a page in the write state is full, it will be marked as read-only. The data of the page in the read-only state will not be changed. There may be multiple pages in the read-only state at the same time.
[0088] Persistent state: The persistent state means that the data of the interface needs to be written to the SSD. Read-only pages will be marked as persistent in two cases: the first case is that the creation time of the oldest page exceeds the threshold from the current time; the second case is that the number of read-only pages exceeds 10. At the same time, only one page in the system is in persistent state.
[0089] Idle state: The idle state means that the page can be written to. After the current page is full, the write module will randomly select a page from the idle pages as the next write page; in addition, when all the data in the page in the persistent state is written to the SSD, the page will be marked as idle.
[0090] 3) Disk access: The disk access module persists the filled areas in the ring buffer to the SSD through PCIe Peer ToPeer communication. The specific logic of disk writing is as follows: first, the disk access proxy module polls the page status and writes the data in the pages with persistent status to the SSD. When enough pages are filled, the disk access proxy module marks the oldest created page as persistent and generates an interrupt on the PCIe bus. The program running in the CPU captures the interrupt, opens the file in the SSD through a system call, obtains its address information and sends it to the disk access proxy module through the PCIe bus. The disk access proxy module writes the filled pages to the address sent by the host and marks these pages as idle.
[0091] On the other hand, the process of executing a read request in the network hardware is as follows Figure 5 As shown, the specific process is:
[0092] 1) Intercept read requests: The packet format of a read request is the same as that of a write request, so the system identifies the read request from the packet and removes it from the normal network data flow in the same way.
[0093] 2) Generate query plan: In order to simplify the system design, the client will complete the parsing of user queries and directly transmit the serialized query expression in the read request data packet. The query plan is serialized into a binary tree represented by an array in the read request data packet. This system only considers queries that can be mapped to a combination of the four operators of scanning, filtering, projection, and aggregation. After completing the parsing of the query request, this system can know whether the current request can be fully executed in the network hardware through the data provided by the metadata directory. If the current request can be executed, the specific location of the data waiting to be queried will be determined by scanning the metadata directory and the page directory, and the encoded query destination address will be stored together with the query metadata in a first-in-first-out queue.
[0094] 3) Execute the request: The executor obtains the query plan from the first-in-first-out queue, then reads the time series data required for the query from the write-ahead log cache and the first-level file after it is written to the disk, and inputs the time series data and the query metadata into the executor. The output of the executor is written to a temporary memory area. This memory area is a group of HBM pages organized in the form of a linked list. After completing the writing, the executor writes the query results to the responder, which reads the query results and constructs a return data packet to complete the execution of the query request.
[0095] (2) Query optimization submodule based on hardware incremental materialized view:
[0096] The query optimization submodule based on hardware incremental materialized views is used to implement incremental materialized views based on programmable network hardware. Figure 6 As shown in the figure, in this submodule, the operating environment of the materialized view consists of three parts: the input module, the state storage module, and the output module. The data input part is responsible for filtering out the logic related to the materialized view write from the write request; the hardware materialized view is used to execute the hardware materialized view calculation logic generated by the user query; the output module is responsible for writing the output of the materialized view into the memory of the programmable logic gate array to support the query.
[0097] The query optimization submodule based on hardware incremental materialized view consists of two parts: 1) materialized view creation tool and 2) materialized view runtime environment. The creation tool is responsible for converting user queries that can be incrementally calculated into a format that can be loaded into programmable network hardware. The runtime environment is a set of components running on programmable network hardware, responsible for the computation execution and state management of hardware materialized views. This section will introduce the system design of these two parts respectively.
[0098] 1) Materialized view creation tool
[0099] The input of the materialized view creation tool is the SQL query statement written by the user. First, the creation tool parses the input into a logical execution plan. And determines whether the logical execution plan can be calculated incrementally. Secondly, the creation tool maps the logical execution plan that can be calculated incrementally into a set of source code files that meet the high-level synthesis specification. Finally, the creation tool calls the high-level synthesis tool to synthesize the generated source code files into a binary bitstream file for configuring the programmable logic gate array, and loads it into the materialized view running environment on the programmable network hardware.
[0100] The first step to determine whether a user query can be incrementally calculated is to check whether the user query is bounded. Bounded here means that the amount of data in the materialized view is a constant and does not increase with the increase of data in the materialized sequence. The second step to determine whether a user query can be incrementally calculated is to check whether the operators in it can be incrementally calculated. When the output query only contains operators that have been implemented in the materialized view running environment, it will be judged as a query that can be accelerated by incremental materialized views.
[0101] 2) Operating environment of materialized views
[0102] like Figure 6 As shown, the operating environment of the incremental materialized view consists of three parts: an input module, a state storage module, and an output module.
[0103] · Input module: The input module is responsible for filtering out logic related to materialized view writing from write requests. Specifically, the input module filters out write requests that may update registered materialized views from the set of all write requests. When data is written according to the method of the time series data writing process, all write requests filtered out in the network will be copied to the input module of the materialized view operating environment at the same time. The input module will determine whether the materialized view has been registered on the object being written based on the sequence identifier in the write request and the information in the metadata storage. Requests that may cause the materialized view status to be updated will be passed to the downstream together with the view identifier.
[0104] ·State storage module: The state storage module is used to execute the hardware materialized view calculation logic generated by the user query. Specifically, the hardware materialized view generated by the user query runs on a partially reconfigurable area of the programmable logic gate array, and the logic in this area can be dynamically updated without affecting other areas. The operating environment of the materialized view reserves three interfaces in the reconfigurable area: the first interface is connected to the first-in first-out queue outside the reconfigurable area, which is used to read the write requests filtered out by the input module; the second interface is connected to the state storage of the materialized view, which is used to read and write the state generated when the materialized view is updated; the third interface is connected to the output module, which is used to store the calculation results of the materialized view in the primary storage.
[0105] Output module: The output module is responsible for writing the output of the materialized view to the memory of the PGA to support queries.
[0106] (3) Hardware-based time series data compression storage submodule:
[0107] The hardware-based time series data compression storage submodule is used to implement the compression-related computing logic for processing time series data in programmable network hardware. This submodule is mainly designed as a combined compressor and a real-time data scanning operator.
[0108] The first part is the merge compressor, which is used to merge and compress several uncompressed files on the disk into a more compact format, and merge and compress several secondary files into tertiary files. Specifically, the merge compressor can be divided into four parts: trigger, feeder module, fan-out module and serializer.
[0109] 1) The trigger will write a task descriptor to the feeder module when the number of secondary files exceeds 40. The task descriptor includes metadata such as the measurement value type of the current sequence and the location information of the secondary files.
[0110] 2) The feeder module requests the contents of the primary file from the disk access agent according to the task descriptor; these contents are fed to the merge sort module in rows. The merge sort module runs a multi-way merge sort algorithm and outputs row-format data that is ordered in time. Each row of data consists of a timestamp and a measurement value, and is 128 bits long.
[0111] 3) The fan-out module has two functions: The first function is to convert row-formatted data into column-formatted data. The fan-out module will split every eight rows of data into two 512-bit data packets and pass them to the compression module, where the timestamp and integer measurement value will be passed to the integer compression module; the floating-point measurement value will be passed to the floating-point compression module. The second function is to count the number of data points, the maximum and minimum values of the measurement value, the average of the measurement value, and other information; after processing a certain number of data points, the fan-out module will generate a special checkpoint data for the downstream module.
[0112] 4) The function of the serializer is to write data to the third-level file. The third-level file is divided into three parts: the first part is the timestamp area, which continuously stores several blocks of compressed timestamps; the second part is the measurement value area, which continuously stores several blocks of measurement value information compressed by the algorithm; the last part is the index area, which stores the timestamp sequence, offset index and pre-calculated results for fast retrieval.
[0113] The second part is the real-time data scanning operator, which is used to execute user queries that require scanning compressed data. Its structure includes two parts: a scheduler and a merging module.
[0114] 1) The scheduler scans the three-level file according to the input execution plan, and completes the scanning of historical data in two stages: reading the index area data and reading the data area. When the scheduler accesses the index area, it first stores the timestamp index in the index area of the three-level file in the HBM memory of the programmable network hardware; then it runs a binary search algorithm on the ordered timestamp index to determine the range of data involved in this query in the file. When the scheduler accesses the data area, it reads different areas in the file according to different input plans. When the input query does not contain aggregation operations, the scheduler will read the timestamp area and measurement value area of the three-level file at the same time, and write the compressed data read into different decompression modules according to the data type. When the input query operation involves aggregation operations, the scheduler will process two types of data blocks separately. The data in the first type of data blocks is completely contained in the scanned area. The scheduler will read the pre-calculated results of these blocks from the three-level file and send them downstream. Only part of the data in the second type of data blocks is within the scope of this scan. The scheduler will send the data of these blocks to the corresponding decompression modules.
[0115] 2) The merge module has two different working modes. When there is no aggregation operation, the merge module merges the column format data output from different decompression modules into row format data and then outputs it; when there is an aggregation operation, the merge module first merges the pre-calculated results, then performs real-time aggregation calculations on the data that is not completely included in the scan range, and finally merges the above two parts of the results and outputs them.
[0116] Analysis Module:
[0117] The main function of the analysis module is to clean and preprocess the raw data obtained by the acquisition module to ensure the quality and availability of the data. The input of this module is the raw time series data from the acquisition module, and the output after cleaning and preprocessing will be used as the input of the data transmission operation of the downstream network module.
[0118] In actual scenarios, the sensors of the acquisition module may cause a certain period of time series data collected to be invalid due to extreme conditions of environmental factors such as temperature and humidity. Therefore, the data in the time series database needs to be cleaned and preprocessed to ensure subsequent use. Specifically, the K-means clustering algorithm is first used to detect outliers in the time series data. The algorithm looks at the data points in the data set and groups similar data points into K clusters, and then distinguishes anomalies by measuring the distance from the data point to its nearest centroid. If the distance is greater than a certain threshold, the data point is marked as an anomaly. Secondly, the anomaly and missing data are interpolated by interpolation. By segmenting the entire interpolation interval and fitting a polynomial function on each segment, the missing data is completed to improve the accuracy and stability of the prediction. The data after the above processing is selected as the input of the data transmission operation of the downstream network module.
[0119] Network Module:
[0120] The network module is responsible for the transmission and processing of big data in the power grid supply chain. Its input is real-time data from terminal devices and edge nodes, which is transmitted to the cloud center node through the network for further processing and analysis; the output is the processed and analyzed data results, which are used to support the carbon reduction decision-making and optimization of enterprises on the chain.
[0121] In this terminal system, we adopt a power grid supply chain big data transmission architecture based on cloud-edge-end collaboration to ensure that the data in the time series database can be effectively sent to the cloud, thereby supporting subsequent data analysis and business decision-making, such as Fig. 9 This architecture sends part of the computing cloud center resources to the edge side close to the device end, which can greatly reduce the communication burden and computing load between the cloud center and the State Grid Corporation, while improving the real-time response of the business.
[0122] In the power grid supply chain scenario, the architecture includes three layers: cloud center server, edge node and intelligent terminal. Among them, the cloud server and edge node are interconnected through a wide area data transmission network, and the intelligent terminal communicates with the nearby edge node through the local area network. At the same time, a dedicated wide area communication network is also deployed to interact with the cloud center. By dynamically optimizing resource scheduling strategies, rationally deploying data storage solutions, and orderly adjusting the execution order of services, the data processing capabilities of the limited network resources of enterprises in the State Grid power grid supply chain can be maximized, unnecessary repeated handling of time series data can be reduced, system energy consumption can be reduced, and the data management level and application efficiency of enterprises can be improved.
[0123] (1) Intelligent terminal: This is the intelligent terminal for carbon emission data of enterprises in the supply chain designed by this patent based on heterogeneous hardware acceleration.
[0124] (2) Edge nodes: including communication base stations, edge server clusters, communication networks, etc., responsible for the collection and preliminary processing of real-time carbon emission data, and capable of data storage, real-time computing and rapid decision-making.
[0125] (3) Cloud center node: It integrates the cloud-edge-end collaborative platform, the Internet of Things platform, and big data services, and is responsible for the aggregation and unified scheduling of computing resources, data, and services of the edge nodes of enterprises in the power grid supply chain. At the same time, it integrates the State Grid power grid supply chain's analysis needs for carbon emission data, provides intelligent big data analysis models and applications for enterprises on the chain, and gives full play to the advantages of large-scale data computing and storage in the cloud center. Through the collaborative management of cloud-edge-end, the interactive integration of real-time data of edge nodes and historical data of the cloud center is realized, ensuring the real-time calculation and secure distribution of end-side data.
[0126] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.
Claims
1. A supply chain intelligent terminal based on heterogeneous hardware acceleration, characterized in that: It includes four parts: acquisition module, storage module, analysis module and network module; The collection module collects carbon emission data of each link in the power grid supply chain in real time during the production, storage, maintenance and scrapping processes through carbon emission gas sensors deployed on the end side; The storage module adopts an optimized time series database system to construct a programmable network hardware read and write request processing submodule, a query optimization submodule based on hardware incremental materialized views, and a hardware-based time series data compression storage submodule to achieve storage and management of a large amount of carbon emission time series data uploaded by the acquisition module; The analysis module ensures the accuracy and validity of carbon emission data by cleaning and preprocessing the data stored in the database; The network module is responsible for transmitting the processed data securely and quickly to the cloud or other terminal devices.
2. A supply chain intelligent terminal based on heterogeneous hardware acceleration as claimed in claim 1, characterized in that: The carbon emission gas sensor is an ULTRAMAT 6 infrared gas analyzer.
3. A supply chain intelligent terminal based on heterogeneous hardware acceleration as claimed in claim 1, characterized in that: The storage module receives the time series data of the content of harmful substances produced by combustion monitored by the sensor of the acquisition module, and stores and manages it through a time series database system based on programmable network hardware acceleration; The time series database system based on programmable network hardware acceleration includes a read and write request processing module based on programmable network hardware, a query optimization module based on hardware incremental materialized view, and a time series data compression storage module based on hardware; The read and write request processing module based on programmable network hardware is used to implement the processing of read and write requests of the time series database in the network hardware, and realizes its functions through the writing process of time series data and the process of executing read requests in the network hardware.
4. A supply chain intelligent terminal based on heterogeneous hardware acceleration as claimed in claim 3, characterized in that: The writing process of the time series data is divided into the following steps: Step 1, traffic filtering: the traffic filtering module matches the write request of time series data, removes it from the normal network data processing flow, and forwards it to the hardware related to the writing of time series data; the traffic filtering module realizes the recognition of filtering requests based on the UDP custom protocol, wherein the user data includes magic number, dictionary code, serialized data point, the data point includes fixed-length measurement value, timestamp and variable-length array, the variable-length array stores string code, identification measurement, metric and time series data label; the data packet is sent repeatedly to simplify the parsing and avoid disorder; after matching the data packet, the write parser parses the data point to be written from the data packet based on the UDP custom protocol; Step 2, write parsing: The write parsing module parses several data points from the write data packet and pushes these data points into a circular buffer through the internal bus. Each element in the buffer queue is a page, and each page has four states: Writing state: When a page is in the writing state, the writing parsing module will write the data point to the page. Only one page is in the writing state at the same time. Read-only state: When a page in the write state is full, it will be marked as read-only. The data of the page in the read-only state will no longer be changed. There may be multiple pages in the read-only state at the same time. Persistent state: The persistent state means that the data of the interface is written to the SSD. The read-only page is marked as persistent in two cases: the first case is that the creation time of the oldest page exceeds the threshold from the current time; the second case is that the number of read-only pages exceeds 10; at the same time, only one page in the system is in persistent state; Idle state: The idle state means that the page can be written with data. After the current page is full, the write module randomly selects a page from the idle pages as the next write page. In addition, when all the data in the page in the persistent state is written to the SSD, the page is marked as idle. Step 3, disk access: The disk access module persists the full area in the ring buffer to the solid-state drive through PCIe Peer ToPeer communication; The specific logic of disk writing is as follows: first, the disk access proxy module polls the page status and writes the data in the pages with persistent status to the SSD. When enough pages are filled, the disk access proxy module marks the oldest created page as persistent and generates an interrupt on the PCIe bus; the program running in the CPU captures the interrupt, opens the file in the SSD through a system call, obtains its address information and sends it to the disk access proxy module through the PCIe bus; the disk access proxy module writes the filled pages to the addresses sent by the host and marks these pages as free; The process of executing a read request in the network hardware specifically includes: 1) Intercept read requests: The packet format of a read request is the same as that of a write request, so the system identifies the read request from the packet in the same way and removes it from the normal network data flow; 2) Generate query plan: The client completes the parsing of the user query and directly transmits the serialized query expression in the read request data packet. The query plan is serialized into a binary tree represented by an array in the read request data packet, and only considers queries that can be mapped to the combination of the four operators of scanning, filtering, projection, and aggregation. After completing the parsing of the query request, the data provided by the metadata directory can be used to determine whether the current request can be fully executed in the network hardware. If the current request can be executed, the specific location of the data waiting to be queried is determined by scanning the metadata directory and the page directory, and the encoded query destination address is stored in a first-in-first-out queue together with the query metadata; 3) Execute the request: The executor obtains the query plan from the first-in-first-out queue, then reads the time series data required for the query from the write-ahead log cache and the first-level file after it is written to the disk, and inputs the time series data and the query metadata into the executor; the output of the executor is written to a temporary memory area; this memory area is a group of HBM pages organized in the form of a linked list; after completing the writing, the executor writes the query results to the responder, which reads the query results and constructs a return data packet to complete the execution of the query request.
5. A supply chain intelligent terminal based on heterogeneous hardware acceleration as claimed in claim 4, characterized in that: The query optimization submodule based on hardware incremental materialized view consists of two parts: a materialized view creation tool and a materialized view operating environment; the creation tool is responsible for converting user queries that can be incrementally calculated into a format that can be loaded into programmable network hardware, and the operating environment is a set of components running on programmable network hardware, responsible for the calculation execution and state management of hardware materialized views; The input of the materialized view creation tool is the SQL query statement written by the user; first, the creation tool parses the input into a logical execution plan; And determine whether the logical execution plan can be incrementally calculated; secondly, create a tool to map the logical execution plan that can be incrementally calculated into a set of source code files that meet high-level synthesis specifications; Finally, the creation tool calls the high-level synthesis tool to synthesize the generated source code file into a binary bitstream file for configuring the programmable logic gate array, and loads it into the materialized view operating environment on the programmable network hardware; The first step of determining whether a user query can be incrementally calculated is to check whether the user query is bounded. Here, bounded means that the amount of data in the materialized view is a constant and does not increase with the increase of data in the materialized sequence. The second step of determining whether a user query can be incrementally calculated is to check whether the operator therein can be incrementally calculated. When the output query only contains operators that have been implemented in the materialized view running environment, it will be determined as a query that can be accelerated by the incremental materialized view. The incremental materialized view operating environment consists of three parts: input module, state storage module, and output module: The input module is responsible for filtering out logic related to materialized view writing from write requests, and the input module filters out write requests that may update registered materialized views from a collection of all write requests; when data is written according to the method of the time series data writing process, all write requests filtered out in the network are simultaneously copied to the input module of the materialized view running environment; the input module determines whether the materialized view has been registered on the object being written based on the sequence identifier in the write request and the information in the metadata storage; and transmits the request that may cause the materialized view status to be updated to the downstream together with the view identifier; The state storage module is used to execute the hardware materialized view calculation logic generated by the user query. The hardware materialized view generated by the user query runs on a partially reconfigurable area of the programmable logic gate array. The logic in the area can be dynamically updated without affecting other areas. The running environment of the materialized view reserves three interfaces in the reconfigurable area: the first interface is connected to the first-in first-out queue outside the reconfigurable area, and is used to read the write request filtered out by the input module; the second interface is connected to the state storage of the materialized view, and is used to read and write the state generated when the materialized view is updated; the third interface is connected to the output module, and is used to store the calculation result of the materialized view in the primary storage; The output module is responsible for writing the output of the materialized view into the memory of the programmable logic gate array to support queries.
6. A supply chain intelligent terminal based on heterogeneous hardware acceleration as claimed in claim 5, characterized in that: The hardware-based time series data compression storage submodule is used to implement the compression-related computing logic for processing time series data in programmable network hardware, and is designed into two parts: the first part is a merge compressor, which is used to merge and compress several uncompressed files on the disk into a more compact format, and merge and compress several secondary files into tertiary files. Specifically, the merge compressor can be divided into four parts: a trigger, a feeder module, a fan-out module, and a serializer; The trigger writes a task descriptor into the feeder module after the number of secondary files exceeds 40; the task descriptor includes metadata such as the measurement value type of the current sequence and the location information of the secondary files; The feeder module requests the content of the primary file from the disk access agent according to the task descriptor; these contents are input into the merge sort module in line units; The merge sort module runs a multi-way merge sort algorithm and outputs row-format data that is ordered in time. Each row of data consists of a timestamp and a measurement value, and is 128 bits long. The fan-out module has two functions: the first function is to convert the row-formatted data into column-formatted data; the fan-out module splits every eight rows of data into two 512-bit data packets and passes them to the compression module, where the timestamp and integer measurement value will be passed to the integer compression module; Floating point type measurements will be passed to the floating point compression module; The second function is to measure the number of data points, the maximum and minimum values of the measured values, the average of the measured values, and other information. After processing a certain number of data points, the fan-out module will generate a special checkpoint data for the downstream module. The function of the serializer is to write data into the third-level file; the file is divided into three parts: the first part is the timestamp area, which continuously stores several blocks of compressed timestamps; The second part is the measurement value area, which continuously stores several blocks of measurement value information compressed by the algorithm; the last part is the index area, which stores the timestamp sequence, offset index and pre-calculated results for fast retrieval; The second part is a real-time data scanning operator, which is used to execute user queries that require scanning compressed data. Its structure includes two parts: a scheduler and a merging module. The scheduler scans the three-level file according to the input execution plan, and completes the scanning of historical data in two stages: reading the index area data and reading the data area. When the scheduler accesses the index area, it first stores the timestamp index in the three-level file index area into the HBM memory of the programmable network hardware. Then, a binary search algorithm is run on the ordered timestamp index to determine the range of the data involved in this query in the file. When the scheduler accesses the data area, it reads different areas in the file according to different input plans. When the input query does not contain an aggregation operation, the scheduler reads the timestamp area and the measurement value area of the three-level file at the same time, and writes the compressed data read into different decompression modules according to the data type. When the input query operation involves aggregation operation, the scheduler processes two types of data blocks separately: the data in the first type of data blocks are completely included in the scanned area, and the scheduler will read the pre-calculated results of these blocks from the third-level file and send them downstream; only part of the data in the second type of data blocks is within the scope of this scan, and the scheduler will send the data of these blocks to the corresponding decompression module. The merge module has two different working modes; when there is no aggregation operation, the merge module merges the column format data output by different decompression modules into row format data and outputs it; when there is an aggregation operation, the merge module first merges the pre-calculated results, and then calculates the data that is not completely included in the scan range in real time for aggregation calculation, and finally merges the above two parts of the results and outputs them.
7. The supply chain intelligent terminal based on heterogeneous hardware acceleration according to claim 1, characterized in that: The analysis module inputs the original time series data from the acquisition module, and the output after cleaning and preprocessing will be used as the input of the data transmission operation of the downstream network module; The cleaning and preprocessing method is as follows: first, detecting outliers in time series data by using a K-means clustering algorithm, checking data points in a data set, and grouping similar data points into K clusters, and then distinguishing anomalies by measuring the distance from a data point to its nearest centroid, and if the distance is greater than a certain threshold, marking the data point as an anomaly; secondly, interpolating abnormal and missing data by using an interpolation method; by segmenting the entire interpolation interval and fitting a polynomial function on each segment, the missing data is completed, the accuracy and stability of the prediction are improved, and the processed data is selected as the input of the data transmission operation of the downstream network module.
8. The supply chain intelligent terminal based on heterogeneous hardware acceleration according to claim 1, characterized in that: The network module inputs real-time data from terminal devices and edge nodes, and transmits it to the cloud center node through the network for further processing and analysis; the output is the processed and analyzed data results, which are used to support carbon reduction decision-making and optimization of enterprises on the chain; A power grid supply chain big data transmission architecture based on cloud-edge-end collaboration is adopted to ensure that the data in the time series database can be effectively sent to the cloud, thereby supporting subsequent data analysis and business decisions. The architecture consists of three layers: cloud center server, edge node and smart terminal. The cloud server and edge node are interconnected through a wide area data transmission network. The smart terminal communicates with nearby edge nodes through the local area network. At the same time, a proprietary wide area communication network is also deployed to interact with the cloud center.