Industrial equipment time series data storage and preprocessing method

Abstract

The invention provides an industrial equipment time series data storage and preprocessing method, and belongs to the field of data storage and data mining. The method comprises the following steps: step 1, configuring a sensor to collect original data of industrial equipment in real time, and sending the original data to a message queue; 2, setting a data import module, analyzing the theme of the message queue based on a data synchronization tool, obtaining first data and storing the first data in a table A; 3, designing a table B corresponding to the table A, and preprocessing and standardizing the first data of the table A to obtain standard data; 4, performing time feature extraction on the standard data to obtain time domain features, and performing partition processing on the standard data according to the time domain features to obtain a partition data list; and 5, obtaining a dimension attribute corresponding to the unit data of each partition data list, and storing the partition data lists into the table C according to a storage logic sequence. Industrial data acquisition and storage are realized, and data processing efficiency and manageability are improved.

Description

Technical Field

[0001] The present invention relates to the fields of data storage and data mining, and in particular to a method for storing and preprocessing time series data of industrial equipment. Background Art

[0002] At present, in the field of intelligent manufacturing and industrial Internet, the log data or measurement data generated by the operation of industrial equipment has time series properties. For example, the vibration amplitude data of the fan sampled at a fixed frequency at a certain time can be obtained through data mining, such as peak value, mean value, variance, waveform and other frequency domain feature values, which can be used for vibration signal classification, fault diagnosis and fault prediction, so as to predict the life of the equipment and perform regular maintenance on the equipment. The traditional storage method uses file servers or cloud objects to store raw data files, or collects them into message queues, or uses time series databases for storage, and then performs data mining through python or Spark. This method requires a lot of storage space and takes up a lot of memory space when reading with Python code, or uses time series databases such as TDengine's SQL capabilities to perform data analysis on time series data through preset functions.

[0003] Therefore, the present invention proposes a method for storing and preprocessing time series data of industrial equipment. Summary of the invention

[0004] The present invention provides a method for storing and preprocessing industrial equipment time series data, which is used to collect industrial equipment data in real time and send it to a message queue, and use a data synchronization tool to store the data in table A of HBase. Then, standard data is obtained through preprocessing and standardization, and a corresponding table B is designed. On the basis of the standardized data, time features are extracted and partitioned to generate a partition data list. Finally, the data is stored according to the dimension attributes, and the partitioned data is stored in table C in the storage logic order. The data processing and storage efficiency is optimized to ensure efficient management and analysis of data.

[0005] In one aspect, the present invention provides a method for storing and preprocessing time series data of industrial equipment, comprising: Step 1: Configure sensors to collect raw data from industrial equipment in real time and send the raw data to the message queue; Step 2: Set a data import module, parse the topic of the message queue based on the data synchronization tool, obtain the first data and store it in Table A of the HBase database; Step 3: Design table B corresponding to table A, and pre-process and standardize the first data of table A to obtain standard data; Step 4: Extract the time features of the standard data to obtain the time domain features, partition the standard data according to the time domain features, and obtain a partition data list; Step 5: Get the dimension attributes corresponding to the cell data of each partition data list, and store the partition data list in table C according to the storage logic order.

[0006] On the other hand, sensors are configured to collect raw data of industrial equipment in real time, including: Obtain the working environment and monitoring requirements of the industrial equipment to select the type of sensor and configure a unique first number for the sensor; Determine the installation location of the sensor based on the original design of the industrial equipment and the surrounding environment, and assign a unique second number to the installation location; According to the corresponding relationship between the first number and the second number, the sensor is configured and installed, and the sensor is initialized and started based on a preset timing sampling frequency, wherein the sensor is used to collect the original data of the industrial equipment in real time.

[0007] On the other hand, sending raw data to the message queue includes: Create a message queue, serialize the original data into a byte stream, and insert the byte stream iteratively according to the bytes; The iteration stops until the byte stream of the original data is completely inserted into the message queue.

[0008] On the other hand, a data import module is set to parse the topic of the message queue based on the data synchronization tool, including: Build a data import module, configure and install a data synchronization tool, parse the queue ID of the message queue based on the data synchronization tool, and generate row key-value pairs of the original data; Create and register consumers in the data synchronization tool, and create a consumption record data table; The consumer consumes the topic of the message queue and inserts the data into the original data table A to generate a row key value.

[0009] On the other hand, the original data is obtained and stored in Table A of the HBase database, including: According to the standard preset time series fields, a table named Table A is built in the HBase database to store the original time series data; According to the result of executing the consumption, first original data is obtained and analyzed; If the first original data are measurement values ​​at the same time point, they are concatenated into one value by string concatenation, wherein the individual measurement values ​​are separated by special symbols; If the first original data are measurement values ​​at different time points, the data at different measurement times are in different rows.

[0010] On the other hand, table B corresponding to table A is designed, and standard data is obtained by preprocessing and standardizing the first data of table A, including: Obtain first data from table A, and convert all data into second data in a preset format; Select a preset number of neighbors K and calculate the KNN distance between any two measured values ​​in the second data as: ;in, represents the distance between the i-th measurement value and the j-th measurement value in the second data, n represents that there are a total of n measurement values ​​in the second data, Indicates the second data The mean of all measurements in , ln( ) represents the logarithmic function; Indicates the second data The variance of all measurements in ; Respectively represent the weights of the i-th measurement value and the j-th measurement value; min and max represent the minimum and maximum values, respectively; Based on the preset number of neighbors K, any measurement value is selected as the middle value, K measurement values ​​near the middle value are screened to form a sample group, and the average distance of the sample group is obtained. If the distance between any measurement value in the sample group and the middle value is greater than the average distance, the corresponding measurement value is determined to be an abnormal value, otherwise, the corresponding measurement value is determined to be normal; The outliers of the second data are removed to obtain third data, and the third data are standardized to obtain standard data; On the other hand, the standard data is subjected to time feature extraction to obtain time domain features, and the standard data is partitioned according to the time domain features to obtain a partition data list, including: Obtain standard data from Table B, and obtain a preset timing sampling frequency of the standard data according to a timing field of the standard data; Extracting time features from the standard data and converting timestamp information of the standard data into specific time features, using the specific time features as time domain features; Define the time interval based on the time domain characteristics, specifically: ;in Indicates the time interval, Indicates the starting time point, represents the time starting point mapping coefficient, represents the time end mapping coefficient, Represents the time domain characteristics, represents the time interval mean of the standard data, represents the maximum time interval of standard data, T( ) represents the event processing function; By time interval The time part of the standard data is partitioned and cut, and each time partition corresponds to a segment of size time interval, wherein the time partition and its corresponding measurement value constitute unit data of the partition data list.

[0011] On the other hand, the dimension attributes corresponding to the cell data of each partition data list are obtained, and the partition data list is stored in table C according to the storage logic order, including: Traverse the partition data list and obtain the time range of each unit data; According to the fields of the standard data corresponding to any unit data in any time interval, the dimension attributes corresponding to all the fields are obtained according to the field name-dimension attribute mapping table, wherein the dimension attributes are the dimension attributes of the unit data; Create a record for each unit data, and store its time interval, unit data value, and dimension attributes in table C in the storage logic order.

[0012] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a method for storing and preprocessing industrial equipment time series data, which is used to collect industrial equipment data in real time and send it to a message queue, and use a data synchronization tool to store the data in table A of HBase. Then, standard data is obtained through preprocessing and standardization, and a corresponding table B is designed. On the basis of the standardized data, time features are extracted and partitioned to generate a partition data list. Finally, the data is stored according to the dimension attributes, and the partitioned data is stored in table C in the storage logic order. The data processing and storage efficiency is optimized to ensure efficient management and analysis of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 It is a flow chart of a method for storing and preprocessing time series data of industrial equipment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] Embodiment 1: like Figure 1 As shown, an embodiment of the present invention provides a method for storing and preprocessing time series data of industrial equipment, including: Step 1: Configure sensors to collect raw data from industrial equipment in real time and send the raw data to the message queue; Step 2: Set a data import module, parse the topic of the message queue based on the data synchronization tool, obtain the first data and store it in Table A of the HBase database; Step 3: Design table B corresponding to table A, and pre-process and standardize the first data of table A to obtain standard data; Step 4: Extract the time features of the standard data to obtain the time domain features, partition the standard data according to the time domain features, and obtain a partition data list; Step 5: Get the dimension attributes corresponding to the cell data of each partition data list, and store the partition data list in table C according to the storage logic order.

[0017] In this embodiment, the sensor is a device used to monitor and collect industrial equipment status or environmental data in real time, including: temperature, pressure, humidity, vibration and other types.

[0018] In this embodiment, industrial equipment refers to machinery, instruments, tools and other equipment used for production, processing, detection or control in the industrial production process.

[0019] In this embodiment, raw data refers to original information that is collected in real time by sensors, instruments and other equipment during the operation of industrial equipment and has not been processed or analyzed.

[0020] In this embodiment, message queue is a technology that allows different services to communicate by sending and receiving messages without requiring a direct synchronous connection.

[0021] In this embodiment, the data import module refers to a component for obtaining data from a message queue and storing the data in a database (eg, HBase).

[0022] In this embodiment, the data synchronization tool is a software tool mainly used to synchronize data between different data sources or systems, such as Kafka.

[0023] In this embodiment, the topic refers to the data stream or message category in the message queue.

[0024] In this embodiment, the HBase database is an open source, distributed column storage database, which is part of the Apache Hadoop ecosystem and is designed to handle large-scale, distributed data storage requirements.

[0025] In this embodiment, table A is an HBase table that stores original data obtained from the message queue.

[0026] In this embodiment, Table B is an HBase table for storing standard data after preprocessing, standardization and partitioning, and includes columns of dimension attributes and columns of time series data after preprocessing.

[0027] In this embodiment, with the help of HBase's coprocessor function, the logic of data preprocessing is placed on the server side, and a large amount of data will not be pulled to the client for processing, occupying too much memory. Through HBase's distributed storage and computing capabilities, higher data preprocessing efficiency is achieved, and data storage and preprocessing functions are performed simultaneously. Subsequent data mining only needs to query the table after data preprocessing, and there is no need to perform data cleaning, deduplication, normalization and other processing steps.

[0028] In this embodiment, data processing of Table A and Table B is performed simultaneously and synchronously. The HBase coprocessor is mounted on the original data Table A. Whenever a new row of data is inserted into Table A, the coprocessor is triggered to run and write the preprocessed data into Table B.

[0029] In this example, preprocessing normalization is used to convert raw data into a standard form suitable for subsequent analysis.

[0030] In this embodiment, standard data refers to data that has been preprocessed and standardized and has a unified format and structure.

[0031] In this embodiment, the time domain feature refers to a data feature related to time, which is extracted from a timestamp or a time field and reflects the time nature, periodicity, trend and other features of the data.

[0032] In this embodiment, partition processing refers to dividing data into different blocks according to certain specific features during data storage, query and processing.

[0033] In this embodiment, the partition data list refers to a series of data units obtained after partitioning the standard data according to the extracted time domain features (such as year, month, day, hour, etc.) in step 4.

[0034] In this embodiment, the unit data refers to the smallest data unit in Table A that has been preprocessed and standardized and partitioned according to time features.

[0035] In this embodiment, the dimension attribute refers to various characteristic fields that can be used to describe or classify data in data processing and storage.

[0036] In this embodiment, the storage logic order refers to a method of storing the data in the partition data list into the target table (table C) according to certain rules.

[0037] In this embodiment, table C is a table for storing dimension attribute data corresponding to the unit data of each partition data list.

[0038] The working principle and beneficial effects of the above technical solution are: by real-time collection and processing of industrial equipment data, using message queues and data synchronization tools to store data in HBase, and after preprocessing, feature extraction and partitioned storage, the data storage efficiency and the accuracy of subsequent analysis are improved, supporting efficient data processing and management.

[0039] Embodiment 2: Based on the above embodiment 1, sensors are configured to collect raw data of industrial equipment in real time, including: Obtain the working environment and monitoring requirements of the industrial equipment to select the type of sensor and configure a unique first number for the sensor; Determine the installation location of the sensor based on the original design of the industrial equipment and the surrounding environment, and assign a unique second number to the installation location; According to the corresponding relationship between the first number and the second number, the sensor is configured and installed, and the sensor is initialized and started based on a preset timing sampling frequency, wherein the sensor is used to collect the original data of the industrial equipment in real time.

[0040] In this embodiment, the working environment refers to the physical and environmental conditions of the device during actual operation, including temperature, humidity, pressure, vibration, gas composition, electromagnetic interference and other factors.

[0041] In this embodiment, the monitoring demand refers to the demand for real-time monitoring and data collection of the operating status, environmental conditions and equipment performance of industrial equipment.

[0042] In this embodiment, the first number is a unique identifier for identifying each sensor.

[0043] In this embodiment, the original design drawing refers to a detailed technical drawing drawn by an engineer during the design and construction phase of an industrial device or system.

[0044] In this embodiment, the installation location refers to a specific location or area where the sensor is actually placed in a device or working environment.

[0045] In this embodiment, the second number is a unique identifier for identifying each sensor location.

[0046] In this embodiment, the preset timing sampling frequency refers to the frequency at which the sensor collects data from the industrial equipment within a specified time interval.

[0047] The working principle and beneficial effects of the above technical solution are: by combining the equipment working environment and monitoring requirements to select the sensor type and configure a unique number, determine the installation location and configure the number, achieve accurate sensor installation and real-time data collection, optimize equipment monitoring efficiency, and ensure the accuracy and reliability of data collection.

[0048] Embodiment 3: On the basis of the above-mentioned embodiment 2, sending the original data to the message queue includes: Create a message queue, serialize the original data into a byte stream, and insert the byte stream iteratively according to the bytes; The iteration stops until the byte stream of the original data is completely inserted into the message queue.

[0049] In this embodiment, serialization refers to the process of converting the state of a data structure into a format that can be stored or transmitted.

[0050] In this embodiment, byte stream refers to a method of processing and transmitting data in a computer system in units of bytes, which is expressed in binary.

[0051] In this embodiment, iterative insertion refers to a process of inserting bytes into the message queue in sequence until all bytes are inserted.

[0052] The working principle and beneficial effects of the above technical solution are: by serializing the original data and iteratively inserting it into the message queue, using the queue identifier and pointer value to control the insertion process, ensuring the orderly insertion of data, avoiding data conflicts and duplications, and improving the reliability and efficiency of data transmission.

[0053] Embodiment 4: On the basis of the above-mentioned embodiment 3, a data import module is set to parse the subject of the message queue based on the data synchronization tool, including: Build a data import module, configure and install a data synchronization tool, parse the queue ID of the message queue based on the data synchronization tool, and generate row key-value pairs of the original data; Create and register consumers in the data synchronization tool, and create a consumption record data table; The consumer consumes the topic of the message queue and inserts the data into the original data table A to generate a row key value.

[0054] In this embodiment, parsing is a process of converting a byte stream into raw data.

[0055] In this embodiment, a row key (Rowkey) in a row key-value pair corresponds to multiple columns, each column stores a value of a dimension attribute or a value after concatenation of time series data, and the preprocessing only processes the columns of the time series data.

[0056] In this embodiment, a consumer refers to a component that processes or consumes data.

[0057] In this embodiment, the consumption record data table is a table that stores time series data after consumption (ie, data processing).

[0058] In this embodiment, traversal refers to analyzing each data record one by one when processing the original time series data.

[0059] The working principle and beneficial effects of the above technical solution are: use the data synchronization tool to parse the message queue and generate row key-value pairs, use consumers to traverse and determine whether the row key value has been consumed, ensure that the data is consumed uniquely and record the consumption status, improve the accuracy and efficiency of data processing, and avoid duplicate consumption.

[0060] Embodiment 5: On the basis of the above-mentioned embodiment 4, the original data is obtained and stored in table A of the HBase database, including: According to the standard preset time series fields, a table named Table A is built in the HBase database to store the original time series data; According to the result of executing the consumption, first original data is obtained and analyzed; If the first original data are measurement values ​​at the same time point, they are concatenated into one value by string concatenation, wherein the individual measurement values ​​are separated by special symbols; If the first original data are measurement values ​​at different time points, the data at different measurement times are in different rows.

[0061] In this embodiment, the standard preset time series field refers to a basic field used to define and identify time series data, such as: timestamp, device identification, data type, etc.

[0062] In this embodiment, the HBase database is an open source, distributed, column-based NoSQL database system used to process large-scale data sets and is particularly suitable for storing and managing non-relational data.

[0063] In this embodiment, the original time series data refers to data collected in time sequence and representing a certain physical virtual phenomenon.

[0064] In this embodiment, the first original data refers to the original data initially acquired during the time series data acquisition process.

[0065] In this embodiment, the measurement value refers to data collected by a sensor, device or system and representing a physical quantity or state, such as temperature, humidity, voltage, air pressure, speed, flow rate, etc.

[0066] In this embodiment, the string concatenation method refers to connecting multiple measurement values ​​together through specific symbols to form a complete string.

[0067] The working principle and beneficial effects of the above technical solution are: time series data is stored through HBase table A, and data is processed according to the measurement time: the measurement values ​​at the same time point are spliced ​​into one value, and the data at different time points are stored in multiple rows, which optimizes the storage and query of time series data and improves the flexibility and efficiency of data processing.

[0068] Embodiment 6: On the basis of the above-mentioned embodiment 5, a table B corresponding to table A is designed, and the first data of table A is preprocessed and standardized to obtain standard data, including: Obtain first data from table A, and convert all data into second data in a preset format; Select a preset number of neighbors K and calculate the KNN distance between any two measured values ​​in the second data as: ;in, represents the distance between the i-th measurement value and the j-th measurement value in the second data, n represents that there are a total of n measurement values ​​in the second data, Indicates the second data The mean of all measurements in , ln( ) represents the logarithmic function; Indicates the second data The variance of all measurements in ; Respectively represent the weights of the i-th measurement value and the j-th measurement value; min and max represent the minimum and maximum values, respectively; Based on the preset number of neighbors K, any measurement value is selected as the middle value, K measurement values ​​near the middle value are screened to form a sample group, and the average distance of the sample group is obtained. If the distance between any measurement value in the sample group and the middle value is greater than the average distance, the corresponding measurement value is determined to be an abnormal value, otherwise, the corresponding measurement value is determined to be normal; The outliers of the second data are removed to obtain third data, and the third data are standardized to obtain standard data; In this embodiment, the second data is data obtained after certain processing and conversion, and the original data comes from Table A.

[0069] In this embodiment, the preset number of neighbors refers to the number of neighbors selected when calculating the KNN (K-nearest neighbor) distance.

[0070] In this embodiment, KNN distance is a core concept in the K-Nearest Neighbor algorithm, which measures the distance between two data points.

[0071] In this embodiment, the middle value refers to a certain measurement value used as a reference point to screen K nearest neighbor data points when performing KNN calculation.

[0072] In this embodiment, the sample group refers to K neighboring measurement values ​​selected from around the selected middle value based on a preset number of neighbors (K) according to the KNN algorithm.

[0073] In this embodiment, an outlier refers to a measurement value in a data set that significantly deviates from the overall data trend.

[0074] In this embodiment, the third data is a result obtained by performing a standardization process on a data set after outliers have been removed.

[0075] In this embodiment, the standard data is data that has been subjected to standardization processing.

[0076] The working principle and beneficial effects of the above technical solution are: the distance between the measured values ​​is calculated by the KNN algorithm, outliers are screened and removed, and standard data is generated based on standardization. This method effectively improves the accuracy and quality of the data, removes outliers, and ensures the reliability and stability of the data analysis results.

[0077] Embodiment 7: On the basis of the above-mentioned embodiment 1, the standard data is subjected to time feature extraction to obtain time domain features, and the standard data is partitioned according to the time domain features to obtain a partition data list, including: Obtain standard data from Table B, and obtain a preset timing sampling frequency of the standard data according to a timing field of the standard data; Extracting time features from the standard data and converting timestamp information of the standard data into specific time features, using the specific time features as time domain features; Define the time interval based on the time domain characteristics, specifically: ;in Indicates the time interval, Indicates the starting time point, represents the time starting point mapping coefficient, represents the time end mapping coefficient, Represents the time domain characteristics, represents the time interval mean of the standard data, represents the maximum time interval of standard data, T( ) represents the event processing function; By time interval The time part of the standard data is partitioned and cut, and each time partition corresponds to a segment of size time interval, wherein the time partition and its corresponding measurement value constitute unit data of the partition data list.

[0078] In this embodiment, the timing field refers to a data field related to time, and is used to indicate the time point at which the data record occurs.

[0079] In this embodiment, the timestamp information refers to the specific time point at which each piece of data is recorded, and exists in the form of a timestamp.

[0080] In this embodiment, the temporal regularity matching degree is to evaluate the matching degree between the extracted temporal features and the preset temporal sampling frequency.

[0081] In this embodiment, the specific time feature refers to a specific data attribute extracted from the timestamp information in the standard data and capable of accurately describing the time dimension, such as year, month, day, minute, hour, etc.

[0082] The working principle and beneficial effect of the above technical solution are: by extracting the time characteristics of standard data and comparing them with the preset sampling frequency, the best time domain characteristics are selected, and the time interval is defined for data partition cutting. This method optimizes the time processing and partitioning of data and improves the efficiency of data time series analysis and processing.

[0083] Embodiment 8: On the basis of the above-mentioned embodiment 1, the dimension attribute corresponding to the unit data of each partition data list is obtained, and the partition data list is stored in table C according to the storage logic order, including: Traverse the partition data list and obtain the time range of each unit data; According to the fields of the standard data corresponding to any unit data in any time interval, the dimension attributes corresponding to all the fields are obtained according to the field name-dimension attribute mapping table, wherein the dimension attributes are the dimension attributes of the unit data; Create a record for each unit data, and store its time interval, unit data value, and dimension attributes in table C in the storage logic order.

[0084] In this embodiment, the field name-dimension attribute mapping table is a mapping structure that associates data fields with their corresponding dimension attributes.

[0085] In this embodiment, the dimension attribute refers to descriptive information related to the data field, such as time, region, product, sales, etc.

[0086] The working principle and beneficial effect of the above technical solution are: by traversing the partition data list, combining the time interval and dimension attribute mapping table, creating a record for each unit data and storing it in table C according to the storage logic. This method improves the structured storage efficiency of data and facilitates subsequent data query and analysis.

[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for storing and preprocessing time series data of industrial equipment, characterized in that: include: Step 1: Configure sensors to collect raw data from industrial equipment in real time and send the raw data to the message queue; Step 2: Set a data import module, parse the topic of the message queue based on the data synchronization tool, obtain the first data and store it in Table A of the HBase database; Step 3: Design table B corresponding to table A, and pre-process and standardize the first data of table A to obtain standard data; Step 4: Extract the time features of the standard data to obtain the time domain features, partition the standard data according to the time domain features, and obtain a partition data list; Step 5: Get the dimension attributes corresponding to the cell data of each partition data list, and store the partition data list in table C according to the storage logic order.

2. The method for storing and preprocessing time series data of industrial equipment according to claim 1, characterized in that: Configure sensors to collect raw data from industrial equipment in real time, including: Obtain the working environment and monitoring requirements of the industrial equipment to select the type of sensor and configure a unique first number for the sensor; Determine the installation location of the sensor based on the original design of the industrial equipment and the surrounding environment, and assign a unique second number to the installation location; According to the corresponding relationship between the first number and the second number, the sensor is configured and installed, and the sensor is initialized and started based on a preset timing sampling frequency, wherein the sensor is used to collect the original data of the industrial equipment in real time.

3. The method for storing and preprocessing time series data of industrial equipment according to claim 2, characterized in that: Send raw data to the message queue, including: Create a message queue, serialize the original data into a byte stream, and insert the byte stream iteratively according to the bytes; The iteration stops until the byte stream of the original data is completely inserted into the message queue.

4. The method for storing and preprocessing time series data of industrial equipment according to claim 3, characterized in that: A data import module is set up to parse the topic of the message queue based on the data synchronization tool, including: Build a data import module, configure and install a data synchronization tool, parse the queue ID of the message queue based on the data synchronization tool, and generate row key-value pairs of the original data; Create and register consumers in the data synchronization tool, and create a consumption record data table; The consumer consumes the topic of the message queue and inserts the data into the original data table A to generate a row key value.

5. The method for storing and preprocessing time series data of industrial equipment according to claim 4, characterized in that: Obtain the original data and store it in Table A of the HBase database, including: According to the standard preset time series fields, a table named Table A is built in the HBase database to store the original time series data; According to the result of executing the consumption, first original data is obtained and analyzed; If the first original data are measurement values ​​at the same time point, they are concatenated into one value by string concatenation, wherein the individual measurement values ​​are separated by special symbols; If the first original data are measurement values ​​at different time points, the data at different measurement times are in different rows.

6. The method for storing and preprocessing time series data of industrial equipment according to claim 5, characterized in that: Table B corresponding to Table A is designed, and standard data is obtained by preprocessing and standardizing the first data of Table A, including: Obtain first data from table A, and convert all data into second data in a preset format; Select a preset number of neighbors K and calculate the KNN distance between any two measured values ​​in the second data as: ;in, represents the distance between the i-th measurement value and the j-th measurement value in the second data, n represents that there are a total of n measurement values ​​in the second data, Indicates the second data The mean of all measurements in , ln( ) represents the logarithmic function; Indicates the second data The variance of all measurements in ; Respectively represent the weights of the i-th measurement value and the j-th measurement value; min and max represent the minimum and maximum values, respectively; Based on the preset number of neighbors K, any measurement value is selected as the middle value, K measurement values ​​near the middle value are screened to form a sample group, and the average distance of the sample group is obtained. If the distance between any measurement value in the sample group and the middle value is greater than the average distance, the corresponding measurement value is determined to be an abnormal value, otherwise, the corresponding measurement value is determined to be normal; The outliers of the second data are removed to obtain the third data, and the third data are standardized to obtain the standard data.

7. The method for storing and preprocessing time series data of industrial equipment according to claim 1, characterized in that: The standard data is subjected to time feature extraction to obtain time domain features, and the standard data is partitioned according to the time domain features to obtain a partition data list, including: Obtain standard data from Table B, and obtain a preset timing sampling frequency of the standard data according to a timing field of the standard data; Extracting time features from the standard data and converting timestamp information of the standard data into specific time features, using the specific time features as time domain features; Define the time interval based on the time domain characteristics, specifically: ;in Indicates the time interval, Indicates the starting time point, represents the time starting point mapping coefficient, represents the time end mapping coefficient, Represents the time domain characteristics, represents the time interval mean of the standard data, represents the maximum time interval of standard data, T( ) represents the event processing function; By time interval The time part of the standard data is partitioned and cut, and each time partition corresponds to a segment of size time interval, wherein the time partition and its corresponding measurement value constitute unit data of the partition data list.

8. The method for storing and preprocessing time series data of industrial equipment according to claim 1, characterized in that: Get the dimension attributes corresponding to the cell data of each partition data list, and store the partition data list in table C according to the storage logic order, including: Traverse the partition data list and obtain the time range of each unit data; According to the fields of the standard data corresponding to any unit data in any time interval, the dimension attributes corresponding to all the fields are obtained according to the field name-dimension attribute mapping table, wherein the dimension attributes are the dimension attributes of the unit data; Create a record for each unit data, and store its time interval, unit data value, and dimension attributes in table C in the storage logic order.

Images