An industrial internet big data storage and analysis device

Abstract

The application discloses an industrial internet big data storage and analysis device, and relates to the technical field of industrial internet big data processing.The device comprises a multi-source data acquisition unit, which acquires heterogeneous data streams and the like and performs standardized processing to output time sequence data blocks; a tensor modeling preprocessing unit, which constructs a four-dimensional tensor and eliminates redundant data; a tensor real-time analysis unit, which allocates resources according to service quality and outputs analysis results; a hierarchical storage scheduling unit, which determines storage positions and strategies; and a service and closed loop optimization unit, which collects data, optimizes parameters, feeds back the first two units, and forms a complete closed loop.The application realizes data standardized processing, tensor modeling, real-time analysis, hierarchical storage scheduling and service closed loop optimization through the multi-source data acquisition unit and the like, solves problems, such as disordered data and uneven distribution of computing power, in traditional industrial big data processing, forms a complete closed loop, improves overall efficiency, and makes industrial internet big data storage and analysis more efficient, economical and practical.

Description

Technical Field

[0001] This invention relates to the field of industrial internet big data processing technology, specifically to an industrial internet big data storage and analysis device. Background Technology

[0002] With the rapid development of industrial internet technology and the comprehensive advancement of industrial digital transformation, the equipment system in industrial production sites is constantly being improved. Various types of equipment continuously generate massive amounts of multi-source heterogeneous data during operation. This data includes various types such as equipment operating status, process parameters, and on-site environmental information, exhibiting characteristics of large volume, significant temporal characteristics, and diverse structural forms. Core businesses in industrial production, such as equipment health monitoring, fault early warning, process optimization, and production control, all rely on this data for efficient processing and in-depth analysis. This places extremely high demands on the accuracy of data collection, the real-time nature of analysis, and the reliability of storage. Currently, the field of industrial big data processing urgently needs to build a complete technical system adapted to industrial scenarios to achieve the fusion processing of multi-source data, the extraction of high-value information, and the rational allocation of system resources. This will support the digital upgrade of the entire industrial production process and solve the core technical challenges in industrial data processing and application.

[0003] Traditional industrial big data storage and analysis technologies have many shortcomings in practical industrial applications, failing to meet the development needs of the modern industrial internet. The data acquisition stage struggles to achieve unified and standardized processing of heterogeneous data, with significant deficiencies in time synchronization, anomaly control, and metadata management, resulting in compromised quality and standardization of raw data. The data preprocessing stage lacks efficient high-dimensional data modeling methods, failing to effectively remove redundant data and significantly increasing the system's computational and storage load. The data analysis stage does not dynamically allocate computing resources based on business needs, making it difficult to guarantee the response speed and service quality of core analysis tasks. The data storage stage employs a crude, hierarchical management model, failing to schedule storage resources based on data value and business needs, leading to an imbalance between storage performance and cost. Furthermore, traditional technologies lack adaptive closed-loop optimization mechanisms, preventing system parameters from dynamically adjusting according to actual operating conditions, resulting in poor overall adaptability, economy, and stability, severely hindering the implementation and promotion of industrial big data applications. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide an industrial internet big data storage and analysis device. This device establishes an integrated technical system encompassing data acquisition, preprocessing, real-time analysis, hierarchical storage, and closed-loop optimization. It can standardize the processing of multi-source heterogeneous industrial data, reduce data volume through tensor modeling and dynamic sparsification, prioritize computing resources based on service quality to ensure efficient execution of core analysis tasks, employ a dual-drive strategy to achieve refined allocation and cost optimization of storage resources, and dynamically adjust system parameters based on a value deviation closed-loop mechanism. This invention effectively solves problems such as low data quality, poor resource utilization, insufficient response efficiency, and weak system adaptability in traditional industrial big data processing, comprehensively improving the overall efficiency of industrial data processing and storage analysis. It provides stable support for applications such as industrial intelligent monitoring, fault early warning, and production optimization, and promotes the digital and intelligent development of the industrial internet.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an industrial internet big data storage and analysis device, the device comprising: Multi-source data acquisition unit: used to acquire heterogeneous data streams and auxiliary data output by sensors of industrial field equipment, perform global time synchronization, data format conversion, outlier filtering and metadata appending operations, and output standardized time-series data blocks; Tensor modeling preprocessing unit: receives the standardized time series data block, constructs a four-dimensional tensor, binds the service quality threshold and priority label to the four-dimensional tensor, calculates the redundancy of each element in the tensor using the service quality weighted dynamic tensor sparsification process, and outputs a sparse tensor with service quality label and average service quality weight parameter after removing redundant data. Tensor Real-Time Analysis Unit: Receives the sparse tensors with service quality labels, allocates system computing resources to perform industrial analysis tasks according to service quality priority labels, and outputs tensor analysis results and service quality achievement rate data; Tiered storage scheduling unit: Receives the original sparse tensor and the sparse tensor with quality of service (QoS) label, calculates the storage tier decision value for each data block using a dual-drive tiered storage decision formula of tensor and QoS, allocates storage locations and backup strategies according to the decision values, and outputs a global tensor storage location index; Service and Closed-Loop Optimization Unit: Receives the tensor analysis results and the global tensor storage location index, provides a standardized data service interface, collects the device's own resource usage data, historical service quality achievement rate data, and tensor access frequency data, calculates the parameter adjustment amount using a value deviation-driven closed-loop parameter self-optimization formula, outputs parameter adjustment instructions, and feeds them back to the tensor modeling preprocessing unit and the hierarchical storage scheduling unit.

[0006] Furthermore, in the multi-source data acquisition unit, the heterogeneous data stream includes vibration, displacement, and rotational speed data output by rotating equipment in the industrial field; current, voltage, power, and frequency data output by power equipment; temperature, pressure, flow rate, liquid level, and concentration data output by process equipment; and auxiliary data including equipment operating status data, production work order information, process parameter settings, industrial field environmental temperature and humidity data, dust concentration data, equipment installation location data, and sensor calibration status data.

[0007] Furthermore, in the multi-source data acquisition unit, global time synchronization is achieved through GPS or BeiDou satellite timing systems; data format conversion transforms all heterogeneous data into a unified internal data structure containing data values, timestamps, device identifiers, and sensor identifiers; outlier filtering removes erroneous data where the data value is greater than 110% of the upper limit of the sensor's physical range or less than 90% of the lower limit of the physical range, null data generated during transmission, and duplicate data where more than 10 consecutive sampling points are identical; linear interpolation is used to complete no more than 3 consecutive missing data points, and more than 3 consecutive missing data points are marked as missing data segments and their duration is recorded; metadata appending adds 10 pieces of information to each data entry, including a unique device identifier, a unique sensor identifier, device model, device installation location, sensor model, sensor accuracy level, sensor calibration date, data acquisition batch number, data source channel, and acquisition timestamp.

[0008] Furthermore, in the tensor modeling preprocessing unit, the four-dimensional tensor includes four dimensions: equipment dimension, parameter dimension, time dimension, and confidence dimension. The equipment dimension is divided into four levels: factory, workshop, production line, and single equipment. Each level corresponds to a unique dimension index, covering all production-related equipment in the industrial site. The parameter dimension is divided into six categories: vibration, electrical, temperature, pressure, flow, and liquid level. Each category corresponds to specific sensor acquisition parameters, and each parameter corresponds to a unique dimension index. The time dimension adopts a variable-length slicing method, supporting four slicing granularities: 10 seconds, 1 minute, 10 minutes, and 1 hour. The slicing granularity can be dynamically adjusted according to the data acquisition frequency. The confidence dimension is divided into three levels: 0.99, 0.95, and 0.90, based on the sensor accuracy level. Each tensor element carries the original data value and the error range label calculated based on the sensor accuracy level. The four dimensions are interconnected to form a complete four-dimensional tensor structure.

[0009] Furthermore, in the tensor modeling preprocessing unit, the service quality-weighted dynamic tensor sparsification process includes: The redundancy matrix R is calculated using the following formula: Where 1 represents a matrix of all 1s with the same dimension as the original four-dimensional tensor. This is a service quality priority weight matrix with the same dimension as the original four-dimensional tensor. Each element corresponds to the service quality priority for different application scenarios, with values ​​ranging from [0.1, 1.0]. This is a confidence weight matrix with the same dimension as the original four-dimensional tensor. Each element corresponds to the confidence level of the tensor element, and the values ​​correspond to the levels of the confidence dimension. The Hadamard product is used to calculate the redundancy value in the redundancy matrix R if the element value is greater than a preset redundancy threshold. The elements in the original four-dimensional tensor are removed, i.e., set to zero, to obtain a sparse tensor. The preset redundancy threshold The value range is [0.6, 0.9].

[0010] Furthermore, in the tensor real-time analysis unit, the service quality priority label is divided into five levels: highest level, upper-middle level, middle level, lower-middle level, and lowest level. The system computing resources include central processing unit resources, memory resources, and input / output resources. When multiple industrial analysis tasks with different priorities are executed concurrently, the system prioritizes and allocates computing resources to each task according to its service quality priority label. The highest-level task can consume up to 60% of CPU resources, 70% of memory resources, and 80% of I / O resources; Medium and high-level tasks can consume up to 30% of CPU resources, 20% of memory resources, and 15% of input / output resources; Intermediate tasks can consume up to 8% of CPU resources, 8% of memory resources, and 4% of I / O resources. Low- to medium-level tasks can consume up to 1% of CPU resources, 1% of memory resources, and 0.8% of I / O resources. The lowest-level task can consume up to 1% of CPU resources, 1% of memory resources, and 0.2% of I / O resources. The industrial analysis tasks include anomaly pattern matching, trend prediction, equipment health assessment, and fault precursor identification. Industrial analysis tasks corresponding to different priority tags are executed sequentially according to the allocated system computing resource limit. Industrial analysis tasks corresponding to high priority tags occupy the allocated resources first, and the remaining resources released after the task is completed are used by industrial analysis tasks corresponding to low priority tags.

[0011] Furthermore, in the hierarchical storage scheduling unit, the mathematical expression of the tensor and quality-of-service dual-driven hierarchical storage decision formula is: in, For the optimal storage tier decision, s=1 corresponds to the NVMe-SSD high-speed tier, s=2 corresponds to the SATA-SSD performance tier, s=3 corresponds to the HDD capacity tier, and s=4 corresponds to the tape archive tier. Let be the comprehensive cost coefficient per unit capacity of the s-th layer of storage, and w represents the average service quality weighting parameter; The real-time value function of tensor data is calculated by the following formula: in, The industrial data value decay coefficient is given by t, where t is the number of days since the data was generated, and f is the data value in the past. Number of visits per day This represents the maximum number of times all tensors in the system can be accessed.

[0012] Furthermore, in the tiered storage scheduling unit, the storage location and backup strategy are determined according to the optimal storage tier decision result. Values ​​are assigned one-to-one: when When, allocate the first-level storage location and use a three-copy backup method; when When, allocate a second-level storage location and use a dual-copy backup method; when When, allocate a third-level storage location and use a single-copy backup method; when At that time, allocate a fourth-level storage location and use erasure coding backup.

[0013] Furthermore, the mathematical expression for the self-optimization formula of the closed-loop parameters driven by the value deviation in the service and closed-loop optimization unit is as follows: in, The parameter adjustment vector specifically includes the service quality priority weight matrix in the tensor modeling preprocessing unit. Element values, confidence weight matrix The level threshold and the data value decay coefficient in the hierarchical storage scheduling unit. ; The learning rate has a value range of (0, 1]. This represents the actual service quality achievement rate of a certain type of application in the previous period. The target service quality achievement rate for this type of application; This represents the actual average value of this type of tensor in the previous period; This represents the expected average value of this type of tensor.

[0014] Compared with existing technologies, this industrial internet big data storage and analysis device has the following advantages: I. This invention standardizes heterogeneous data streams from industrial sites through a multi-source data acquisition unit, achieving global time synchronization, format unification, and anomaly data regularization, ensuring the consistency and validity of input data. It constructs a four-dimensional tensor structure based on a tensor modeling preprocessing unit, and combines service quality-weighted dynamic sparsity processing to eliminate redundant data, reducing data processing volume while retaining core effective information and improving the efficiency of subsequent analysis and storage. The tensor real-time analysis unit allocates system computing resources according to service quality priority, allowing high-value analysis tasks to receive priority computing power support, ensuring the execution efficiency and response speed of core tasks such as industrial anomaly identification and health assessment. It adapts to the dual requirements of real-time and accurate data processing in industrial scenarios, solving the problems of data disorder, uneven computing power distribution, and low analysis efficiency in traditional industrial big data processing, and comprehensively improving the overall efficiency of real-time industrial data analysis.

[0015] Second, this invention employs a layered storage scheduling unit with a decision-making logic driven by both tensors and quality of service (QoS). This unit matches appropriate storage tiers and backup strategies to data with different values ​​and QoS requirements, balancing performance demands and cost investment in data storage. This achieves refined scheduling and efficient utilization of storage resources. The service and closed-loop optimization unit dynamically adjusts core parameters based on device operation data, QoS achievement rate, and tensor access frequency through a value-biased self-optimization logic. This forms a complete closed loop of data processing, analysis, storage, and optimization, continuously improving the stability and adaptability of device operation. The entire solution connects the entire process of industrial big data from acquisition, modeling, analysis to storage and optimization, solving the pain points of traditional storage and analysis systems such as resource waste, poor adaptability, and inability to self-optimize. This makes industrial internet big data storage and analysis more efficient, economical, and better suited to the actual application needs of industrial sites.

[0016] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1 A flowchart of an industrial internet big data storage and analysis device; Figure 2 Flowchart of the preprocessing unit for tensor modeling; Figure 3 This is a flowchart of the internal process of the tensor real-time analysis unit. Detailed Implementation

[0019] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0020] Reference Figure 1 One embodiment of the present invention proposes an industrial internet big data storage and analysis device. The device is composed of a multi-source data acquisition unit, a tensor modeling preprocessing unit, a tensor real-time analysis unit, a hierarchical storage scheduling unit, and a service and closed-loop optimization unit connected in sequence. Each unit independently executes its corresponding function and the data can be communicated bidirectionally and the parameters can be adjusted in linkage. It can complete the entire process of heterogeneous data acquisition, standardization processing, tensor construction, dynamic sparsification, real-time analysis, intelligent hierarchical storage, and system parameter self-optimization in industrial sites. It is suitable for the big data storage and analysis needs of various industrial scenarios such as discrete manufacturing, process chemical industry, and energy and power.

[0021] In this embodiment, the operation process of each unit of the device is as follows: Multi-source data acquisition unit: The multi-source data acquisition unit is used to acquire heterogeneous data streams and auxiliary data output by sensors of industrial field equipment, and sequentially performs global time synchronization, data format conversion, outlier filtering and metadata appending operations, and finally outputs standardized time-series data blocks.

[0022] Specific execution steps: Data type acquisition: The acquired heterogeneous data streams include: vibration, displacement, and speed data output from rotating equipment in the industrial field; current, voltage, power, and frequency data output from power equipment; and temperature, pressure, flow rate, liquid level, and concentration data output from process equipment. Auxiliary data acquired includes: equipment operating status data, production work order information, process parameter setpoints, industrial environmental temperature and humidity data, dust concentration data, equipment installation location data, and sensor calibration status data.

[0023] Global time synchronization: GPS or BeiDou satellite timing system is used to achieve global time synchronization. Satellite timing signal is used as a unified time reference. A timestamp accurate to the millisecond level is added to all collected data to eliminate timing deviations between different sensors, different devices and different acquisition channels, and ensure that all collected data are completely aligned on the time axis.

[0024] Data format conversion: All types of heterogeneous data are uniformly converted into a fixed internal data format. The converted data contains four fixed fields: data value, timestamp, device identifier, and sensor identifier. All collected data are reconstructed according to this structure to eliminate data format differences and provide a unified data input basis for subsequent tensor construction.

[0025] Outlier filtering: Performs multi-level outlier data processing according to preset rules. Remove erroneous data whose values ​​are greater than 110% of the upper limit of the sensor's physical range or less than 90% of the lower limit of the physical range; Remove null values ​​caused by network packet loss or electromagnetic interference during data transmission; Remove duplicate data where 10 or more consecutive sampling points have the same value; For no more than three consecutive missing data points, linear interpolation is used to complete the numerical values. For more than three consecutive missing data points, mark them as missing data segments and record the start time, end time and total duration of the missing data.

[0026] Metadata appended: For each piece of valid data that has been filtered, 10 metadata items are appended, including the unique device identifier, unique sensor identifier, device model, device installation location, sensor model, sensor accuracy level, sensor calibration date, data acquisition batch number, data source channel, and acquisition timestamp. The metadata is stored in a bound manner with the data value, enabling traceability and correlation of the data throughout its entire lifecycle.

[0027] Exemplary illustration: In the industrial production workshop, rotating equipment, power equipment, and process equipment operate synchronously. Multi-source data acquisition units simultaneously collect heterogeneous data streams from various equipment, including vibration, speed, current, temperature, pressure, and flow rate, as well as auxiliary data such as equipment operating status, production work orders, ambient temperature and humidity, and installation location. Global time synchronization is achieved through the BeiDou satellite timing system, assigning millisecond-level precise timestamps to all collected data and eliminating timing discrepancies between different acquisition channels. All collected heterogeneous data is uniformly converted into a standard internal format containing data values, timestamps, equipment identifiers, and sensor identifiers. The vibration sensor for the rotating equipment has a physical range of 0~25mm / s, and collected over-range data of 28.6mm / s. Data exceeding 110% of the upper limit was directly identified as erroneous and discarded. Data from three consecutive sampling points of this sensor was missing; corresponding values ​​were calculated and completed using linear interpolation. For the process equipment pressure sensor, eight consecutive sampling points had no data transmission; this was marked as a missing data segment, and the start and end times and the 4-second duration of the missing data were recorded. Simultaneously, null values ​​generated during transmission and duplicate data identical to 12 consecutive sampling points were discarded. Finally, for all filtered and completed valid data, 10 metadata items, including the device's unique identifier, sensor's unique identifier, installation location, accuracy class, calibration date, and batch number, were added to form a complete and traceable standardized time-series data block, which was then stably output to the tensor modeling preprocessing unit. Tensor modeling preprocessing unit: The tensor modeling preprocessing unit receives standardized time-series data blocks output by the multi-source data acquisition unit, constructs a four-dimensional tensor, binds service quality thresholds and priority labels to the four-dimensional tensor, calculates the redundancy of each element in the tensor using a service quality-weighted dynamic tensor sparsification process, removes redundant data, and outputs a sparse tensor with service quality labels and average service quality weight parameters.

[0028] Specific execution steps, such as Figure 2 As shown: Four-dimensional tensor construction: Based on standardized time-series data blocks, a four-dimensional tensor is constructed, including device dimension, parameter dimension, time dimension, and confidence dimension. The rules for dividing each dimension are as follows: Equipment dimension: Divided into four levels: factory, workshop, production line, and single equipment. Each level is assigned a unique dimension index, which fully covers all production-related equipment in the industrial site. Parameter dimensions: Divided into six major categories: vibration, electrical, temperature, pressure, flow, and liquid level. Each major category corresponds to specific sensor-collected parameters, and each parameter is assigned a unique dimension index. Time dimension: A variable length slicing method is adopted, supporting four slice granularities: 10 seconds, 1 minute, 10 minutes, and 1 hour. The slice granularity is dynamically adjusted according to the data acquisition frequency. High-frequency acquisition data uses small-granularity slices, while low-frequency acquisition data uses large-granularity slices. Confidence level: Divided into three levels: 0.99, 0.95, and 0.90, based on the sensor accuracy level. Each tensor element carries the original data value and the error range label calculated based on the sensor accuracy level.

[0029] Service Quality Tag Binding: Service quality thresholds and priority tags are bound to the constructed four-dimensional tensor. The service quality thresholds are set according to the response speed and accuracy requirements of industrial application scenarios. Priority tags are used to distinguish the importance of data analysis and storage. The tags are bound one by one to the tensor elements and are transmitted synchronously with the tensor.

[0030] Quality-of-Service (QoS) Weighted Dynamic Tensor Sparsity: The redundancy matrix R is calculated using a pre-defined formula, which is: Where 1 represents a matrix of all 1s with the same dimension as the original four-dimensional tensor. This is a service quality priority weight matrix with the same dimension as the original four-dimensional tensor. Each element corresponds to the service quality priority for different application scenarios, with values ​​ranging from [0.1, 1.0]. This is a confidence weight matrix with the same dimension as the original four-dimensional tensor. Each element corresponds to the confidence level of the tensor element, and the values ​​correspond to the levels of the confidence dimension. The Hadamard product is used to calculate the redundancy value in the redundancy matrix R if the element value is greater than a preset redundancy threshold. The elements in the original four-dimensional tensor are removed, i.e., set to zero, to obtain a sparse tensor. The preset redundancy threshold The value range is [0.6, 0.9].

[0031] Average service quality weight parameter calculation: The average service quality priority weight of all valid elements in the sparse tensor is calculated to generate the average service quality weight parameter, which is output synchronously to the tensor real-time analysis unit and the hierarchical storage scheduling unit along with the sparse tensor with service quality label.

[0032] Exemplary illustration: Following the standardized time-series data block output from the previous step, the tensor modeling preprocessing unit first divides the equipment dimension into four levels: factory, workshop, production line, and single equipment. A unique dimension index is assigned to each piece of equipment. Parameter dimensions are then divided into six categories: vibration, electrical, temperature, etc. Based on the frequency of on-site data acquisition, the time dimension granularity is set to 1 minute. Combined with the sensor accuracy level, the confidence level is divided into three levels: 0.99, 0.95, and 0.90, completing the full construction of the four-dimensional tensor. Based on the key monitoring requirements of the equipment, corresponding service quality thresholds and priority labels (highest, medium, and high) are bound to the tensor. Subsequently, the redundancy of tensor elements is calculated according to the formula, where the service quality priority weights of the tensor elements representing the key operating states of the equipment are... The weight matrix value is 0.85, and the confidence weight matrix value is 0.95. After Hadamard product operation and redundancy calculation, the redundancy is 0.1925, which is less than the preset redundancy threshold of 0.7. Therefore, it is considered valid data and retained. However, the tensor elements representing non-critical auxiliary parameters have a service quality priority weight matrix value of 0.2 and a confidence weight matrix value of 0.90. The calculated redundancy is 0.82, which is greater than the preset redundancy threshold. Therefore, it is considered redundant data and is removed by setting it to zero. After completing the sparsification of the entire tensor, the service quality priority weights of all valid elements are counted, and the average value is calculated to be 0.79. The average service quality weight parameter is generated and synchronously and stably output to the next two units along with the sparse tensor with service quality labels.

[0033] Tensor Real-Time Analysis Unit: The tensor real-time analysis unit receives sparse tensors with service quality labels output by the tensor modeling preprocessing unit, allocates system computing resources to perform industrial analysis tasks according to the service quality priority labels, and outputs tensor analysis results and service quality achievement rate data.

[0034] Specific execution steps, such as Figure 3 As shown: Service Quality Priority and Resource Allocation: Service quality priority is divided into five levels: highest, upper-middle, middle, lower-middle, and lowest. System computing resources include CPU resources, memory resources, and input / output resources. The resource allocation ratios for each level are as follows: Highest level: Allocate 60% of CPU resources, 70% of memory resources, and 80% of I / O resources; Medium to high level: Allocate 30% of CPU resources, 20% of memory resources, and 15% of I / O resources; Intermediate: Allocate 8% of CPU resources, 8% of memory resources, and 4% of I / O resources; Low to medium level: Allocate 1% of CPU resources, 1% of memory resources, and 0.8% of I / O resources; Lowest level: Allocate 1% of CPU resources, 1% of memory resources, and 0.2% of I / O resources.

[0035] Industrial analysis task execution: Industrial analysis tasks include four categories: abnormal pattern matching, trend prediction, equipment health assessment, and fault precursor identification. Tasks of different priorities are executed sequentially according to the allocated resources. High-priority tasks take priority in using the allocated resources, and the remaining resources released after the task is completed are automatically supplied to low-priority tasks.

[0036] Service Quality Achievement Rate Statistics: Real-time monitoring of task execution response time, data accuracy, and analysis completeness; comparison of actual execution indicators with preset service quality thresholds; calculation of service quality achievement rate; generation of service quality achievement rate data; and synchronous output of the data, along with tensor analysis results, to the hierarchical storage scheduling unit and the service and closed-loop optimization unit.

[0037] Exemplary illustration: Following the sparse tensor with service quality labels input from the previous step, the tensor real-time analysis unit first identifies the priority labels bound to the tensor, marking the equipment fault precursor identification task as the highest priority. It allocates 60% of the CPU resources, 70% of the memory resources, and 80% of the I / O resources to this task according to the allocation ratio, prioritizing the initiation of this critical analysis task. This allows for the rapid matching of abnormal equipment operation patterns and the identification of fault precursor features. After the highest priority task is completed, the remaining computing resources are automatically allocated to production data trend prediction tasks marked as medium to high priority, completing data trend analysis and equipment health assessment according to the corresponding resource allocation ratio. The entire process involves real-time monitoring of task execution response time, analysis accuracy, and data completeness. The actual execution results are compared one by one with preset service quality thresholds to statistically determine the overall completion status of tasks such as fault precursor identification and trend prediction. The calculated service quality achievement rate is 97.5%, and the complete tensor analysis results and service quality achievement rate data are synchronously output to the hierarchical storage scheduling unit and the service and closed-loop optimization unit.

[0038] Tiered storage scheduling unit: The hierarchical storage scheduling unit receives the original sparse tensor and the tensor analysis results output by the tensor real-time analysis unit. It uses a dual-driven hierarchical storage decision formula based on tensor and quality of service to calculate the storage level decision value of each data block. Based on the decision value, it allocates storage locations and backup strategies and outputs a global tensor storage location index.

[0039] Specific execution steps: Storage tier decision value calculation: The optimal storage tier is calculated using a dual-driven tensor and quality of service (QoS) tiered storage decision formula. The formula is: in, For the optimal storage tier decision, s=1 corresponds to the NVMe-SSD high-speed tier, s=2 corresponds to the SATA-SSD performance tier, s=3 corresponds to the HDD capacity tier, and s=4 corresponds to the tape archive tier. Let be the comprehensive cost coefficient per unit capacity of the s-th layer of storage, and w represents the average service quality weighting parameter; The real-time value function of tensor data is calculated by the following formula: in, The industrial data value decay coefficient is given by t, where t is the number of days since the data was generated, and f is the data value in the past. Number of visits per day This represents the maximum number of times all tensors in the system can be accessed.

[0040] Storage location and backup strategy allocation: Allocation of corresponding strategies based on the optimal storage tier decision results. NVMe-SSD high-speed layer, three-copy backup; SATA-SSD performance layer, dual-copy backup; HDD capacity tier, single copy backup; : Tape archiving layer, erasure code backup.

[0041] Global tensor storage location index generation: Record the storage level, physical address, and backup information of each sparse tensor, generate a global tensor storage location index, and archive the index file, tensor data, and analysis results synchronously. The index data is synchronously output to the service and closed-loop optimization unit.

[0042] Exemplary illustration: Following the input of the raw sparse tensor and tensor analysis results from the previous step, the tiered storage scheduling unit first retrieves the average service quality weight parameter. Combining this with the number of days since data generation, the number of accesses in the past 30 days, the maximum number of system accesses, and the data value decay coefficient, it calculates the real-time value of the tensor data using a real-time value function. This value is then substituted into the tiered storage decision formula to calculate storage decision values ​​for the NVMe-SSD high-speed layer, SATA-SSD performance layer, HDD capacity layer, and tape archive layer, respectively. After comparison, the optimal storage layer is determined to be the NVMe-SSD high-speed layer. A high-speed layer storage location is allocated to the sparse tensor according to the corresponding strategy, employing a three-copy backup method to ensure data access speed and security. Simultaneously, the storage layer, physical storage address, number of backups, and backup location of the tensor are recorded in detail. All tensor storage information is integrated and summarized to generate a complete global tensor storage location index. The index file, tensor data, and analysis results are archived synchronously, and the index data is synchronously output to the service and closed-loop optimization unit.

[0043] Service and Closed-Loop Optimization Unit: The service and closed-loop optimization unit receives tensor analysis results and global tensor storage location index, provides standardized data service interface, collects the device's own operating data, calculates parameter adjustment amount using value deviation-driven closed-loop parameter self-optimization formula, outputs parameter adjustment instructions and feeds them back to the tensor modeling preprocessing unit and hierarchical storage scheduling unit.

[0044] Specific execution steps: Data services and operational data acquisition: Provides standardized data service interfaces to support data calls, analysis queries, and result export for upper-layer applications; collects real-time device operation data, including CPU, memory, input / output resource utilization, historical data on service quality achievement rate, and tensor access frequency data.

[0045] Closed-loop parameter self-optimization calculation: The parameter adjustment is calculated using a value-bias-driven closed-loop parameter self-optimization formula. The formula is: in, The parameter adjustment vector specifically includes the service quality priority weight matrix in the tensor modeling preprocessing unit. Element values, confidence weight matrix The level threshold and the data value decay coefficient in the hierarchical storage scheduling unit. ; The learning rate has a value range of (0, 1]. This represents the actual service quality achievement rate of a certain type of application in the previous period. [Target service quality achievement rate for this type of application]; This represents the actual average value of this type of tensor in the previous period; This represents the expected average value of this type of tensor.

[0046] Parameter adjustment and feedback execution: Adjustment instructions are generated based on the parameter adjustment amount. When Δθ is positive, the corresponding parameter is increased; when Δθ is negative, the corresponding parameter is decreased. The adjustment instructions are sent synchronously to the tensor modeling preprocessing unit and the hierarchical storage scheduling unit. The new parameters take effect immediately, forming a closed-loop optimization mechanism.

[0047] Exemplary illustration: Continuing from the tensor analysis results and global tensor storage location index input from the previous stage, the service and closed-loop optimization unit provides stable data retrieval, query, and result export services to upper-level industrial applications through standardized data service interfaces. Simultaneously, it collects real-time data on the CPU, memory, and input / output resource usage throughout the device's operation, as well as historical service quality achievement rate data and tensor data access frequency data. With a learning rate of 0.7, it retrieves the actual service quality achievement rate and target service quality achievement rate of the highest-level analysis application from the previous cycle, along with the actual average value and expected average value of the tensors. These are then substituted into the closed-loop parameter self-optimization formula to calculate a parameter adjustment amount of 0.0021. Based on this adjustment amount, a parameter adjustment instruction is generated, increasing the value of the corresponding element in the service quality priority weight matrix, fine-tuning the confidence weight matrix level threshold, and decreasing the data value decay coefficient. This adjustment instruction is simultaneously sent to the tensor modeling preprocessing unit and the hierarchical storage scheduling unit. The adjusted parameters take effect immediately, continuously optimizing the data preprocessing, analysis, and storage process for the next round, forming a complete closed-loop self-optimization operation mechanism.

[0048] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0049] The following examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0050] This application employs an industrial internet big data storage and analysis device to store and analyze multi-source heterogeneous industrial data from process industry production sites. In some specific embodiments, industrial big data storage and analysis is conducted on process industry production sites. This site data includes vibration, displacement, and speed sensor data for rotating equipment; current, voltage, power, and frequency sensor data for power equipment; temperature, pressure, flow, level, and concentration sensor data for process equipment; and auxiliary data such as equipment operating status, production work orders, process parameters, site environment, equipment location, and sensor calibration status. The industrial internet big data storage and analysis device provided in this application completes the entire process of data processing, analysis, storage, and optimization. The specific execution unit's work content is as follows: Multi-source data acquisition unit: The multi-source data acquisition unit is responsible for the comprehensive acquisition and standardization of multi-source heterogeneous raw data from the process industry production site. It first performs integrity verification on the sensor data and auxiliary data of all equipment on site, and then sequentially completes the data acquisition channel connectivity test, data format validity identification, data transmission stability determination, data content non-empty determination, and data timing integrity determination. It comprehensively investigates problems such as acquisition interruption, format abnormality, transmission packet loss, content blankness, and timing disorder in the data. After testing, the data on site has no problems such as disconnected acquisition channels, unrecognizable format, transmission packet loss, content blankness, and timing deviation. The verification record and valid verification data are formed to define the effective range for subsequent data processing. Subsequently, global time synchronization was achieved through the BeiDou satellite timing system, adding millisecond-level precise timestamps to all collected data to eliminate timing deviations from different sensors, devices, and acquisition channels. All heterogeneous data was uniformly converted into a standard internal data structure containing data values, timestamps, device identifiers, and sensor identifiers, completing format normalization. Outlier filtering was performed according to preset rules, removing invalid data that was out of range, null values, or consecutively repeated. Linear interpolation was used to complete a small number of consecutively missing data, and missing segments were marked and their durations were recorded for a large number of consecutively missing data. Ten metadata items, including a unique device identifier, a unique sensor identifier, installation location, accuracy level, calibration date, and acquisition batch number, were added to each valid data point, generating a traceable and associative standardized time-series data block, providing a unified data foundation for subsequent tensor construction.

[0051] Tensor modeling preprocessing unit: The tensor modeling preprocessing unit receives standardized time-series data blocks output by the multi-source data acquisition unit and completes four-dimensional tensor construction, service quality label binding, and dynamic sparsification processing. A four-dimensional tensor is constructed based on standardized time-series data blocks, comprising equipment, parameter, time, and confidence dimensions. The equipment dimension is categorized into four levels: factory, workshop, production line, and individual equipment, each assigned a unique index. The parameter dimension is categorized into six types: vibration, electrical, temperature, pressure, flow, and level, each also assigned a unique index. The time dimension is sliced ​​to 1 minute based on the frequency of on-site data acquisition. Confidence levels are divided into three levels based on sensor accuracy, forming a complete four-dimensional tensor structure. Service quality thresholds and hierarchical priority labels are bound to the tensor according to key monitoring needs in the industrial field, clarifying the importance of different data for analysis and storage. A service quality-weighted dynamic tensor sparsification algorithm is used to calculate the redundancy of tensor elements. Redundant data is identified and eliminated through a redundancy matrix, and tensor elements with redundancy exceeding the threshold are set to zero, resulting in a sparse tensor with service quality labels. The average service quality priority weight of the effective elements in the sparse tensor is calculated to generate an average service quality weight parameter, which is output synchronously with the sparse tensor to subsequent analysis and storage units.

[0052] Tensor Real-Time Analysis Unit: The tensor real-time analysis unit receives sparse tensors with service quality labels output by the tensor modeling preprocessing unit, and completes hierarchical resource allocation and real-time industrial data analysis. First, it identifies the service quality priority labels bound to the sparse tensors. Based on a five-level standard (highest, upper-middle, middle, lower-middle, lowest), it allocates CPU, memory, and I / O system resources to different priority analysis tasks. Highest-priority tasks prioritize core computing resources to ensure the execution efficiency of critical analysis tasks. Based on the allocated computing resources, it sequentially performs four types of industrial analysis tasks: anomaly pattern matching, production trend prediction, equipment health assessment, and fault precursor identification. Remaining resources released after high-priority tasks are completed are automatically allocated to lower-priority tasks, improving system resource utilization. Throughout the process, it monitors the response time, data accuracy, and analysis completeness of the analysis tasks in real time, compares the actual execution indicators with preset service quality thresholds, statistically analyzes task completion, calculates the service quality achievement rate, and generates tensor analysis results and service quality achievement rate data, which are simultaneously output to the hierarchical storage scheduling unit and the service closed-loop optimization unit.

[0053] Tiered storage scheduling unit: The tiered storage scheduling unit receives the raw sparse tensors and the analysis results output by the tensor real-time analysis unit, and completes intelligent tiered storage decision-making and storage strategy configuration. It retrieves the average quality of service (QoS) weight parameter, combines it with data generation time, historical access frequency, maximum system access count, and data value decay coefficient, and calculates the real-time value of tensor data using a dual-driven tiered storage decision formula based on tensors and QoS. It then calculates storage decision values ​​for the NVMe-SSD high-speed layer, SATA-SSD performance layer, HDD capacity layer, and tape archive layer respectively, determining the optimal storage tier. Based on the optimal storage tier results, it allocates corresponding storage locations and backup strategies: the high-speed layer uses three-copy backup, the performance layer uses two-copy backup, the capacity layer uses one-copy backup, and the archive layer uses erasure coding backup, balancing data access performance, storage cost, and data security. It records the storage tier, physical storage address, and backup information for each sparse tensor in detail, integrates all tensor storage information to generate a global tensor storage location index, and archives the index file, tensor data, and analysis results synchronously. The index data is synchronously output to the service and closed-loop optimization unit.

[0054] Service and Closed-Loop Optimization Unit: The service and closed-loop optimization unit receives tensor analysis results and global tensor storage location indexes, and completes data services, operation monitoring, and parameter self-optimization. It provides standardized data service interfaces to external users, offering stable data access, analysis, query, and result export services for upper-level industrial applications, meeting the business needs of industrial production control, equipment maintenance, and other aspects. It collects real-time data on the CPU, memory, and input / output resource usage throughout the entire device operation process, as well as historical service quality achievement rate data and tensor data access frequency data, providing a comprehensive understanding of the system's operational status. Employing a value-biased closed-loop parameter self-optimization algorithm, it calculates parameter adjustments based on the previous cycle's actual service quality achievement rate, target service quality achievement rate, actual tensor average value, and expected average value. Based on the parameter adjustment, it generates optimization instructions to adjust the service quality priority weight matrix and confidence level threshold of the tensor modeling preprocessing unit, as well as the data value decay coefficient of the hierarchical storage scheduling unit. These optimization instructions are then synchronously pushed to the corresponding units, and the new parameters take effect immediately. Simultaneously, the parameter optimization results are fed back to the front-end processing stage, dynamically adjusting the relevant configurations for data acquisition and tensor construction. The system summarizes core operational indicators weekly and generates a closed-loop optimization iteration report monthly, forming a continuous self-optimization mechanism to continuously improve data storage and analysis efficiency, service quality stability, and system resource utilization.

[0055] This application provides an industrial internet big data storage and analysis device that effectively improves the standardization of multi-source heterogeneous data acquisition, the efficiency of tensor modeling preprocessing, the accuracy of real-time analysis, the intelligence of hierarchical storage, and the sustainability of closed-loop optimization in the industrial internet. It reduces data redundancy, resource waste, and storage cost losses. Furthermore, through hierarchical resource scheduling, dual-drive storage decision-making, and value deviation self-optimization functions, it provides equipment support and algorithm optimization foundation for big data storage and analysis in different industrial scenarios such as discrete manufacturing, energy and power, and equipment operation and maintenance. This enables the intelligent, efficient, and stable operation of industrial internet big data from multi-source acquisition, tensor processing, real-time analysis, hierarchical storage to closed-loop optimization.

[0056] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An industrial internet big data storage and analysis device, characterized in that, The device includes: Multi-source data acquisition unit: used to acquire heterogeneous data streams and auxiliary data output by sensors of industrial field equipment, perform global time synchronization, data format conversion, outlier filtering and metadata appending operations, and output standardized time-series data blocks; Tensor modeling preprocessing unit: receives the standardized time series data block, constructs a four-dimensional tensor, binds the service quality threshold and priority label to the four-dimensional tensor, calculates the redundancy of each element in the tensor using the service quality weighted dynamic tensor sparsification process, and outputs a sparse tensor with service quality label and average service quality weight parameter after removing redundant data. Tensor Real-Time Analysis Unit: Receives the sparse tensors with service quality labels, allocates system computing resources to perform industrial analysis tasks according to service quality priority labels, and outputs tensor analysis results and service quality achievement rate data; Tiered storage scheduling unit: Receives the original sparse tensor and the sparse tensor with quality of service (QoS) label, calculates the storage tier decision value for each data block using a dual-drive tiered storage decision formula of tensor and QoS, allocates storage locations and backup strategies according to the decision values, and outputs a global tensor storage location index; Service and Closed-Loop Optimization Unit: Receives the tensor analysis results and the global tensor storage location index, provides a standardized data service interface, collects the device's own resource usage data, historical service quality achievement rate data, and tensor access frequency data, calculates the parameter adjustment amount using a value deviation-driven closed-loop parameter self-optimization formula, outputs parameter adjustment instructions, and feeds them back to the tensor modeling preprocessing unit and the hierarchical storage scheduling unit.

2. The industrial internet big data storage and analysis device according to claim 1, characterized in that, In the multi-source data acquisition unit, the heterogeneous data stream includes vibration, displacement, and rotational speed data output by rotating equipment in the industrial field; current, voltage, power, and frequency data output by power equipment; temperature, pressure, flow rate, liquid level, and concentration data output by process equipment; and auxiliary data including equipment operating status data, production work order information, process parameter setpoints, industrial field environmental temperature and humidity data, dust concentration data, equipment installation location data, and sensor calibration status data.

3. The industrial internet big data storage and analysis device according to claim 1, characterized in that, In the multi-source data acquisition unit, global time synchronization is achieved through GPS or BeiDou satellite timing system; data format conversion converts all heterogeneous data into a unified internal data structure containing data value, timestamp, device identifier, and sensor identifier; outlier filtering removes erroneous data with data value greater than 110% of the upper limit of the sensor's physical range or less than 90% of the lower limit of the physical range, null data generated during transmission, and duplicate data with more than 10 consecutive sampling points being exactly the same; linear interpolation is used to complete no more than 3 consecutive missing data points, and more than 3 consecutive missing data points are marked as missing data segments and the missing duration is recorded.

4. The industrial internet big data storage and analysis device according to claim 1, characterized in that, In the tensor modeling preprocessing unit, the four-dimensional tensor includes four dimensions: equipment dimension, parameter dimension, time dimension, and confidence dimension. The equipment dimension is divided into four levels: factory, workshop, production line, and single equipment. Each level corresponds to a unique dimension index, covering all production-related equipment in the industrial site. The parameter dimension is divided into six categories: vibration, electrical, temperature, pressure, flow, and liquid level. Each category corresponds to specific sensor acquisition parameters, and each parameter corresponds to a unique dimension index. The time dimension adopts a variable-length slicing method, supporting four slicing granularities: 10 seconds, 1 minute, 10 minutes, and 1 hour. The slicing granularity can be dynamically adjusted according to the data acquisition frequency. The confidence dimension is divided into three levels: 0.99, 0.95, and 0.90, based on the sensor accuracy level. Each tensor element carries the original data value and the error range label calculated based on the sensor accuracy level. The four dimensions are interconnected to form a complete four-dimensional tensor structure.

5. The industrial internet big data storage and analysis device according to claim 1, characterized in that, The service quality weighted dynamic tensor sparsification process in the tensor modeling preprocessing unit includes: The redundancy matrix R is calculated using the following formula: Where 1 represents a matrix of all 1s with the same dimension as the original four-dimensional tensor. This is a service quality priority weight matrix with the same dimension as the original four-dimensional tensor. Each element corresponds to the service quality priority for different application scenarios, with values ​​ranging from [0.1, 1.0]. This is a confidence weight matrix with the same dimension as the original four-dimensional tensor. Each element corresponds to the confidence level of the tensor element, and the values ​​correspond to the levels of the confidence dimension. The Hadamard product is used to calculate the redundancy value in the redundancy matrix R if the element value is greater than a preset redundancy threshold. The elements in the original four-dimensional tensor are removed, i.e., set to zero, to obtain a sparse tensor. The preset redundancy threshold The value range is [0.6, 0.9].

6. The industrial internet big data storage and analysis device according to claim 1, characterized in that, In the tensor real-time analysis unit, the service quality priority label is divided into five levels: highest level, upper-middle level, middle level, lower-middle level, and lowest level. The system computing resources include central processing unit resources, memory resources, and input / output resources. When multiple industrial analysis tasks with different priorities are executed concurrently, the system prioritizes and allocates computing resources to each task according to its service quality priority label. The highest-level task can consume up to 60% of CPU resources, 70% of memory resources, and 80% of I / O resources; Medium and high-level tasks can consume up to 30% of CPU resources, 20% of memory resources, and 15% of input / output resources; Intermediate tasks can consume up to 8% of CPU resources, 8% of memory resources, and 4% of I / O resources. Low- to medium-level tasks can consume up to 1% of CPU resources, 1% of memory resources, and 0.8% of I / O resources. The lowest-level task can consume up to 1% of CPU resources, 1% of memory resources, and 0.2% of I / O resources. The industrial analysis tasks include anomaly pattern matching, trend prediction, equipment health assessment, and fault precursor identification. Industrial analysis tasks corresponding to different priority tags are executed sequentially according to the allocated system computing resource limit. Industrial analysis tasks corresponding to high priority tags occupy the allocated resources first, and the remaining resources released after the task is completed are used by industrial analysis tasks corresponding to low priority tags.

7. The industrial internet big data storage and analysis device according to claim 1, characterized in that, In the hierarchical storage scheduling unit, the mathematical expression of the tensor and quality-of-service dual-driven hierarchical storage decision formula is: in, For the optimal storage tier decision, s=1 corresponds to the NVMe-SSD high-speed tier, s=2 corresponds to the SATA-SSD performance tier, s=3 corresponds to the HDD capacity tier, and s=4 corresponds to the tape archive tier. Let be the comprehensive cost coefficient per unit capacity of the s-th layer of storage, and w represents the average service quality weighting parameter; The real-time value function of tensor data is calculated by the following formula: in, The industrial data value decay coefficient is given by t, where t is the number of days since the data was generated, and f is the data value in the past. Number of visits per day This represents the maximum number of times all tensors in the system can be accessed.

8. The industrial internet big data storage and analysis device according to claim 1, characterized in that, In the tiered storage scheduling unit, storage location and backup strategy are determined according to the optimal storage tier decision result. Values ​​are assigned one-to-one: when When, allocate the first-level storage location and use a three-copy backup method; when When, allocate a second-level storage location and use a dual-copy backup method; when When, allocate a third-level storage location and use a single-copy backup method; when At that time, allocate a fourth-level storage location and use erasure coding backup.

9. The industrial internet big data storage and analysis device according to claim 1, characterized in that, The mathematical expression for the self-optimization formula of the closed-loop parameters driven by the value deviation in the service and closed-loop optimization unit is as follows: in, The parameter adjustment vector specifically includes the service quality priority weight matrix in the tensor modeling preprocessing unit. Element values, confidence weight matrix The level threshold and the data value decay coefficient in the hierarchical storage scheduling unit. ; The learning rate has a value range of (0, 1]. This represents the actual service quality achievement rate of a certain type of application in the previous period. The target service quality achievement rate for this type of application; This represents the actual average value of this type of tensor in the previous period; This represents the expected average value of this type of tensor.

Images