Offshore installation process data organization method, system and storage medium based on data portrait

Abstract

The application discloses a marine device process data organization method, system and storage medium based on data portraits, and the system comprises a data caching unit, a data mirroring unit, a buffering and data pool unit, a data cleaning unit, a feature extraction unit, a feature index generation unit and a superimposed storage and compression unit; the data caching unit is used for copying an original data stream into two data streams, the data cleaning unit is used for synchronously performing data cleaning on the second data stream; the feature extraction unit is used for generating a diagnostic evaluation result from the cleaned data, and adding a corresponding label to the diagnostic evaluation result; the feature index generation unit is used for generating and outputting a structured metadata object containing a time window boundary, a statistical feature and a diagnostic evaluation label; and the superimposed storage and compression unit is used for forming a data storage record which is inseparable from an original data body and the structured metadata object. The application can guarantee data integrity and improve early fault diagnosis efficiency.

Description

Technical Field

[0001] This application relates to the field of marine technology, and in particular to a method, system, and storage medium for organizing process data of marine equipment based on data profiling. Background Technology

[0002] Chinese invention patent CN121008519A (publication date November 25, 2025, applicant Hangzhou Jiwang Communication Technology Co., Ltd.) discloses a smart factory equipment data acquisition system based on extended access. The system includes an adapter, a data acquisition module, a data processing module, and a communication transmission network. The adapter connects to factory equipment to adapt to different interfaces and protocols; the data acquisition module collects operational data and obtains production shift data; the data processing module performs selective transmission and time-series synchronization processing on the operational data; the communication transmission network includes transmission links and relay nodes. Each relay node has a built-in power supply unit, network access unit, frequency prediction unit, and path optimization unit, enabling power supply monitoring, transmission mode selection, load level determination, high-risk node offloading, and backup path generation. This method aims to solve the problems of coverage, transmission reliability, and efficiency in equipment data acquisition in smart factories.

[0003] Although the above technologies enable heterogeneous device access through adapters and improve transmission stability through dynamic load adjustment and path optimization of relay nodes, they still have the following shortcomings: 1. Data asset integrity is easily compromised: The system only focuses on the collection, transmission and storage of data. Subsequent evaluation information is physically or logically separated from the original data, which makes it easy to lose evaluation information during subsequent data migration and archiving, thus compromising the integrity of data assets.

[0004] 2. Limited Search Dimensions: This system only supports limited searches based on timestamps, device IDs, or network node status, and cannot perform searches based on data content features (e.g., "find all data segments with kurtosis greater than 5.5" or "find all waveforms rated as 'early wear'"). Users cannot directly locate abnormal data through statistical features or diagnostic labels.

[0005] 3. Lack of feature indexes required for intelligent diagnosis: The system does not convert the statistical characteristics or diagnostic evaluation results of the collected data into index items that can be efficiently retrieved. In intelligent diagnosis scenarios, it is still necessary to first query alarm records to obtain time points or data identifiers, and then read the raw data segment by segment, resulting in high I / O overhead and poor real-time performance.

[0006] 4. Simple storage structure: The compressed data packets are separated from the evaluation metadata, and the data assets lack semantic integrity, which is not conducive to subsequent data governance and intelligent analysis.

[0007] In summary, this method still belongs to the traditional "acquisition-transmission-storage" pipeline mode, which does not achieve deep integration of evaluation information and data volume, nor does it provide feature-based reverse retrieval capabilities. Summary of the Invention

[0008] In view of this, the technical problem to be solved by this application is to provide a method, system and storage medium for organizing process data of marine equipment based on data profiling, so as to solve the problems of data asset integrity being easily compromised, retrieval dimensions being limited, retrieval overhead being high, and being unfavorable to subsequent data governance and intelligent analysis in the prior art.

[0009] To address the aforementioned problems, one aspect of this application proposes a data profiling-based system for organizing marine plant process data, comprising a data caching unit, a data mirroring unit, a buffer and data pool unit, a data cleaning unit, a feature extraction unit, a feature index generation unit, and an overlay storage and compression unit; wherein: The data caching unit receives the high-speed input raw data stream from the data source and performs temporary high-speed buffering to smooth traffic peaks. The data mirroring unit copies the raw data stream into two data streams without loss, using batch IDs to uniquely identify each batch of raw data. The first data stream serves as a backup of the raw data and is sent to the buffer and data pool unit, while the second data stream serves as a processing and evaluation stream and is sent to the data cleaning unit. The buffer and data pool unit includes a first-level buffer and a second-level buffer, with the first-level buffer temporarily storing the first data stream. The data cleaning unit synchronously performs data cleaning on the second data stream to generate cleaned data. The feature extraction unit synchronously or asynchronously processes the cleaned data... The system calculates and statistically analyzes row data features, generates diagnostic evaluation results based on preset evaluation criteria for each data feature, adds corresponding tags to the diagnostic evaluation results, and sends them to the second-level buffer for temporary storage. The feature index generation unit converts the statistical and diagnostic evaluation results and tags into feature index items, and then outputs a structured metadata object containing time window boundaries, statistical features, and diagnostic evaluation tags. The overlay storage and compression unit obtains the original backup stream from the buffer and data pool unit, obtains the structured metadata object from the feature index generation unit, and bundles and compresses the original backup stream and the structured metadata object and writes them to the storage medium to form an inseparable data storage record of the original data body and the structured metadata object.

[0010] Another aspect of this application proposes a method for organizing process data of marine equipment based on data profiling, applied to the above system, including the following steps: S1. The data buffer unit receives the raw data stream from the high-speed input of the data source and performs temporary high-speed buffering to smooth traffic peaks; S2. The data mirroring unit uses batch ID to uniquely identify each batch of original data and copies it into two data streams without loss. The first data stream serves as a backup stream of the original data and is sent to the first-level buffer of the buffer and data pool unit for temporary storage. The second data stream serves as a processing and evaluation stream and is sent to the data cleaning unit. S3. The data cleaning unit synchronously performs data cleaning on the second data stream to generate cleaned data; S4. The feature extraction unit synchronously or asynchronously calculates and statistically analyzes the cleaned data features, generates diagnostic evaluation results based on the preset evaluation criteria for each data feature, adds corresponding labels to the diagnostic evaluation results, and sends them to the second-level buffer of the buffer and data pool unit for temporary storage. S5. The feature index generation unit converts the statistical and diagnostic evaluation results and labels into feature index items, and then outputs a structured metadata object containing time window boundaries, statistical features, and diagnostic evaluation labels. S6. The stacked storage and compression unit obtains the original backup stream from the buffer and data pool unit, obtains the structured metadata object from the feature index generation unit, and bundles and compresses the original backup stream and the structured metadata object and writes them into the storage medium to form an inseparable data storage record of the original data body and the structured metadata object.

[0011] Another aspect of this application proposes a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the data profiling-based marine equipment process data organization method of the above embodiments.

[0012] This application discloses a method, system, and storage medium for organizing process data of offshore equipment based on data profiling. It integrates data cleaning, status assessment, feature extraction, and enhanced storage functions, and encapsulates each unit in the entire processing flow into an independent microservice module, providing standardized process data writing interfaces and feature-index-based query interfaces. This invention is particularly suitable for scenarios such as intelligent diagnostics of offshore equipment, equipment health management, industrial IoT, and big data analysis, enabling rapid location and accurate analysis of abnormal process data. Attached Figure Description

[0013] Figure 1 The diagram shown is a schematic diagram of the structure of a marine equipment process data organization system based on data profiling provided in one embodiment of this application; Figure 2 The image shown is from the timeline. Figure 1 The diagram shows the data structure after system processing. Figure 3 The diagram shown is a schematic flowchart of a method for organizing process data of offshore equipment based on data profiling, provided in one embodiment of this application. Detailed Implementation

[0014] To make the technical problems, technical solutions, and beneficial effects to be solved by this application clearer and more understandable, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the scope of this application.

[0015] In this application, the term "data cleaning" is the core step in data preprocessing, which refers to the review, verification, transformation or correction of raw data to remove "dirty data" (such as erroneous, missing, duplicate or irrelevant parts), thereby improving data quality and usability.

[0016] In this application, the term "DC Offset Removal" refers to the process of removing the constant bias portion of a signal during data cleaning in order to restore and highlight the dynamic variation of the signal itself and avoid interference from the reference error in subsequent feature extraction.

[0017] In this application, the terms "Data Profiling & User Profiling" refer to the process of digitizing and labeling information about objects by collecting, organizing, and analyzing multidimensional data. This typically includes: Feature Extraction, which uses algorithms to reduce the dimensionality or complexity of data and extract the most representative key variables; and Clustering, which uses algorithms to group data objects with similar characteristics into one category.

[0018] In this application, the term "feature index item" refers to the transformation of complex feature data (such as text, images, geographic locations, or behaviors) into structured identifiers that can be quickly located, filtered, and calculated by the system.

[0019] Because existing technologies do not generate data quality scores, statistical features (such as mean, variance, skewness, kurtosis, information entropy, etc.) or diagnostic evaluation labels during data cleaning or processing, and do not store this evaluation information in an inseparable manner overlaid with the original data, the evaluation information remains physically or logically separated from the original data. This leads to the easy loss of evaluation information during subsequent data migration and archiving, damage to the integrity of data assets, and difficulty in achieving retrieval based on data content features. This application proposes the following solution.

[0020] One aspect of this application proposes a marine plant process data organization system based on data profiling, comprising a data caching unit, a data mirroring unit, a buffer and data pool unit, a data cleaning unit, a feature extraction unit, a feature index generation unit, and an overlay storage and compression unit; wherein: The data caching unit receives the high-speed input raw data stream from the data source and performs temporary high-speed buffering to smooth traffic peaks. The data mirroring unit copies the raw data stream into two data streams without loss, using batch IDs to uniquely identify each batch of raw data. The first data stream serves as a backup of the raw data and is sent to the buffer and data pool unit, while the second data stream serves as a processing and evaluation stream and is sent to the data cleaning unit. The buffer and data pool unit includes a first-level buffer and a second-level buffer, with the first-level buffer temporarily storing the first data stream. The data cleaning unit synchronously performs data cleaning on the second data stream to generate cleaned data. The feature extraction unit synchronously or asynchronously processes the cleaned data... The system calculates and statistically analyzes row data features, generates diagnostic evaluation results based on preset evaluation criteria for each data feature, adds corresponding tags to the diagnostic evaluation results, and sends them to the second-level buffer for temporary storage. The feature index generation unit converts the statistical and diagnostic evaluation results and tags into feature index items, and then outputs a structured metadata object containing time window boundaries, statistical features, and diagnostic evaluation tags. The overlay storage and compression unit obtains the original backup stream from the buffer and data pool unit, obtains the structured metadata object from the feature index generation unit, and bundles and compresses the original backup stream and the structured metadata object and writes them to the storage medium to form an inseparable data storage record of the original data body and the structured metadata object.

[0021] In one embodiment, the raw data stream received by the data caching unit includes real-time or batch data streams from one or more of IoT devices, central control systems, host systems, offshore power management systems, temperature sensors, pressure sensors, level sensors, vibration sensors, and MQTT message queues.

[0022] In one embodiment, the data cleaning unit includes one or more operations such as format standardization, missing value handling, noise reduction, resampling or sliding window segmentation, outlier replacement, removal of DC component, and labeling.

[0023] In one embodiment, the data features extracted by the feature extraction unit include one or more of the following: mean, median, variance, standard deviation, skewness, kurtosis, information entropy, sample entropy, RMS / RMS, and peak factor; the diagnostic evaluation results include one or more of the following: data quality score, anomaly confidence, pattern label, and scalable business label.

[0024] In one embodiment, the feature index generation unit converts statistical and diagnostic evaluation results, as well as tags, into feature index items by: retaining the encoding required for range queries or retaining the original values ​​for statistical features in continuous numerical form; retaining string or enumeration mappings for discrete tag data; and constructing an inverted index or lightweight hash table to accelerate query matching.

[0025] In one embodiment, the data caching unit employs a first-in-first-out (FIFO) and least recently used (LRU) strategy to prevent data backlog.

[0026] In one embodiment, the bundling operation performed by the overlay storage and compression unit includes associating and aligning the data block ID of the structured metadata object or the time window ID of the original data backup stream, and assembling the compressed binary data body with the feature index item and evaluation information as an indivisible record; the compression operation includes performing a lossless compression algorithm on the data body.

[0027] Another aspect of this application proposes a method for organizing process data of marine equipment based on data profiling, applied to the above system, including the following steps: S1. The data buffer unit receives the raw data stream from the high-speed input of the data source and performs temporary high-speed buffering to smooth traffic peaks; S2. The data mirroring unit uses batch ID to uniquely identify each batch of original data and copies it into two data streams without loss. The first data stream serves as a backup stream of the original data and is sent to the first-level buffer of the buffer and data pool unit for temporary storage. The second data stream serves as a processing and evaluation stream and is sent to the data cleaning unit. S3. The data cleaning unit synchronously performs data cleaning on the second data stream to generate cleaned data; S4. The feature extraction unit synchronously or asynchronously calculates and statistically analyzes the cleaned data features, generates diagnostic evaluation results based on the preset evaluation criteria for each data feature, adds corresponding labels to the diagnostic evaluation results, and sends them to the second-level buffer of the buffer and data pool unit for temporary storage. S5. The feature index generation unit converts the statistical and diagnostic evaluation results and labels into feature index items, and then outputs a structured metadata object containing time window boundaries, statistical features, and diagnostic evaluation labels. S6. The stacked storage and compression unit obtains the original backup stream from the buffer and data pool unit, obtains the structured metadata object from the feature index generation unit, and bundles and compresses the original backup stream and the structured metadata object and writes them into the storage medium to form an inseparable data storage record of the original data body and the structured metadata object.

[0028] In one embodiment, the method further includes step S7: executing a retrieval request with statistical feature thresholds or diagnostic evaluation labels as query conditions, directly matching and scanning the index metadata within the storage medium, and returning the corresponding original data body.

[0029] Another aspect of this application proposes a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the data profiling-based marine equipment process data organization method of the above embodiments.

[0030] Example 1 This embodiment provides a data profiling-based system for organizing process data of offshore equipment, such as... Figure 1 As shown, among them: The data caching unit 10 is used to receive the raw data stream from the high-speed input of the data source and to perform temporary high-speed buffering to smooth traffic peaks; Specifically, the data caching unit 10 serves as a data entry point, receiving real-time or batch data streams from one or more data sources. These data sources include IoT devices, central control systems, host systems, offshore power management systems, sensors for temperature / pressure / level / vibration, MQTT message queues, etc. Internally, the data caching unit 10 employs a high-performance, first-in-first-out (FIFO) memory queue for temporary data storage. It also supports a scalable flow control mechanism. When the cache utilization reaches a preset threshold, the system can automatically expand the data cache within resource limits to address insufficient cache space and enhance system robustness. When the cache utilization reaches the system's maximum threshold, data reception is paused and logs are recorded to prevent memory overflow, thereby improving system stability. The scalability feature supports batch aggregation by time window or data count, reducing the processing frequency of downstream units.

[0031] The data mirroring unit 20 is used to copy the original data stream into two data streams without loss, using batch IDs to uniquely identify each batch of original data. The first data stream 201 serves as a backup stream and is sent to the buffer and data pool unit 30, while the second data stream 202 serves as a processing and evaluation stream and is sent to the data cleaning unit 40. The buffer and data pool unit 30 includes a first-level buffer 301 and a second-level buffer 302. The first-level buffer 301 is used to temporarily store the first data stream 201, and the second-level buffer 302 is used to temporarily store the structured metadata objects generated by the feature extraction unit 50. The buffer and data pool unit 30 is the "data and metadata hub" within the module, employing a two-level buffer structure to achieve adaptive matching of upstream and downstream processing speeds. The first data stream 201 temporarily stored in the first-level buffer 301 is asynchronously pulled by the overlay storage and compression unit 70 and supports time-based or data volume-based eviction strategies. These strategies include First In First Out (FIFO) and Least Recently Used (LRU) to prevent infinite backlog. The first data stream 201, after being fed into the first-level buffer 301 of the buffer and data pool unit 30, serves as the source of raw data for subsequent storage. The second data stream 202 is sent from the data mirroring unit 20 to the data cleaning unit 40 and the feature extraction unit 50 for data cleaning, statistical feature extraction, and data evaluation. This data is then read by the feature index generation unit 60 and used by the overlay storage and compression unit 70 to obtain corresponding metadata. The data cleaning unit 40 and the feature extraction unit 50 operate synchronously or asynchronously. The first data stream 201 and the second data stream 202 use a "batch ID" mechanism to ensure strict alignment between the two data streams. The mirroring operation does not change the data content, does not introduce additional time delays, and guarantees strict semantic consistency between the two data streams. The data mirroring unit 20 can be configured for synchronous or asynchronous replication modes to adapt to different reliability requirements.

[0032] The buffer and data pool unit 30 also serves as the aggregation and decoupling center for raw data and metadata, simultaneously serving the data mirroring, data cleaning and feature extraction, and overlay storage and compression stages. Furthermore, it ensures that the processing speed of the data cleaning and feature extraction unit, the construction speed of the feature index generation unit, and the write speed of the final storage unit are independent of each other, preventing brief fluctuations in any stage from disrupting the entire pipeline.

[0033] The data cleaning unit 40 is used to synchronously perform data cleaning on the second data stream to generate cleaned data.

[0034] Specifically, the data cleaning unit 40 performs one or more of the following operations on the second data stream 202 from the mirror unit 20: format standardization (e.g., unifying timestamps from different sources to UTC millisecond precision), missing value handling (interpolation, deletion, or marking), denoising (low-pass filtering, median filtering, wavelet denoising), resampling or sliding window segmentation, and outlier replacement or marking.

[0035] The feature extraction unit 50 is used to synchronously or asynchronously calculate and statistically analyze the cleaned data features, generate diagnostic evaluation results based on the preset evaluation criteria of each data feature, add corresponding tags to the diagnostic evaluation results, and send them to the second-level buffer 302 for temporary storage. Specifically, the feature extraction unit 50 performs synchronous calculations during or immediately after the data cleaning process by the data cleaning unit 40: Statistical characteristics include: first-order statistics (mean, median), second-order statistics (variance, standard deviation), higher-order statistics (skewness, kurtosis), information characteristics (information entropy, sample entropy), and time-domain or frequency-domain characteristics (RMS, peak factor, etc.).

[0036] Diagnostic evaluation results include: data quality score (0-100 points), anomaly confidence (probability or level), pattern labels (normal, suspected drift, mutation, periodic anomaly, etc.), and scalable business labels (injected through the intelligent diagnostic model interface).

[0037] Key features: Data cleaning and profile evaluation are performed simultaneously, meaning that after the same data flows through this unit, both "clean data" and "feature / evaluation information" are obtained simultaneously, without the need for secondary reading.

[0038] Output: Cleaned data (optional, raw data can be passed through when cleaning is not enabled) and a structured metadata object describing the batch of data, including time window boundaries, statistical features, and diagnostic evaluation labels.

[0039] The feature index generation unit 60 is used to convert statistical and diagnostic evaluation results and labels into feature index items, and then output a structured metadata object containing time window boundaries, statistical features, and diagnostic evaluation labels; The feature index generation unit 60 receives metadata objects from the data cleaning and feature extraction unit, and transforms the continuous numerical statistical features and evaluation information into a data structure (feature index item) that can be efficiently retrieved. Specifically, this includes: For continuous features (such as kurtosis = 5.8), support the encoding required for range queries or preserve the original values; Preserve a string or enumeration mapping for discrete labels (such as "severely worn"); Optionally, an inverted index or a lightweight hash table can be built to speed up subsequent query matching.

[0040] Output: Feature index items, aligned with the original data (associated via time window ID or data block ID).

[0041] The stacked storage and compression unit 70 is used to obtain the original backup stream from the buffer and data pool unit, obtain the structured metadata object from the feature index generation unit, and bundle and compress the original backup stream and the structured metadata object and write them to the storage medium to form an inseparable data storage record of the original data body and the structured metadata object.

[0042] The overlay storage and compression unit 70 is connected to the data mirroring unit and the feature index generation unit, respectively, and performs the following operations: Input: Obtain raw data (or cleaned data, depending on configuration) from the first-level buffer of the buffer and data pool unit; obtain the corresponding feature index items and evaluation information from the feature index generation unit (or directly through the second-level buffer).

[0043] Compression operation: Perform lossless compression algorithms (such as Gzip, ZSTD, etc.) on the data body. The compressor can be selected adaptively according to the data type.

[0044] Overlay storage: The compressed binary data body, feature index items, and evaluation information are assembled as an indivisible record. Example record format (representation of the storage abstraction layer, non-restricted implementation): json { "data_blob": "<compressed_binary> #Data body: Compressed binary data "index_meta": #Additional index metadata { "time_range": [start_ts, end_ts], # Time range: [start timestamp, end timestamp] "stat_features": {"rms":1.2, "kurtosis":5.8, "entropy":3.4}, #Statistical features: {RMS (root mean square value), kurtosis, information entropy, "diagnosis": # Diagnostic Evaluation "quality_score": 92, #quality score "anomaly_label": "Severe Wear", #anomaly label "confidence": 0.96 #confidence level} } } Atomicity guarantee: The data body and the attached index metadata are written successfully or fail simultaneously, ensuring that they are indivisible. This record can be stored in local files, object storage (such as Huawei OceanStor storage), distributed storage (such as Huawei Cloud, Tencent Cloud), or time-series databases (such as TDengine, IoTDB).

[0045] Preferably, a columnar storage format that supports schema evolution (such as Apache Parquet) is used to store feature index items as file metadata, thereby achieving physical indivisibility.

[0046] After being processed by the superimposed storage and compression unit 70, the data will form as follows: Figure 2 The data overlay storage structure is shown.

[0047] Where: The left side is the original data block (after compression). The right side contains additional index metadata, including: timestamp range, statistical features, diagnostic evaluation labels, and other business labels are optional.

[0048] This system also provides two types of interfaces to the outside world: Write interface: Receives raw data and internally completes all the pipeline processing described above. Supports synchronous return (returns upon successful write) or asynchronous return (only confirms receipt).

[0049] Query Interface (Core Differentiating Feature): Provides a data query API interface, supporting query condition examples: Single query condition diagnosis.anomaly_label="Severe Wear and Tear" stat_features.kurtosis > 5.5 stat_features.rms between 1.0 and 2.0 Combined query conditions: diagnosis.quality_score < 60 AND stat_features.entropy > 4.0 Implementation mechanism: Scan the `index_meta` region in storage (preferably using index copies in the cache layer). Upon a successful match, return the corresponding `data_blob` directly, without relying on external timestamps or ID mapping tables. The returned result can be the decompressed form of the raw data or in any desired format.

[0050] Optionally, this system can also adopt a pure index mode, which only saves index metadata and allows external references to the original data; Optionally, this system can also adopt a delayed profiling mode, in which the feature extraction unit generates data profiles asynchronously. Optionally, this system can also adopt a distributed collaborative mode (consistent hashing cluster deployment, supporting asynchronous backfilling and version control). All of the above fall within the scope of this patent protection.

[0051] This system ultimately achieves the following significant and beneficial technical effects: 1. Achieve "evaluation following data," fundamentally resolving the disconnect between data and evaluation, and ensuring the integrity of data assets.

[0052] In traditional data processing workflows, data cleaning, quality assessment, and anomaly diagnosis often operate independently, resulting in evaluation information (such as data quality scores, anomaly labels, and statistical characteristics) being physically separated from the original data. This leads to serious problems: first, when the original data is archived, migrated, or deleted, the corresponding evaluation information is easily lost or misplaced, resulting in incomplete data assets; second, subsequent analysis requires simultaneous queries to multiple systems, increasing system complexity and the probability of errors.

[0053] This invention uses superimposed storage and compression units to bundle the original data volume and its corresponding feature index and evaluation information into the same storage medium as an indivisible atomic record.

[0054] The processing method provided by this invention is as follows: During data writing, the original data (or cleaned data) and metadata such as real-time generated statistical features and diagnostic labels are assembled into a complete record, which is submitted, succeeded, or failed simultaneously. This record ensures a strong correlation between the original data and the evaluation information at the physical storage level. Regardless of how the data is transferred, backed up, or migrated, the evaluation information always accompanies the data ontology.

[0055] This technological effect upgrades data assets from "raw numerical values" to "knowledge units with complete semantic labels," laying a solid foundation for subsequent data governance, auditing and tracing, and intelligent analysis, and completely avoiding information loss and consistency problems caused by the separation of data and evaluation.

[0056] 2. Supports reverse retrieval without time dimension: Breaks the dependence of traditional retrieval on timestamps, greatly improving query flexibility.

[0057] In existing technologies, data retrieval relies almost entirely on timestamps, device IDs, or predefined fixed tags. To find "all segments with variance exceeding a threshold" or "all waveforms evaluated as 'suspected sensor drift'," users must first obtain a list of matching time points from other systems (such as an alarm database), and then use these time points to retrieve the raw data one by one from the historical database. This approach is not only cumbersome, but also makes it impossible for users to locate relevant data using timestamps when abnormal events do not trigger alarms (such as early degradation).

[0058] This invention achieves feature-based reverse retrieval through the following mechanism: The feature index generation unit transforms the statistical features (mean, variance, skewness, information entropy, kurtosis, etc.) and diagnostic evaluation results (quality score, anomaly label, etc.) output by the data cleaning and feature extraction unit into index terms that can be retrieved efficiently.

[0059] The query interface directly accepts search requests based on statistical feature thresholds or diagnostic evaluation labels (e.g., stat_features.kurtosis > 5.5 or diagnosis.anomaly_label="severe wear").

[0060] During query execution, the system directly scans the index metadata area in the stored records (which can be accelerated with memory caching). Once a match is found, the corresponding original data body is returned immediately without going through any intermediate mapping tables or secondary queries.

[0061] The practical value of this technological effect: The semantics of queries have been greatly enriched: users can search based on the "content characteristics" of the data itself, rather than just "spatial and temporal location".

[0062] Early anomalies can be located: even if the device has not reached the severity level that would trigger an alarm, it can be detected as long as the characteristic indicators deviate from the normal range, thus enabling true predictive maintenance.

[0063] Reduced coupling between systems: No need to maintain the mapping relationship between independent alarm databases and historical databases, simplifying the system architecture.

[0064] 3. Significantly improves intelligent diagnostic efficiency: Reduces cross-system queries, lowers I / O overhead, and achieves millisecond-level anomaly localization.

[0065] In intelligent diagnostic scenarios (such as industrial equipment fault diagnosis and anomaly detection), traditional solutions require repeated cross-queries of multiple data sources. Take crude oil generator vibration diagnosis as an example: Traditional approach: The diagnostic system first queries the alarm database to obtain a list of timestamps for all "severe wear and tear" events (assuming 100 time points); then, for each time point, it initiates an independent data read request to the historical database, totaling 101 I / O operations. If the data is stored in remote object storage or a distributed file system, each I / O operation is accompanied by network latency and overhead.

[0066] The present invention proposes the following solution: The diagnostic system directly sends a request to the query interface of this module with the diagnostic tag "severe wear". Internally, the module directly scans the index metadata at the storage layer, matching all relevant data blocks at once and returning the original data in batches. The entire process requires only one I / O scan (or a very few batch reads) to complete all data acquisition.

[0067] The above comparison shows that the present invention can reduce the number of cross-system queries from N to 1. When N is large (e.g., thousands of anomalies), the I / O overhead reduction can reach more than 90%, and the advantage becomes more obvious as the amount of data increases.

[0068] Furthermore, since the feature index is physically bound to the original data, the retrieval process can complete conditional filtering without loading the complete data block, further reducing disk reads and network transmission. For example, for vibration data with a sampling rate of 1kHz and 1000 points in a single window, under the query condition of 'kurtosis > 5.5', the average response time of this invention can be shortened by 80% to 95% compared to traditional solutions, greatly improving the real-time performance and interactive experience of the intelligent diagnostic system.

[0069] 4. High module cohesion and low coupling: Facilitates independent deployment, flexible scaling, and ecosystem integration.

[0070] This invention integrates seven core functional units—data caching, data mirroring, buffering and data pooling, data cleaning, feature extraction, feature index generation, and overlay storage and compression—into a unified microservice module, exposing only standardized data write and query interfaces (such as RESTful APIs) to the outside world. This design has the following significant advantages: High cohesion: All logic related to "data-enhanced storage and feature retrieval" is encapsulated within the same module, reducing maintenance and upgrade complexity. Data exchange between functional units is accomplished through internal buffers and data pools, avoiding dependencies on external middleware.

[0071] Loose coupling: The module interacts with upper-layer applications (such as diagnostic systems and data analysis platforms) only through lightweight interfaces, without requiring the upper-layer applications to understand the internal implementation details. At the same time, the module abstracts the underlying storage media (local files, object storage, distributed key-value pairs, time-series databases), so changing the storage solution does not affect the interface behavior.

[0072] Independent deployment and elastic scaling: As a microservice, this module can be deployed in containers (such as Docker + Kubernetes), dynamically adjusting the number of instances based on data traffic. In edge computing environments such as offshore devices, it can also run as a single instance, with controllable resource consumption.

[0073] Easy to integrate: The module provides standard APIs that can be seamlessly embedded into existing data middleware, IoT platforms or industrial internet platforms as middleware for data preprocessing and indexing enhancement, significantly reducing the technical threshold for enterprises to build intelligent diagnostic systems.

[0074] This module not only achieves the superimposed storage and feature-based reverse retrieval of "raw data + evaluation information" in terms of technical functionality, but also possesses the deployment flexibility and maintainability of the cloud-native era at the software engineering level, and has broad application prospects.

[0075] It should be noted that the above-described device embodiments and method embodiments belong to the same concept. For details of their implementation process, please refer to the method embodiments. Furthermore, the technical features in the method embodiments are all applicable to the device embodiments, and will not be repeated here.

[0076] Example 2 This embodiment proposes a method for organizing process data of offshore equipment based on data profiling, which is applied to the above system, such as... Figure 3 As shown, it includes the following steps: S1. The data buffer unit receives the raw data stream from the high-speed input of the data source and performs temporary high-speed buffering to smooth traffic peaks; S2. The data mirroring unit uses batch ID to uniquely identify each batch of original data and copies it into two data streams without loss. The first data stream serves as a backup stream of the original data and is sent to the first-level buffer of the buffer and data pool unit for temporary storage. The second data stream serves as a processing and evaluation stream and is sent to the data cleaning unit. S3. The data cleaning unit synchronously performs data cleaning on the second data stream to generate cleaned data; S4. The feature extraction unit synchronously or asynchronously calculates and statistically analyzes the cleaned data features, generates diagnostic evaluation results based on the preset evaluation criteria for each data feature, adds corresponding labels to the diagnostic evaluation results, and sends them to the second-level buffer of the buffer and data pool unit for temporary storage. S5. The feature index generation unit converts the statistical and diagnostic evaluation results and labels into feature index items, and then outputs a structured metadata object containing time window boundaries, statistical features, and diagnostic evaluation labels. S6. The stacked storage and compression unit obtains the original backup stream from the buffer and data pool unit, obtains the structured metadata object from the feature index generation unit, and bundles and compresses the original backup stream and the structured metadata object and writes them into the storage medium to form an inseparable data storage record of the original data body and the structured metadata object.

[0077] In one embodiment, the method further includes step S7: when a retrieval request is executed with a statistical feature threshold or a diagnostic evaluation label as the query condition, the index metadata is directly matched and scanned within the storage medium to return the corresponding original data body.

[0078] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a hard disk, ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0079] Application Example 1: Vibration Diagnosis of Crude Oil Generator Bearings The vibration sensor reports generator bearing vibration data at a frequency of 1kHz. The data caching unit receives the data using a memory queue; the data mirroring unit copies the waveform into two channels; the data cleaning and feature extraction unit removes the DC component from 1000 points per window, calculates features such as RMS and kurtosis, and outputs labels "Normal," "Early Wear," and "Severe Wear" based on the kurtosis; the feature index generation unit converts the window start time, RMS=1.2, kurtosis=5.8, and the evaluation "Severe Wear" into index entries; the overlay storage and compression unit compresses the raw data using ZSTD and stores it in combination with the index metadata. When the diagnostic system queries the "Severe Wear" label, the query interface directly scans the index metadata and returns the corresponding waveform, without needing to check the timestamp first.

[0080] Implementation steps: Data access The vibration sensor reports generator bearing vibration data at a frequency of 1 kHz. The microservice module's data caching unit uses a memory queue to receive data, preventing data loss during sudden traffic spikes.

[0081] Data mirroring and offloading The data mirroring unit (103) copies the original vibration waveform data into two copies. One copy is temporarily stored in the buffer and data pool unit; the other copy is sent to the next level.

[0082] Data cleaning and evaluation The data cleaning and feature extraction unit removes the DC component from the data and performs sliding window segmentation (1000 points per window).

[0083] Image calculation: Calculate the effective value (root mean square value), peak factor, and kurtosis index (statistical characteristics) for each window.

[0084] Diagnostic evaluation: Built-in lightweight diagnostic model, outputs evaluation results based on kurtosis index: "normal", "early wear", "severe wear".

[0085] Feature Index Construction The feature index generation unit converts the above results into index entries: {window start time, effective value (root mean square value) = 1.2, kurtosis = 5.8, rating = severe wear}.

[0086] Overlay storage and compression The superimposed storage and compression unit performs lossless compression on the original 1000 vibration data points (e.g., using the ZSTD algorithm).

[0087] The compressed binary data is combined with the index entry generated in step 4 to form a new data object for storage. The storage record format is as follows: JSON structure: { "data_blob": "<compressed binary data>", "index_meta": { "time_range": [1767278222, 1767278402], "stat_features": {"rms":1.2, "kurtosis":5.8}, "diagnosis": "severe wear and tear" } } Search and use When the diagnostic system sends a request: "Please find all raw vibration data for 'severe wear' in the past 24 hours," the microservice module's query interface directly scans the `index_meta.diagnosis` field in storage and quickly returns the corresponding `data_blob`. Compared to existing technologies, there's no need to first query the alarm database for the time point and then check the historical database for waveforms; a single scan directly identifies the problem.

[0088] Application Example 2: Air compressor fault diagnosis: Every 10 seconds, the system collects data on the air compressor crankcase vibration acceleration, exhaust temperature, cooling water inlet and outlet temperature difference, motor current, and exhaust pressure. When the effective value (RMS) of the vibration signal calculated by the data cleaning and feature extraction unit exceeds the normal operating baseline by 20%, and the kurtosis index exceeds the threshold for three consecutive windows (e.g., kurtosis > 4.0), the module automatically overlays and stores the vibration waveform data for that period with the evaluation label "valve plate wear or increased bearing clearance". If an abnormal increase in exhaust temperature and aggravated current fluctuations are further detected, a label "compressor severe wear" is generated. The subsequent diagnostic system can directly locate the original vibration and temperature waveforms of all abnormal periods by querying "evaluation = compressor severe wear" or "stat_features.kurtosis > 4.0", without relying on timestamps for step-by-step troubleshooting, thus achieving early warning and rapid location of air compressor faults.

[0089] Application Example 3: For crude oil export pump fault diagnosis, the following routine data acquisition is performed: Second-by-second data acquisition is conducted on the vibration acceleration of the drive-end bearing, pump casing temperature, outlet pressure, motor current, and lubricating oil pressure. When the effective value (RMS) of the vibration signal calculated by the data cleaning and feature extraction units continuously rises above 30% of the normal operating baseline, and the kurtosis index exceeds a threshold (e.g., kurtosis > 4.0), the module automatically overlays and stores the vibration waveform data for that period with the evaluation label "early bearing wear or impeller imbalance." If further abnormal increases in pump casing temperature, intensified outlet pressure fluctuations, and periodic pulsations in motor current are detected, a label "severe pump failure (e.g., shaft bending, impeller damage)" is generated. The subsequent diagnostic system can directly locate the original vibration, pressure, and temperature waveforms for all abnormal periods by querying "evaluation = severe pump failure" or "stat_features.kurtosis > 5.0," eliminating the need to rely on timestamp-based segment-by-segment searches, thus significantly improving the response speed and location accuracy of the export pump fault.

[0090] While flowcharts presented for this technology may suggest a specific execution order, the execution order can differ from that illustrated. For example, the order of two or more boxes may be rearranged relative to the order shown. Furthermore, two or more boxes shown consecutively may be executed in parallel or partially in parallel. In some configurations, one or more boxes shown in the flowchart may be omitted or skipped. Any number of counters, status variables, warning signals, or messages may be added to the logic flow for purposes such as enhancing utility, accounting, performance, measurement, or fault handling, or for similar reasons.

[0091] Some of the functional units described in this specification have been labeled as modules to more specifically emphasize their implementation independence. For example, a module may be implemented as hardware circuitry comprising custom VLSI circuitry or gate arrays, such as logic chips, transistors, or other discrete semiconductor components. Modules may also be implemented in programmable hardware devices such as field-programmable gate arrays, programmable array logic, programmable logic devices, etc. Modules may also be implemented in software for execution by various types of processors. The executable code identifying a module may, for example, comprise one or more blocks of computer instructions that may be organized into objects, procedures, or functions. However, the executable code identifying a module does not need to be physically located together, but may include different instructions stored in different locations that include the module and, when logically combined, achieve the module's intended purpose.

[0092] In practice, an executable code module can be a single instruction or multiple instructions, and can even be distributed across several different code segments, different programs, and across several memory devices. Similarly, operational data can be identified and represented within a module, and can be embodied in any suitable form and organized within any suitable data structure. Operational data can be collected as a single dataset or distributed across different locations, including across different storage devices. Modules can be passive or active, including agents operable to perform desired functions.

[0093] The techniques described herein can also be stored on computer-readable storage media, including volatile and non-volatile, removable and non-removable media implemented using any techniques for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other computer storage medium that can be used to store the desired information and the described techniques.

[0094] The apparatus described herein may also include communication connections or networking devices and networking connections that allow the apparatus to communicate with other devices. A communication connection is an example of a communication medium. A communication medium is typically embodied in computer-readable instructions, data structures, program modules, and other data in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information delivery medium. Communication media include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency, infrared, and other wireless media. As used herein, the term "computer-readable medium" includes communication media.

[0095] In describing this technology, the following terms will be used: the singular forms “a,” “an,” and “described” include plural references unless the context clearly indicates otherwise. Thus, for example, a reference to an item includes a reference to one or more items. The term “some” refers to one, two, or more, and generally applies to a selection of some or all quantities. The term “multiple” refers to two or more items. The term “about” means that quantities, dimensions, sizes, formulations, parameters, shapes, and other characteristics need not be precise, but may be approximated and / or larger or smaller as needed, reflecting acceptable tolerances, conversion factors, rounding, measurement errors, and other factors known to those skilled in the art. The term “substantially” means that the listed characteristics, parameters, or values ​​do not need to be implemented precisely, but deviations or variations (including, for example, tolerances, measurement errors, measurement accuracy limitations, and other factors known to those skilled in the art) may occur in quantities that do not preclude the effects that the characteristic is intended to provide. Numerical data may be represented or presented herein in range format. It should be understood that such range formats are used for convenience and simplicity only, and therefore should be flexibly interpreted to include not only the values ​​explicitly listed as range limits, but also all individual values ​​or subranges covered within the range, just as each value and subrange is explicitly listed.

[0096] As an example, the numerical range "1 to 5" should be interpreted as including not only the explicitly listed values ​​of 1 to 5, but also individual values ​​and subranges within the indicated range. Thus, included within the numerical range are individual values ​​such as 2, 3, and 4, and subranges such as 1-3, 2-4, and 3-5, etc. The same principle applies to ranges that list only a single value (e.g., "greater than 1") regardless of the width or characteristics of the range being described. For convenience, multiple items may be presented in a common list. However, these lists should be understood as if each member of the list were individually identified as a unique and distinct member. Therefore, any individual member of such a list should not be construed as equivalent to any other member of the same list solely based on its presentation in the common group, without any indication to the contrary.

[0097] Furthermore, when the terms “and” and “or” are used with a list of items, they will be interpreted broadly, as any one or more of the listed items may be used alone or in combination with other listed items. The term “alternatively” means selecting one from two or more alternatives, and the selection is not limited to only the listed options, nor to selecting only a single option from the listed options, unless the context explicitly indicates otherwise. As used herein, the term “coupled” does not require components to be directly connected to each other. Rather, the term is intended to also include configurations with indirect connections, where one or more other components may be included between the coupled components. Additionally, as used herein, including in the claims, the “or” used in a list of items prefixed with “at least one of…” indicates a disjunctive list, such that a list of, for example, “at least one of A, B, or C” means A or B or C or AB or AC or BC or ABC (i.e., A and B and C). Furthermore, the term “exemplary” does not imply that the stated example is superior to or better than other examples. As used herein, a “group” of elements means “one or more” of these elements, unless it is explicitly required that the group has more than one element or is explicitly permitted to be a zero group.

[0098] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the invention. Those skilled in the art can implement the present invention in various modifications without departing from its scope and spirit; for example, a feature of one embodiment can be used in another embodiment to obtain yet another embodiment. Any modifications, equivalent substitutions, and improvements made within the scope of the present invention's technical concept should be within the scope of the present invention.

Claims

1. Structured metadata objects for window boundaries, statistical features, and diagnostic evaluation labels; The overlay storage and compression unit is used to obtain the original backup stream from the buffer and data pool unit, obtain the structured metadata object from the feature index generation unit, and bundle and compress the original backup stream and the structured metadata object and write them into the storage medium to form an inseparable data storage record of the original data body and the structured metadata object.

2. The system according to claim 1, characterized in that: The raw data stream includes real-time or batch data streams from one or more IoT devices, central control systems, host systems, offshore power management systems, temperature sensors, pressure sensors, level sensors, vibration sensors, and MQTT message queues.

3. The system according to claim 1, characterized in that: The data cleaning includes one or more operations such as data format standardization, missing value handling, noise reduction, resampling or sliding window segmentation, outlier replacement, removal of DC component, and labeling.

4. The system according to claim 1, characterized in that: The data features include one or more of the following: mean, median, variance, standard deviation, skewness, kurtosis, information entropy, sample entropy, RMS / RMS, and peak factor; the diagnostic evaluation results include one or more of the following: data quality score, anomaly confidence, pattern label, and scalable business label.

5. The system according to claim 1, characterized in that: The process of converting statistical and diagnostic evaluation results, as well as tags, into feature index items includes: preserving the encoding required for range queries or preserving the original values ​​for statistical features in continuous numerical form; preserving string or enumeration mappings for discrete tag data; and constructing inverted indexes or lightweight hash tables to accelerate query matching.

6. The system according to claim 1, characterized in that: The data caching unit employs a first-in-first-out (FIFO) and least recently used (LRU) strategy to prevent data backlog.

7. The system according to claim 1, characterized in that: The bundling operation includes associating and aligning the data block ID of the structured metadata object or the time window ID of the original data backup stream, and assembling the compressed binary data body with the feature index item and evaluation information as an indivisible record; the compression operation includes performing a lossless compression algorithm on the data body.

8. A method for organizing process data of marine equipment based on data profiling, applied to the system of claim 1, comprising the following steps: S1. The data buffer unit receives the raw data stream from the high-speed input of the data source and performs temporary high-speed buffering to smooth traffic peaks; S2. The data mirroring unit uses batch ID to uniquely identify each batch of original data and copies it into two data streams without loss. The first data stream serves as a backup stream of the original data and is sent to the first-level buffer of the buffer and data pool unit for temporary storage. The second data stream serves as a processing and evaluation stream and is sent to the data cleaning unit. S3. The data cleaning unit synchronously performs data cleaning on the second data stream to generate cleaned data; S4. The feature extraction unit synchronously or asynchronously calculates and statistically analyzes the cleaned data features, generates diagnostic evaluation results based on the preset evaluation criteria for each data feature, adds corresponding tags to the diagnostic evaluation results, and sends them to the second-level buffer of the buffer and data pool unit for temporary storage. S5. The feature index generation unit converts the statistical and diagnostic evaluation results and labels into feature index items, and then outputs a structured metadata object containing time window boundaries, statistical features, and diagnostic evaluation labels; S6. The stacked storage and compression unit obtains the original backup stream from the buffer and data pool unit, obtains the structured metadata object from the feature index generation unit, and bundles and compresses the original backup stream and the structured metadata object and writes them into the storage medium to form an inseparable data storage record of the original data body and the structured metadata object.

9. The method according to claim 8, characterized in that... It also includes step S7: executing a retrieval request with statistical feature thresholds or diagnostic evaluation labels as query conditions, directly matching and scanning the index metadata within the storage medium, and returning the corresponding original data body.

10. A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the method for organizing process data of marine equipment based on data profiling as described in claim 8 or 9.

Images