Data processing methods, systems and electronic equipment
By extracting unified metadata from the process industry and using technologies such as Apache Paimon and StarRocks for data storage and retrieval, unified management of multimodal data is accelerated, the problem of data silos is solved, and efficient cross-type data association and real-time querying are achieved.
Patent Information
- Application Number
- CN202611315768.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-27
- Publication Date
- 2026-11-17
AI Technical Summary
In the process industry, the decentralized storage of multimodal data leads to difficulties in data association, a lack of a unified data governance foundation, insufficient real-time performance and query performance, and high risks of metadata synchronization delay and consistency.
By acquiring multimodal data from the process industry production process, unified metadata is extracted and stored in a metadata table. Structured data is stored in a structured table, and unstructured data is stored in an unstructured table. Apache Paimon is used as the lakehouse storage platform, StarRocks is used as the query acceleration layer, and Flink is used to generate metadata, thereby realizing unified registration and association of data.
It enables unified storage and rapid retrieval of multimodal data, eliminates data silos, supports unified retrieval and deep correlation analysis of cross-type data, and improves the real-time visibility and data governance efficiency of the data lake.
Smart Images

Figure CN123115218A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically, to a data processing method, system, and electronic device. Background Technology
[0002] Process industries (such as petrochemicals and steel smelting) generate massive amounts of multimodal data during production, including time-series data, configuration data, alarm information, operation logs, and unstructured files such as flowcharts and PDF documents. These data are characterized by multimodality, massive volume, and heterogeneity.
[0003] Currently, the management of the aforementioned data generally adopts a distributed storage model, such as storing time-series data in dedicated real-time databases, unstructured files in object storage or file systems, and structured data in relational databases. This siloed architecture has the following main drawbacks:
[0004] First, multimodal data is fragmented, making correlation analysis difficult. Because different types of data (time series data, structured parsed data, and unstructured files) are stored separately and lack a unified data object identifier and correlation model, users need to manually retrieve and piece together data across systems when performing cross-type queries (such as querying drawings and alarm records associated with a certain tag number), which is inefficient and prone to errors.
[0005] Secondly, real-time performance and query performance are insufficient. Most related technical solutions rely on offline batch processing for data import into the lake, resulting in delays of up to hours from data generation to queryability. This fails to meet the demands of industrial real-time monitoring, online diagnostics, and anomaly alarms for second- or minute-level response times. Furthermore, in terms of metadata retrieval and directory display, related solutions typically rely on recursive list operations in object storage to simulate folder hierarchies. When the number of files reaches millions or tens of millions, a single list operation takes an extremely long time and is prone to triggering API rate-limiting policies from cloud service providers. While some solutions can introduce relevant search engines to improve query speed, the overall latency from data being written from the source to the lake storage to index updates remains in the minutes range. Moreover, when handling deep pagination (e.g., paging beyond page 100), query performance drops drastically, severely impacting the user experience.
[0006] Third, there are risks associated with metadata synchronization delays and data consistency. In storage-compute separation architectures, the metadata refresh mechanisms of related technical solutions often rely on periodic fetching or cache invalidation strategies, resulting in changes not taking effect in real time. For example, if a user uploads a new file or creates a new partition and queries it immediately, the data cannot be retrieved; they must wait for several minutes or even longer cache refresh cycles to see it. This metadata synchronization delay severely impacts the real-time visibility of the data lake, restricting applications that rely on real-time data for decision-making and reducing the availability of the data platform. Furthermore, related technical solutions often lack transaction coordination mechanisms between the lake storage and the retrieval engine. When using dual-write or asynchronous compensation methods, if one side fails to write, it can easily lead to permanent inconsistencies between the retrieval results and the data within the lake, and the lack of automatic detection and repair mechanisms increases the security risks of data management.
[0007] There is currently no effective solution to the above problems. Summary of the Invention
[0008] This application provides a data processing method, system, and electronic device to at least solve the technical problem that multimodal data in the process industry is usually scattered and stored in multiple isolated systems, resulting in difficulties in data correlation and a lack of a unified data governance foundation.
[0009] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: acquiring multimodal data from a process industry production process, wherein the multimodal data includes at least raw data, structured data, and unstructured data; extracting metadata from the multimodal data to obtain unified metadata; storing the unified metadata in a metadata table, storing structured data in a structured table, and storing unstructured data in an unstructured table, wherein the structured table and the unstructured table are associated with the metadata table through data object identifiers.
[0010] Optionally, metadata is extracted from the multimodal data, including: object storage of the multimodal data; receiving event messages generated in the object storage, wherein the event messages are used to indicate that there are file addition, modification or deletion operations in the multimodal data; parsing the object path information in the event messages through a stream processing engine, wherein the object path information includes at least the path structure field and data object level in the event messages; generating data object identifiers based on the object path information and the corresponding file attribute information; and generating unified metadata based on the event messages, object path information, file attribute information and data object identifiers.
[0011] Optionally, storing unified metadata in a metadata table, storing structured data in a structured table, and storing unstructured data in an unstructured table includes: writing unified metadata into a metadata table, wherein the field types of the metadata table include at least one of the following: data object basic fields, path structure fields, business affiliation fields, lifecycle fields, and parsing association fields; parsing multimodal data to obtain data parsing results and identifying the data structure types in the data parsing results; based on the data structure types, writing structured data from the multimodal data into a structured table and writing unstructured data from the multimodal data into an unstructured table, wherein the structured table is used to store parsed data with a fixed structure and includes parsing table fields associated with the metadata table, and the unstructured table is used to store unstructured parsed data written in binary form and includes content identifier fields associated with the metadata table.
[0012] Optionally, the structured data in the multimodal data is written into a structured table, including: determining the business type of the data objects in the structured data; writing the structured data into the corresponding parsing table in the structured table according to the business type, wherein the parsing table includes at least one of a configuration parsing table, an alarm parsing table, an operation log parsing table, and a historical trend parsing table; wherein, in the process of writing the structured data into the historical trend parsing table, the project name and dataset are used as partitioning fields for partitioning, and trend data records are generated based on historical sequence identifiers, wherein the project name and dataset support partitioning the structured data according to the project dimension and the dataset dimension, and the historical sequence identifier is used to identify the time-series data points in the structured data.
[0013] Optionally, the method further includes: organizing data objects in multimodal data using a hierarchical model, wherein the hierarchical model includes: a raw layer for storing raw data in multimodal data; a parsing layer for storing structured and unstructured data after parsing multimodal data; a unified metadata layer for storing unified metadata corresponding to multimodal data; a directory layer for storing path hierarchy relationships in multimodal data; and other object layers for storing data objects in multimodal data that cannot be identified according to preset rules.
[0014] Optionally, the metadata table may also include target extended fields to support platform access control, lifecycle management, and data object governance.
[0015] Optionally, the method further includes: synchronizing the changed data of the metadata table to the query acceleration database and establishing a metadata copy, wherein the query acceleration database is used to carry the metadata copy to support the retrieval query of the data object; and in response to the data retrieval request of the target object, performing a retrieval in the metadata copy, the structured table or the unstructured table to obtain the data retrieval result.
[0016] Optionally, the query acceleration table uses the data object identifier corresponding to the unified metadata or a combination of the data object identifier and the project name and dataset as a unique identifier, and performs update operations through the primary key update mechanism when there are file addition, modification or deletion operations in the multimodal data; in the query acceleration table, the process industry information in the multimodal data is divided into independent fields, wherein the independent fields include at least one of project name, dataset, object level, data parsing type, control station and server name; and in the query acceleration table, a target index is configured for the preset string field in the multimodal data, wherein the target index is used to guide similar queries on the preset string field.
[0017] Optionally, in response to a data retrieval request for a target object, a retrieval is performed in a metadata copy, a structured table, or an unstructured table, including: if the data retrieval request indicates that the target object is a queried data object, a retrieval is performed based on the data object identifier in the metadata copy to determine the target data object corresponding to the data retrieval request; if the data retrieval request indicates that the target object is queried data structure content, a retrieval is performed based on the parsing table field in the structured table to determine the target structured data content corresponding to the data retrieval request; or a retrieval is performed based on the content identifier field in the unstructured table to determine the target unstructured data content corresponding to the data retrieval request.
[0018] According to another aspect of the embodiments of this application, a data processing system is also provided, comprising: a data object layer for storing multimodal data from process industry production processes, wherein the multimodal data includes at least raw data, structured data, and unstructured data; a lake processing layer for extracting metadata from the multimodal data to obtain unified metadata; a lake storage layer for storing unified metadata in a metadata table, storing structured data in a structured table, and storing unstructured data in an unstructured table, wherein the structured table and the unstructured table are associated with the metadata table through data object identifiers; a retrieval and query layer for synchronizing changed data from the metadata table to a query acceleration database and establishing a metadata copy, wherein the query acceleration database is used to carry the metadata copy to support data object retrieval and query; and, in response to a data retrieval request for a target object, performing a retrieval in the metadata copy, the structured table, or the unstructured table to obtain data retrieval results.
[0019] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and used to execute the data processing method described above.
[0020] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described data processing method by running the computer program.
[0021] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described data processing method.
[0022] In this embodiment, multimodal data from the process industry production process is acquired, including at least raw data, structured data, and unstructured data. Metadata is extracted from the multimodal data to obtain unified metadata. The unified metadata is stored in a metadata table, the structured data is stored in a structured table, and the unstructured data is stored in an unstructured table. The structured table and the unstructured table are associated with the metadata table through data object identifiers. This achieves the goal of unified registration, hierarchical storage, and cross-modal data association of multimodal data in the process industry. This eliminates multimodal data silos, supports unified retrieval and deep association analysis of cross-type data, and solves the technical problem that multimodal data in the process industry is usually scattered and stored in multiple isolated systems, resulting in difficulties in data association and a lack of a unified data governance foundation. Attached Figure Description
[0023] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0024] Figure 1 This is a hardware structure diagram of a computer terminal for implementing a data processing method according to an embodiment of this application;
[0025] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application;
[0026] Figure 3 This is a schematic diagram of a metadata registration process according to an embodiment of this application;
[0027] Figure 4 This is a schematic diagram of a multimodal data ingestion and storage process according to an embodiment of this application;
[0028] Figure 5 This is a schematic diagram of a data retrieval process according to an embodiment of this application;
[0029] Figure 6 This is a structural diagram of a data processing system according to an embodiment of this application. Detailed Implementation
[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0032] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:
[0033] Process industries refer to industrial sectors characterized by continuous or semi-continuous production processes, including petrochemicals, coal chemicals, steel smelting, pharmaceuticals, energy, and power. These industries typically rely on industrial control systems such as DCS (Distributed Control System), PLC (Programmable Logic Controller), and SCADA (Supervisory Control and Data Acquisition) to continuously generate various types of data, including time-series data, alarm data, operation logs, configuration information, and process diagrams.
[0034] Multimodal data lake: refers to a data lake system capable of uniformly storing and managing various data formats such as structured, semi-structured, and unstructured data. In this application, the multimodal data lake not only stores raw files and parsing results, but also establishes relationships between different data objects through a unified metadata layer, enabling time-series data, configuration data, alarm information, operation logs, drawings, and documents to be retrieved and used uniformly.
[0035] Apache Paimon: A lakehouse table format for managing data tables on top of object storage or distributed file systems, supporting capabilities such as primary key tables, partitioning, snapshots, streaming and batch read / write, and access from Flink and Spark. In this application, Paimon serves as the lakehouse storage foundation, used to carry unified metadata, structured parsing results, and unstructured content data.
[0036] Paimon JDBC Catalog is a metadata management method used by Paimon to store Paimon database, table structure, and table metadata in a relational database. In this application, MySQL can be used as the metadata management library for Paimon JDBC Catalog to manage lake repository table definitions, while the actual data files are still stored in object storage or the file system.
[0037] StarRocks is a distributed analytical database suitable for high-concurrency, low-latency interactive query and analysis scenarios. In this application, StarRocks primarily serves as a unified metadata real-time incremental synchronization and query acceleration layer, supporting metadata filtering, directory queries, paginated display, aggregated statistics, and fast platform-side retrieval.
[0038] Flink: A distributed stream processing engine suitable for processing real-time or near real-time data streams. In this application, Flink can be used to consume object storage events in Kafka, parse object paths and file attributes, generate unified metadata records, and write them to the Paimon table.
[0039] BLOB (Binary Large Object): Used to store binary content such as images, documents, drawings, and attachments. In this application, unstructured data is written to an unstructured content table in BLOB format and associated with a unified metadata table through blob_record_id, thereby achieving unified management and retrieval of unstructured files.
[0040] This application provides a data processing method that can be run in... Figure 1 The computer terminal shown is described below.
[0041] The data processing method embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware structure block diagram of a computer terminal for implementing a data processing method is shown. Figure 1As shown, the computer terminal 10 may include a processor 102, a memory 104 for storing data, and a transmission module 106 for communication functions. The processor 102 may include one or more processors. For ease of explanation, Figure 1 The diagram illustrates multiple processors, including a first processor 102a, a second processor 102b, ..., an nth processor 102n. The processor 102 may be a processing device including, but not limited to, a microcontroller unit (MCU) or a field-programmable gate array (FPGA).
[0042] In addition, a computer terminal may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.
[0043] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).
[0044] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned data processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0045] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a radio frequency (RF) module, used for data interaction with external networks or devices via wired and / or wireless network connections.
[0046] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows the user to interact with the user interface of the computer terminal 10. In addition, the computer terminal 10 can also connect to a cursor control device and a keyboard via an input / output interface. The cursor control device (e.g., a mouse, trackball, or touchpad) is mainly used to transmit directional information and command selection to the terminal and control the movement of the cursor on the display; the keyboard is mainly used to receive letters, numbers, and other control commands input by the user, thereby enabling interaction between the user and the computer terminal 10.
[0047] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.
[0048] In the above operating environment, this application provides a data processing method embodiment. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than that shown here.
[0049] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application, such as... Figure 2 As shown in this embodiment, a unified data lake storage system is provided for the storage design of multimodal data in process industry scenarios, including the following steps:
[0050] Step S202: Obtain multimodal data from the process industry production process, wherein the multimodal data includes at least raw data, structured data and unstructured data;
[0051] Step S204: Extract metadata from multimodal data to obtain unified metadata.
[0052] Step S206 involves storing unified metadata in a metadata table, storing structured data in a structured table, and storing unstructured data in an unstructured table, wherein the structured table and the unstructured table are associated with the metadata table through data object identifiers.
[0053] In this embodiment of the application, a unified data lake retrieval system is also provided for multimodal data retrieval applications in process industry scenarios, including the following steps:
[0054] Step S208: Synchronize the changed data of the metadata table to the query acceleration database and establish a metadata copy. The query acceleration database is used to carry the metadata copy to support the retrieval query of the data object. In response to the data retrieval request of the target object, a retrieval is performed in the metadata copy, structured table or unstructured table to obtain the data retrieval result.
[0055] Through steps S202 to S208, the goal of uniformly registering, hierarchically storing, and establishing cross-modal data associations for multimodal data in the process industry is achieved. This eliminates multimodal data silos, supports unified retrieval and deep correlation analysis of cross-type data, and solves the technical problem that multimodal data in the process industry is usually scattered and stored in multiple isolated systems, resulting in difficulties in data association and a lack of a unified data governance foundation. The following is a detailed explanation.
[0056] The core idea of this application's embodiments is as follows: using object storage as the foundation for carrying multimodal data in the process industry, using the UMD (Unified Metadata) layer as the registration, management, and association center for all data objects, using Apache Paimon as the lake warehouse table format to carry unified metadata and parsing layer data, using Kafka and Flink to build a passive message-driven inbound link, using inbound tools to support the active import of batch data and historical time-series data, and using StarRocks (i.e., the aforementioned query acceleration database) as the unified metadata real-time incremental synchronization and fast query layer, thereby realizing unified storage, unified association, fast retrieval, and multi-engine querying of multimodal data in the process industry.
[0057] In step S202 above, the main responsibility is to obtain raw data, structured data and unstructured data generated by the process industry from the data object layer, as the source of data entering the lake, and to provide a multi-source heterogeneous data foundation for subsequent unified management and multi-modal storage.
[0058] Specifically, the data object layer is used to carry various data objects generated during the production and data management processes in the process industry, and is the input source for the entire data lake system. The multimodal data in this layer includes, but is not limited to, raw data, parsed data (such as structured data, semi-structured data, and unstructured data), directory objects with path hierarchy relationships, and other data objects that cannot be identified according to existing rules at the moment.
[0059] Raw data refers to data objects that have not undergone business parsing or structure transformation before entering the data lake, including original files, original directories, original historical data files, and original drawings and documents in object storage. When raw data enters the data lake, it is first registered in the unified metadata layer, which records information such as its path, file name, file size, hash, project, and dataset.
[0060] Parsed data refers to the data results obtained after the raw data has been parsed, cleaned, or converted in format by an external parsing process. It is linked to the original data through unified metadata. Parsed data can be structured data, such as configuration files, alarms, operation logs, historical trend data, etc.; it can also be unstructured content, such as flowcharts, audit documents, or attachments that need to be managed uniformly after parsing.
[0061] In step S204 above, the identification, transformation and registration of data objects before they enter the data lake are mainly completed through the lake entry processing layer, generating unified metadata with data object identifiers, so that multimodal data from different sources and in different formats can be written according to a unified data lake table structure.
[0062] In this embodiment, extracting metadata from multimodal data includes: object storage of the multimodal data; receiving event messages generated in the object storage, wherein the event messages are used to indicate that there are file addition, modification, or deletion operations in the multimodal data; parsing object path information in the event messages through a stream processing engine, wherein the object path information includes at least the path structure field and data object level in the event messages; generating data object identifiers based on the object path information and corresponding file attribute information; and generating unified metadata based on the event messages, object path information, file attribute information, and data object identifiers.
[0063] This process does not require all multimodal data to undergo content parsing before being imported into the data lake. Instead, it first incorporates the data objects themselves into a unified metadata layer for management. This unified metadata layer records the data object's source, path, type, business affiliation, lifecycle status, and relationships with subsequent parsing results. The specific implementation process is as follows: Figure 3 As shown, it includes the following steps:
[0064] S1: The data object enters the object storage.
[0065] Multimodal data in the process industry is stored in object storage. Data objects can come from production systems, file upload systems, parsing systems, platform import tools, or other external systems. Each file or directory tag in object storage is a data object to be registered.
[0066] S2: Object storage generates event messages.
[0067] When a file is added, modified, or deleted in object storage, object storage generates an event message. This event message includes at least the bucket name, object key, file size, object hash, last modified time, and storage type. The event message is then sent to the Kafka message queue.
[0068] S3: Flink consumes Kafka messages.
[0069] The Flink stream processing engine continuously consumes object events from Kafka and performs format validation and field extraction on the event content to obtain basic object path information such as bucket_name, s3_object_key, file_size_bytes, file_hash, last_modified_time, and storage_class.
[0070] S4: Path resolution and object classification.
[0071] Based on the path structure in the object key, fields such as project_name, dataset, parent_path, relative_path, path_level, file_name, and file_ext are parsed. Then, according to path rules and file types, the data objects are divided into object levels: raw, parsed, folder, and other. Here, raw represents the raw data layer object, mainly used to record the original files; parsed represents the parsing layer object, used to record parsed result files or data that can be written to the parsing layer table; folder represents the directory layer object, used to retain the path hierarchy in object storage; and other represents data objects that cannot be identified according to existing rules temporarily, used to avoid missing data assets.
[0072] S5: Generate a unique data object identifier.
[0073] The `object_id` is generated based on file fields such as object path information, project name, dataset, object level, file hash, and file size. The `object_id` is used to uniquely identify a data object within the unified metadata layer and serves as the basis for linking subsequent structured parsing data, unstructured BLOB data, and fast retrieval results.
[0074] S6: Generate unified metadata.
[0075] Based on the event message, object path information, file attribute information, and data object identifier mentioned above, a complete unified metadata record is constructed to complete the metadata registration of multimodal data.
[0076] In the above process, by combining object storage event notification, Kafka message queue and Flink stream processing engine, passive, automated and real-time metadata extraction of multimodal data entering the lake is realized, ensuring that data objects can be perceived and retrieved by the metadata layer within seconds after entering storage, which significantly improves the real-time visibility and data governance efficiency of the data lake.
[0077] In step S206 above, unified metadata is stored in the metadata table through the lake warehouse storage layer, and structured and unstructured data are stored in the corresponding parsing table or BLOB table respectively. The association is established through object ID, thereby realizing the hierarchical storage and logical association of multimodal data, breaking down data silos, and ensuring the logical traceability and consistency of original data, parsing results and content data.
[0078] Optionally, storing unified metadata in a metadata table, storing structured data in a structured table, and storing unstructured data in an unstructured table includes: writing unified metadata into a metadata table, wherein the field types of the metadata table include at least one of the following: data object basic fields, path structure fields, business affiliation fields, lifecycle fields, and parsing association fields; parsing multimodal data to obtain data parsing results and identifying the data structure types in the data parsing results; based on the data structure types, writing structured data from the multimodal data into a structured table and writing unstructured data from the multimodal data into an unstructured table, wherein the structured table is used to store parsed data with a fixed structure and includes parsing table fields associated with the metadata table, and the unstructured table is used to store unstructured parsed data written in binary form and includes content identifier fields associated with the metadata table.
[0079] Specifically, the lakehouse storage layer is the core of data persistence in this embodiment, used to store unified metadata, structured data, and unstructured data. This lakehouse storage layer uses Apache Paimon as the lakehouse table format, carrying the UMD unified metadata table (i.e., the aforementioned metadata table), the structured parsing layer table (i.e., the aforementioned structured table), and the unstructured BLOB content table (i.e., the aforementioned unstructured table). The unified metadata table records information such as the object identifier, path information, project name, dataset, object level, data type, target table name, BLOB association identifier, and lifecycle status of data objects, serving as the unified entry point for data retrieval and association. The structured parsing layer table stores data that can be expressed in a structured table, such as configurations, alarm information, operation logs, and historical trends. The unstructured BLOB table stores unstructured content such as flowcharts, audit documents, and document attachments, and is associated with the unified metadata table through blob_record_id. Through this layer design, the system can simultaneously manage structured and unstructured data while maintaining a traceable relationship between both types of data and the unified metadata.
[0080] In this embodiment of the application, the process of storing multimodal data in the lake is as follows: Figure 4 As shown, it includes the following steps:
[0081] S1: Write the unified metadata into the metadata table.
[0082] The system writes the generated unified metadata records into the unified metadata table in Apache Paimon. The unified metadata table includes at least the following types of fields:
[0083] The first category consists of basic fields of the data object, including object_id, bucket_name, s3_uri, s3_object_key, file_name, file_ext, file_size_bytes, file_hash, etc.
[0084] The second category consists of path structure fields, including parent_path, relative_path, path_level, dataset_root, source_dir_name, etc.
[0085] The third category is business attribution fields, including project_name, dataset, object_layer, parsed_data_type, parsed_sub_type, dcs_type, etc.
[0086] The fourth category is lifecycle fields, including ingest_time, last_modified_time, archive_status, etc.
[0087] The fifth category is parsing related fields, including target_table_name, blob_record_id, source_type, import_batch_id, etc.
[0088] Once unified metadata is written, data objects are included in the data lake management scope. Subsequent systems can use this unified metadata for file retrieval, directory display, object tracking, parsing result association, lifecycle status management, and integration with the platform's permission system. It should be noted that specific user entities, administrative permissions, role authorization, and other platform permission capabilities are implemented on the platform side. This application's embodiment primarily provides the storage fields and data organization foundation required to support permission filtering and data isolation.
[0089] S2: Receives the parsed result data and identifies the data type.
[0090] Receives parsing results generated by an external parsing module. The parsing result data includes, but is not limited to, configuration data, alarm information, operation logs, historical trend data, flowcharts, audit documents, and document attachments.
[0091] Subsequently, `parsed_data_type` and `parsed_sub_type` are identified based on the path, file type, parsing type field, and import parameters. For example, configuration data, alarm information, operation logs, and historical trend data are considered structured or semi-structured data; flowcharts, audit documents, and document attachments are considered unstructured data.
[0092] S3: Write structured data into a structured table.
[0093] Optionally, the structured data in the multimodal data is written into a structured table, including: determining the business type of the data objects in the structured data; writing the structured data into the corresponding parsing table in the structured table according to the business type, wherein the parsing table includes at least one of a configuration parsing table, an alarm parsing table, an operation log parsing table, and a historical trend parsing table; wherein, in the process of writing the structured data into the historical trend parsing table, the project name and dataset are used as partitioning fields for partitioning, and trend data records are generated based on historical sequence identifiers, wherein the project name and dataset support partitioning the structured data according to the project dimension and the dataset dimension, and the historical sequence identifier is used to identify the time-series data points in the structured data.
[0094] It should be noted that, in this embodiment of the application, different parsing layer organization methods are adopted to address the characteristics of large differences in structured data types and large amounts of historical trend data in the process industry. That is, different structured parsing data are written into the corresponding Paimon parsing table according to the data type.
[0095] Instead of forcibly writing data with different structures such as configuration, alarms, operation logs, and historical trends into the same wide table, the system establishes dedicated parsing layer tables according to data types. This ensures that the field model, primary key design, partitioning method, and query path of each data type match its business characteristics, thereby reducing empty fields, redundant fields, and invalid scans. For example, configuration data is written to the corresponding configuration parsing table according to different point types or configuration types; alarm information is written to the alarm parsing table according to alarm type; operation logs are written to the operation log parsing table; and historical trend data is written to the historical trend parsing table. Each parsing table retains related fields such as object_id, project_name, and dataset for association with the unified metadata table.
[0096] For historical trend data, a historical trend parsing table structure designed for large-scale time-series data is adopted. This historical trend table includes at least the following fields: history_series_id, object_id, project_name, dataset, import_batch_id, time, tag_name, and value. The history_series_id (historical sequence identifier) identifies time-series data points, while project_name and dataset are used for partitioning and organizing data to accommodate the storage and querying of hundreds of millions to billions of time-series data points within a single project or dataset. Specifically, Flink tasks read the parsed structured files, convert the data into a unified Paimon writing format, and write it to the historical trend table in parallel. During the writing process, partitioning is performed using fields such as project_name and dataset, import_batch_id identifies an inbound batch, and history_series_id identifies specific time-series data points. This allows hundreds of millions to billions of data points within the same project and dataset to be tracked by batch, pruned by partition, and queried by tag number and time.
[0097] This layered parsing design retains fields such as `object_id`, `project_name`, and `dataset` that link the parsing table and the metadata table, while also providing specialized modeling for specific industrial data types. Platforms or computing tasks can first locate data objects through the metadata table and then access the structured content in the corresponding parsing layer table, avoiding direct scanning of all raw files or mixed wide tables. For large volumes of historical trend data, the combination of Flink parallel writing and Paimon lake warehouse table storage improves the efficiency of large-scale data ingestion into the lake and ensures batch traceability and subsequent multi-engine query capabilities for the ingestion results.
[0098] S4: Write unstructured data into an unstructured table.
[0099] For unstructured data such as flowcharts, audit documents, and document attachments, BLOB storage is used. The system generates a blob_record_id for the unstructured content and writes the file content into an unstructured BLOB content table. This BLOB content table includes at least the following fields: blob_record_id, object_id, project_name, dataset, parsed_data_type, ingest_time, and content_blob.
[0100] To reduce invalid scans during unstructured content retrieval and reading, the system stores file type, file size, hash value, path, business category, and blob_record_id of unstructured data in a unified metadata table. During retrieval, the platform can prioritize querying the metadata table, accessing the BLOB content table only when specific content needs to be read, thereby reducing the impact of reading large fields on query performance.
[0101] In the above process, by constructing a unified metadata table containing five types of fields, and writing structured data into a dedicated parsing table and unstructured data into a BLOB table according to the data structure type, fine-grained hierarchical storage and strong association management of multimodal data are realized. This effectively solves the problems of unstructured data only having paths and not being able to associate content, as well as the problem of query performance degradation caused by mixed storage of multiple types of data in traditional solutions. It provides a unified and efficient data storage foundation for subsequent fast retrieval, deep association analysis and data governance.
[0102] In this embodiment, the request to import multimodal data into the data lake is initiated by the platform, task scheduling system, or data processing workflow. The import request includes information such as project name, dataset, data type, source file path, target table type, and import batch. Upon receiving the import request, the system accesses object storage or other storage locations based on the input path and reads the parsed result file. For historical trend data, columnar files such as Parquet can be read; for configuration data, alarm information, and operation logs, structured or semi-structured parsing results can be read; for unstructured data such as flowcharts, audit documents, and document attachments, the corresponding file content can be read.
[0103] After successfully reading the data, field standardization and type conversion are performed on the parsed results from different sources and in different formats. Standardization includes unifying field names, converting data types, unifying the format of time fields, completing project names and dataset fields, generating import batches, generating object identifiers, and identifying target tables. For structured or semi-structured data, the Paimon parsing table to be written is identified based on data type fields such as `parsed_data_type` and `parsed_sub_type`, and a `target_table_name` is generated. For unstructured data, a `blob_record_id` is generated to establish the relationship between the unified metadata table and the unstructured BLOB content table.
[0104] After standardization, data is written to the corresponding data lake tables according to data type. For example, configuration data is written to the configuration parsing table, alarm information to the alarm parsing table, operation logs to the operation log parsing table, and historical trend data to the historical trend table. Unstructured data such as flowcharts, audit documents, and document attachments are written to unstructured BLOB content tables. When writing various types of data, primary keys and partitioning fields are set according to the corresponding table structure to ensure the efficiency of subsequent queries, filtering, and correlation analysis.
[0105] In the aforementioned data entry process, the system supports writing various data types into the data lake, including historical trends, configurations, alarm information, operation logs, flowcharts, audit documents, and document attachments. Although different data types are written to different parsing layer tables or BLOB content tables, they are all registered and associated through the unified metadata table UMD, thus maintaining a consistent data object management approach, storage approach, and retrieval entry point.
[0106] In this embodiment, to avoid difficulties in expansion and querying caused by different types of data being mixed in a single table structure, the data lake adopts a hierarchical model to organize data objects. This hierarchical model includes: a raw layer for storing the raw data in the multimodal data; a parsing layer for storing the structured and unstructured data after parsing the multimodal data; a unified metadata layer for storing the unified metadata corresponding to the multimodal data; a directory layer for storing the path hierarchy relationships in the multimodal data; and an other object layer for storing data objects in the multimodal data that cannot be identified according to preset rules. A detailed analysis follows:
[0107] Raw Layer: This layer stores raw data objects before they enter the data lake. The system records basic attributes of these raw data objects through a unified metadata table, including object storage path, file name, file type, file size, file hash, write time, modification time, and lifecycle status. The raw layer does not directly store the actual parsed content; instead, it establishes associations with subsequent layers through data object identifiers, providing a basis for data tracking, auditing, and backtracking.
[0108] The parsing layer stores parsed data processed by the external parsing process, including structured and unstructured tables. Structured tables store configuration data, alarm information, operation logs, historical trend data, and other data with a fixed structure. The system writes data to corresponding dedicated parsing tables (such as configuration parsing tables, alarm parsing tables, historical trend tables, etc.) based on the data type. Each type of parsing table retains fields associated with the unified metadata table (such as Object ID, Project Name, Dataset) to enable multimodal data association and retrieval. Unstructured tables store unstructured content in binary form, such as flowcharts, audit documents, and document attachments. The system generates a unique BLOB record ID (such as blob_record_id), writes the file content to the unstructured BLOB content table, and establishes an association with the corresponding record in the unified metadata table through blob_record_id.
[0109] Unified Metadata Layer: Used to register and associate all metadata objects. The unified metadata table serves as the system's unified retrieval entry point, recording basic information, path information, business information, lifecycle information, and target table association information for each data object. Users or the platform can first locate data objects through the metadata table, and then query structured tables based on `target_table_name`, or query unstructured tables based on `blob_record_id`. The unified metadata layer enables raw data, parsed data, structured data, and unstructured data to be managed and retrieved under the same object model.
[0110] Directory layer: Used to maintain the hierarchical relationship of paths in object storage. The system stores the directory structure through fields such as parent_path, relative_path, and path_level, so that the platform does not need to repeatedly recursively scan the object storage path when building directory trees, displaying file levels, or querying data objects by path. By writing directory relationships into unified metadata, the system can improve the efficiency of directory query and file location.
[0111] Other object layer: Used to register data objects that cannot be identified according to preset rules. For data objects that are temporarily unidentifiable, not parsed, or do not belong to existing categories, the system still writes them into the unified metadata table and marks them as "other". This design can avoid the omission of data assets and provides a basis for subsequent supplementary parsing rules, reclassification, or reprocessing.
[0112] In the aforementioned process, by constructing a layered model comprising a raw layer, a parsing layer, a unified metadata layer, a directory layer, and other object layers, fine-grained classification, storage, and logical association of multimodal data can be achieved. This layered architecture not only avoids the mixed storage of data with different structures, reducing system complexity and maintenance costs, but also enables a global view and fast retrieval through the unified metadata layer, optimizes file navigation performance through the directory layer, and ensures the integrity of data assets through other object layers, thereby improving the scalability, query efficiency, and data governance capabilities of the data lake system.
[0113] In this embodiment, primary keys and partitioning can be designed according to different data types. A detailed analysis follows:
[0114] The metadata table uses `object_id` as the core identifier for data objects and organizes them using fields such as `project_name`, `object_layer`, and `dataset`. `project_name` and `dataset` allow for data location by project and dataset; `object_layer` distinguishes between the raw layer, parsed layer, directory layer, and other object layers; and `parsed_data_type` and `parsed_sub_type` differentiate specific business data types.
[0115] Structured tables can be designed based on their respective business primary keys. For example, configuration data can be organized using fields such as configuration item identifier, project name, dataset, control domain, and control station; alarm information can be organized using fields such as alarm event identifier, project name, dataset, and server; and operation logs can be organized using fields such as operation log identifier, project name, dataset, server, and log sequence number. The historical trend table, designed for large-scale time-series data, can use `project_name` and `dataset` as partitioning fields to reduce cross-project and cross-dataset scanning. Simultaneously, `history_series_id` is used to identify time-series records, while retaining fields such as `time`, `tag_name`, and `value` to support queries by time, tag number, project, and dataset dimensions.
[0116] Unstructured tables can use blob_record_id as the identifier for content records, while retaining related fields such as object_id, project_name, dataset, and parsed_data_type. When a user retrieves unstructured data, the system first queries the unified metadata table to locate the data object, and then reads the corresponding BLOB content based on the blob_record_id.
[0117] Through the above-described primary key and partition design, the embodiments of this application can simultaneously support data organization of multiple projects, multiple datasets, multiple data types, and multiple object levels, ensuring that structured data, unstructured data, raw data, and parsed data in the process industry can be uniformly registered, stored, and retrieved, and providing a unified data foundation for subsequent rapid retrieval, data tracking, lifecycle management, and platform-side access control.
[0118] In this embodiment of the application, the metadata table also includes target extended fields for supporting platform access control, lifecycle management, and data object governance.
[0119] Specifically, the unified metadata table not only includes the aforementioned basic fields, path fields, business fields, lifecycle fields, and parsing association fields, but also further includes several target extended fields. These are designed to provide the upper-layer platform with fine-grained access control, full lifecycle state management, and comprehensive data object governance capabilities. These target extended fields include, but are not limited to:
[0120] Access control and isolation fields include tenant_id, access_level, and data_classification. By recording the tenant, project, and security level of the data, the platform can implement row-level or multi-tenant data isolation based on these fields. This ensures that different tenants or users with different roles can only access data objects within their authorized scope, meeting the stringent data security compliance requirements in industrial scenarios.
[0121] Lifecycle management fields include retention_policy, expiration_time, archive_status, and delete_status. The system uses these fields to automate the lifecycle management of data objects. For example, it can automatically mark historical trend data that has not been accessed for a long time as archived to save storage costs, or automatically trigger a data cleanup process after expiration, achieving efficient data storage and compliant handling.
[0122] Data governance and lineage fields include data_owner (data owner), quality_score (data quality score), lineage_info (lineage information JSON), and audit_log_id (audit log ID). By recording the data owner, quality assessment results, and the lineage relationship between the data source and processing, the platform can support data asset inventory, quality monitoring, problem tracing, and compliance auditing, thereby improving the transparency and credibility of data management.
[0123] By extending permission, lifecycle, and governance-related fields in the unified metadata table, the storage management capabilities of the data lake are extended to the upper business logic layer. This enables the platform to directly implement fine-grained permission isolation, automated lifecycle flow, and intelligent data governance based on a unified metadata view. This avoids the high coupling and maintenance costs caused by hard-coding filtering conditions at the application layer or relying on complex external permission components, and significantly improves the security, maintainability, and data asset value of the industrial data platform.
[0124] In step S208 above, the main responsibility is to synchronize metadata changes to the query acceleration library in real time to create a replica, and prioritize retrieval through the replica. Specific content is obtained by accessing the underlying storage as needed, thereby leveraging the query acceleration library to achieve second-level fast retrieval and deep pagination display of metadata, solving the latency problem of large-scale data retrieval, while avoiding the performance bottleneck caused by directly scanning the underlying lakehouse storage. The overall process is as follows: Figure 5 As shown, it includes the following steps:
[0125] S1: Read the change data from the metadata table.
[0126] The system reads new, modified, or deleted records from the Paimon metadata table and generates incremental change data to be synchronized. To support real-time consumption of unified metadata changes downstream, the metadata table can be configured with the Paimon primary key table's changelog generation capability, enabling downstream tasks to read changes such as data object insertions, updates, and deletions in changelog form. For input data with complete change semantics, an input-type changelog generation method can be used; for scenarios where the input data itself is incomplete but a complete change log still needs to be generated, a lookup or full-compaction-type changelog generation method can be used. The Paimon changelog is read through Flink or other synchronization tasks, converting records such as new objects, field updates, archive status changes, and target table join changes in the unified metadata table into incremental change data that can be written to StarRocks (the aforementioned query acceleration database).
[0127] S2: Synchronize the changed data to StarRocks in real time.
[0128] In this embodiment, the query acceleration table uses the data object identifier corresponding to the unified metadata or the combination of the data object identifier and the project name and dataset as a unique identifier, and performs the update operation through the primary key update mechanism when there are file addition, modification or deletion operations in the multimodal data.
[0129] Specifically, in terms of table model design, StarRocks can use a primary key table or other table models suitable for update scenarios, using object_id or a combination of object_id and project_name, dataset, etc., as a unique identifier. When a synchronization task receives a new or updated record, StarRocks performs a UPSERT operation based on the primary key, writing the latest metadata status to the query acceleration table. When a synchronization task receives archived, deleted, or invalid records, the system can perform logical updates through status fields such as archive_status, or perform deletion processing according to business needs. Through the primary key update mechanism, StarRocks always retains the latest metadata view suitable for platform queries.
[0130] In the query acceleration table, process industry information in multimodal data can be divided into independent fields, including at least one of project name, dataset, object level, data parsing type, control station and server name; and in the query acceleration table, target indexes can be configured for preset string fields in multimodal data, wherein the target indexes are used to guide similar queries on preset string fields.
[0131] Specifically, in terms of field organization, the StarRocks table can be partitioned, sorted, and indexed based on the platform's frequently used query conditions. For example, query paths can be organized by fields such as `project_name` and `datase`, enabling the platform to quickly filter by project, dataset, object level, and parsed data type. For string fields such as `file_name` and `parent_path`, N-gram Bloom Filter indexes can be created based on the query scenario to accelerate LIKE queries, fuzzy filename matching, path fragment retrieval, and string similarity queries like `ngram_search`. Through this design, the system can reduce large-scale string field scanning and improve the efficiency of directory and file retrieval.
[0132] Furthermore, the rapid metadata retrieval in this embodiment is not a simple replication of data lake metadata, but rather semantic modeling tailored to the characteristics of process industry data. The system decomposes industrial business information such as project, dataset, object layer, parsing type, control station, server, and log number in the original path into independent fields. This allows the platform to prioritize narrowing the query scope using highly selective fields such as project_name, dataset, object_layer, parsed_data_type, control_station, and server_name, avoiding a large-scale scan of the complete path or filename.
[0133] Meanwhile, fields such as object_id, file_hash, file_size_bytes, last_modified_time, archive_status, target_table_name, and blob_record_id are used to establish unique identifiers, status indicators, and content associations for data objects. For changes such as additions, overwrites, archives, and deletions, StarRocks maintains the latest valid metadata view through a primary key update mechanism. The platform defaults to retrieving valid objects, thereby reducing the interference of duplicate, expired, and archived objects on query results and improving retrieval accuracy.
[0134] In terms of string retrieval, indexes such as N-gram Bloom Filters are set only for high-frequency retrieval fields such as file_name, parent_path, relative_path, and s3_object_key to accelerate fuzzy matching of filenames, path fragment retrieval, and similar string queries, thereby reducing invalid scans and improving the efficiency of directory queries, file retrieval, and content location.
[0135] S3: The platform prioritizes querying StarRocks.
[0136] Preferably, when the data retrieval request indicates that the target object is the queried data object, the retrieval is performed based on the data object identifier in the metadata copy to determine the target data object corresponding to the data retrieval request.
[0137] Specifically, when a target object (such as a user) performs data object retrieval, directory display, file query, data type filtering, or pagination query, the platform prioritizes querying the unified metadata copy in StarRocks. StarRocks is used for high-frequency metadata queries, directory construction, deep pagination display, and object location tasks, while Paimon continues to serve as the main storage for the unified data lake, retaining complete metadata and content data. The platform can quickly identify the target data object in StarRocks first, and then access the specific content table in Paimon based on the associated fields in the unified metadata.
[0138] S4: Access structured or unstructured content tables based on associated fields.
[0139] Optionally, when the data retrieval request indicates that the target object is the content of the queried data structure, the retrieval is performed based on the parsing table fields in the structured table to determine the target structured data content corresponding to the data retrieval request; or the retrieval is performed based on the content identifier fields in the unstructured table to determine the target unstructured data content corresponding to the data retrieval request.
[0140] Specifically, when a user needs to view structured content, they can query the structured parsing table in Paimon based on `target_table_name` and `object_id`; when a user needs to view unstructured file content, they can query the BLOB content table in Paimon based on `blob_record_id`. This approach separates metadata retrieval from content reading, improving retrieval speed and preventing large unstructured fields from impacting the performance of regular metadata queries.
[0141] S5: Supports multi-engine queries and operations.
[0142] It's worth noting that StarRocks is primarily used for high-frequency metadata queries and interactive retrieval, while Paimon is mainly used for unified storage of lakehouse data and multi-engine access. In addition to the platform's retrieval service, Flink SQL and Spark can also directly read Paimon tables for real-time processing, batch computation, quality verification, statistical analysis, and data processing.
[0143] Overall, the embodiments of this application innovatively construct a system based on Apache to address the pain points of scattered storage, difficulty in association, and retrieval delays of multimodal data in the process industry. The lakehouse architecture, with Paimon as the underlying storage platform, a unified metadata layer as the core index, and StarRocks as the query acceleration layer, is designed with a multimodal unified metadata model for the process industry in mind. By generating globally unique object_ids, it logically and deeply associates raw data, structured parsing results, unstructured BLOB content, and directory hierarchy, breaking down data silos. Simultaneously, it utilizes Flink to consume object storage events in real time, synchronizing metadata changes incrementally to StarRocks in real time. This achieves separation of metadata retrieval from underlying content storage, solving the problems of poor performance in large-scale directory tree scanning and high metadata synchronization latency in traditional solutions. Furthermore, for large-scale time-series data such as historical trends, it adopts a partitioned storage strategy based on project name and dataset, combined with a layered model (raw layer, parsing layer, unified metadata layer, directory layer, and other object layers) and extended permission / governance fields. This enables fine-grained organization of multimodal data, second-level fast retrieval, fine-grained permission isolation, and full lifecycle management, significantly improving the real-time performance, scalability, and data asset utilization of the process industry data lake.
[0144] According to embodiments of this application, a data processing system is provided. It should be noted that the data processing system of this application can be used to execute the data processing method provided in the embodiments of this application. The data processing system provided in the embodiments of this application will be described below.
[0145] Figure 6 This is a structural diagram of a data processing system provided according to an embodiment of this application. For example... Figure 6 As shown, the system includes:
[0146] The data object layer 60 is used to store multimodal data from the process industry production process, wherein the multimodal data includes at least raw data, structured data and unstructured data.
[0147] In this embodiment, the data object layer is used to carry various data objects generated during the production and data management processes of the process industry, and serves as the input source for the entire data lake system. This layer contains multimodal data, including but not limited to raw data, parsed data (such as structured data, semi-structured data, and unstructured data), directory objects with hierarchical path relationships, and other data objects that cannot be identified according to existing rules. Raw data refers to data objects that have not undergone business parsing or structure transformation before entering the data lake, including original files, original directories, original historical data files, and original drawings / documents in object storage. When raw data enters the data lake, it is first registered in the unified metadata layer, recording its path, filename, file size, hash, project, and dataset information. Parsed data refers to the data results obtained after external parsing processes parse, clean, or convert the raw data; it is associated with the raw data through unified metadata. Parsed data can be structured data, such as configuration files, alarms, operation logs, and historical trend data; or it can be unstructured content, such as flowcharts, audit documents, or attachments that need to be uniformly managed after parsing. By treating the above data as a unified data object, a foundation can be laid for subsequent unified registration, unified storage, and unified retrieval.
[0148] The lake processing layer 62 is used to extract metadata from multimodal data to obtain unified metadata.
[0149] In this embodiment, the data lake processing layer is mainly used to identify, transform, and register data objects before they enter the data lake. The platform or task can initiate a data lake entry request, passing parameters such as project name, dataset, data type, source file path, target table type, and import batch. Flink tasks can consume object storage event messages to perform path parsing and metadata registration for newly added or modified data objects. This layer can parse fields such as project_name, dataset, parent_path, relative_path, file_name, and file_ext based on the object path and business rules, dividing data objects into raw layer objects, parsed layer objects, unified metadata layer objects, directory objects, or other objects. For the parsed data, the system further performs operations such as field standardization, data type conversion, target table identification, object_id generation, and blob_record_id generation, enabling data from different sources and in different formats to be written according to a unified data lake table structure.
[0150] Lakeware storage layer 64 is used to store unified metadata in a metadata table, structured data in a structured table, and unstructured data in an unstructured table. The structured table and the unstructured table are associated with the metadata table through data object identifiers.
[0151] In this embodiment, the lakehouse storage layer is the core of data persistence, used to store unified metadata, structured data, and unstructured data. This layer uses Apache Paimon as the lakehouse table format, carrying a unified metadata table, a structured parsing layer table, and unstructured BLOB content tables. The unified metadata table records information such as the object identifier, path information, project name, dataset, object level, data type, target table name, BLOB association identifier, and lifecycle status of data objects, serving as a unified entry point for data retrieval and association. The structured parsing layer table stores data that can be expressed in a structured table, such as configurations, alarm information, operation logs, and historical trends. The unstructured BLOB table stores unstructured content such as flowcharts, audit documents, and document attachments, and is associated with the unified metadata table through `blob_record_id`. Through this layer design, the system can simultaneously manage structured and unstructured data while maintaining a traceable relationship between both types of data and the unified metadata.
[0152] The retrieval query layer 66 is used to synchronize the changed data of the metadata table to the query acceleration database and establish a metadata copy. The query acceleration database is used to carry the metadata copy to support the retrieval query of data objects. In response to the data retrieval request of the target object, the retrieval is performed in the metadata copy, structured table or unstructured table to obtain the data retrieval results.
[0153] In this embodiment, the retrieval query layer supports rapid retrieval on the platform side and query analysis by the computing engine. The system synchronizes real-time incremental changes to the unified metadata table in Paimon to StarRocks, which handles high-frequency metadata filtering, directory queries, paginated display, aggregation statistics, and object location. When querying data objects, the platform prioritizes accessing the unified metadata copy in StarRocks, quickly locating the data object based on fields such as project_name, dataset, object_layer, parsed_data_type, file_name, and parent_path. When specific content needs to be read, the platform queries the structured parsed table in Paimon based on target_table_name, or queries the unstructured BLOB content table based on blob_record_id. Simultaneously, Flink SQL and Spark can directly access the Paimon lakehouse tables for real-time processing, batch analysis, quality verification, and large-scale statistical calculations. Through this layer design, the system supports both interactive rapid retrieval on the platform side and multi-engine access for data analysis and computing tasks.
[0154] Through the data object layer, lake entry processing layer, lake storage layer, and retrieval query layer of the aforementioned data processing system, the goal of uniformly registering, hierarchically storing, and establishing cross-modal data associations for multimodal data in the process industry is achieved. This realizes the technical effect of eliminating multimodal data silos, supporting unified retrieval and deep association analysis of cross-type data, and thus solves the technical problem that multimodal data in the process industry is usually scattered and stored in multiple isolated systems, resulting in difficulties in data association and a lack of a unified data governance foundation.
[0155] This application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and used to execute the above-described data processing method.
[0156] It should be noted that the aforementioned electronic equipment is used to perform Figure 2 The data processing method shown above is also applicable to this electronic device, and will not be repeated here.
[0157] This application also provides a non-volatile storage medium that includes a stored computer program, wherein the device containing the non-volatile storage medium executes the above-described data processing method by running the computer program.
[0158] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The data processing method shown above is also applicable to this non-volatile storage medium, and will not be repeated here.
[0159] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the above-described data processing method.
[0160] It should be noted that the above-mentioned computer program product is used to execute Figure 2 The data processing method shown above is also applicable to this computer program product, and will not be repeated here.
[0161] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0162] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0163] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.
[0164] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0165] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0166] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.
[0167] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A data processing method, characterized in that, include: Acquire multimodal data from the production process in the process industry, wherein the multimodal data includes at least raw data, structured data, and unstructured data; Metadata is extracted from the multimodal data to obtain unified metadata; The unified metadata is stored in a metadata table, the structured data is stored in a structured table, and the unstructured data is stored in an unstructured table, wherein the structured table and the unstructured table are associated with the metadata table through data object identifiers.
2. The method according to claim 1, characterized in that, Metadata is extracted from the multimodal data, including: The multimodal data is stored as an object; Receive event messages generated in the object storage, wherein the event messages are used to indicate that there is a file addition, modification or deletion operation in the multimodal data; The object path information in the event message is parsed by the stream processing engine, wherein the object path information includes at least the path structure field and the data object hierarchy in the event message; A data object identifier is generated based on the object path information and the corresponding file attribute information; Generate unified metadata based on the event message, the object path information, the file attribute information, and the data object identifier.
3. The method according to claim 1, characterized in that, Storing the unified metadata in a metadata table, storing the structured data in a structured table, and storing the unstructured data in an unstructured table includes: The unified metadata is written into a metadata table, wherein the field types of the metadata table include at least one of the following: basic data object fields, path structure fields, business affiliation fields, lifecycle fields, and parsing association fields; The multimodal data is parsed to obtain data parsing results, and the data structure type in the data parsing results is identified; Based on the data structure type, structured data in the multimodal data is written into a structured table, and unstructured data in the multimodal data is written into an unstructured table. The structured table is used to store parsed data with a fixed structure and includes a parsed table field associated with the metadata table. The unstructured table is used to store unstructured parsed data written in binary form and includes a content identifier field associated with the metadata table.
4. The method according to claim 3, characterized in that, Writing the structured data from the multimodal data into a structured table includes: Determine the business type of the data objects in the structured data; The structured data is written into the corresponding parsing table in the structured table according to the business type, wherein the parsing table includes at least one of the configuration parsing table, alarm parsing table, operation log parsing table and historical trend parsing table; In the process of writing the structured data into the historical trend parsing table, the data is organized into partitions using the project name and dataset as partitioning fields, and trend data records are generated based on historical sequence identifiers. The project name and dataset support the division of the structured data according to the project dimension and dataset dimension, and the historical sequence identifier is used to identify the time-series data points in the structured data.
5. The method according to claim 1, characterized in that, The method further includes: organizing the data objects in the multimodal data using a hierarchical model, wherein the hierarchical model includes: The raw layer is used to store the raw data in the multimodal data; The parsing layer is used to store the structured and unstructured data after parsing the multimodal data; A unified metadata layer is used to store unified metadata corresponding to the multimodal data; The directory layer is used to store the path hierarchy relationships in the multimodal data; Other object layers are used to store data objects in the multimodal data that cannot be identified according to preset rules.
6. The method according to claim 1, characterized in that, The metadata table also includes target extended fields for supporting platform access control, lifecycle management, and data object governance.
7. The method according to claim 1, characterized in that, The method further includes: The changed data in the metadata table is synchronized to the query acceleration database and a metadata copy is established. The query acceleration database is used to carry the metadata copy to support the retrieval and query of data objects. In response to a data retrieval request from a target object, a search is performed in the metadata copy, the structured table, or the unstructured table to obtain data retrieval results.
8. The method according to claim 7, characterized in that, The query acceleration table uses the data object identifier corresponding to the unified metadata or the combination of the data object identifier and the project name and dataset as a unique identifier, and performs an update operation through the primary key update mechanism when there are file addition, modification or deletion operations in the multimodal data; In the query acceleration table, the process industry information in the multimodal data is divided into independent fields, wherein the independent fields include at least one of project name, dataset, object level, data parsing type, control station name, and server name; and In the query acceleration table, a target index is configured for a preset string field in the multimodal data, wherein the target index is used to guide similar queries on the preset string field.
9. The method according to claim 1, characterized in that, In response to a data retrieval request for a target object, a retrieval is performed in the metadata copy, the structured table, or the unstructured table, including: When the data retrieval request indicates that the target object is a queried data object, a retrieval is performed based on the data object identifier in the metadata copy to determine the target data object corresponding to the data retrieval request; When the data retrieval request indicates that the target object is the content of the queried data structure, a retrieval is performed based on the parsing table fields in the structured table to determine the target structured data content corresponding to the data retrieval request; or The target unstructured data content corresponding to the data retrieval request is determined by searching the content identifier field in the unstructured table.
10. A data processing system, characterized in that, include: The data object layer is used to store multimodal data from the process industry production process, wherein the multimodal data includes at least raw data, structured data and unstructured data; The lake entry processing layer is used to extract metadata from the multimodal data to obtain unified metadata; The lake warehouse storage layer is used to store the unified metadata in a metadata table, store the structured data in a structured table, and store the unstructured data in an unstructured table, wherein the structured table and the unstructured table are associated with the metadata table through data object identifiers; The retrieval and query layer is used to synchronize the changed data of the metadata table to the query acceleration database and establish a metadata copy. The query acceleration database is used to carry the metadata copy to support the retrieval and query of data objects. In response to the data retrieval request of the target object, the retrieval is performed in the metadata copy, the structured table or the unstructured table to obtain the data retrieval results.
11. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; The processor, connected to the memory, is used to execute the data processing method according to any one of claims 1 to 8.
12. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the data processing method according to any one of claims 1 to 8 by running the computer program.
13. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the data processing method according to any one of claims 1 to 8.
