Multi-modal data processing method and device, electronic equipment, medium and program product

Abstract

The invention provides a multi-modal data processing method and device, electronic equipment, a medium and a program product, and the method comprises the steps: receiving structured data and unstructured data from different sources based on a unified data access interface, and building a preliminary correlation between the structured data and the unstructured data; screening out a target data subset meeting business requirements on the enterprise side according to the preliminary association result, and generating a standardized data packet; uploading the standardized data packet to a shared network disk; and downloading and analyzing the standardized data packet from the shared network disk at the group side, and carrying out bidirectional dynamic association on the enhanced metadata extracted from the unstructured data file and the loaded structured data to form a multi-modal data set for large model training. According to the method, cross-organization, high-reliability and low-manual-intervention data circulation and bidirectional dynamic association between the enterprise side and the group side are realized, and the multi-modal data fusion efficiency, accuracy and availability in large model training are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of rail transit technology, and in particular to a multimodal data processing method, apparatus, electronic device, medium, and program product. Background Technology

[0002] In the intelligent operation and safety management of rail transit, a large amount of structured data and unstructured multimodal data need to be processed simultaneously. However, existing technologies generally adopt a separate architecture to collect and process these two types of data separately, resulting in difficulties in aligning data time sequences, inconsistent feature semantics, incomplete metadata extraction, and a lack of effective cross-modal correlation mechanisms. Especially when data is shared and collaborated between the group and its subordinate operating companies, the lack of a unified data format standard and efficient transmission mechanism further exacerbates the data fragmentation problem.

[0003] The aforementioned deficiencies severely restrict the construction of high-quality multimodal training datasets, making it difficult to support AI applications in rail transit driven by intelligent diagnosis, predictive maintenance, and large models. There is an urgent need for a technical solution that can achieve deep fusion, dynamic correlation, and efficient cross-organizational flow of structured and unstructured data. Summary of the Invention

[0004] This invention provides a multimodal data processing method, apparatus, electronic device, medium, and program product to overcome the shortcomings of existing technologies that cannot effectively integrate structured and unstructured data and cannot be transferred across organizations, thus providing a high-quality multimodal data foundation for intelligent operation and maintenance, fault diagnosis, and large model training.

[0005] On one hand, this invention provides a multimodal data processing method, comprising: receiving structured and unstructured data from different sources based on a unified data access interface, and establishing a preliminary association between the structured and unstructured data; selecting a target data subset that meets business needs on the enterprise side based on the preliminary association results, and generating a standardized data package based on the target data subset; the standardized data package includes a data table structure definition file, a structured data export file, and its associated unstructured data file; uploading the standardized data package to a shared cloud drive; downloading and parsing the standardized data package from the shared cloud drive on the group side, creating a data table based on the data table structure definition file to load the structured data export file, and performing a bidirectional dynamic association between the enhanced metadata extracted from the unstructured data file and the loaded structured data to form a multimodal dataset for large model training.

[0006] Furthermore, establishing the preliminary association between the structured data and the unstructured data includes: performing multimodal content recognition on the unstructured data to extract the device identification information and timestamp information contained therein; comparing the extracted device identification information with the device identification field in the structured data, and, provided that the device identifications are consistent, matching the extracted timestamp information with the time field in the structured data to determine one or more structured data records with the smallest time difference as the association object.

[0007] Furthermore, the step of generating a standardized data package based on the target data subset includes: automatically analyzing the table structure of the target data subset and generating a data table structure definition file; exporting the structured data in the target data subset as a structured data export file, and organizing the unstructured data files associated with the records in the structured data export file into a specified subdirectory according to a predefined directory structure to obtain the standardized data package.

[0008] Furthermore, the step of downloading and parsing standardized data packets from the shared cloud drive on the group side includes: migrating the unstructured data file to an internal unstructured storage system supported by the transfer; calling the multimodal processing module to perform deep parsing on the unstructured data file to generate enhanced metadata; wherein the enhanced metadata includes one or more of the following: device identifier, timestamp, spatial location, device status, abnormal event type, and operation behavior category.

[0009] Furthermore, the step of dynamically associating the enhanced metadata extracted from the unstructured data file with the loaded structured data in a two-way manner includes: establishing a two-way association in a graph database or relational database based on the device identifier, timestamp, or spatial location commonly contained in the enhanced metadata and the loaded structured data; the two-way association is used to enable the synchronization verification of the associated unstructured data to be triggered when the structured data is updated, or to update the association status of the structured data record in reverse when the unstructured data annotation changes.

[0010] Furthermore, the structured data includes one or more of the following: train operation logs, sensor time-series data, and equipment status records; the unstructured data includes one or more of the following: monitoring images, infrared thermal images, fault report documents, operation manuals, alarm audio, and maintenance videos.

[0011] Furthermore, when receiving structured and unstructured data on the enterprise side, a unified scheduler is used to align the time windows for the collection tasks of structured and unstructured data.

[0012] Furthermore, the unified data access interface supports multiple protocols, including RESTful API, Kafka, and SFTP, for simultaneously accessing structured and unstructured data. The structured data includes at least temperature, pressure, rotational speed, equipment status logs, and GPS tracks collected by equipment sensors, while the unstructured data includes at least infrared thermal images captured by equipment monitoring cameras, fault report documents uploaded by operators, and equipment operation audio.

[0013] Furthermore, when generating the standardized data packet, integrity checks are performed on the unstructured data file to remove corrupted, encrypted, or format-unsupported files, and a list of filtered files and the reasons are recorded in the data table structure definition file.

[0014] Furthermore, the structured data export file adopts UTF-8 encoded CSV format and retains the original field order and null value representation.

[0015] Furthermore, after parsing the standardized data packets on the group side, before migrating the unstructured data files to the internal unstructured storage system, the hash value is first verified to be consistent with the record when the enterprise side uploaded the data, in order to prevent data tampering or damage during the transmission process.

[0016] Furthermore, the multimodal processing module supports a plug-in model loading mechanism, allowing dynamic switching between image recognition, speech analysis, or text understanding models based on business scenarios to adapt to different device types or industry terminology systems.

[0017] Furthermore, the bidirectional dynamic association is stored in the form of bidirectional edges in a graph database, wherein the structured data records and the enhanced metadata are nodes to each other, and the edge attributes include association confidence, time alignment deviation, and spatial distance.

[0018] Furthermore, when key business fields in structured data change, a process for re-annotating associated unstructured data is automatically triggered, and a versioned association is established between the newly generated enhanced metadata version and the original structured record.

[0019] Furthermore, when the enhanced metadata of unstructured data is updated due to model upgrades, the associated structured data records are automatically marked as pending review, and change traceability information, including old labels, new labels, and change times, is added to the dataset.

[0020] Furthermore, the shared cloud drive adopts an access control strategy based on organizational isolation, with standardized data packets from different subsidiaries uploaded to their respective exclusive subdirectories.

[0021] Furthermore, after forming a multimodal dataset on the group side, a unique global identifier is generated for each sample, and structured data, enhanced metadata, unstructured file paths, and bidirectional associations are packaged into sample units for training large models. Secondly, the present invention also provides a multimodal data processing device, comprising: a preliminary association module, used to receive structured and unstructured data from different sources based on a unified data access interface, and establish a preliminary association between the structured data and the unstructured data; a data packet generation module, used to filter out a target data subset that meets business needs on the enterprise side based on the preliminary association results, and generate a standardized data packet based on the target data subset; the standardized data packet includes a data table structure definition file, a structured data export file and its associated unstructured data file; a data packet uploading module, used to upload the standardized data packet to a shared cloud drive; and a bidirectional dynamic association module, used to download and parse the standardized data packet from the shared cloud drive on the group side, create a data table based on the data table structure definition file to load the structured data export file, and perform bidirectional dynamic association between the enhanced metadata extracted from the unstructured data file and the loaded structured data to form a multimodal dataset for large model training.

[0022] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal data processing method as described above.

[0023] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal data processing method as described above.

[0024] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal data processing method as described above.

[0025] The multimodal data processing method provided by this invention receives structured and unstructured data from different sources through a unified data access interface and establishes a preliminary association between the structured and unstructured data. Subsequently, based on the preliminary association results, a target data subset that meets business needs is selected on the enterprise side, and a standardized data package is generated based on this target data subset. The standardized data package includes a data table structure definition file, a structured data export file, and its associated unstructured data files. Next, the standardized data package is uploaded to a shared cloud drive. Finally, the standardized data package is downloaded and parsed on the group side. A data table is created based on the data table structure definition file to load the structured data export file. Enhanced metadata extracted from the unstructured data files is dynamically and bidirectionally associated with the loaded structured data to form a multimodal dataset for large model training. This method achieves synchronous collection and preliminary association of structured and unstructured data through a unified access interface, effectively solving the problems of time sequence asynchrony, resource redundancy and fusion difficulties caused by the separation and processing of multi-source heterogeneous data in existing technologies. At the same time, through standardized data packets and shared network disk mechanism, it realizes cross-organizational, highly reliable, low-manual-intervention data flow and bidirectional dynamic association between the enterprise side and the group side, significantly improving the efficiency, accuracy and availability of multimodal data fusion in large model training. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the multimodal data processing method provided in an embodiment of the present invention.

[0028] Figure 2 This is a schematic diagram of the overall process of the multimodal data processing method provided in the embodiments of the present invention.

[0029] Figure 3 This is a schematic diagram of the structure of the multimodal data processing device provided in the embodiment of the present invention.

[0030] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] It should be noted that existing technologies suffer from multiple systemic defects in the fusion processing of structured and unstructured data: The data acquisition stage generally employs a separate architecture, resulting in independent processing flows for structured and unstructured data, frequent timing discrepancies, low resource utilization, and poor scalability; the feature extraction stage relies on complex normalization operations due to significant differences in semantic space between modalities, easily introducing information distortion, especially with limited effectiveness in cross-modal mapping of industrial sensor data and image / video data; metadata extraction over-relies on predefined rule engines, making it difficult to adapt to the complex and variable characteristics of unstructured data (such as novel annotations in engineering drawings or free-format documents), and small sample sizes... The data fusion methods suffer from several shortcomings. First, they lack robustness in various scenarios. Second, data fusion strategies are often statically weighted or rule-driven, lacking the dynamic adaptability to changes in data quality (such as sensor failures) or scenario requirements, leading to decreased accuracy. Third, association management typically only supports one-way queries (e.g., from unstructured data to structured data), and the updates to association relationships are lagging, failing to meet real-time decision-making needs. For example, when the identity of a video surveillance target changes, related text records and audio data cannot be updated synchronously. Furthermore, large-scale data fusion incurs high computational overhead, existing methods are highly dependent on computing power and storage resources, and lack sufficient privacy protection mechanisms (such as data encryption and access control), making it difficult to support the application requirements of high-concurrency or high-security scenarios. These deficiencies collectively result in low data processing efficiency and insufficient reliability, limiting the practical effectiveness of multimodal data in high-value scenarios such as industrial predictive maintenance and financial risk control.

[0033] In view of this, the present invention proposes a multimodal data processing method, specifically, Figure 1 A flowchart illustrating the multimodal data processing method provided in an embodiment of the present invention is shown.

[0034] like Figure 1As shown, the method includes: S110, receiving structured and unstructured data from different sources based on a unified data access interface, and establishing a preliminary association between the structured and unstructured data; S120, selecting a target data subset that meets business needs on the enterprise side based on the preliminary association results, and generating a standardized data package based on the target data subset; the standardized data package includes a data table structure definition file, a structured data export file, and its associated unstructured data file; S130, uploading the standardized data package to a shared cloud drive; S140, downloading and parsing the standardized data package from the shared cloud drive on the group side, creating a data table based on the data table structure definition file to load the structured data export file, and performing a bidirectional dynamic association between the enhanced metadata extracted from the unstructured data file and the loaded structured data to form a multimodal dataset for large model training.

[0035] The following will provide a detailed description of steps S110-S140 and related steps.

[0036] S110: Receive structured and unstructured data from different sources based on a unified data access interface, and establish a preliminary association between the structured and unstructured data.

[0037] It's easy to understand that a unified data access interface is used to receive structured and unstructured data from different sources and establish a preliminary connection between them. The structured and unstructured data from different sources constitute multi-source heterogeneous data, or multimodal data.

[0038] The unified data access interface supports multiple protocols (such as RESTful API, Kafka, and SFTP) for simultaneously accessing structured and unstructured data. Structured data includes time-series data such as temperature, pressure, rotational speed, equipment status logs, and GPS tracks collected by equipment sensors. Unstructured data includes infrared thermal images captured by equipment monitoring cameras, fault report documents uploaded by operators, and equipment operation audio.

[0039] Initial association is based on key fields, including but not limited to timestamps, device IDs, and spatial location information. For example, when a structured data record has a timestamp of "2024-XX-XX 11:11:32" and a device ID of "Vehicle A", an initial association is automatically established between this record and an unstructured file (such as "Vehicle A_XX_Emergency Braking Photo.jpg") whose filename contains "Vehicle A" and whose creation time is closest to the timestamp "2024-XX-XX 11:11:32". This forms a cross-modal data pair.

[0040] After receiving structured and unstructured data from different sources based on a unified data access interface in step S110, and establishing a preliminary association between the structured and unstructured data, step S120 is further executed.

[0041] S120, based on the preliminary association results, a target data subset that meets business needs is selected on the enterprise side, and a standardized data package is generated based on the target data subset; the standardized data package includes a data table structure definition file, a structured data export file and its associated unstructured data file.

[0042] Specifically, based on the preliminary association results of step S110, a target data subset that meets specific business needs is selected on the enterprise side. For example, if operations and maintenance personnel need to analyze all high-risk fault events within the past week, they can execute the filtering logic through a low-code data development module to extract structured data records that meet the conditions and their associated unstructured file lists from the full set of preliminary association data, thereby obtaining the target data subset. The filtering logic can be configured based on SQL or visual rules, and is not specifically limited here.

[0043] Subsequently, a standardized data package is generated based on the target data subset. This standardized data package consists of three parts: first, a data table structure definition file, or DDL file, which describes the field names, types, and constraints of the target structured data (the structured data within the selected target data subset); second, a structured data export file, typically in CSV format, containing cleaned and aligned time-series and business data; and third, unstructured data files associated with the target structured data, such as original files of images, videos, and documents, whose file paths are consistent with the referenced fields in the structured data records to ensure accurate matching later.

[0044] Next, proceed to step S130.

[0045] S130, Upload the standardized data packet to the shared cloud drive.

[0046] Specifically, the standardized data packets obtained in step S120 are organized according to a preset directory structure and then uploaded to the shared cloud drive. The shared cloud drive serves as a secure data exchange medium between the enterprise and the group, supporting breakpoint resume and access control to ensure the integrity and security of data transmission.

[0047] Further, proceed to step S140.

[0048] S140, the standardized data packet is downloaded and parsed on the group side, a data table is created according to the data table structure definition file to load the structured data export file, and the enhanced metadata extracted from the unstructured data file is dynamically correlated bidirectionally with the loaded structured data to form a multimodal dataset for large model training.

[0049] Specifically, the standardized data packets are downloaded and parsed from the shared cloud drive on the group side.

[0050] First, the corresponding data tables are automatically created based on the data table structure definition file. Second, the structured data export file is loaded into the created data tables. Simultaneously, a multimodal content recognition engine, such as a processing module integrating OCR, object detection, and speech recognition capabilities, is invoked to perform deep analysis of the unstructured data files, extracting enhanced metadata. For example, device model and fault codes are extracted from fault report documents, device component status labels are identified from images, and frames are extracted from videos and annotated with abnormal behaviors.

[0051] Subsequently, the enhanced metadata is dynamically linked to the loaded structured data in a two-way manner. This two-way dynamic link means that, on the one hand, unstructured data files and their enhanced metadata can be directly referenced through foreign keys based on structured data records; on the other hand, when the enhanced metadata of unstructured data files is updated (such as when AI recognition tags are added), the associated structured data can be automatically updated in reverse or a re-fusion process can be triggered.

[0052] Through the aforementioned steps, a well-structured, modally aligned, and semantically rich multimodal dataset is finally formed, which can be directly used for training large models, significantly improving the model's generalization ability and accuracy in tasks such as equipment failure prediction and root cause analysis.

[0053] The multimodal dataset includes structured data, reference information for unstructured data files (i.e., access paths or unique identifiers used for quick retrieval of unstructured data files), enhanced metadata, and bidirectional associations between structured data and enhanced metadata. Enhanced metadata exists as structured semantic labels in the multimodal dataset.

[0054] It should be noted that in this embodiment, "group" represents the headquarters, and "enterprise" represents local branches or business units. Data is collected by local branches, and the headquarters centrally constructs a large model for intelligent customer service, risk prediction, supply chain optimization, etc.

[0055] In this embodiment, structured and unstructured data from different sources are received through a unified data access interface, and a preliminary association is established between the structured and unstructured data. Subsequently, based on the preliminary association results, a target data subset that meets business needs is selected on the enterprise side, and a standardized data package is generated based on the target data subset. The standardized data package includes a data table structure definition file, a structured data export file, and its associated unstructured data file. Next, the standardized data package is uploaded to a shared cloud drive. Finally, the standardized data package is downloaded and parsed on the group side, a data table is created based on the data table structure definition file to load the structured data export file, and the enhanced metadata extracted from the unstructured data file is dynamically associated with the loaded structured data in a two-way manner to form a multimodal dataset for large model training. This method achieves synchronous collection and preliminary association of structured and unstructured data through a unified access interface, effectively solving the problems of time sequence asynchrony, resource redundancy and fusion difficulties caused by the separation and processing of multi-source heterogeneous data in existing technologies. At the same time, through standardized data packets and shared network disk mechanism, it realizes cross-organizational, highly reliable, low-manual-intervention data flow and bidirectional dynamic association between the enterprise side and the group side, significantly improving the efficiency, accuracy and availability of multimodal data fusion in large model training.

[0056] Based on the above embodiments, the process of establishing a preliminary association between structured data and unstructured data in step S110 will be described in detail below.

[0057] Establishing a preliminary association between structured and unstructured data includes: performing multimodal content recognition on unstructured data to extract the device identification information and timestamp information contained therein; comparing the extracted device identification information with the device identification field in the structured data; and, provided that the device identification is consistent, matching the extracted timestamp information with the time field in the structured data to determine one or more structured data records with the smallest time difference as the association object.

[0058] It's easy to understand that in a multimodal data processing system deployed on the enterprise side, structured and unstructured data from different sources are first received through a unified data access interface. The structured data originates from industrial databases or real-time data streams, including sensor time-series records generated during equipment operation. Each record contains equipment identification information (such as equipment ID), a time field (accurate to seconds or milliseconds), and multiple process parameters (such as temperature, pressure, vibration values, etc.). Unstructured data includes images collected on-site (such as infrared thermal images of equipment, close-up photos of faulty areas), videos (such as operation process recordings, abnormal event playback), documents (such as PDF fault reports, Word inspection records), and audio (such as recordings of abnormal equipment noises).

[0059] To establish a preliminary connection between structured and unstructured data, multimodal content recognition is required for the unstructured data. This recognition process is executed by a multimodal content recognition engine, which automatically extracts structured semantic information from different types of unstructured data. Specifically, this includes: for image files, using object detection and OCR (Optical Character Recognition) technologies to identify equipment nameplates, labels, or screen display content contained in the image, extracting equipment identification information (such as "pump-203") and visible timestamp information (such as "2024-XX-XX 14:25:18" displayed on the screen); for document files (such as fault reports), using text parsing and named entity recognition to extract key fields such as equipment number and occurrence time from the document title or body; and for video or audio files, using speech-to-text, keyframe analysis, or metadata reading to obtain the event occurrence time and associated equipment information.

[0060] After extracting the above information, the timestamp information extracted from the unstructured data is matched with the time field in the structured data. The matching logic is as follows: assuming the device identification information is consistent, the absolute value of the time difference between the timestamp of the unstructured data and the time field of each structured data record is calculated, and the one or more structured data records with the smallest time difference are identified as the associated objects.

[0061] For example, if the device identifier "Pump-203" and the timestamp information "2024-XX-XX 14:25:00" are extracted from a fault report named "Pump-203_Overheat Alarm.pdf", then all records with the device ID "Pump-203" are filtered in the structured data, and one or more records whose time field is closest to "14:25:00" (such as "14:24:58", "14:25:02") are identified and established as the initial associated objects of the fault report.

[0062] In addition, users can verify, adjust, or enhance the initial correlation results through the data development module. The data development module here refers to a visual tool that supports low-code configuration of data processing and correlation logic.

[0063] For example, operations and maintenance personnel can drag and drop field mapping relationships in the graphical interface to manually correct mismatches caused by time drift, or add business rules, such as "only associate records with a fault level of 'high'". The confirmed or optimized relationships are then automatically written to a data storage system, such as a distributed relational database like Doris or a graph database, forming a persistent cross-modal relationship view. This provides a reliable foundation for subsequent target data subset filtering and standardized data package generation steps.

[0064] This embodiment performs multimodal content recognition on unstructured data, extracts the device identification information and timestamp information contained therein, and matches the extracted timestamp information with the time field in the structured data to determine one or more structured data records with the smallest time difference as the association object, thereby establishing a preliminary association between structured data and unstructured data. This achieves a precise preliminary association between structured and unstructured data in the semantic and temporal dimensions, effectively solving the fusion failure problem caused by modal fragmentation and temporal misalignment in the prior art.

[0065] Based on the above embodiments, the process of generating standardized data packets based on the target data subset in step S120 will be described in detail below.

[0066] The process of generating standardized data packages based on a subset of target data includes: automatically analyzing the table structure of the subset of target data and generating a data table structure definition file; exporting the structured data in the subset of target data as a structured data export file; and organizing the unstructured data files associated with the records in the structured data export file into a specified subdirectory according to a predefined directory structure to obtain a standardized data package.

[0067] It is easy to understand that, on the enterprise side, the first step is to use data tools to collect time-series data (such as temperature, pressure, speed, etc. collected by sensors, as shown in Table 1 below) and business structured data (such as fault work orders and equipment ledgers, as shown in Table 2 below) from various source systems into big data computing components (such as distributed databases Hive or Doris).

[0068] Table 1

[0069] Table 2

[0070] Subsequently, leveraging the computing power of big data computing components and combining it with specific business needs, such as filtering all high-risk equipment failure events within the past 30 days, the system correlates and filters structured data (i.e., the aforementioned time-series data and business structured data) with multimodal metadata (unstructured data). This process supports two operation modes: one is to achieve precise conditional filtering by writing SQL statements; the other is to extract a subset of data that meets the conditions by dragging and dropping fields and setting rules through the data development module, and store the results separately in a temporary table or dedicated dataset to form the target data subset.

[0071] Unstructured data includes, for example, vehicle A_XX_fault report.doc, vehicle A_XX_emergency braking photo.jpg, and vehicle A_XX_video 10 minutes before the fault.mp4. The metadata for unstructured data is shown in Table 3 below.

[0072] Table 3

[0073] The preliminary screening results are shown in Table 4 below.

[0074] Table 4

[0075] Next, based on the aforementioned filtering results, a list of unstructured data files associated with the target data subset is further determined. Specifically, if the structured data record already contains an explicit unstructured data reference field, such as "fault attachment=vehicle A_XX_emergency braking photo.jpg", the corresponding filename is directly extracted; if there is no explicit association, such as only a time-series record without an attachment field, then based on the time field and device identification information in the structured data record, unstructured data files with matching time ranges (such as within 10 seconds before and after) and consistent devices are filtered from the multimodal metadata table (which records the metadata of all unstructured data files, including filename, creation time, and device to which they belong), thereby completing the association relationship.

[0076] Subsequently, a standardized data packet generation operation is performed. The first step is to automatically analyze the table structure of the target data subset, including field names, data types, primary keys, and constraint information, and generate a data table structure definition file, i.e., a DDL file, which is used to reconstruct the same data table structure at the receiving end (i.e., the group side).

[0077] The second step involves exporting the structured data from the target data subset into a structured data export file, typically in CSV format, preserving the original field order and values. Simultaneously, based on a predefined directory structure, the unstructured data files associated with each record in the structured data export file are organized into designated subdirectories. This predefined directory structure is defined by the cloud storage transfer protocol; for example, the top-level directory might be "Fault Analysis Data," with an attachment subdirectory for storing unstructured data files. For instance, all images and videos related to the "Car A" fault event are copied or moved to the "Fault Analysis Data / Attachments / " path, ensuring the file location logically matches the reference path in the structured data records.

[0078] Finally, the generated DDL file, CSV file, and organized unstructured data subdirectories are packaged together to form a complete standardized data package, and uploaded to a shared cloud drive according to the preset naming convention (such as "Enterprise A_2024XXXX_Fault Analysis Package") for download and use by the group.

[0079] This embodiment automatically analyzes the table structure of the target data subset, generates a data table structure definition file, and exports the structured data in the target data subset as a structured data export file. Based on a predefined directory structure, it organizes the unstructured data files associated with the records in the structured data export file into a specified subdirectory, resulting in a standardized data package. This achieves standardized encapsulation of cross-modal data on the enterprise side, which not only avoids formatting errors or file omissions caused by manual processing, but also ensures that the group side can seamlessly load and merge data based on a unified structure, significantly improving the reliability, efficiency, and reproducibility of cross-organizational data flow.

[0080] Based on the above embodiments, the process of generating enhanced metadata in step S140 will be described in detail below.

[0081] On the group side, standardized data packets are downloaded and parsed from the shared cloud drive. This process includes: migrating unstructured data files to an internal unstructured storage system that supports data transfer; calling the multimodal processing module to perform deep parsing on the unstructured data files to generate enhanced metadata; wherein, the enhanced metadata includes one or more of the following: device identifier, timestamp, spatial location, device status, abnormal event type, and operation behavior category.

[0082] It's easy to understand that on the group side, the first step is to download the standardized data package uploaded by the enterprise from the shared cloud drive, and then perform integrity verification and structure parsing on it.

[0083] After parsing, the unstructured data files (such as images, videos, documents, audio, etc.) contained in the standardized data packets are first migrated to the internal unstructured storage system supported by the transfer. The unstructured storage system is designed for efficient reading, version management and secure access of multimodal data within the group, ensuring that unstructured data files can be stably accessed in the future.

[0084] Then, deep parsing is performed on the migrated unstructured data files. The multimodal processing module is a unified processing engine integrating multiple artificial intelligence analysis capabilities, used to perform automated deep parsing on unstructured data such as audio, video, images, and text. This process includes, but is not limited to: object detection and image classification for image files to identify equipment components, surface conditions, and abnormal areas; keyframe extraction and behavior recognition for video files to analyze operator actions or changes in equipment operating status; optical character recognition and natural language understanding for document files to extract key event descriptions and parameter information from the text; and voice activity detection and acoustic feature analysis for audio files to identify abnormal equipment noise patterns or alarm voice content.

[0085] Enhanced metadata is generated from unstructured data through multimodal parsing operations. This enhanced metadata is not simply the attributes of the original unstructured data file, but rather high-value structured information extracted through semantic understanding. Specifically, it includes one or more of the following: device identifier, timestamp, spatial location, device status, anomaly event type, and operational behavior category.

[0086] For example, from an image titled "pump-203_overheat.jpg", the equipment identifier "pump-203" can be identified, the event time can be inferred to be the time the image was captured "2024-XX-XX 14:25:00", the spatial location can be "workstation No. 3 in workshop A", the equipment status can be "abnormal increase in surface temperature", and the abnormal event type can be "bearing overheat warning".

[0087] For example, from an operation video, the operation behavior category can be extracted as "emergency stop operation", the associated equipment identifier is "vehicle A", the timestamp is "2024-XX-XX 11:11:32", and the spatial location is "assembly line B segment".

[0088] The generated enhanced metadata is stored in the group's data storage system in the form of structured records, and a two-way dynamic association is established with the subsequently loaded structured data to build a high-quality, semantically rich multimodal dataset.

[0089] This embodiment migrates unstructured data files to an internally supported unstructured storage system and calls a multimodal processing module to perform deep parsing on the unstructured data files, generating enhanced metadata. The enhanced metadata includes one or more of the following: device identifier, timestamp, spatial location, device status, abnormal event type, and operation behavior category, which significantly improves the usability of unstructured data.

[0090] Based on the above embodiments, the process of establishing bidirectional dynamic association in step S140 will be described in detail below.

[0091] The enhanced metadata extracted from the unstructured data file is dynamically linked bidirectionally with the loaded structured data. This includes establishing a bidirectional association in a graph database or relational database based on the device identifier, timestamp, or spatial location shared by the enhanced metadata and the loaded structured data. The bidirectional association is used to enable synchronous verification of the associated unstructured data when the structured data is updated, or to update the association status of the structured data record in reverse when the unstructured data label changes.

[0092] On the group side, after parsing the standardized data packets, a corresponding data table is created based on the data table structure definition file. The structured data export file is then loaded into this data table to form the loaded structured data. At the same time, the unstructured data file is deeply parsed through the multimodal processing module to generate enhanced metadata, as mentioned above. Its content includes one or more of the following: device identifier, timestamp, spatial location, device status, abnormal event type, and operation behavior category.

[0093] Subsequently, a bidirectional dynamic association operation is performed. Specifically, key association fields commonly found in both the enhanced metadata and the loaded structured data are identified, including device identifiers, timestamps, or spatial locations. For example, a structured data record contains the device identifier "pump-203", the time field "2024-XX-XX 14:25:02", and GPS coordinates "(33.12, 40.15)", while an enhanced metadata record also contains the device identifier "pump-203", the inferred timestamp "2024-XX-XX 14:25:00", and the spatial location "(33.12, 40.16)". Based on preset matching rules (such as the device identifiers must be identical, the time difference must not exceed 5 seconds, and the spatial distance must be less than 10 meters), it is determined that the two have the conditions for association.

[0094] Building upon this foundation, bidirectional relationships are established in graph databases or relational databases. These bidirectional relationships exist in the form of explicit links. On one hand, foreign key or reference fields are added to structured data records, pointing to the corresponding enhanced metadata records and their original unstructured data file paths. On the other hand, the enhanced metadata records also store reverse pointers, linking to one or more structured data records.

[0095] The purpose of the two-way association is as follows: When structured data is updated (for example, when operations personnel change the "fault level" of a fault record from "medium" to "high"), the association verification mechanism is automatically triggered to check whether the associated unstructured data (such as fault images and videos) is still consistent with the updated business context. If necessary, manual review or re-labeling is prompted. When the labeling of unstructured data changes (for example, after the AI ​​model iterates, a new category of "not wearing a safety helmet" is identified in a video), the "operation compliance status" field in the associated structured data record is automatically updated through a reverse pointer, and the record is marked as "requires review", realizing information feedback from the unstructured side to the structured side.

[0096] In one specific embodiment, the structured data and unstructured data files after bidirectional association are shown in Table 5 below.

[0097] Table 5

[0098] Based on the above, a multimodal dataset for training large models can be obtained. This dataset not only contains the original structured metrics and time-series information, but also integrates semantic information from unstructured data from multiple sources such as images, videos, and documents, and maintains dynamic consistency among all modal data. For example, in a predictive maintenance scenario for equipment, a single multimodal data record can simultaneously present vibration values, infrared hotspot regions, operator emergency stop actions, and fault report summaries, providing high-fidelity, high-dimensional input samples for training large models.

[0099] This embodiment establishes a bidirectional association in a graph database or relational database based on the device identifier, timestamp, or spatial location jointly contained in the enhanced metadata and the loaded structured data. The bidirectional association is used to enable the synchronous verification of the associated unstructured data when the structured data is updated, or to update the association status of the structured data record in reverse when the unstructured data label (corresponding to the enhanced metadata parsed from the unstructured data) changes. This achieves deep, real-time, and maintainable integration of structured and unstructured data, effectively overcoming the defects of static, unidirectional, and easily failed associations in the prior art, and significantly improving the reliability of multimodal datasets in intelligent analysis and decision support.

[0100] Based on the above embodiments, the following will further describe in detail the process of collecting structured and unstructured data in step S110 and the content covered therein.

[0101] The system receives structured and unstructured data from different sources through a unified data access interface. The structured data includes one or more of the following: train operation logs, sensor time-series data, and equipment status records. The unstructured data includes one or more of the following: monitoring images, infrared thermal images, fault report documents, operation manuals, alarm audio, and maintenance videos.

[0102] Specifically, in the intelligent operation and maintenance scenario of rail transit, structured and unstructured data from different sources are received through a unified data access interface. The structured data includes one or more of the following: train operation logs, sensor time-series data (such as temperature, pressure, speed, GPS trajectory), and equipment status records; the unstructured data includes one or more of the following: monitoring images, infrared thermal images, fault report documents, operation manuals, alarm audio, and operation and maintenance videos.

[0103] Structured data is collected periodically or in real-time by the Starsea data acquisition module (a component used to configure standardized acquisition tasks from various database systems such as MySQL and Oracle) and written to storage engines in the data lake (such as Doris and Hive).

[0104] Unstructured data is collected through the Xinghai Multimodal Data Center processing module. Specifically, the data source is first defined (referring to a file storage system such as SFTP, Ceph, FTP, or HDFS, including its connection address, access key, and other necessary information). Then, multimodal data processing tasks are configured to read image, video, document, or audio files from the specified data source and perform corresponding preprocessing. For example, images may undergo corruption filtering, format conversion, quality enhancement, or OCR recognition; videos may undergo frame extraction analysis; documents may undergo text extraction; or audio may undergo speech activity detection. During processing, built-in data analysis operators (such as image violation detection) can be invoked to generate detection tags and write them to the data storage engine.

[0105] The aforementioned data acquisition tasks can be configured to be executed periodically or run continuously in real time, ensuring that multi-source heterogeneous data is continuously and efficiently integrated into a unified processing flow.

[0106] This embodiment integrates heterogeneous data sources through a unified data access interface, which is compatible with traditional structured acquisition systems and supports automated preprocessing of multimodal unstructured data.

[0107] Based on the above embodiments, when receiving structured and unstructured data on the enterprise side, a unified scheduler is used to align the time windows of the data collection tasks for structured and unstructured data.

[0108] In traditional technical solutions, structured and unstructured data are often accessed by the processing system through independent acquisition channels, resulting in uncontrollable offsets in the acquisition time of the two types of data. For example, if an equipment emergency stop occurs at 11:11:32, the corresponding sensor record may be written to the database at 11:11:32.050, while the emergency braking image captured by the on-site camera may not be uploaded until 11:11:33.200. If there is no coordination mechanism between the two acquisition tasks, it will be difficult to accurately match the multimodal data of the event in the subsequent correlation stage, thus affecting the fusion quality.

[0109] To address this issue, this embodiment introduces a unified scheduler on the enterprise side, which coordinates the execution timing of collection tasks from different data sources. This scheduler aligns the time windows for structured and unstructured data collection tasks, limiting the collection triggers or data ingestion actions of both types of data to the same logical time interval, thereby ensuring that data originating from the same physical event has similar timestamps within the system.

[0110] Specifically, the unified scheduler synchronously triggers structured and unstructured data acquisition tasks based on a preset acquisition cycle or event-driven signals. For example, at the start of each 5-second acquisition cycle, the unified scheduler simultaneously sends a "retrieve latest sensor records" command to the structured data acquisition module and a "scan for newly added unstructured files in the specified directory" command to the multimodal data acquisition module. Upon receiving the commands, both modules read data generated within the most recent time window (e.g., [t-5s, t]) from their respective data sources and assign a unified logical time tag (e.g., cycle start time t) to this data, or retain the original timestamp but record the alignment window identifier to which it belongs.

[0111] This mechanism ensures that even if the original data have slight differences in physical generation time, their ingestion context within the system is constrained to the same time window, thus significantly reducing timing drift caused by asynchronous acquisition. For example, in the aforementioned emergency stop event, although the actual image upload time is 11:11:33.200, it falls within the alignment window of 11:11:30–11:11:35, and is therefore considered to be source data from the same sensor record (11:11:32.050) within the same window, laying the foundation for subsequent precise correlation based on device identifiers and timestamps.

[0112] Furthermore, the unified scheduler supports dynamically adjusting the time window width to adapt to the needs of different business scenarios. For rapidly changing industrial processes (such as motor vibration monitoring), millisecond-level windows can be configured; for manually operated events (such as inspection report uploads), the window width can be widened to minutes. This flexibility ensures the applicability and effectiveness of the time window alignment strategy in various application scenarios.

[0113] Based on the above embodiments, the unified data access interface further supports multiple protocols, including RESTful API, Kafka, and SFTP, for simultaneously accessing structured and unstructured data; wherein, the structured data includes at least temperature, pressure, rotational speed, equipment status logs, and GPS trajectory collected by equipment sensors, and the unstructured data includes at least infrared thermal images captured by equipment monitoring cameras, fault report documents uploaded by operators, and equipment operation audio.

[0114] In multimodal data processing systems deployed on the enterprise side, a unified data access interface is set up to efficiently aggregate heterogeneous data from different business systems and sensing terminals. This interface is a standardized and scalable data entry service used to shield the protocol differences of the underlying data sources and provide consistent data reception capabilities to the outside world.

[0115] In this embodiment, the unified data access interface is configured to support multiple communication and transmission protocols, specifically including RESTful API, Kafka, and SFTP. RESTful API is a lightweight interface specification based on the HTTP protocol, suitable for real-time push or query-based data interaction; Kafka is a distributed stream processing platform suitable for high-throughput, low-latency real-time data stream access; SFTP is a secure file transfer protocol suitable for secure batch file upload and synchronization.

[0116] By supporting the above-mentioned protocols, the unified data access interface can flexibly adapt to the output methods of different types of data sources, thereby enabling simultaneous access to structured and unstructured data. That is, under the same system architecture, the two types of data can be received and initially aggregated in parallel, avoiding the need to build separate independent channels.

[0117] Specifically, when the data source is an industrial control system, an IoT platform, or a database, the resulting structured data typically exists in a format with clearly defined fields and regular formats. In this embodiment, the structured data includes at least the following types: temperature (a key parameter reflecting the thermal state of the equipment), pressure (a physical quantity characterizing the load on a fluid or mechanical system), rotational speed (a quantitative indicator of the operating speed of rotating equipment) collected by equipment sensors, equipment status logs (time-series text records of events such as equipment start-up, shutdown, alarms, and mode switching), and GPS trajectories (a sequence of location coordinates reported by vehicle-mounted or mobile devices for spatial behavior analysis). This structured data can be pushed in real time to a unified data access interface via Kafka in the form of message streams, or it can be pulled or reported on demand via a RESTful API.

[0118] On the other hand, when the data source is a field monitoring system, mobile terminal, or manual operation terminal, the unstructured data generated is usually in the form of raw files, lacking a fixed field structure. In this embodiment, the unstructured data includes at least: infrared thermal images captured by the equipment monitoring camera (images of the equipment surface temperature distribution generated by infrared imaging technology, used to identify hot spots or abnormal heat dissipation), fault report documents uploaded by operators (usually text files in PDF, Word, or other formats, recording fault phenomena, handling processes, and cause analysis), and equipment operation audio (snippets of equipment operation sounds recorded by microphones or acoustic sensors, which can be used for abnormal noise detection or voice alarm recognition). This type of unstructured data is usually large in volume and has a low update frequency, making it suitable for periodic uploading to a unified data access interface in the form of file packages via the SFTP protocol. It can also be uploaded in chunks via a RESTful API in scenarios with network access.

[0119] Upon receiving the aforementioned data types, the unified data access interface first performs preliminary classification and caching based on protocol type and data identifier, providing raw input for subsequent multimodal correlation analysis. For example, a structured record sent via Kafka containing "Car A, Time 2024-XX-XX 11:11:32, Temperature 40℃" and an infrared thermal image uploaded via SFTP named "Car A_Emergency Braking_2024XX111132.jpg", despite originating from different protocols, are both received by the unified data access interface and marked as potential related objects within the same time window.

[0120] This embodiment effectively solves the data silo problem caused by protocol fragmentation in traditional solutions by adopting a unified data access interface that supports multiple protocols such as RESTful API, Kafka, and SFTP. It realizes the fusion and convergence of structured and unstructured data at the access layer, providing basic support for subsequent establishment of cross-modal associations, generation of standardized data packets, and construction of high-quality multimodal training datasets.

[0121] Based on the above embodiments, when generating the standardized data packet, integrity verification is performed on the unstructured data file to remove corrupted, encrypted, or format-unsupported files, and a list of filtered files and the reasons are recorded in the data table structure definition file.

[0122] During the process of generating standardized data packages based on target data subsets on the enterprise side, in order to ensure that the group side can stably and reliably load and parse multimodal data subsequently, preprocessing quality control of associated unstructured data files is required. Specifically, before organizing unstructured data files into standardized data packages, integrity verification is first performed on each file. This involves determining whether the file can be opened normally, whether the content is complete, and whether it conforms to the expected type through methods such as file header identification, checksum verification, and format compliance checks.

[0123] Integrity verification covers the detection of the following three types of abnormal situations: First, corrupted files, which are files whose content is missing or whose structure is damaged due to transmission interruption, storage media failure, or write errors, and cannot be read normally by standard applications; Second, encrypted files, which are files that are protected by passwords or processed by commercial encryption mechanisms and whose content cannot be directly parsed without providing a decryption key; Third, files with unsupported formats, which are file types that are not in the preset whitelist, such as only supporting JPEG and PNG images, MP4 and AVI videos, PDF and DOCX documents, and WAV and MP3 audio, while receiving files with unknown extensions such as ".exe" or ".dat" is considered unsupported.

[0124] For unstructured data files that are determined to be any of the above-mentioned abnormal types after integrity verification, they will be removed from the list of files to be packaged, that is, excluded from the standardized data package and will not participate in subsequent uploading and sharing, so as to avoid transmitting invalid or unusable data to the group side, thereby ensuring the reliability of data flow and the stability of downstream processing.

[0125] Meanwhile, to enhance the transparency and traceability of the data processing, a filter list is generated by summarizing the relevant information of all rejected unstructured data files, and this list, along with the corresponding reasons for rejection, is written into the data table structure definition file. The filter list is embedded at the end of the data table structure definition file as a comment or additional metadata segment, for example, starting with "--Filtered Non-Structured Files:", and listing the file name, original path, rejection time, and reason line by line.

[0126] Through the above mechanism, on the one hand, invalid unstructured data can be effectively prevented from polluting standardized data packets, thus improving the success rate of data loading on the group side; on the other hand, by explicitly recording filtering information in the data table structure definition file, operation and maintenance personnel or data engineers can quickly understand the integrity status and potential missing items of the data packet content without accessing the original enterprise side system, which facilitates problem investigation and business decision-making.

[0127] Based on the above embodiments, the structured data export file adopts UTF-8 encoded CSV format and retains the original field order and null value representation.

[0128] During the process of generating standardized data packets based on a subset of target data on the enterprise side, structured data needs to be exported from the internal storage system into a universal file format that can be transmitted and parsed across platforms. To this end, this embodiment specifies that the exported structured data file adopts the UTF-8 encoded CSV format, that is, data rows are organized by comma-separated values, and the text content is serialized using the UTF-8 character encoding standard to support lossless expression of multilingual characters, special symbols, and industry terms.

[0129] Specifically, the CSV format, due to its simple structure and strong compatibility, is widely supported in various databases, data analysis tools, and large model training frameworks. Using UTF-8 encoding ensures that the file content will not suffer from garbled characters or missing characters under different operating systems or regional settings. For example, when the device status log contains the Chinese fault description "bearing overheat" or the German part name "Antriebswelle," UTF-8 encoding can completely preserve its semantic information, avoiding parsing errors caused by encoding inconsistencies.

[0130] Furthermore, to ensure the consistency of data semantics and the accuracy of downstream processing, this embodiment emphasizes preserving the original field order and the null value representation during the export process. On the one hand, preserving the field order allows the group side to directly map and load CSV content by column position after creating tables based on the data table structure definition file, without the need for additional field matching logic. On the other hand, the null value representation avoids semantic distortion introduced by value filling. For example, mistakenly filling in "0" for uncollected vibration values ​​may be misjudged as the equipment being stationary, while retaining it as a null value clearly indicates "no valid measurement".

[0131] This embodiment effectively solves common problems in cross-organizational data exchange, such as encoding conflicts, field misalignment, and misreading of null values, by adopting the CSV format encoded in UTF-8 and strictly preserving the original field order and null value representation. It significantly improves the accuracy, reproducibility, and automation of loading structured data on the group side.

[0132] Based on the above embodiments, further, after parsing the standardized data packets on the group side, before migrating the unstructured data files to the internal unstructured storage system, the hash value is first verified to be consistent with the record when the enterprise side uploaded the data, in order to prevent data tampering or damage during the transmission process.

[0133] Specifically, when generating standardized data packets on the enterprise side, a hash value is calculated for each unstructured data file, and this hash value is written into the metadata list of the standardized data packet. After downloading and parsing the standardized data packets on the group side, before writing the unstructured data files into the internal unstructured storage system, the hash value of each file is recalculated and compared with the original hash value recorded in the metadata list. If the two match, it is determined that the file has not been tampered with or corrupted during transmission and is allowed to be stored; otherwise, the file is rejected and a verification failure alarm is recorded to ensure that subsequent multimodal processing is based on a real and complete data source.

[0134] Based on the above embodiments, the multimodal processing module further supports a plug-in model loading mechanism, which allows dynamic switching of image recognition, speech analysis, or text understanding models according to business scenarios to adapt to different device types or industry terminology systems.

[0135] During the deep analysis of unstructured data files at the group level, the semantic understanding requirements vary across different industries, equipment types, and business scenarios. A single, fixed analysis model is insufficient to meet the diverse requirements for recognition accuracy and semantic coverage. Therefore, this embodiment introduces a plug-in model loading mechanism in the multimodal processing module. This mechanism encapsulates various AI analysis models into functional plug-ins that can be independently deployed, registered, and invoked, with the system dynamically loading them as needed.

[0136] Specifically, a model registry is first established to centrally manage all available analysis model plugins. Each plugin is clearly labeled with its applicable type, including but not limited to: image recognition models (used to identify components, states, or abnormal areas from images such as infrared thermal images and equipment exterior photos), voice analysis models (used to perform voice activity detection, keyword recognition, or acoustic anomaly classification on equipment operating audio or alarm recordings), and text understanding models (used to extract structured information such as equipment numbers, fault codes, and handling measures from documents such as fault reports and inspection records). In addition, each model plugin is also associated with its compatible equipment type (such as "high-speed rail bogie," "wind turbine gearbox," and "chemical reactor") and industry terminology system (such as "axle temperature alarm" in the railway industry, "sleeve oil leakage" in the power industry, and "main shaft vibration exceeding limits" in the manufacturing industry).

[0137] In actual processing, when the multimodal processing module receives an unstructured data file to be parsed, it first reads the business context information from its metadata, such as the subsidiary, equipment category, data source system, or task tag. Based on this context, it automatically matches and loads the most suitable model plugin from the model registry. For example, if the file originates from a rail transit subsidiary and the equipment type is "traction motor," then an image recognition model specifically trained on motor infrared images is loaded; if the file is a PDF maintenance log uploaded by a petrochemical plant, then a text understanding model with a built-in petroleum industry terminology library is invoked.

[0138] The plug-in mechanism in this embodiment supports dynamic switching at runtime without requiring service restarts or modifications to the core code. When a new device type is added or a new industry domain is entered, the corresponding new model plug-in only needs to be registered with the system for it to take effect immediately. Simultaneously, multiple model plug-ins can coexist in parallel, supporting the processing needs of mixed scenarios within the same dataset.

[0139] This embodiment, through the aforementioned plug-in model loading mechanism, achieves flexible expansion and precise adaptation of multimodal semantic parsing capabilities. It effectively solves the problems of insufficient generalization ability and high term misidentification rate of traditional fixed models in cross-device and cross-industry applications, significantly improves the accuracy and business relevance of enhanced metadata, and provides solid support for building high-quality, high-value large model training datasets.

[0140] Based on the above embodiments, the bidirectional dynamic association is further stored in the form of bidirectional edges in a graph database, wherein the structured data records and enhanced metadata are nodes to each other, and the edge attributes include association confidence, time alignment deviation and spatial distance.

[0141] After completing the parsing of standardized data packets, loading of structured data, and deep parsing of unstructured data on the group side, the structured data needs to be deeply integrated with the enhanced metadata generated by the multimodal processing module. To this end, this embodiment uses a graph database as the underlying storage engine, persistently storing bidirectional dynamic relationships in the form of bidirectional edges.

[0142] Specifically, each structured data record is modeled as a node in the graph database; simultaneously, each enhanced metadata record obtained from parsing unstructured data is also modeled as another independent node. When it is determined that the two originate from the same physical event or business scenario, a bidirectional edge is created between the two nodes, thus forming a symmetrical association structure of "structured data record ↔ enhanced metadata".

[0143] It is worth mentioning that the bidirectional edges in this embodiment not only indicate the existence of association, but also carry rich edge attributes, which are used to quantify and describe the quality and context of the association. The edge attributes include at least the following three items: (1) Association confidence, which represents the evaluation score of the correctness of the current association, usually ranging from 0.0 to 1.0, calculated by matching rules, model output probability or multi-source consistency, and used for subsequent retrieval ranking or confidence filtering; (2) Time alignment deviation, which represents the absolute difference between the time field in the structured data record and the timestamp inferred from the enhanced metadata, in seconds, reflecting the synchronization accuracy of the two types of data in the time dimension; (3) Spatial distance, which represents the physical or logical distance between the device location (such as GPS coordinates or workstation number) recorded in the structured data and the unstructured data collection location, in meters or hierarchical hops, used to measure spatial consistency.

[0144] For example, a bidirectional edge is established between a structured record "Car A recorded a speed of 80 rpm at 11:11:32 on 2024-XX-XX" and an enhanced metadata record "Video frame identifies that Car A performed an emergency stop operation at 11:11:33". The edge attributes may be: association confidence 0.92 (because the device ID is the same and the time is close), time alignment deviation 1 second, and spatial distance 0 meters (both belong to assembly line B segment).

[0145] This embodiment, through the aforementioned graph database storage method, not only achieves deep coupling between structured and unstructured semantic information but also preserves key metadata of the association process. This enables subsequent large-scale model training or intelligent analysis tasks to efficiently obtain cross-modal context based on graph traversal and utilize edge attributes for weighted fusion or anomaly detection. Furthermore, the bidirectional edge structure naturally supports dynamic updates. When enhanced metadata is corrected due to model iteration, only the corresponding nodes and edge attributes need to be updated, without reconstructing the entire association system, significantly improving the maintainability and evolutionary capability of the multimodal dataset.

[0146] Based on the above embodiments, when key business fields in structured data change, a re-annotation process for associated unstructured data is automatically triggered, and a versioned association is established between the newly generated enhanced metadata version and the original structured record.

[0147] Specifically, if operations personnel change the "fault level" of a structured data record from "medium" to "high" on the group side, after identifying that the field belongs to a critical business field, the process of re-labeling the unstructured data (such as fault images and videos) associated with that record is immediately triggered. The multimodal processing module calls the latest model to re-parse the unstructured data and generate a new version of enhanced metadata. Both the old and new versions of enhanced metadata are retained and associated with the same structured record through version numbers, forming a versioned association to ensure that the data evolution process is traceable and the analysis results are updated synchronously with the business context.

[0148] Based on the above embodiments, when the enhanced metadata of unstructured data is updated due to model upgrade, the associated structured data records are automatically marked as pending review, and change traceability information, including old labels, new labels and change time, is added to the dataset.

[0149] Specifically, when the new version of the image recognition model deployed on the group side generates an anomaly type label for a device's infrared image that differs from the old version (e.g., changing from "slight overheating" to "severe overheating"), the enhanced metadata is automatically marked as updated, and all related structured data records are searched, with their status fields set to "pending review." Simultaneously, a change traceability entry is added to the multimodal dataset, clearly recording the old label, new label, model version number, and change time. This ensures the timeliness and reliability of the multimodal dataset, preventing the loss of historical conclusions due to model iterations and the resulting lack of traceability.

[0150] Based on the above embodiments, the shared cloud drive further adopts an access control strategy based on organizational isolation, with standardized data packets from different subsidiaries uploaded to their respective exclusive subdirectories.

[0151] Specifically, shared cloud storage is a network storage space used for the secure exchange of standardized data packets between the enterprise and the group level, and it is configured with an access control policy based on organizational isolation. Under this policy, each subsidiary (such as Business Unit A, Branch Office B, or other local branches) is assigned a unique and exclusive subdirectory path (e.g., " / enterprise_A / ", " / enterprise_B / "), which only has the permission to upload data to its own subdirectory and read its own files, and cannot access the directory content of other subsidiaries.

[0152] Once a subsidiary generates a standardized data package, it is automatically uploaded to its corresponding dedicated subdirectory. The group-level entity has global read permissions and can retrieve data packages from each subdirectory for unified processing as needed. This effectively prevents cross-organizational data scrambling, unauthorized access, or accidental overwriting, ensuring the security and orderliness of data flow in a multi-tenant environment.

[0153] Based on the above embodiments, after forming a multimodal dataset on the group side, a unique global identifier is generated for each sample, and structured data, enhanced metadata, unstructured file paths, and bidirectional association relationships are packaged into sample units for training large models.

[0154] Specifically, after completing the loading of structured data, in-depth parsing of unstructured data, and construction of bidirectional dynamic associations, a unique global identifier is assigned to each fused sample in the multimodal dataset on the group side. This identifier is a globally unique and persistent string ID in the entire dataset, used for precise indexing and tracking of individual samples.

[0155] Subsequently, the original structured data contained in the sample, the enhanced metadata generated by the AI ​​model, the unstructured file path pointing to the original file, and the bidirectional relationship describing the logic between the two are integrated and encapsulated into a logically complete sample unit. The sample unit is the basic input unit for large model training tasks and is usually serialized in JSON, Parquet, or a custom binary format.

[0156] The sample units in this embodiment can be directly called by the large model training framework, ensuring that the model synchronously perceives numerical values, semantics, original media and related context during the training process, which significantly improves its understanding and generalization ability in complex tasks such as fault prediction and root cause analysis.

[0157] In some other embodiments, Figure 2 A schematic diagram of the overall flow of the multimodal data processing method provided in an embodiment of the present invention is shown.

[0158] like Figure 2 As shown, on the enterprise side, structured data (including time-series data and business structured data) and unstructured data are collected based on a unified data access interface. Then, multimodal content recognition is performed on the unstructured data to extract metadata, such as device identification information and timestamp information. The extracted metadata is then compared with fields in the structured data to establish preliminary associations. Next, structured and unstructured data that meet the criteria are extracted from the full set of preliminary association data to form a target data subset. Finally, a standardized data package is generated based on the target data subset and uploaded to a shared cloud drive.

[0159] In shared cloud storage, standardized data packages include data table structure definition files (DDL), structured data export files (CSV), and unstructured data files.

[0160] On the group side, data tables are created based on the data table structure definition file, and structured data export files are loaded into the created data tables. Simultaneously, the group-side multimodal processing module performs deep parsing on the unstructured data files to generate enhanced metadata. Subsequently, a bidirectional dynamic association is established between the enhanced metadata and the loaded structured data, integrating them to obtain a multimodal dataset for training large models.

[0161] Corresponding to the multimodal data processing methods described in the preceding embodiments, this invention also provides a multimodal data processing apparatus. Specifically, Figure 3 A schematic diagram of the structure of the multimodal data processing device provided in an embodiment of the present invention is shown.

[0162] like Figure 3 As shown, the device includes: a preliminary association module 310, used to receive structured and unstructured data from different sources based on a unified data access interface, and establish a preliminary association between the structured and unstructured data; a data packet generation module 320, used to filter out a target data subset that meets business needs on the enterprise side based on the preliminary association results, and generate a standardized data packet based on the target data subset; the standardized data packet includes a data table structure definition file, a structured data export file, and its associated unstructured data file; a data packet upload module 330, used to upload the standardized data packet to a shared cloud drive; and a bidirectional dynamic association module 340, used to download and parse the standardized data packet from the shared cloud drive on the group side, create a data table based on the data table structure definition file to load the structured data export file, and perform bidirectional dynamic association between the enhanced metadata extracted from the unstructured data file and the loaded structured data to form a multimodal dataset for large model training.

[0163] In this embodiment, the preliminary association module 310 receives structured and unstructured data from different sources through a unified data access interface and establishes a preliminary association between the structured and unstructured data. Subsequently, the data packet generation module 320 filters out a target data subset that meets business needs on the enterprise side based on the preliminary association results and generates a standardized data packet based on the target data subset. The standardized data packet includes a data table structure definition file, a structured data export file, and its associated unstructured data file. Next, the data packet upload module 330 uploads the standardized data packet to a shared network drive. Finally, the bidirectional dynamic association module 340 downloads and parses the standardized data packet on the group side, creates a data table based on the data table structure definition file to load the structured data export file, and performs a bidirectional dynamic association between the enhanced metadata extracted from the unstructured data file and the loaded structured data to form a multimodal dataset for large model training. This device enables the synchronous acquisition and preliminary association of structured and unstructured data through a unified access interface, effectively solving the problems of asynchronous timing, resource redundancy, and fusion difficulties caused by the separation and processing of multi-source heterogeneous data in existing technologies. At the same time, through standardized data packets and a shared network disk mechanism, it realizes cross-organizational, highly reliable, and low-manual-intervention data flow and bidirectional dynamic association between the enterprise side and the group side, significantly improving the efficiency, accuracy, and availability of multimodal data fusion in large model training.

[0164] It should be noted that the multimodal data processing device provided in the embodiments of the present invention and the multimodal data processing methods described in the above embodiments can be referred to each other, and will not be repeated here.

[0165] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communications bus 440. The processor 410 can call logic instructions in the memory 430 to execute a multimodal data processing method, which includes: receiving structured and unstructured data from different sources based on a unified data access interface, and establishing a preliminary association between the structured and unstructured data; selecting a target data subset that meets business needs on the enterprise side based on the preliminary association results, and generating a standardized data package based on the target data subset; the standardized data package includes a data table structure definition file, a structured data export file, and its associated unstructured data file; uploading the standardized data package to a shared cloud drive; downloading and parsing the standardized data package from the shared cloud drive on the group side, creating a data table based on the data table structure definition file to load the structured data export file, and performing a bidirectional dynamic association between the enhanced metadata extracted from the unstructured data file and the loaded structured data to form a multimodal dataset for large model training.

[0166] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multimodal data processing method provided by the above methods. The method includes: receiving structured and unstructured data from different sources based on a unified data access interface, and establishing a preliminary association between the structured and unstructured data; selecting a target data subset that meets business needs on the enterprise side based on the preliminary association results, and generating a standardized data package based on the target data subset; the standardized data package includes a data table structure definition file, a structured data export file, and its associated unstructured data file; uploading the standardized data package to a shared cloud drive; downloading and parsing the standardized data package from the shared cloud drive on the group side, creating a data table based on the data table structure definition file to load the structured data export file, and performing a bidirectional dynamic association between the enhanced metadata extracted from the unstructured data file and the loaded structured data to form a multimodal dataset for large model training.

[0168] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the multimodal data processing method provided by the above methods. The method includes: receiving structured and unstructured data from different sources based on a unified data access interface, and establishing a preliminary association between the structured and unstructured data; selecting a target data subset that meets business needs on the enterprise side based on the preliminary association results, and generating a standardized data package based on the target data subset; the standardized data package includes a data table structure definition file, a structured data export file, and its associated unstructured data file; uploading the standardized data package to a shared cloud drive; downloading and parsing the standardized data package from the shared cloud drive on the group side, creating a data table based on the data table structure definition file to load the structured data export file, and performing a bidirectional dynamic association between the enhanced metadata extracted from the unstructured data file and the loaded structured data to form a multimodal dataset for large model training.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal data processing method, characterized in that, include: Based on a unified data access interface, structured and unstructured data from different sources are received, and a preliminary association is established between the structured and unstructured data. Based on the preliminary correlation results, a target data subset that meets business needs is selected on the enterprise side, and a standardized data package is generated based on the target data subset; The standardized data package includes a data table structure definition file, a structured data export file, and its associated unstructured data file; Upload the standardized data packet to a shared cloud drive; On the group side, the standardized data package is downloaded and parsed from the shared cloud drive. A data table is created based on the data table structure definition file to load the structured data export file. The enhanced metadata extracted from the unstructured data file is dynamically correlated bidirectionally with the loaded structured data to form a multimodal dataset for training large models.

2. The multimodal data processing method according to claim 1, characterized in that, The initial association between the structured data and the unstructured data includes: Multimodal content recognition is performed on the unstructured data to extract the device identification information and timestamp information contained therein; The extracted device identification information is compared with the device identification field in the structured data. If the device identification is consistent, the extracted timestamp information is matched with the time field in the structured data to determine one or more structured data records with the smallest time difference as the associated objects.

3. The multimodal data processing method according to claim 1, characterized in that, The generation of standardized data packets based on the target data subset includes: Automatically analyze the table structure of the target data subset and generate a data table structure definition file; The structured data in the target data subset is exported as a structured data export file, and the unstructured data files associated with the records in the structured data export file are organized into a specified subdirectory according to a predefined directory structure to obtain the standardized data package.

4. The multimodal data processing method according to claim 1, characterized in that, The process of downloading and parsing standardized data packets from the shared cloud drive on the group side includes: The unstructured data files are migrated to an internally supported unstructured storage system. The multimodal processing module is invoked to perform deep parsing on the unstructured data file, generating enhanced metadata; The enhanced metadata includes one or more of the following: device identifier, timestamp, spatial location, device status, abnormal event type, and operation behavior category.

5. The multimodal data processing method according to claim 4, characterized in that, The step of dynamically associating the enhanced metadata extracted from the unstructured data file with the loaded structured data in a two-way manner includes: Based on the device identifier, timestamp, or spatial location jointly contained in the enhanced metadata and the loaded structured data, a bidirectional association is established in a graph database or relational database; The bidirectional association is used to enable the synchronization verification of associated unstructured data when the structured data is updated, or to enable the association status of the structured data record to be updated in reverse when the unstructured data label is changed.

6. The multimodal data processing method according to any one of claims 1-5, characterized in that, The structured data includes one or more of the following: train operation logs, sensor time-series data, and equipment status records. The unstructured data includes one or more of the following: monitoring images, infrared thermal images, fault report documents, operation manuals, alarm audio, and maintenance videos.

7. The multimodal data processing method according to claim 1, characterized in that, When receiving structured and unstructured data on the enterprise side, a unified scheduler is used to align the time windows for the collection tasks of structured and unstructured data.

8. The multimodal data processing method according to claim 1, characterized in that, The unified data access interface supports multiple protocols, including RESTful API, Kafka, and SFTP, for simultaneously accessing structured and unstructured data. The structured data includes at least temperature, pressure, rotational speed, equipment status logs, and GPS trajectory data collected by the equipment sensors, while the unstructured data includes at least infrared thermal images captured by the equipment monitoring camera, fault report documents uploaded by operators, and equipment operation audio.

9. The multimodal data processing method according to claim 2, characterized in that, When generating the standardized data packet, integrity checks are performed on the unstructured data file to remove corrupted, encrypted, or format-unsupported files, and a list of filtered files and the reasons are recorded in the data table structure definition file.

10. The multimodal data processing method according to claim 3, characterized in that, The structured data export file is in UTF-8 encoded CSV format and retains the original field order and null value representation.

11. The multimodal data processing method according to claim 4, characterized in that, After parsing standardized data packets on the group side, before migrating unstructured data files to the internal unstructured storage system, the hash value is first verified to be consistent with the record when the enterprise side uploaded the data, in order to prevent data tampering or damage during the transmission process.

12. The multimodal data processing method according to claim 4, characterized in that, The multimodal processing module supports a plug-in model loading mechanism, allowing dynamic switching between image recognition, speech analysis, or text understanding models based on business scenarios to adapt to different device types or industry terminology systems.

13. The multimodal data processing method according to claim 5, characterized in that, The bidirectional dynamic association is stored in the form of bidirectional edges in a graph database, where structured data records and enhanced metadata are nodes to each other, and edge attributes include association confidence, time alignment deviation, and spatial distance.

14. The multimodal data processing method according to claim 5, characterized in that, When key business fields in structured data change, a process for re-annotating associated unstructured data is automatically triggered, and the newly generated enhanced metadata version is linked to the original structured record in a versioned manner.

15. The multimodal data processing method according to claim 5, characterized in that, When the enhanced metadata of unstructured data is updated due to model upgrades, the associated structured data records are automatically marked as pending review, and change traceability information, including old labels, new labels, and change time, is added to the dataset.

16. The multimodal data processing method according to claim 1, characterized in that, The shared cloud drive adopts an access control policy based on organizational isolation, with standardized data packets from different subsidiaries uploaded to their respective exclusive subdirectories.

17. The multimodal data processing method according to claim 1, characterized in that, After forming a multimodal dataset on the group side, a unique global identifier is generated for each sample, and structured data, enhanced metadata, unstructured file paths, and bidirectional associations are packaged into sample units for training large models.

18. A multimodal data processing device, characterized in that, include: The preliminary association module is used to receive structured and unstructured data from different sources based on a unified data access interface, and to establish a preliminary association between the structured data and the unstructured data. The data packet generation module is used to filter out a target data subset that meets business needs on the enterprise side based on the preliminary association results, and generate a standardized data packet based on the target data subset; the standardized data packet includes a data table structure definition file, a structured data export file and its associated unstructured data file; The data packet upload module is used to upload the standardized data packets to the shared cloud drive; The bidirectional dynamic association module is used to download and parse the standardized data packet from the shared cloud drive on the group side, create a data table according to the data table structure definition file to load the structured data export file, and perform bidirectional dynamic association between the enhanced metadata extracted from the unstructured data file and the loaded structured data to form a multimodal dataset for large model training.

19. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal data processing method as described in any one of claims 1 to 17.

20. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multimodal data processing method as described in any one of claims 1 to 17.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the multimodal data processing method as described in any one of claims 1 to 17.

Images