An artificial intelligence-oriented industrial production multi-modal data publishing method and system

Abstract

This invention discloses a method and system for publishing multimodal data in industrial production for artificial intelligence, belonging to the field of industrial intelligence and data processing technology. The method includes: collecting multimodal data and performing hardware-level time synchronization and preprocessing to obtain a synchronized data stream; extracting multimodal features from the synchronized data stream in parallel and using an attention mechanism to fuse networks and generate a unified feature vector; constructing a five-dimensional quality assessment system and intelligently enhancing it; performing structured annotation and version management; adapting and publishing datasets according to AI tasks; implementing multi-level data anonymization and fine-grained access control, embedding invisible digital watermarks, and generating audit logs; and enabling multi-terminal access through a publishing platform. This invention solves the problems of difficult data alignment, poor quality, and low reusability, achieving secure data sharing and providing a high-quality, standardized data infrastructure for industrial AI applications.

Description

Technical Field

[0001] This invention relates to the technical field of industrial intelligence and data processing, and in particular to a method and system for publishing multimodal data in industrial production oriented towards artificial intelligence. Background Technology

[0002] With the deepening of Industry 4.0 and intelligent manufacturing, massive amounts of multimodal data are generated in industrial production processes, including sensor time-series data, visual image data, acoustic monitoring data, and text process logs. This multimodal data is a key resource for training industrial AI models (such as fault diagnosis, quality inspection, and process optimization), and related technologies have formed a preliminary technical system around the three major directions of "data acquisition," "data fusion," and "data openness."

[0003] In terms of data acquisition and preprocessing, existing solutions mainly rely on traditional industrial buses and general sensor networks to achieve real-time acquisition and storage of single-modal data. In recent years, with the development of edge computing technology, some solutions have attempted to perform preliminary cleaning and noise reduction at the data acquisition end, but these solutions mostly focus on single data types and lack a systematic design for time synchronization and heterogeneous alignment of multimodal data. In the field of multimodal data fusion, existing technologies mainly include three types of methods: early fusion, late fusion, and hybrid fusion. With the intervention of deep learning technology, feature fusion schemes based on neural networks have gradually emerged, achieving correlation modeling of multi-source data through attention mechanisms, graph neural networks, and other methods. However, these solutions are mostly geared towards general scenarios and have not been optimized for the complex situations unique to industrial sites, such as high noise, strong interference, and missing data, and their fusion efficiency is difficult to meet real-time requirements. In the field of data openness and release, existing industrial data platforms mostly adopt traditional database export and file sharing methods, lacking structured release standards for AI training. Although some advanced industrial internet platforms provide API interfaces, their data release formats are singular, lack metadata descriptions, and lack version management, resulting in low data reuse rates and difficulty in supporting flexible adaptation to different AI tasks.

[0004] In summary, the existing technology has the following obvious defects and shortcomings in practical applications: (i) Difficulty in aligning multimodal data and insufficient guarantee of time consistency. The sampling frequencies of different sensors in industrial fields vary greatly (kHz level for vibration sensors, Hz level for thermal imaging cameras), and the data formats are heterogeneous (binary stream, JSON, image, audio). Furthermore, the lack of hardware-level time synchronization mechanisms makes it difficult to accurately align multi-source data in terms of time sequence. Traditional solutions often rely on software timestamp interpolation, and the synchronization error often exceeds 10ms, which seriously affects the accuracy of correlation analysis and causal inference of multimodal data.

[0005] (ii) Inconsistent data quality and a lack of systematic evaluation and enhancement mechanisms. Industrial environments are complex, and data is susceptible to electromagnetic interference, dust obstruction, and temperature and humidity fluctuations, leading to noise interference, signal loss, and frequent outliers. Existing data cleaning methods are mostly based on simple threshold filtering or statistical imputation, lacking a deep understanding of the physical meaning and technological mechanisms of industrial data, making it difficult to accurately remove interference while retaining valid information. Furthermore, the imbalanced sample problem—scarce fault samples and redundant normal samples—directly affects the training effect and generalization ability of AI models.

[0006] (III) Data release lacks AI task adaptability and has a low reuse rate. Currently, industrial data release often adopts a "one-size-fits-all" approach, providing raw datasets that have not been adapted for specific tasks. Different AI tasks (such as classification, detection, and prediction) have significantly different requirements for data features, annotation granularity, and sample distribution. Data scientists need to invest a lot of time in feature re-extraction, sample re-screening, and format re-conversion before use, resulting in a data reuse rate of less than 30%. In addition, the lack of incremental release and version management mechanisms makes it impossible to support continuous learning and iterative optimization of models.

[0007] (iv) Data security and compliance risks constrain open sharing. Industrial data contains sensitive information such as core process parameters, equipment operating status, and production efficiency. Traditional data release methods are relatively weak in terms of security control, making it difficult to achieve fine-grained access control, dynamic data anonymization, and complete audit trails. Enterprises have many concerns when sharing data, and cross-departmental and cross-enterprise data collaboration is difficult to carry out, which restricts the construction of the industrial data ecosystem.

[0008] (v) Lack of end-to-end collaborative optimization, resulting in insufficient release of data value. Existing technical solutions mostly focus on single-point optimization, with data collection, processing, fusion, annotation, and publishing being isolated from each other, lacking end-to-end collaborative design. For example, data collection does not consider the feature requirements of subsequent AI training, and data annotation does not take into account the characteristics of the model architecture, resulting in low data engineering efficiency and long deployment cycles and high costs for AI applications. Summary of the Invention

[0009] The purpose of this invention is to provide a method and system for publishing multimodal data in industrial production for artificial intelligence. It ensures data consistency through hardware-level time synchronization, improves feature quality through adaptive fusion, solves sample imbalance through intelligent enhancement, improves data reuse rate through task adaptation, and promotes data openness through multi-level security control, thus providing a high-quality, standardized, safe and reliable data infrastructure for industrial AI applications.

[0010] To achieve the above objectives, this invention provides a method for publishing multimodal data in industrial production based on artificial intelligence, comprising the following steps: S1. Deploy multi-type sensor networks in industrial sites to collect multimodal data, and simultaneously collect process parameters and text logs from PLC, MES, and SCADA systems in real time through industrial protocol gateways; S2. Perform time synchronization on the multimodal data, and at the same time perform targeted preprocessing operations on the collected raw data to obtain a synchronized data stream; S3. Perform parallel multimodal feature extraction on the synchronous data stream, use an attention mechanism to fuse the network, dynamically allocate fusion weights according to the information entropy of each modality feature, and generate a unified feature vector; S4. Construct a five-dimensional quality assessment index system that includes completeness, consistency, accuracy, timeliness, and interpretability. Automatically classify the quality based on a machine learning classification model and trigger enhancement processes for data with lower quality. S5 provides multiple annotation modes, supports multiple types of annotation content, builds a data version management system, records data update information, and realizes full-link data traceability; S6. Dynamically construct training datasets based on downstream AI task types, support multiple data publishing formats and mainstream AI framework interfaces, and implement an incremental publishing mechanism; S7. Implement multi-level data desensitization, build a fine-grained access control system, embed invisible digital watermarks, and generate audit logs; S8. Through the publishing platform, it enables multi-terminal access, provides multiple types of API services, and builds an industrial AI data ecosystem service system.

[0011] Preferably, in step S2, time synchronization of multimodal data specifically includes: A hardware synchronization module based on the IEEE 1588 PTPv2 protocol is adopted to provide a unified time reference for all data sources, ensuring that the time alignment error of multimodal data is ≤1ms. Based on the event-triggered soft alignment mechanism, when equipment start-up / shutdown, fault alarm, or process switching events are detected, the time window alignment of multi-source data is automatically triggered. Correct for time deviations caused by network latency or equipment failure.

[0012] Preferably, in step S2, the targeted preprocessing operations performed on the collected raw data specifically include wavelet threshold denoising of vibration data, median filtering and Gaussian filtering of image data, spectral subtraction denoising of acoustic data, handling of missing values ​​through linear interpolation and KNN interpolation, anomaly detection and correction by combining the 3σ principle and the isolated forest algorithm, and storing the preprocessed raw data in a distributed cache system, supporting breakpoint resume, and outputting a time-synchronized synchronous data stream.

[0013] Preferably, in step S3, the parallel extraction of multimodal features from the synchronous data stream specifically includes: Extract time-domain, frequency-domain, and time-frequency-domain features from sensor data; A pre-trained convolutional neural network is used to extract deep features from visual data, which are then combined with traditional features to enhance interpretability. Extract MFCC features, spectral centroid, zero-crossing rate, and short-time energy from acoustic data; We extract word vectors from text data using a pre-trained language model for the industrial field, and then combine keyword weighting with topic modeling to extract key information.

[0014] Preferably, in step S4, the enhanced process specifically includes: A conditional generative adversarial network is used to generate diverse operating condition samples, expanding fault data and edge operating condition data; Simulation data is generated based on the process mechanism model to supplement samples of extreme working conditions and dangerous scenarios. Oversampling techniques are applied to address the imbalanced sample problem and construct a balanced training set.

[0015] Preferably, step S5 specifically includes manual annotation on a Web annotation platform, semi-automatic annotation based on pre-trained model suggestions, and fully automatic annotation based on rule engines and self-supervised learning, supporting four types of content: classification labels, regression values, spatial annotations, and temporal annotations.

[0016] Preferably, in step S6, the incremental publishing mechanism supports incremental data publishing by time window, by event trigger, and by version update. After a data change is detected, only the incremental part is published to avoid duplicate transmission.

[0017] Preferably, in step S7, the multi-level data desensitization specifically includes static desensitization, dynamic desensitization, and differential privacy protection. Static desensitization masks, hashes, and generalizes sensitive fields, dynamic desensitization dynamically hides sensitive fields based on user roles, and differential privacy protection adds controllable noise to protect statistical features. Fine-grained access control systems combine role-based access control and attribute-based access control. Role-based access control defines several levels of role permissions, while attribute-based access control dynamically authorizes access based on user attributes, environmental conditions, and data tags. Access to the corresponding data can only be authorized when both permissions are verified.

[0018] Preferably, it also includes a data-model bidirectional feedback enhancement step: Real-time collection of performance data of downstream AI models in practical applications, including recognition accuracy, false alarm rate, and generalization ability; Based on performance data, the model intelligently identifies the weak links in data quality. If the model's recognition accuracy under a specific working condition is lower than the preset threshold, the data characteristics under that working condition are analyzed to identify whether there are insufficient samples or missing features. Based on the recognition results, targeted data enhancement or supplementary collection is automatically triggered, forming a closed loop for continuous optimization of data quality.

[0019] A multimodal data publishing system for industrial production oriented towards artificial intelligence, comprising: The data acquisition and synchronization module is used to deploy sensor networks and perform hardware time synchronization and data acquisition. The feature extraction and fusion module is used to perform feature extraction and adaptive fusion on multimodal data; The quality assessment and enhancement module is used to perform data quality assessment and intelligent enhancement processing. The annotation and version management module is used to provide annotation services and perform data version control; The task adaptation and publishing engine module is used to dynamically build and publish datasets based on AI task requirements; The security control and access management module is used to perform data anonymization, access control, and security auditing. The platform and service module is used to provide multi-terminal access interfaces and data ecosystem services.

[0020] Therefore, the present invention employs the above-described method and system for publishing multimodal data in industrial production oriented towards artificial intelligence, which has the following advantages: (1) In this invention, a three-level time synchronization system is used. The first level adopts the IEEE 1588 PTPv2 hardware clock synchronization module to provide a nanosecond-level unified time reference for all field devices, ensuring that the timestamp error of devices with different sampling frequencies, such as vibration sensors, thermal imaging cameras, and ultrasonic detectors, is ≤1ms. The second level is based on an event-triggered soft alignment mechanism. When key events such as equipment start-up, shutdown, fault alarm, and process switching are detected, the time window alignment of multi-source data is automatically triggered to ensure accurate matching of event-related data. The third level adopts an adaptive interpolation algorithm to intelligently correct small time deviations caused by network latency or equipment failure. The error can be stably controlled within the range of 0.8-1.2ms, providing unprecedented time accuracy for subsequent feature fusion and causal analysis, enabling the AI ​​model to accurately capture the subtle correlation between equipment status and process parameters.

[0021] (2) In this invention, a five-dimensional quality assessment and closed-loop enhancement system is constructed. When the system detects that the number of certain fault samples is insufficient, the data enhancement process based on conditional generative adversarial network is automatically triggered. While preserving the physical mechanism, diverse simulation samples are generated to accelerate the training and convergence speed of the AI ​​model, improve the generalization accuracy, and reduce the false alarm rate.

[0022] (3) In this invention, the task feature matching algorithm based on deep learning automatically identifies AI task requirements and dynamically optimizes data features, sample distribution and annotation format to provide a high-quality training dataset that is "ready to use out of the box".

[0023] (4) In this invention, a three-level dynamic security control framework of "static desensitization + dynamic permissions + digital watermark" is designed to maximize data availability while protecting core secrets.

[0024] (5) In this invention, a data value quantification evaluation model is established by enhancing the closed loop through data-model bidirectional feedback, and the intelligent analysis system that back-infers data optimization from model performance is used to achieve continuous optimization of data quality and spiral growth of business value.

[0025] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a multimodal data publishing method for industrial production oriented towards artificial intelligence, according to the present invention. Figure 2 This is a schematic diagram of a three-level dynamic security control for secure sharing of industrial data provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Specific model specifications need to be selected and determined according to the actual specifications of the device, etc. The specific selection calculation method adopts existing technology in the art, and therefore will not be described in detail.

[0028] Example like Figure 1 As shown, this invention provides a method for publishing multimodal data in industrial production oriented towards artificial intelligence, comprising the following steps: S1. Deploy a multi-type sensor network in the industrial field to collect multi-modal data. The multi-sensor network includes vibration sensors (sampling frequency ≥10kHz), temperature sensors (range -40℃~200℃), high-definition industrial cameras (resolution ≥1920×1080), infrared thermal imagers, ultrasonic sensors, noise sensors, etc. At the same time, process parameters and text logs of PLC (Programmable Logic Controller), MES (Manufacturing Execution System), and SCADA (Supervisory Control and Data Acquisition) systems are collected in real time through an industrial protocol gateway (supporting Modbus, OPC UA, MQTT, Profinet, etc.). S2. Time synchronization of multimodal data is achieved using a hardware synchronization module based on the IEEE 1588 PTPv2 protocol. This provides a unified time reference for all data sources, ensuring that the time alignment error of multimodal data is ≤1ms. Based on an event-triggered soft alignment mechanism, time window alignment of multi-source data is automatically triggered when equipment start-up / shutdown, fault alarm, or process switching events are detected, correcting time deviations caused by network latency or equipment failure. Simultaneously, targeted preprocessing operations are performed on the collected raw data. These operations include wavelet threshold denoising for vibration data, median and Gaussian filtering for image data, and spectral subtraction denoising for acoustic data. Missing values ​​are handled through linear interpolation and KNN interpolation. Anomaly detection and correction are achieved by combining the 3σ principle and the isolated forest algorithm. The preprocessed raw data is stored in a distributed cache system, supporting breakpoint resumption, and outputting a time-synchronized synchronous data stream. According to on-site industrial measurements, in a typical workshop environment with an electromagnetic interference intensity of 50V / m, the multimodal data alignment error of traditional solutions is 8-15ms, while this solution can stably control the error within the range of 0.8-1.2ms, providing unprecedented time accuracy for subsequent feature fusion and causal analysis, enabling AI models to accurately capture the subtle correlation between equipment status and process parameters.

[0029] S3. Perform parallel multimodal feature extraction on the synchronous data stream. Specifically, extract time-domain, frequency-domain, and time-frequency-domain features from sensor data; extract deep features from visual data using a pre-trained convolutional neural network, and enhance interpretability by combining traditional features; extract MFCC (Mel frequency cepstral coefficients), spectral centroid, zero-crossing rate, and short-time energy from acoustic data; and extract word vectors from text data based on a pre-trained language model in the industrial field, combine keyword weighting and topic modeling to extract key information, and use an attention mechanism to fuse the network, dynamically allocating fusion weights according to the information entropy of each modality feature to generate a 384-dimensional unified feature vector. S4. Construct a five-dimensional quality assessment index system including completeness (missing rate ≤5%), consistency (time alignment error ≤10ms), accuracy (sensor calibration error ≤1%), timeliness (collection to release delay ≤1s), and interpretability (expert score ≥4 points). Automatic quality grading is performed based on a machine learning classification model. In this embodiment, a comprehensive quality score is calculated based on a rule engine and an XGBoost classification model, categorized into four levels: Excellent (score ≥90), Good (80 ≤ score <90), Average (70 ≤ score <80), and Poor (score <70). For data at the Average and Poor levels, an enhancement process is triggered, including using a Conditional Generative Adversarial Network (CGAN) to generate diverse operating condition samples, expanding fault and edge operating condition data; generating simulation data based on a process mechanism model to supplement extreme operating condition and hazardous scenario samples; applying oversampling techniques such as SMOTE and ADASYN to address sample imbalance issues; and constructing a balanced training set. When the system detects an insufficient number of samples for a certain type of fault (e.g., early bearing crack samples account for only 0.5% of the total samples), it automatically triggers a data augmentation process based on conditional generative adversarial networks. This generates diverse simulation samples while preserving the physical mechanisms, increasing the amount of data for small sample categories by 5-8 times. Simultaneously, addressing the common "dirty data" problem in industrial data, the system can intelligently identify and correct outliers based on historical operating patterns and an expert knowledge base, avoiding information loss caused by simple deletion. Practical applications show that after adopting this solution, the training convergence speed of the AI ​​model is accelerated by 40%, the generalization accuracy under unknown operating conditions is improved by 18-32%, and the false alarm rate is reduced by more than 60%.

[0030] S5 offers three annotation modes: manual annotation on a web annotation platform, semi-automatic annotation based on suggestions from pre-trained models such as Mask R-CNN, and fully automatic annotation based on rule engines and self-supervised learning. It supports four types of content: classification labels (fault type, quality level), regression values ​​(remaining lifetime, energy consumption), spatial annotations (bounding boxes, semantic segmentation), and temporal annotations (event start and end times, abnormal periods). It builds a data version management system based on DVC (Data Version Control) and Git, recording the collection parameters, preprocessing methods, annotation versions, and enhancement strategies for each data update. It records data update information and enables full-link data traceability, visually displays the complete data process from collection to publication, and supports problem tracing and root cause analysis.

[0031] S6. Dynamically construct training datasets based on downstream AI task types (classification, detection, regression, anomaly recognition). (Classification tasks automatically select relevant features and balanced samples; detection tasks automatically generate bounding box annotation files; time-series tasks automatically construct sliding window sample sequences.) Supports five data publishing formats: TFRecord (adapted to TensorFlow), HDF5 (adapted to scientific research analysis), CSV (adapted to traditional machine learning), JSON-LD (adapted to knowledge graphs), and Parquet (adapted to big data platforms), as well as interfaces to mainstream AI frameworks (TensorFlow Dataset interface, PyTorch DataLoader interface, MindSporeDataset interface, PaddlePaddle Dataset interface). Implements an incremental publishing mechanism that supports incremental data publishing by time window, by event trigger, and by version update. After detecting data changes, only the incremental part is published to avoid duplicate transmission.

[0032] S7. Implement three-layer data masking, including static masking, dynamic masking, and differential privacy protection. Static masking masks, hashes, and generalizes sensitive fields; dynamic masking dynamically hides sensitive fields based on user roles; and differential privacy protection adds controllable noise to protect statistical features. A fine-grained access control system is constructed, combining role-based access control (RBAC) and attribute-based access control (APAC). RBAC defines the permissions of roles such as administrators, researchers, engineers, and visitors. AAPAC dynamically authorizes access based on user attributes, environmental conditions, and data tags. Access to the corresponding data is only authorized when both authorization verifications pass. An invisible digital watermark is embedded to track data usage paths, allowing for tracing leaks back to specific users and times. All data access, download, and modification operations are recorded, generating a complete audit log that supports compliance checks and report generation for GDPR, Cybersecurity Classified Protection 2.0, and ISO / IEC 27001.

[0033] like Figure 2As shown, this solution innovatively designs a three-tiered dynamic security control framework to maximize data availability while protecting core confidentiality: The first layer is the data anonymization layer, employing a dual mechanism of "static generalization + dynamic hiding" to classify sensitive information such as equipment serial numbers, process formulas, and production capacity data. Public datasets retain only generalized information such as equipment type and process category; collaborative R&D datasets provide detailed anonymized parameters; internal datasets retain complete information but implement strict access control. The second layer is the access control layer, based on a three-dimensional authorization model of "role + attribute + environment." It not only considers the user's role (researcher / engineer / administrator) but also dynamically adjusts permissions based on data sensitivity level, usage scenario, and time factors. For example, the same user can access complete data during normal working hours but can only access anonymized data during non-working hours. The third layer is the traceability and auditing layer, which embeds unique identifiers at each stage of data release using invisible digital watermarking technology, achieving traceability and auditability throughout the entire data usage process.

[0034] In a real-world deployment at a large manufacturing enterprise, the three-tiered dynamic security control framework of this invention expanded the scope of enterprise data sharing from the original three internal departments to ecosystem partners including eight suppliers and two universities, improving data utilization efficiency by 300% while achieving 100% traceability of security incidents and preventing any data breaches.

[0035] S8. Develop a responsive web platform that supports access from multiple devices including PCs, tablets, and mobile phones, providing functions such as data retrieval, visualization, online annotation, and batch download; offer multiple API services: RESTful API (standard data query), GraphQL API (flexible field selection), WebSocket (real-time data push), and OAuth 2.0 authentication; build a data ecosystem service system: establish an industrial AI benchmark model library (including pre-trained weights and fine-tuning scripts), host online algorithm competitions and leaderboards, and build a technical community to promote data co-construction and model sharing; provide complete data usage documentation, tutorial videos, API references, and best practice cases to lower the barrier to entry for users.

[0036] It also includes a two-way data-model feedback enhancement step: In the forward process, high-quality multimodal data, after intelligent publishing, supports the training and optimization of the AI ​​model; in the feedback process, the model's performance data in practical applications (such as recognition accuracy, false positives, and generalization ability) is collected in real time and fed back to the data publishing system. Based on this feedback information, the system intelligently identifies weaknesses in data quality. If the model's recognition accuracy is low under certain conditions, the system automatically analyzes the data characteristics under those conditions to identify problems such as insufficient samples, missing features, or noise interference, and then triggers targeted data augmentation or supplementary data collection. More importantly, the system establishes a data value quantification assessment model, which can evaluate the value contribution of different data sources, features, and annotations to the final AI application from multiple dimensions such as accuracy contribution, efficiency improvement, and cost savings, providing a quantitative basis for enterprise data asset management and investment decisions. During the 12-month operation cycle of a smart factory, the closed-loop system, through continuous data optimization, gradually increased the accuracy of fault prediction for key equipment from the initial 87% to 96%, extended the mean time between failures by 35%, and reduced maintenance costs by 28%, achieving a double helix growth of data value and business value.

[0037] A multimodal data publishing system for industrial production oriented towards artificial intelligence, comprising: The data acquisition and synchronization module is used to deploy sensor networks and perform hardware time synchronization and data acquisition. The feature extraction and fusion module is used to perform feature extraction and adaptive fusion on multimodal data; The quality assessment and enhancement module is used to perform data quality assessment and intelligent enhancement processing. The annotation and version management module is used to provide annotation services and perform data version control; The task adaptation and publishing engine module is used to dynamically build and publish datasets based on AI task requirements; The security control and access management module is used to perform data anonymization, access control, and security auditing. The platform and service module is used to provide multi-terminal access interfaces and data ecosystem services.

[0038] Therefore, this invention employs the aforementioned method and system for publishing multimodal industrial production data oriented towards artificial intelligence, constructing an integrated industrial multimodal data publishing system encompassing "collection-fusion-evaluation-labeling-publishing-security," overcoming the limitations of existing technologies such as "fragmented processes, poor adaptability, and weak security." Through hardware-level time synchronization to ensure data consistency, adaptive fusion to improve feature quality, intelligent enhancement to address sample imbalance, task adaptation to increase data reuse rate, and multi-level security control to promote data openness, it provides a high-quality, standardized, secure, and reliable data infrastructure for industrial AI applications.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for publishing multimodal data in industrial production oriented towards artificial intelligence, characterized in that: Includes the following steps: S1. Deploy multi-type sensor networks in industrial sites to collect multimodal data, and simultaneously collect process parameters and text logs from PLC, MES, and SCADA systems in real time through industrial protocol gateways; S2. Perform time synchronization on the multimodal data, and at the same time perform targeted preprocessing operations on the collected raw data to obtain a synchronized data stream; S3. Perform parallel multimodal feature extraction on the synchronous data stream, use an attention mechanism to fuse the network, dynamically allocate fusion weights according to the information entropy of each modality feature, and generate a unified feature vector; S4. Construct a five-dimensional quality assessment index system that includes completeness, consistency, accuracy, timeliness, and interpretability. Automatically classify the quality based on a machine learning classification model and trigger enhancement processes for data with lower quality. S5 provides multiple annotation modes, supports multiple types of annotation content, builds a data version management system, records data update information, and realizes full-link data traceability; S6. Dynamically construct training datasets based on downstream AI task types, support multiple data publishing formats and mainstream AI framework interfaces, and implement an incremental publishing mechanism; S7. Implement multi-level data desensitization, build a fine-grained access control system, embed invisible digital watermarks, and generate audit logs; S8. Through the publishing platform, it enables multi-terminal access, provides multiple types of API services, and builds an industrial AI data ecosystem service system.

2. The method for publishing multimodal data in industrial production oriented towards artificial intelligence according to claim 1, characterized in that: Step S2, specifically, includes time synchronization of multimodal data: A hardware synchronization module based on the IEEE 1588 PTPv2 protocol is adopted to provide a unified time reference for all data sources, ensuring that the time alignment error of multimodal data is ≤1ms. Based on the event-triggered soft alignment mechanism, when equipment start-up / shutdown, fault alarm, or process switching events are detected, the time window alignment of multi-source data is automatically triggered. Correct for time deviations caused by network latency or equipment failure.

3. The method for publishing multimodal data in industrial production oriented towards artificial intelligence according to claim 2, characterized in that: In step S2, targeted preprocessing operations are performed on the collected raw data, including wavelet threshold denoising for vibration data, median filtering and Gaussian filtering for image data, spectral subtraction denoising for acoustic data, handling of missing values ​​through linear interpolation and KNN interpolation, anomaly detection and correction by combining the 3σ principle and the isolated forest algorithm, and storing the preprocessed raw data in a distributed cache system, supporting breakpoint resume and outputting a time-synchronized synchronous data stream.

4. The method for publishing multimodal data in industrial production oriented towards artificial intelligence according to claim 3, characterized in that: Step S3, specifically the parallel extraction of multimodal features from the synchronous data stream, includes: Extract time-domain, frequency-domain, and time-frequency-domain features from sensor data; A pre-trained convolutional neural network is used to extract deep features from visual data, which are then combined with traditional features to enhance interpretability. Extract MFCC features, spectral centroid, zero-crossing rate, and short-time energy from acoustic data; We extract word vectors from text data using a pre-trained language model for the industrial field, and then combine keyword weighting with topic modeling to extract key information.

5. The method for publishing multimodal data in industrial production oriented towards artificial intelligence according to claim 4, characterized in that: In step S4, the enhanced process specifically includes: A conditional generative adversarial network is used to generate diverse operating condition samples, expanding fault data and edge operating condition data; Simulation data is generated based on the process mechanism model to supplement samples of extreme working conditions and dangerous scenarios. Oversampling techniques are applied to address the imbalanced sample problem and construct a balanced training set.

6. The method for publishing multimodal data in industrial production oriented towards artificial intelligence according to claim 5, characterized in that: Step S5 specifically includes manual annotation on the Web annotation platform, semi-automatic annotation based on pre-trained model suggestions, and fully automatic annotation based on rule engines and self-supervised learning, supporting four types of content: classification labels, regression values, spatial annotations, and temporal annotations.

7. A method for publishing multimodal data in industrial production oriented towards artificial intelligence, as described in claim 6, characterized in that: In step S6, the incremental release mechanism supports incremental data release by time window, by event trigger, and by version update. After a data change is detected, only the incremental part is released to avoid duplicate transmission.

8. A method for publishing multimodal data in industrial production oriented towards artificial intelligence, as described in claim 7, characterized in that: In step S7, multi-level data desensitization specifically includes static desensitization, dynamic desensitization, and differential privacy protection. Static desensitization masks, hashes, and generalizes sensitive fields. Dynamic desensitization dynamically hides sensitive fields based on user roles. Differential privacy protection adds controllable noise to protect statistical features. Fine-grained access control systems combine role-based access control and attribute-based access control. Role-based access control defines several levels of role permissions, while attribute-based access control dynamically authorizes access based on user attributes, environmental conditions, and data tags. Access to the corresponding data can only be authorized when both permissions are verified.

9. A method for publishing multimodal data in industrial production oriented towards artificial intelligence, as described in claim 8, characterized in that: It also includes a data-model bidirectional feedback enhancement step: Real-time collection of performance data of downstream AI models in practical applications, including recognition accuracy, false alarm rate, and generalization ability; Based on performance data, the model intelligently identifies the weak links in data quality. If the model's recognition accuracy under a specific working condition is lower than the preset threshold, the data characteristics under that working condition are analyzed to identify whether there are insufficient samples or missing features. Based on the recognition results, targeted data enhancement or supplementary collection is automatically triggered, forming a closed loop for continuous optimization of data quality.

10. A multimodal data publishing system for industrial production oriented towards artificial intelligence, used to implement the multimodal data publishing method for industrial production oriented towards artificial intelligence as described in any one of claims 1-9, characterized in that: include: The data acquisition and synchronization module is used to deploy sensor networks and perform hardware time synchronization and data acquisition. The feature extraction and fusion module is used to perform feature extraction and adaptive fusion on multimodal data; The quality assessment and enhancement module is used to perform data quality assessment and intelligent enhancement processing. The annotation and version management module is used to provide annotation services and perform data version control; The task adaptation and publishing engine module is used to dynamically build and publish datasets based on AI task requirements; The security control and access management module is used to perform data anonymization, access control, and security auditing. The platform and service module is used to provide multi-terminal access interfaces and data ecosystem services.

Images