Multimodal production line state inference method, system, device, and medium

Abstract

The application relates to the technical field of artificial intelligence, and particularly provides a multi-modal production line state reasoning method, system, device and medium, which comprises the following steps: collecting multi-modal sensor data generated by multiple devices on a production line; performing timestamp alignment on the multi-modal sensor data from different devices, and fusing to generate a multi-dimensional time sequence feature matrix; inputting the multi-dimensional time sequence feature matrix into a pre-deployed lightweight prediction model in an edge processor, and outputting a production line state prediction result; wherein the lightweight prediction model is obtained by compressing a teacher model deployed on a cloud through a knowledge distillation method; and the lightweight prediction model comprises a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network. The application reduces the response time, can accurately locate a cross-device conductive fault, and is significantly superior to a traditional independent device diagnosis mode.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a multimodal production line state reasoning method, system, equipment, and medium. Background Technology

[0002] In industrial manufacturing scenarios, production lines typically consist of multiple key pieces of equipment operating in tandem (such as CNC machine tools, robotic arms, and conveyor systems). Existing monitoring systems mostly adopt a single-device independent monitoring mode, collecting only single physical quantity parameters such as vibration and temperature, and triggering alarms by setting fixed thresholds.

[0003] This traditional method has three significant drawbacks: First, single-modal sensor data cannot comprehensively characterize the equipment's operating status; for example, current fluctuations in a robotic arm need to be analyzed in conjunction with visual positioning data. Second, threshold alarm mechanisms have a severe lag in identifying complex faults; by the time parameters reach the threshold, production line shutdowns have often already occurred. Third, existing methods do not consider the process coupling relationships between equipment (such as material flow timing dependencies), leading to the neglect of abnormal transmission effects between equipment during fault localization. Furthermore, while complex cloud-based analysis models can handle multi-source data, they are limited by network latency and edge computing resource constraints, making it difficult to meet the high real-time requirements of operational condition monitoring.

[0004] The aforementioned defects result in a high false alarm rate and a long average fault response time in the current production line status monitoring. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of the prior art, the present invention provides a multimodal production line state reasoning method, system, equipment and medium to solve the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a multimodal production line state reasoning method, comprising: Collect multimodal sensor data generated by multiple devices on the production line; Timestamp alignment is performed on multimodal sensor data from different devices, and the data is fused to generate a multidimensional time-series feature matrix; The multidimensional temporal feature matrix is ​​input into a lightweight prediction model pre-deployed in the edge processor, and the production line status prediction result is output. The lightweight prediction model is obtained by compressing a teacher model deployed in the cloud through a knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network.

[0007] In one optional implementation, multimodal sensor data generated by multiple devices on the production line are collected, including: Multimodal sensors are pre-deployed on multiple devices in the production line and communication connections are established with the multimodal sensors on multiple devices; the multimodal sensors include industrial cameras, acoustic sensor arrays, and infrared thermal imagers.

[0008] In one optional implementation, multimodal sensor data from different devices are timestamped and fused to generate a multidimensional temporal feature matrix, including: For multimodal sensor data from a single device, significant event points are extracted for each modality, and the time difference between significant event points across modalities is added as a penalty term to the dynamic time warping loss function. The dynamic time warping loss function is then used to align the timestamps of the multimodal sensor data from a single device. The timestamps of each device are mapped to a low-dimensional manifold space, the relative time relationship between devices is preserved using t-SNE, and the timestamps between devices are aligned using a pre-constructed joint optimization loss function and the relative time relationship between devices. A 3D tensor is constructed using aligned multimodal sensor data from different devices, and core features are extracted from the 3D tensor using Tucker to obtain a multidimensional temporal feature matrix.

[0009] In an optional implementation, for multimodal sensor data from a single device, salient event points are extracted for each modality, and the time difference between salient event points across modalities is added as a penalty term to a dynamic time warping loss function. The dynamic time warping loss function is then used to align the timestamps of the multimodal sensor data from the single device, including: Detect instantaneous abrupt changes in each modality's data and mark the data points corresponding to instantaneous abrupt changes exceeding a preset abrupt change threshold as significant event points; Based on salient event points in each modal data, cross-modal event anchor pairs are established between different modal data, and the alignment path is forced to pass through the cross-modal event anchor pairs. ); Construct a dynamic time warping loss function:

[0010] Where π is the alignment path and K is the number of matched event anchor pairs. For the data point corresponding to time point i in the first mode, λ represents the data point corresponding to time point j in the second modality; λ is a parameter dynamically adjusted through adversarial learning. The optimal cross-modal event anchor pair is determined by minimizing the dynamic time warping loss function; Based on the optimal cross-modal event anchor pair, the timestamps of a device's multimodal sensor data are aligned using interpolation.

[0011] In an optional implementation, the timestamps of each device are mapped to a low-dimensional manifold space, t-SNE is used to preserve the relative time relationships between devices, and the timestamps between devices are aligned using a pre-constructed joint optimization loss function and the relative time relationships between devices, including: Map the device timestamp to the manifold space to obtain low-dimensional coordinates; Calculate the manifold time offset between low-dimensional coordinates of different devices; The manifold space is adjusted by Procrustes analysis to make the reference event points of all devices coincide. The low-dimensional coordinates of each device are calibrated after adjustment, and the calibrated low-dimensional coordinates are then converted to a global time reference.

[0012] In one optional implementation, the teacher model includes: The three-dimensional multimodal feature extraction layer includes an input layer, a 3D convolutional kernel group, and spatiotemporal pooling. The input layer is used to receive raw sensor data. The 3D convolutional kernel group adopts grouped 3D convolution, with each group processing different modal features. The spatiotemporal pooling includes dynamic temporal pooling and spatial max pooling. The dynamic knowledge graph embedding layer is used to determine the topology embedding vector of each device based on the device's 3D coordinates and the connection relationships between devices using a graph embedding network; it uses spatiotemporal graph convolution to determine the device state propagation risk encoding based on historical fault propagation paths; and it uses a tree-structured LSTM network to determine the device's process compliance characteristics based on a process dependency tree. A multi-head cross-attention fusion layer is used to associate the features extracted by the 3D multimodal feature extraction layer with the features extracted by the dynamic knowledge graph embedding layer to obtain spatiotemporal-graph joint features; The spatiotemporal graph prediction layer is used to determine the production line status based on spatiotemporal-graph joint features using a spatiotemporal gated graph network.

[0013] In an optional implementation, the method further includes: The prediction results and corresponding multimodal sensor data are pushed to the cloud via a message queue to fine-tune the teacher model in the cloud.

[0014] Secondly, the present invention provides a multimodal production line state reasoning system, comprising: The data acquisition module is used to collect multimodal sensor data generated by multiple devices on the production line. The preprocessing module is used to align the timestamps of multimodal sensor data from different devices and fuse them to generate a multidimensional time-series feature matrix. The prediction module is used to input the multidimensional time-series feature matrix into a lightweight prediction model pre-deployed in the edge processor and output the production line status prediction result. The lightweight prediction model is obtained by compressing a teacher model deployed in the cloud through a knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network.

[0015] Thirdly, a device is provided, comprising: Memory, used to store multimodal production line state reasoning programs; A processor, used to implement the steps of the multimodal production line state reasoning method as provided in the first aspect when executing the multimodal production line state reasoning program.

[0016] Fourthly, a computer-readable medium is provided, on which a multimodal production line state reasoning program is stored, wherein when the multimodal production line state reasoning program is executed by a processor, the steps of the multimodal production line state reasoning method as provided in the first aspect are implemented.

[0017] The beneficial effects of this invention are as follows: The multimodal production line state reasoning method, system, equipment, and medium provided by this invention, by fusing multimodal data from multiple devices and aligning them with timestamps, constructs a multidimensional matrix containing spatiotemporal correlation features between devices, solving the problem of data dimension fragmentation in traditional single-device threshold detection; secondly, by employing knowledge distillation technology to transfer the capabilities of the cloud-based teacher model to a lightweight edge model, the model reasoning time is reduced while ensuring prediction accuracy, meeting the needs of real-time monitoring; finally, through the LSTM-GAN joint network architecture, LSTM is used to capture the temporal dependencies between devices, while GAN adversarial training enhances the model's ability to identify latent abnormal patterns, reducing response time and accurately locating cross-device transmissive faults, which is significantly better than the traditional independent device diagnosis mode.

[0018] Furthermore, the design principle of this invention is reliable, the structure is simple, and it has a very wide range of application prospects. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0022] Figure 3This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0025] The multimodal production line state reasoning method provided in this embodiment of the invention is executed by a computer device, and correspondingly, the multimodal production line state reasoning system runs in the computer device.

[0026] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The executing entity can be a multimodal production line state reasoning system. Depending on different requirements, the order of steps in this flowchart can be changed, and some steps can be omitted.

[0027] like Figure 1 As shown, the method includes: S1. Collect multimodal sensor data generated by multiple devices on the production line; S2. Timestamp-align multimodal sensor data from different devices and fuse them to generate a multidimensional time-series feature matrix; S3. Input the multidimensional temporal feature matrix into a lightweight prediction model pre-deployed in the edge processor and output the production line status prediction result; The lightweight prediction model is obtained by compressing a teacher model deployed in the cloud through a knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network.

[0028] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0029] Multimodal sensors are pre-deployed on multiple devices in the production line and communication connections are established with the multimodal sensors on multiple devices; the multimodal sensors include industrial cameras, acoustic sensor arrays, and infrared thermal imagers.

[0030] Before deploying multimodal sensors, a comprehensive evaluation of multiple pieces of equipment on the production line is necessary. Based on equipment type (e.g., CNC machine tools, robotic arms, conveyor belts), operating conditions (speed, temperature, vibration range), and monitoring requirements (fault warning, performance optimization), a suitable sensor combination should be determined. For example, CNC machine tools, which focus on monitoring tool wear and spindle vibration, can be equipped with high-resolution industrial cameras and high-precision accelerometers; robotic arms, which need to monitor joint temperature and motion trajectory, are best suited for infrared thermal imagers and inertial measurement units.

[0031] Industrial cameras: Select area scan cameras with a frame rate of no less than 120fps and a resolution of 2000×1500 pixels or higher, equipped with autofocus and optical image stabilization, and supporting GigEVision or CameraLink high-speed interfaces to ensure clear imaging of high-speed moving parts.

[0032] Acoustic sensor array: It adopts an 8-channel or higher MEMS microphone array with a frequency response range of 20Hz-20kHz. It has beamforming and noise suppression algorithms, which can accurately locate the sound source and realize the capture and analysis of abnormal noise of the equipment.

[0033] Infrared thermal imager: Uses an uncooled focal plane array (FPA) detector with a resolution of no less than 320×240 pixels, a temperature measurement range of -20℃ to 500℃, a temperature measurement accuracy of ±2% or ±2℃, and supports real-time temperature data output and thermal image generation.

[0034] Industrial cameras: Based on the key monitoring parts of the equipment (such as the cutting area of ​​the tool, the surface of the conveyor belt), a bracket is used to fix the camera to ensure that the optical axis of the camera is perpendicular to the monitoring surface, and the working distance is kept at 0.5-3 meters to avoid obstruction and reflection interference.

[0035] Acoustic sensor array: Arranged in a ring or straight line around the device, 0.3-1 meter away from the device surface, with the center of the array aligned with the main sound-emitting area of ​​the device, and windproof covers and shock-absorbing pads are used to reduce the impact of environmental noise and mechanical vibration.

[0036] Infrared thermal imager: Installed above or to the side of the equipment to ensure that the thermal imager's field of view covers the key heat-generating components of the equipment (such as motors and bearings), avoiding interference from high-temperature radiation and environmental heat sources. An infrared filter is added to the lens to improve the accuracy of temperature measurement.

[0037] All sensors must be equipped with housings with an IP67 or higher protection rating to withstand harsh environments such as high temperatures, humidity, and dust on the production line. Sensor cables are shielded, with waterproof aviation connectors at the joints. The cables are protected by metal cable trays and conduits to prevent electromagnetic interference and mechanical damage.

[0038] Construct a star topology network with industrial Ethernet as the backbone, and employ redundant dual-port switches (such as Siemens SCALANCE X series) to ensure communication reliability. Sensors access the network via Ethernet, RS-485, or wireless communication (such as 5G, Wi-Fi 6), and critical equipment is equipped with dual-link backup to avoid single points of failure.

[0039] Industrial cameras use GigE Vision or GenICam standard protocols to achieve high-speed transmission of image data and cross-platform compatibility.

[0040] The acoustic sensor array uses the TCP / IP protocol to encapsulate audio data, and works with the Real-Time Transport Protocol (RTP) to ensure low latency and integrity of the audio stream.

[0041] Infrared thermal imagers upload temperature data via Modbus TCP or OPC UA protocols, supporting seamless integration with industrial automation systems.

[0042] Deploy a network time synchronization protocol (NTP or IEEE 1588 Precision Clock Protocol) to ensure time consistency of data acquisition from all sensors. Employ flow control and QoS (Quality of Service) strategies to prioritize data transmission from critical sensors and avoid network congestion.

[0043] Configuring edge computing units (such as the NVIDIA Jetson series) on the device side enables preprocessing and feature extraction of sensor data. By deploying lightweight deep learning models (such as MobileNet and YOLO), image recognition, sound classification, and temperature anomaly detection can be performed at the edge, reducing data transmission pressure and cloud processing load.

[0044] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0045] S201. For multimodal sensor data of a single device, extract significant event points for each modality, and add the time difference of significant event points across modalities as a penalty term to the dynamic time warping loss function, and use the dynamic time warping loss function to align the timestamps of the multimodal sensor data of a single device.

[0046] (1) Detect the instantaneous mutation of each modal data and mark the data point corresponding to the instantaneous mutation value exceeding the preset mutation threshold as a significant event point.

[0047] Before performing transient mutation detection, data for each modality needs to be preprocessed to eliminate the influence of noise and outliers on the detection results. Common preprocessing methods include: Smoothing: Using methods such as moving average and Gaussian filtering to smooth the data reduces high-frequency noise. For example, Gaussian filtering can be used to smooth image data acquired by industrial cameras.

[0048] Normalization: Normalizing data from different modalities to the same range, such as [0, 1] or [-1, 1], to enable uniform mutation detection. Min-max normalization or z-score normalization methods can be used.

[0049] Transient mutation detection methods: Difference method: Calculate the difference between adjacent data points and use the absolute value of the difference as the instantaneous change value. If this value exceeds a preset change threshold, the data point is marked as a significant event point. For example, for audio data collected by an acoustic sensor array, the difference in audio amplitude between adjacent time points can be calculated.

[0050] (2) Based on the salient event points in each modal data, establish cross-modal event anchor point pairs between different modal data, and force the alignment path to pass through the cross-modal event anchor point pairs. ); Establishing cross-modal event anchor pairs: Time matching: For salient event points in different modalities, find pairs of salient event points with similar times by comparing their timestamps. A time tolerance can be set; when the time difference between two salient event points is less than this tolerance, they are treated as a cross-modal event anchor pair.

[0051] Feature matching: In addition to time matching, matching can also be performed by combining the feature information of the data. For example, for image data acquired by industrial cameras and audio data acquired by acoustic sensors, features of the images (such as color and texture) and features of the audio (such as frequency and timbre) can be extracted, and cross-modal event anchor pairs can be established by comparing feature similarity.

[0052] Forced alignment paths: During subsequent dynamic time warping, ensure that the alignment path passes through these cross-modal event anchor pairs. Additional weights or constraints can be assigned to paths passing through anchor pairs during dynamic programming to guarantee the path's validity.

[0053] (3) Construct the dynamic time warping loss function:

[0054] wherein, π is the alignment path, K is the number of matched event anchor point pairs, is the data point corresponding to time point i in the first modality, is the data point corresponding to time point j in the second modality; λ is a parameter dynamically adjusted through adversarial learning; Adversarial learning can be implemented by a generator and a discriminator. The generator attempts to generate a suitable alignment path to minimize the loss function; the discriminator attempts to determine whether the generated alignment path is reasonable. Through continuous adversarial training, the parameter λ is dynamically adjusted, so that the loss function can better balance the distance between data points and the matching of cross-modality event anchor point pairs.

[0055] (4) Determining the optimal cross-modality event anchor point pairs by minimizing the dynamic time warping loss function.

[0056] A dynamic programming algorithm is used to solve the problem of minimizing the dynamic time warping loss function. The specific steps are as follows: Initialize a two-dimensional matrix D, where (D [i][j] ) represents the minimum loss between the first i data points in the first modality and the first j data points in the second modality.

[0057] Fill the elements of matrix D according to the definition of the loss function. In the filling process, the constraints of cross-modality event anchor point pairs are considered.

[0058] Starting from the last element of matrix D, backtrack to find the optimal alignment path π.

[0059] (5) Based on the optimal cross-modality event anchor point pairs, using an interpolation method to align the timestamps of multimodal sensor data of one device.

[0060] Linear interpolation: for data between adjacent cross-modality event anchor point pairs, linear interpolation can be used for timestamp alignment. Assuming (i₁,j₁) and (i₂,j₂) are two adjacent cross-modality event anchor point pairs, for a time point i (i₁ < i < i₂) in the first modality, the corresponding time point j in the second modality can be calculated by linear interpolation: .

[0061] S202. Mapping the timestamps of each device to a low-dimensional manifold space, using t-SNE to retain the relative temporal relationship between devices, and aligning the timestamps between devices by using a pre-constructed joint optimization loss function and said relative temporal relationship between devices.

[0062] The device timestamps are mapped to the manifold space to obtain low-dimensional coordinates; the manifold time offset between the low-dimensional coordinates of different devices is calculated; the manifold space is adjusted through Procrustes analysis to make the reference event points of all devices coincide; the adjusted low-dimensional coordinates of each device are calibrated, and the calibrated low-dimensional coordinates are transformed to the global time reference.

[0063] Specifically, it includes the following steps: 1. Map the device timestamp to the manifold space to obtain low-dimensional coordinates. We selected isomap, locally linear embedding (LLE), or t-distributed random neighborhood embedding (t-SNE) algorithms for manifold mapping. For time series data, Isomap effectively preserves the global geometric structure of the data, LLE is suitable for handling locally linear structured data, and t-SNE performs well in data visualization and dimensionality reduction. Taking Isomap as an example, the specific steps are as follows: Construct an adjacency graph: Calculate the Euclidean distance between each data point in the device timestamp sequence and its k nearest neighbors, and construct an undirected weighted graph with edge weights equal to the distance between the two points.

[0064] Calculate the shortest path: Use Dijkstra's or Floyd-Warshall's algorithm to calculate the shortest path distance between any two points in the graph and obtain the geodesic distance matrix.

[0065] Eigenvalue decomposition: Multidimensional scaling (MDS) is performed on the geodesic distance matrix to obtain a low-dimensional coordinate representation. Typically, the dimension is reduced to 2-3 dimensions for easier analysis and visualization.

[0066] Before mapping, the timestamp data is standardized to eliminate dimensional differences. If the data is noisy, wavelet denoising or median filtering can be used for smoothing to ensure the accuracy of the manifold mapping. Simultaneously, missing timestamp data is filled using linear interpolation or model-based prediction methods.

[0067] 2. Calculate the manifold time offset between the low-dimensional coordinates of different devices.

[0068] In a low-dimensional coordinate space, representative feature points are selected, such as equipment operating state switching points or periodic peak points. Key feature points can be identified by detecting local extrema, abrupt slope changes, or using clustering algorithms in the low-dimensional coordinate sequence. For example, SIFT (Scale Invariant Feature Transform) or SURF (Accelerated Robust Feature Transform) algorithms can be used to extract stable feature points in low-dimensional coordinates.

[0069] Feature point matching: Matching pairs of feature points are found by calculating the Euclidean distance or cosine similarity of low-dimensional coordinate feature points from different devices. For each pair of matching points, the difference on the time axis is calculated, and the average value is taken as the manifold time offset.

[0070] Dynamic Time Warping (DTW): If the number or order of feature points is inconsistent, the DTW algorithm is used to find the best alignment path for the low-dimensional coordinate sequence and calculate the time offset based on the alignment result.

[0071] 3. Adjust the manifold space through Procrustes analysis to make the reference event points of all devices coincide.

[0072] Procrustes analysis is a method for comparing two sets of shape or coordinate data, using rotation, scaling, and translation operations to make the two sets of data overlap as much as possible. In this scenario, the low-dimensional coordinates of one device are used as a reference template, and the low-dimensional coordinates of other devices are adjusted accordingly.

[0073] Data centerization: Calculate the mean of the low-dimensional coordinates of each device, subtract the mean from the coordinate data, and shift the data center to the origin.

[0074] Rotation and scaling: Singular value decomposition (SVD) is used to calculate the optimal rotation matrix and scaling factor to make the low-dimensional coordinates of the target device as similar as possible to the shape of the reference template.

[0075] Translation operation: Calculate the translation vector and translate the adjusted low-dimensional coordinates to completely coincide with the reference template, ensuring that the reference event points of all devices are aligned in the manifold space.

[0076] 4. Calibrate the low-dimensional coordinates of each device after adjustment, and convert the calibrated low-dimensional coordinates to the global time reference.

[0077] Error correction: Analyze the residual error of the adjusted low-dimensional coordinates, fit the error curve using the least squares method, and correct the coordinates. For example, if systematic bias exists, correction coefficients can be calculated using a linear regression model to compensate for the coordinates.

[0078] Consistency verification: Use hypothesis testing methods (such as t-test or analysis of variance) to verify the consistency of low-dimensional coordinates of each device after calibration, and ensure the time synchronization effect.

[0079] Reference selection: Select a stable and reliable time source as the global time reference, such as atomic clock time or GPS time.

[0080] Coordinate transformation: Based on the manifold time offset and calibration parameters, the low-dimensional coordinates are mapped back to the time axis and aligned with the global time reference. Accurate transformation from manifold space to global time can be achieved by establishing linear or nonlinear mapping models.

[0081] Through the above steps, the device timestamps can be effectively processed and synchronized in the manifold space, providing an accurate time reference for the collaborative operation of multiple devices on the production line.

[0082] S203. Construct a 3D tensor using the aligned multimodal sensor data from different devices, and extract core features from the 3D tensor using Tucker to obtain a multidimensional temporal feature matrix.

[0083] 1. Data preprocessing and 3D tensor construction.

[0084] Before constructing the 3D tensor, the aligned multimodal sensor data from different devices needs further processing. For numerical data (such as temperature data from infrared thermal imagers and sound pressure data from acoustic sensors), standardization methods are used to scale the data to the same scale. Common standardization methods include Z-score standardization and min-max standardization to eliminate the impact of dimensional differences on subsequent analysis.

[0085] For image data (images captured by industrial cameras), the image size needs to be standardized. Bilinear interpolation or bicubic interpolation algorithms should be used to adjust the image to a fixed resolution, such as 224×224 pixels. Simultaneously, the image data should be normalized to the [0,1] interval to facilitate fusion with other modal data. Furthermore, if missing data exists, linear interpolation, K-nearest neighbor interpolation, or prediction methods based on machine learning models (such as random forests or neural networks) can be used to fill in the gaps.

[0086] Defining the meaning of tensor dimensions: The three dimensions of a 3D tensor correspond to the device dimension, modal dimension, and time dimension, respectively. Assume a production line has M devices, each equipped with N types of modal sensors (industrial cameras, acoustic sensor arrays, infrared thermal imagers, etc.), and the time series data collected is of length T.

[0087] Data population method: For each modality of data for each device, the data is populated into the tensor sequentially in chronological order. For example, the elements of the tensor for the i-th device (1≤i≤M), j-th modality (1≤j≤N), and k-th time point (1≤k≤T) correspond to the measurement value or image data of the j-th modality of the i-th device at the k-th time point (if it is image data, it needs to be flattened first). Finally, a 3D tensor X of size M×N×T is constructed.

[0088] 2. Tucker decomposes and extracts core features.

[0089] Tucker decomposition is a high-order tensor decomposition method that decomposes the original tensor into a core tensor and multiple factor matrices. Its core idea is to extract key information from the tensor through low-rank approximation, reducing data dimensionality while preserving main features. For a 3D tensor X, Tucker decomposition can be represented as: , where g is the core tensor, and U, V, and W are the factor matrices corresponding to the device dimension, modality dimension, and time dimension, respectively.

[0090] Determining the decomposition rank: Before performing Tucker decomposition, the ranks r1, r2, and r3 of the core tensor in each dimension need to be pre-defined (r1≤M, r2≤N, r3≤T). The choice of rank directly affects the feature extraction effect and computational complexity. The optimal rank can be determined through cross-validation or methods based on information criteria (such as AIC, BIC).

[0091] Calculating the factor matrices: The factor matrices U, V, and W are calculated using Higher-Order Singular Value Decomposition (HOSVD). Taking the device dimension as an example, the tensor X is expanded along the device dimension to obtain matrix X. (1) , for X (1) Perform Singular Value Decomposition (SVD): Take the first r1 singular vectors to form the factor matrix U. Similarly, calculate the factor matrices V and W for the modal and temporal dimensions.

[0092] Calculate the core tensor: Based on the obtained factor matrices U, V, and W, use the formula... Calculate the core tensor g, which has a size of r1×r2×r3, and whose elements contain the core feature information of the original tensor.

[0093] Generate a multidimensional time series feature matrix: Flatten the core tensor g, expand it into a one-dimensional vector in a certain order (such as row first, column second), and then rearrange it into a two-dimensional matrix to obtain the multidimensional time series feature matrix F. Each row of the matrix represents a feature vector at a time point, and the number of columns is r1×r2×r3. This matrix can be used for subsequent tasks such as fault diagnosis and equipment status prediction.

[0094] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0095] The teacher model includes: The three-dimensional multimodal feature extraction layer includes an input layer, a 3D convolutional kernel group, and spatiotemporal pooling. The input layer is used to receive raw sensor data. The 3D convolutional kernel group adopts grouped 3D convolution, with each group processing different modal features. The spatiotemporal pooling includes dynamic temporal pooling and spatial max pooling. The dynamic knowledge graph embedding layer is used to determine the topology embedding vector of each device based on the device's 3D coordinates and the connection relationships between devices using a graph embedding network; it uses spatiotemporal graph convolution to determine the device state propagation risk encoding based on historical fault propagation paths; and it uses a tree-structured LSTM network to determine the device's process compliance characteristics based on a process dependency tree. A multi-head cross-attention fusion layer is used to associate the features extracted by the 3D multimodal feature extraction layer with the features extracted by the dynamic knowledge graph embedding layer to obtain spatiotemporal-graph joint features; The spatiotemporal graph prediction layer is used to determine the production line status based on spatiotemporal-graph joint features using a spatiotemporal gated graph network.

[0096] The student model includes: Lightweight Feature Extractor: Bidirectional LSTM network: Time-dimension grouping: Input multimodal sensor data in groups according to physical characteristics; The hidden layer dimension is compressed to 1 / 4 of that of the teacher model; Add a temporal attention pooling layer to replace spatiotemporal pooling; Adversarial training module: Generator: Multilayer Temporal Convolutional Network (TCN); Discriminator: Multi-scale structures include: Original signal discrimination branch; Frequency domain feature discrimination branch; Moments statistic discriminant branch.

[0097] Knowledge distillation process 1. Feature space alignment.

[0098] Cross-modal projection: Establish the mapping relationship between the teacher's 3D convolutional features (C×T×H×W) and the student's LSTM hidden state (D×T).

[0099] Loss function: SWD (Sliced ​​Wasserstein Distance) measures the difference in feature distributions.

[0100] 2. Dynamic knowledge transfer.

[0101] Spectral encoding distillation: Embedding teachers' knowledge graphs as a soft objective:

[0102] Where τ is a temperature coefficient, dynamically adjusted:

[0103] Risk propagation simulation: Constructing a pseudo-adjacency matrix to guide student models:

[0104] Counter-distillation: Generator objective function:

[0105] Feature matching loss:

[0106] Implement two-stage distillation: Feature preheating stage: Freeze the teacher model; Optimize only the feature alignment loss ; The Lookahead optimizer is used to accelerate convergence.

[0107] Countering the Refining Stage: Jointly optimize the student model and discriminator.

[0108] Loss function combination:

[0109] Introducing a gradient penalty term to stabilize training:

[0110] Compression acceleration strategy: Structured pruning: Block-level pruning is performed on LSTM cells to preserve hidden connections at important time steps; Redundant gating units were removed based on Hessian sensitivity analysis. Quantitative deployment: Hybrid precision quantization is employed. Feature extractor: 8-bit dynamic quantization; Adversarial module: 16-bit floating-point retention of generation quality; The compiler is optimized using the TVM graph during deployment.

[0111] Based on the above embodiments, in order to further improve the accuracy of fault prediction, the model is continuously fine-tuned in a feasible manner.

[0112] The prediction results and corresponding multimodal sensor data are pushed to the cloud via a message queue to fine-tune the teacher model in the cloud. The fine-tuned teacher model is then used to fine-tune the student model.

[0113] In some embodiments, the multimodal production line state reasoning system may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the multimodal production line state reasoning system may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality of multimodal production line state reasoning.

[0114] In this embodiment, the multimodal production line state reasoning system can be divided into multiple functional modules based on the functions it performs, such as... Figure 2 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0115] The data acquisition module is used to collect multimodal sensor data generated by multiple devices on the production line. The preprocessing module is used to align the timestamps of multimodal sensor data from different devices and fuse them to generate a multidimensional time-series feature matrix. The prediction module is used to input the multidimensional time-series feature matrix into a lightweight prediction model pre-deployed in the edge processor and output the production line status prediction result. The lightweight prediction model is obtained by compressing a teacher model deployed in the cloud through a knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network.

[0116] Figure 3 The multimodal production line state reasoning method provided in the embodiments of this application can be applied to equipment. Those skilled in the art will understand that the equipment structure involved in the embodiments of this invention does not constitute a limitation on the equipment. Equipment may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, equipment includes, but is not limited to, laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Equipment may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0117] The device 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0118] The memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is able to perform some or all of the steps in the above method embodiments.

[0119] The processor 310 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0120] The communication unit 330 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0121] The present invention also provides a computer medium, wherein the computer medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0122] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0123] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0124] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0125] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0127] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A multimodal production line state reasoning method, characterized in that, include: Collect multimodal sensor data generated by multiple devices on the production line; Timestamp alignment is performed on multimodal sensor data from different devices, and the data is fused to generate a multidimensional time-series feature matrix; The multidimensional temporal feature matrix is ​​input into a lightweight prediction model pre-deployed in the edge processor, and the production line status prediction result is output. The lightweight prediction model is obtained by compressing a teacher model deployed in the cloud through a knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network. Timestamp alignment is performed on multimodal sensor data from different devices, and the data is fused to generate a multidimensional time-series feature matrix, including: For multimodal sensor data from a single device, significant event points are extracted for each modality, and the time difference between significant event points across modalities is added as a penalty term to the dynamic time warping loss function. The dynamic time warping loss function is then used to align the timestamps of the multimodal sensor data from a single device. The timestamps of each device are mapped to a low-dimensional manifold space, the relative time relationship between devices is preserved using t-SNE, and the timestamps between devices are aligned using a pre-constructed joint optimization loss function and the relative time relationship between devices. A 3D tensor is constructed using aligned multimodal sensor data from different devices, and core features are extracted from the 3D tensor using Tucker to obtain a multidimensional temporal feature matrix. For multimodal sensor data from a single device, significant event points are extracted for each modality, and the time difference between significant event points across modalities is added as a penalty term to the dynamic time warping loss function. The timestamps of the multimodal sensor data from a single device are then aligned using the dynamic time warping loss function, including: Detect instantaneous abrupt changes in each modality's data and mark the data points corresponding to instantaneous abrupt changes exceeding a preset abrupt change threshold as significant event points; Based on salient event points in each modal data, cross-modal event anchor pairs are established between different modal data, and the alignment path is forced to pass through the cross-modal event anchor pairs. ); Construct a dynamic time warping loss function: Where π is the alignment path and K is the number of matched event anchor pairs. For the data point corresponding to time point i in the first mode, Let λ be the data point corresponding to time point j in the second modality; λ is a parameter dynamically adjusted through adversarial learning so that the loss function can balance the distance between data points and the matching of cross-modal event anchor pairs. The optimal cross-modal event anchor pair is determined by minimizing the dynamic time warping loss function; Based on the optimal cross-modal event anchor point pair, the timestamps of a device's multimodal sensor data are aligned using interpolation. The timestamps of each device are mapped to a low-dimensional manifold space. The relative time relationships between devices are preserved using t-SNE. The timestamps between devices are aligned using a pre-constructed joint optimization loss function and the relative time relationships between devices, including: Map the device timestamp to the manifold space to obtain low-dimensional coordinates; Calculate the manifold time offset between low-dimensional coordinates of different devices; The manifold space is adjusted through Procrustes analysis to make the reference event points of all devices coincide. The adjusted low-dimensional coordinates of each device are calibrated, and the calibrated low-dimensional coordinates are converted to a global time reference. The teacher model includes: The three-dimensional multimodal feature extraction layer includes an input layer, a 3D convolutional kernel group, and spatiotemporal pooling. The input layer is used to receive raw sensor data. The 3D convolutional kernel group adopts grouped 3D convolution, with each group processing different modal features. The spatiotemporal pooling includes dynamic temporal pooling and spatial max pooling. The dynamic knowledge graph embedding layer is used to determine the topology embedding vector of each device based on the device's 3D coordinates and the connection relationships between devices using a graph embedding network; it uses spatiotemporal graph convolution to determine the device state propagation risk encoding based on historical fault propagation paths; and it uses a tree-structured LSTM network to determine the device's process compliance characteristics based on a process dependency tree. A multi-head cross-attention fusion layer is used to associate the features extracted by the 3D multimodal feature extraction layer with the features extracted by the dynamic knowledge graph embedding layer to obtain spatiotemporal-graph joint features; The spatiotemporal graph prediction layer is used to determine the production line status based on spatiotemporal-graph joint features using a spatiotemporal gated graph network.

2. The method according to claim 1, characterized in that, Collect multimodal sensor data generated by multiple devices on the production line, including: Multimodal sensors are pre-deployed on multiple devices in the production line and communication connections are established with the multimodal sensors on multiple devices; the multimodal sensors include industrial cameras, acoustic sensor arrays, and infrared thermal imagers.

3. The method according to claim 1, characterized in that, The method further includes: The prediction results and corresponding multimodal sensor data are pushed to the cloud via a message queue to fine-tune the teacher model in the cloud.

4. A multimodal production line state reasoning system, characterized in that, include: The data acquisition module is used to collect multimodal sensor data generated by multiple devices on the production line. The preprocessing module is used to align the timestamps of multimodal sensor data from different devices and fuse them to generate a multidimensional time-series feature matrix. The prediction module is used to input the multidimensional time-series feature matrix into a lightweight prediction model pre-deployed in the edge processor and output the production line status prediction result. The lightweight prediction model is obtained by compressing a teacher model deployed in the cloud through a knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network. Timestamp alignment is performed on multimodal sensor data from different devices, and the data is fused to generate a multidimensional time-series feature matrix, including: For multimodal sensor data from a single device, significant event points are extracted for each modality, and the time difference between significant event points across modalities is added as a penalty term to the dynamic time warping loss function. The dynamic time warping loss function is then used to align the timestamps of the multimodal sensor data from a single device. The timestamps of each device are mapped to a low-dimensional manifold space, the relative time relationship between devices is preserved using t-SNE, and the timestamps between devices are aligned using a pre-constructed joint optimization loss function and the relative time relationship between devices. A 3D tensor is constructed using aligned multimodal sensor data from different devices, and core features are extracted from the 3D tensor using Tucker to obtain a multidimensional temporal feature matrix. For multimodal sensor data from a single device, significant event points are extracted for each modality, and the time difference between significant event points across modalities is added as a penalty term to the dynamic time warping loss function. The timestamps of the multimodal sensor data from a single device are then aligned using the dynamic time warping loss function, including: Detect instantaneous abrupt changes in each modality's data and mark the data points corresponding to instantaneous abrupt changes exceeding a preset abrupt change threshold as significant event points; Based on salient event points in each modal data, cross-modal event anchor pairs are established between different modal data, and the alignment path is forced to pass through the cross-modal event anchor pairs. ); Construct a dynamic time warping loss function: Where π is the alignment path and K is the number of matched event anchor pairs. For the data point corresponding to time point i in the first mode, Let λ be the data point corresponding to time point j in the second modality; λ is a parameter dynamically adjusted through adversarial learning so that the loss function can balance the distance between data points and the matching of cross-modal event anchor pairs. The optimal cross-modal event anchor pair is determined by minimizing the dynamic time warping loss function; Based on the optimal cross-modal event anchor point pair, the timestamps of a device's multimodal sensor data are aligned using interpolation. The timestamps of each device are mapped to a low-dimensional manifold space. The relative time relationships between devices are preserved using t-SNE. The timestamps between devices are aligned using a pre-constructed joint optimization loss function and the relative time relationships between devices, including: Map the device timestamp to the manifold space to obtain low-dimensional coordinates; Calculate the manifold time offset between low-dimensional coordinates of different devices; The manifold space is adjusted through Procrustes analysis to make the reference event points of all devices coincide. The adjusted low-dimensional coordinates of each device are calibrated, and the calibrated low-dimensional coordinates are converted to a global time reference. The teacher model includes: The three-dimensional multimodal feature extraction layer includes an input layer, a 3D convolutional kernel group, and spatiotemporal pooling. The input layer is used to receive raw sensor data. The 3D convolutional kernel group adopts grouped 3D convolution, with each group processing different modal features. The spatiotemporal pooling includes dynamic temporal pooling and spatial max pooling. The dynamic knowledge graph embedding layer is used to determine the topology embedding vector of each device based on the device's 3D coordinates and the connection relationships between devices using a graph embedding network; it uses spatiotemporal graph convolution to determine the device state propagation risk encoding based on historical fault propagation paths; and it uses a tree-structured LSTM network to determine the device's process compliance characteristics based on a process dependency tree. A multi-head cross-attention fusion layer is used to associate the features extracted by the 3D multimodal feature extraction layer with the features extracted by the dynamic knowledge graph embedding layer to obtain spatiotemporal-graph joint features; The spatiotemporal graph prediction layer is used to determine the production line status based on spatiotemporal-graph joint features using a spatiotemporal gated graph network.

5. An electronic device, characterized in that, include: Memory, used to store multimodal production line state reasoning programs; A processor, configured to implement the steps of the multimodal production line state reasoning method as described in any one of claims 1-3 when executing the multimodal production line state reasoning program.

6. A computer-readable medium storing a computer program, characterized in that, The readable medium stores a multimodal production line state reasoning program, which, when executed by a processor, implements the steps of the multimodal production line state reasoning method as described in any one of claims 1-3.

Images