Multi-modal production line state reasoning method, system, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and particularly provides a multi-modal production line state reasoning method, system and device and a medium, and the method comprises the steps: collecting multi-modal sensor data generated by a plurality of devices on a production line; performing timestamp alignment on the multi-modal sensor data from different devices, and fusing to generate a multi-dimensional time sequence feature matrix; inputting the multi-dimensional time sequence characteristic matrix into a lightweight prediction model pre-deployed in an edge processor, and outputting a production line state prediction result; wherein the lightweight prediction model is obtained by compressing a teacher model deployed from a cloud through a knowledge distillation method, and the lightweight prediction model comprises a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network. According to the method, the response time is shortened, cross-device conductivity faults can be accurately positioned, and the method is obviously superior to a traditional independent device diagnosis mode.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a multimodal production line state reasoning method, system, equipment and medium. Background Art

[0002] In industrial manufacturing, production lines typically consist of multiple critical pieces of equipment (such as CNC machine tools, robotic arms, and conveyors) operating in concert. Existing monitoring systems often employ a single-device, independent monitoring model, collecting only a single physical parameter, such as vibration or temperature, and triggering alarms based on fixed thresholds.

[0003] This traditional approach suffers from three significant drawbacks: First, single-modal sensor data cannot fully characterize the equipment's operating status. For example, current fluctuations in a robotic arm require combined analysis with visual positioning data. Second, the threshold alarm mechanism has a significant lag in identifying complex faults, often causing production line downtime by the time parameters reach the threshold. Third, existing methods fail to consider process coupling between equipment (such as material flow timing dependencies), leading to the neglect of abnormal transmission effects between equipment during fault location. Furthermore, while complex cloud-based analysis models can process multi-source data, they are limited by network latency and edge computing resources, making them difficult to meet the demanding real-time requirements for operating condition monitoring.

[0004] The above defects result in a high false alarm rate in current production line status monitoring and a long average fault response time. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multimodal production line state reasoning method, system, device and medium to solve the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a multimodal production line state reasoning method, comprising: Collect multimodal sensor data generated by multiple devices on the production line; Align timestamps of multimodal sensor data from different devices and fuse them to generate a multi-dimensional time series feature matrix; Inputting the multi-dimensional time series feature matrix into a lightweight prediction model pre-deployed in the edge processor, and outputting a production line status prediction result; The lightweight prediction model is obtained by compressing the teacher model deployed in the cloud through the knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network.

[0007] In an optional embodiment, collecting multimodal sensor data generated by multiple devices on a production line includes: Multimodal sensors are pre-deployed for multiple devices on the production line, and communication connections are established with the multimodal sensors of multiple devices; multimodal sensors include industrial cameras, acoustic sensor arrays, and infrared thermal imagers.

[0008] In an optional embodiment, timestamp alignment is performed on multimodal sensor data from different devices, and fusion is performed to generate a multidimensional time series feature matrix, including: For multimodal sensor data from a single device, significant event points are extracted for each modal data, and the time difference of the significant event points across modalities is added as a penalty term to the dynamic time warping loss function. The dynamic time warping loss function is used to align the timestamps of the multimodal sensor data of a single device; Map the timestamps of each device to a low-dimensional manifold space, use t-SNE to preserve the relative temporal relationship between devices, and align the timestamps between devices using a pre-built joint optimization loss function and the relative temporal relationship between devices; A 3D tensor is constructed using aligned multimodal sensor data from different devices, and Tucker is used to extract core features from the 3D tensor to obtain a multidimensional time series feature matrix.

[0009] In an optional embodiment, for multimodal sensor data of a single device, significant event points are extracted for each modal data, and the time difference of the significant event points across modalities is added as a penalty term to the dynamic time warping loss function. The dynamic time warping loss function is used to align the timestamps of the multimodal sensor data of the single device, including: Detect the instantaneous mutation of each modal data and mark the corresponding data point when the instantaneous mutation value exceeds the preset mutation threshold as a significant event point; Based on the significant event points in each modal data, cross-modal event anchor pairs are established between different modal data, and the alignment path is forced to pass through the cross-modal event anchor pairs ( ); Construct the dynamic time warping loss function:

[0010] Among them, π is the alignment path, K is the number of matched event anchor points, is the data point corresponding to time point i in the first mode, is the data point corresponding to time point j in the second modality; λ is the parameter dynamically adjusted through adversarial learning; Determining the optimal cross-modal event anchor pair by minimizing the dynamic time warping loss function; Based on the optimal cross-modal event anchor point pair, timestamps of multimodal sensor data of a device are aligned using an interpolation method.

[0011] In an optional embodiment, the timestamps of each device are mapped to a low-dimensional manifold space, the relative time relationship between the devices is preserved using t-SNE, and the timestamps between the devices are aligned using a pre-built joint optimization loss function and the relative time relationship between the devices, including: Map the device timestamp to the manifold space to obtain low-dimensional coordinates; Calculate the manifold time offset between low-dimensional coordinates of different devices; Adjust the manifold space through Procrustes analysis so that the reference event points of all devices coincide; The adjusted low-dimensional coordinates of each device are calibrated, and the calibrated low-dimensional coordinates are converted to a global time reference.

[0012] In an optional embodiment, the teacher model includes: A three-dimensional multimodal feature extraction layer, comprising an input layer for receiving raw sensor data, a 3D convolution kernel group, and spatiotemporal pooling. The input layer employs grouped 3D convolution, with each group processing features of a different modality. The spatiotemporal pooling comprises dynamic temporal pooling and spatial maximum pooling. The dynamic knowledge graph embedding layer uses a graph embedding network to determine the topological embedding vector of each device based on the device's three-dimensional coordinates and the connection relationship between devices. It uses spatiotemporal graph convolution to determine the device status propagation risk code based on the historical fault diffusion path. It also uses a tree-shaped LSTM network to determine the compliance characteristics of the equipment process based on the process dependency tree. The multi-head cross attention fusion layer is used to associate the features extracted by the 3D multimodal feature extraction layer with the features extracted by the dynamic knowledge graph embedding layer to obtain the spatiotemporal-graph joint features; The spatiotemporal graph prediction layer is used to determine the production line status based on the spatiotemporal-graph-atlas joint features using the spatiotemporal gated graph network.

[0013] In an optional embodiment, the method further comprises: The prediction results and corresponding multimodal sensor data are pushed to the cloud through a message queue to fine-tune the teacher model in the cloud.

[0014] In a second aspect, the present invention provides a multimodal production line state reasoning system, comprising: The data acquisition module is used to collect multimodal sensor data generated by multiple devices on the production line; The preprocessing module is used to align the timestamps of multimodal sensor data from different devices and fuse them to generate a multi-dimensional time series feature matrix; A prediction module, configured to input the multi-dimensional time series feature matrix into a lightweight prediction model pre-deployed in the edge processor and output a production line status prediction result; The lightweight prediction model is obtained by compressing the teacher model deployed in the cloud through the knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network.

[0015] According to a third aspect, a device is provided, comprising: A memory for storing a multimodal production line state reasoning program; A processor is configured to implement the steps of the multimodal production line state reasoning method provided in the first aspect when executing the multimodal production line state reasoning program.

[0016] In a fourth aspect, a computer-readable medium is provided, on which a multimodal production line state reasoning program is stored. When the multimodal production line state reasoning program is executed by a processor, the steps of the multimodal production line state reasoning method provided in the first aspect are implemented.

[0017] The beneficial effects of the present invention are that the multimodal production line state inference method, system, equipment and medium provided by the present invention, by fusing the multimodal data of multiple devices and aligning the timestamps, construct a multidimensional matrix containing the spatiotemporal correlation characteristics between devices, thereby solving the problem of data dimension fragmentation in traditional single-device threshold detection; secondly, the knowledge distillation technology is used to migrate the capabilities of the cloud-based teacher model to the lightweight model on the edge side, and while ensuring the prediction accuracy, the model inference time is reduced to meet the real-time monitoring needs; finally, through the LSTM-GAN joint network architecture, LSTM is used to capture the temporal dependency between devices, and at the same time, GAN adversarial training is used to enhance the model's ability to recognize implicit abnormal patterns, thereby reducing the response time and accurately locating cross-device conductive faults, which is significantly better than the traditional independent device diagnosis mode.

[0018] In addition, the present invention has a reliable design principle, a simple structure and a very broad application prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention.

[0021] Figure 2 FIG. 4 is a schematic block diagram of a system according to an embodiment of the present invention.

[0022] Figure 3A schematic structural diagram of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0025] The multimodal production line state reasoning method provided by the embodiment of the present invention is executed by a computer device. Accordingly, the multimodal production line state reasoning system runs in the computer device.

[0026] Figure 1 is a schematic flow chart of a method according to an embodiment of the present invention. Figure 1 The execution subject can be a multi-modal production line state reasoning system. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0027] like Figure 1 As shown, the method includes: S1. Collect multimodal sensor data generated by multiple devices on the production line; S2. Align the timestamps of multimodal sensor data from different devices and fuse them to generate a multidimensional time series feature matrix; S3. Input the multi-dimensional time series feature matrix into the lightweight prediction model pre-deployed in the edge processor, and output the production line status prediction result; The lightweight prediction model is obtained by compressing the teacher model deployed in the cloud through the knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network.

[0028] In an embodiment of the present invention, based on step S1, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0029] Multimodal sensors are pre-deployed for multiple devices on the production line, and communication connections are established with the multimodal sensors of multiple devices; multimodal sensors include industrial cameras, acoustic sensor arrays, and infrared thermal imagers.

[0030] Before deploying multimodal sensors, a comprehensive assessment of multiple pieces of equipment on the production line is required. The appropriate sensor combination should be determined based on the equipment type (e.g., CNC machine tools, robotic arms, conveyors), operating conditions (speed, temperature, vibration range), and monitoring requirements (fault warning, performance optimization). For example, for CNC machine tools, where tool wear and spindle vibration are key considerations, a high-resolution industrial camera and a high-precision accelerometer might be appropriate. For robotic arms, joint temperature and motion trajectory are key considerations, so an infrared thermal imager and an inertial measurement unit might be appropriate.

[0031] Industrial cameras: Choose area scan cameras with a frame rate of at least 120 fps and a resolution of 2000 × 1500 pixels or higher. These cameras should be equipped with autofocus and optical image stabilization, and support GigEVision or CameraLink high-speed interfaces to ensure clear imaging of high-speed moving parts.

[0032] Acoustic sensor array: Utilizes an 8-channel or higher MEMS microphone array with a frequency response range of 20Hz-20kHz. It features beamforming and noise suppression algorithms, allowing for precise location of sound sources and the capture and analysis of abnormal equipment noise.

[0033] Infrared thermal imager: Use an uncooled focal plane array (FPA) detector with a pixel count of no less than 320×240, a temperature measurement range of -20°C to 500°C, a temperature measurement accuracy of ±2% or ±2°C, and support real-time temperature data output and thermal map generation.

[0034] Industrial cameras: Based on the key monitoring parts of the equipment (such as the tool cutting area and the conveyor belt surface), use a bracket to fix the camera to ensure that the optical axis of the camera is perpendicular to the monitoring surface and the working distance is maintained at 0.5-3 meters to avoid obstruction and reflection interference.

[0035] Acoustic sensor array: Installed in a circular or linear arrangement around the device, 0.3-1 meter away from the device surface, with the center of the array aligned with the device's main sound-emitting area. Windshields and shock-absorbing pads are used to reduce the impact of environmental noise and mechanical vibration.

[0036] Infrared thermal imager: Installed above or on the side of the equipment, ensure that the thermal imager's field of view covers the key heat-generating components of the equipment (such as motors and bearings), avoid interference from high-temperature radiation and environmental heat sources, and install an infrared filter on the lens to improve temperature measurement accuracy.

[0037] All sensors must be equipped with enclosures with an IP67 rating or higher to withstand the harsh environments of the production line, including high temperature, humidity, and dust. Sensor cables are shielded, with waterproof aviation plugs at the connectors. Metal wire troughs and conduit are used for line protection to prevent electromagnetic interference and mechanical damage.

[0038] Build a star-topology network with Industrial Ethernet as the backbone, using redundant dual-port switches (such as the Siemens SCALANCE X series) to ensure communication reliability. Sensors access the network via Ethernet, RS-485, or wireless communications (such as 5G and Wi-Fi 6). Dual-link backup is deployed for critical equipment to avoid single points of failure.

[0039] Industrial cameras use GigE Vision or GenICam standard protocols to achieve high-speed transmission of image data and cross-platform compatibility.

[0040] The acoustic sensor array uses the TCP / IP protocol to encapsulate audio data and cooperates with the Real-time Transport Protocol (RTP) to ensure low latency and integrity of the audio stream.

[0041] The infrared thermal imager uploads temperature data via Modbus TCP or OPC UA protocols, enabling seamless integration with industrial automation systems.

[0042] Deploy a network time synchronization protocol (NTP or IEEE 1588 Precision Time Protocol) to ensure consistent time for data collected by each sensor. Use flow control and QoS (Quality of Service) strategies to prioritize critical sensor data transmission and avoid network congestion.

[0043] Configure edge computing units (such as the NVIDIA Jetson series) on the device side to preprocess and extract features from sensor data. Deploy lightweight deep learning models (such as MobileNet and YOLO) to perform image recognition, sound classification, and temperature anomaly detection at the edge, reducing data transmission pressure and cloud processing load.

[0044] In an embodiment of the present invention, based on step S2, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0045] S201. For the multimodal sensor data of a single device, significant event points are extracted for each modal data, and the time difference of the significant event points across modalities is added as a penalty term to the dynamic time warping loss function, and the dynamic time warping loss function is used to align the timestamps of the multimodal sensor data of a single device.

[0046] (1) Detect the instantaneous mutation of each modal data and mark the corresponding data point when the instantaneous mutation value exceeds the preset mutation threshold as a significant event point.

[0047] Before performing transient mutation detection, the data of each modality needs to be preprocessed to eliminate the impact of noise and outliers on the detection results. Common preprocessing methods include: Smoothing: Use methods such as moving average and Gaussian filtering to smooth data and reduce high-frequency noise. For example, for image data collected by industrial cameras, Gaussian filtering can be used to smooth the image.

[0048] Normalization: Normalize data from different modalities to the same range, such as [0, 1] or [-1, 1], to enable uniform mutation detection. Min-max normalization or z-score normalization can be used.

[0049] Transient mutation detection method: Difference method: Calculate the difference between adjacent data points and use the absolute value of the difference as the instantaneous mutation value. If the value exceeds a preset mutation threshold, the data point is marked as a significant event point. For example, for audio data collected by an acoustic sensor array, the difference in audio amplitude between adjacent time points can be calculated.

[0050] (2) Based on the significant event points in each modal data, a cross-modal event anchor pair is established between different modal data, and the alignment path is forced to pass through the cross-modal event anchor pair ( ); Establishment of cross-modal event anchor pairs: Temporal matching: For significant event points in different modal data, by comparing their timestamps, we find pairs of significant event points with similar times. We can set a time tolerance. When the time difference between two significant event points is less than the tolerance, they are considered as a cross-modal event anchor pair.

[0051] Feature matching: In addition to temporal matching, matching can also be performed based on the data's feature information. For example, for image data captured by industrial cameras and audio data collected by acoustic sensors, image features (such as color and texture) and audio features (such as frequency and timbre) can be extracted, and cross-modal event anchor pairs can be established by comparing feature similarities.

[0052] Enforce alignment paths: During the subsequent dynamic time warping process, ensure that the alignment paths pass through these cross-modal event anchor pairs. During the dynamic planning process, additional weights or constraints can be assigned to paths passing through these anchor pairs to ensure path legitimacy.

[0053] (3) Constructing dynamic time warping loss function:

[0054] Among them, π is the alignment path, and K is the number of pairs of matching event anchors. is the data point corresponding to time point i in the first modality. is the data point corresponding to time point j in the second modality; λ is a parameter dynamically adjusted through adversarial learning. Adversarial learning can be achieved through a generator and a discriminator. The generator attempts to generate a suitable alignment path to minimize the loss function; the discriminator attempts to determine whether the generated alignment path is reasonable. Through continuous adversarial training, the parameter λ is dynamically adjusted so that the loss function can better balance the distance between data points and the matching of cross-modal event anchor pairs.

[0055] (4) Determine the optimal cross-modal event anchor pairs by minimizing the dynamic time warping loss function.

[0056] Use the dynamic programming algorithm to solve the problem of minimizing the dynamic time warping loss function. The specific steps are as follows: Initialize a two-dimensional matrix D, where (D [i][j] ) represents the minimum loss between the first i data points in the first modality and the first j data points in the second modality.

[0057] Fill the elements of matrix D according to the definition of the loss function. During the filling process, consider the constraints of cross-modal event anchor pairs.

[0058] Starting from the last element of matrix D, backtrack to find the optimal alignment path π.

[0059] (5) Based on the optimal cross-modal event anchor pairs, use interpolation to align the timestamps of multi-modal sensor data of a device.

[0060] Linear interpolation: For the data between adjacent cross-modal event anchor pairs, the method of linear interpolation can be used for timestamp alignment. Suppose (i1, j1) and (i2, j2) are two adjacent cross-modal event anchor pairs. For time point i (i1 < i < i2) in the first modality, the corresponding time point j in the second modality can be calculated by linear interpolation: .

[0061] S202. Map the timestamps of each device to a low-dimensional manifold space, use t-SNE to preserve the relative time relationship between devices, and align the timestamps between devices using the pre-constructed joint optimization loss function and the relative time relationship between devices.

[0062] Map device timestamps to manifold space to obtain low-dimensional coordinates; calculate the manifold time offset between the low-dimensional coordinates of different devices; adjust the manifold space through Procrustes analysis so that the reference event points of all devices coincide; calibrate the adjusted low-dimensional coordinates of each device and convert the calibrated low-dimensional coordinates to the global time reference.

[0063] Specifically, the following steps are included: 1. Map the device timestamp to the manifold space to obtain low-dimensional coordinates For manifold mapping, use the isomap, locally linear embedding (LLE), or t-distributed stochastic neighbor embedding (t-SNE) algorithms. For time series data, isomap effectively preserves the data's global geometric structure, LLE is suitable for processing locally linear data, and t-SNE excels at data visualization and dimensionality reduction. Using isomap as an example, the specific steps are as follows: Construct an adjacency graph: Calculate the Euclidean distance between each data point in the device timestamp sequence and its k nearest neighbors, and construct an undirected weighted graph where the edge weight is the distance between the two points.

[0064] Calculate the shortest path: Use the Dijkstra or Floyd-Warshall algorithm to calculate the shortest path distance between any two points in the graph and obtain the geodesic distance matrix.

[0065] Eigenvalue decomposition: Multidimensional scaling (MDS) is performed on the geodesic distance matrix to obtain a low-dimensional coordinate representation through eigenvalue decomposition, usually reducing the dimension to 2-3 dimensions for easy analysis and visualization.

[0066] Before mapping, the timestamp data is normalized to eliminate dimensional differences. If the data is noisy, wavelet denoising or median filtering can be used to smooth it to ensure the accuracy of the manifold mapping. Missing timestamp data can also be filled using linear interpolation or model-based prediction methods.

[0067] 2. Calculate the manifold time offset between low-dimensional coordinates of different devices.

[0068] In the low-dimensional coordinate space, select representative feature points, such as equipment operating state switching points and periodic peak points. Key feature points can be identified by detecting local extreme values ​​or slope mutations in the low-dimensional coordinate sequence, or by using clustering algorithms. For example, SIFT (Scale Invariant Feature Transform) or SURF (Speeded Up Robust Features) algorithms can be used to extract stable feature points in the low-dimensional coordinate space.

[0069] Feature point matching: This method finds matching pairs of feature points by calculating the Euclidean distance or cosine similarity of low-dimensional coordinate feature points on different devices. For each pair of matching points, the difference along the time axis is calculated and the average is taken as the manifold time offset.

[0070] Dynamic Time Warping (DTW): If the number or order of feature points is inconsistent, the DTW algorithm is used to find the optimal alignment path for the low-dimensional coordinate sequence and calculate the time offset based on the alignment result.

[0071] 3. Adjust the manifold space through Procrustes analysis so that the reference event points of all devices coincide.

[0072] Procrustes analysis is a method used to compare two sets of shape or coordinate data, using rotation, scaling, and translation to make them as close as possible. In this scenario, the low-dimensional coordinates of one device are used as a reference template to adjust the low-dimensional coordinates of other devices.

[0073] Data centering: Calculate the mean of the low-dimensional coordinates of each device, subtract the mean from the coordinate data, and shift the data center to the origin.

[0074] Rotation and Scaling: Use singular value decomposition (SVD) to calculate the optimal rotation matrix and scaling factors to make the low-dimensional coordinates of the target device as similar as possible to the shape of the reference template.

[0075] Translation operation: Calculate the translation vector and translate the adjusted low-dimensional coordinates to completely coincide with the reference template, ensuring that the reference event points of all devices are aligned in the manifold space.

[0076] 4. Calibrate the adjusted low-dimensional coordinates of each device and convert the calibrated low-dimensional coordinates to the global time reference.

[0077] Error Correction: Analyze the residual error of the adjusted low-dimensional coordinates and use the least squares method to fit the error curve to correct the coordinates. For example, if there is a systematic deviation, the correction coefficient can be calculated through a linear regression model to compensate for the coordinates.

[0078] Consistency test: Use hypothesis testing methods (such as t-test or analysis of variance) to verify the consistency of the low-dimensional coordinates of each device after calibration to ensure time synchronization.

[0079] Reference selection: Select a stable and reliable time source as the global time reference, such as atomic clock time or GPS time.

[0080] Coordinate transformation: Based on the manifold time offset and calibration parameters, the low-dimensional coordinates are mapped back to the time axis and aligned with the global time reference. Accurate transformation from manifold space to global time can be achieved by establishing linear or nonlinear mapping models.

[0081] Through the above steps, the device timestamps can be effectively processed and synchronized in the manifold space, providing an accurate time reference for the coordinated operation of multiple devices on the production line.

[0082] S203. Construct a 3D tensor using the aligned multimodal sensor data of different devices, and use Tucker to extract core features from the 3D tensor to obtain a multidimensional time series feature matrix.

[0083] 1. Data preprocessing and 3D tensor construction.

[0084] Before constructing a 3D tensor, the aligned multimodal sensor data from different devices must be further processed. For numerical data (such as temperature data from infrared cameras and sound pressure data from acoustic sensors), normalization methods are used to scale the data to the same scale. Common normalization methods include Z-score normalization and min-max normalization to eliminate the impact of dimensional differences on subsequent analysis.

[0085] Image data (captured by industrial cameras) must be resized to a fixed resolution, such as 224×224 pixels, using bilinear or bicubic interpolation. Image data must also be normalized to the range [0, 1] to facilitate integration with other modal data. Missing data can be filled using linear interpolation, K-nearest neighbor interpolation, or prediction methods based on machine learning models (such as random forests or neural networks).

[0086] Determining the meaning of tensor dimensions: The three dimensions of a 3D tensor correspond to the device dimension, the modal dimension, and the time dimension. Assume that a production line has M devices, each equipped with N modal sensors (such as industrial cameras, acoustic sensor arrays, and infrared thermal imagers), and the length of the acquired time series is T.

[0087] Data filling method: For each modality data of each device, the tensor is filled sequentially in chronological order. For example, the element in the tensor with the i-th device (1≤i≤M), the j-th modality (1≤j≤N), and the k-th time point (1≤k≤T) corresponds to the measurement value or image data (if the image data is image data, it must be flattened first) of the j-th modality of the i-th device at the k-th time point. The result is a 3D tensor X of size M×N×T.

[0088] 2. Tucker decomposition extracts core features.

[0089] Tucker decomposition is a high-order tensor decomposition method that decomposes the original tensor into a core tensor and multiple factor matrices. Its core idea is to extract key information from the tensor through low-rank approximation, reducing the data dimension while retaining the main features. For a 3D tensor X, Tucker decomposition can be expressed as: , where g is the core tensor, and U, V, and W are factor matrices corresponding to the device dimension, modality dimension, and time dimension, respectively.

[0090] Determine the decomposition rank: Before performing Tucker decomposition, you must predetermine the rank r1, r2, and r3 of the core tensor in each dimension (r1 ≤ M, r2 ≤ N, r3 ≤ T). The choice of rank directly affects the feature extraction effect and computational complexity. The optimal rank can be determined through cross-validation or methods based on information criteria (such as AIC and BIC).

[0091] Calculate the factor matrix: Use high-order singular value decomposition (HOSVD) to calculate the factor matrices U, V, and W. Taking the device dimension as an example, expand the tensor X in the device dimension to obtain the matrix X (1) , for X (1) Perform singular value decomposition (SVD): , take the first r1 singular vectors to form the factor matrix U. Similarly, calculate the factor matrices V and W of the modal dimension and time dimension.

[0092] Calculate the core tensor: According to the obtained factor matrices U, V, W, through the formula Calculate the core tensor g, the size of the core tensor g is r1×r2×r3, and its elements contain the core feature information of the original tensor.

[0093] Generate a multidimensional time series feature matrix: Flatten the core tensor g, expand it into a one-dimensional vector in a certain order (such as rows first, then columns), and then rearrange it into a two-dimensional matrix to obtain a multidimensional time series feature matrix F. Each row of the matrix represents the feature vector of a time point, and the number of columns is r1×r2×r3. This matrix can be used for subsequent fault diagnosis, equipment status prediction, and other tasks.

[0094] In an embodiment of the present invention, based on step S3, a possible embodiment will be given below to illustrate its specific implementation scheme in a non-limiting manner.

[0095] The teacher model includes: A three-dimensional multimodal feature extraction layer, comprising an input layer for receiving raw sensor data, a 3D convolution kernel group, and spatiotemporal pooling. The input layer employs grouped 3D convolution, with each group processing features of a different modality. The spatiotemporal pooling comprises dynamic temporal pooling and spatial maximum pooling. The dynamic knowledge graph embedding layer uses a graph embedding network to determine the topological embedding vector of each device based on the device's three-dimensional coordinates and the connection relationship between devices. It uses spatiotemporal graph convolution to determine the device status propagation risk code based on the historical fault diffusion path. It also uses a tree-shaped LSTM network to determine the compliance characteristics of the equipment process based on the process dependency tree. The multi-head cross attention fusion layer is used to associate the features extracted by the 3D multimodal feature extraction layer with the features extracted by the dynamic knowledge graph embedding layer to obtain the spatiotemporal-graph joint features; The spatiotemporal graph prediction layer is used to determine the production line status based on the spatiotemporal-graph-atlas joint features using the spatiotemporal gated graph network.

[0096] Student models include: Lightweight feature extractor: Bidirectional LSTM network: Time dimension grouping: multimodal sensor data are grouped and input according to physical characteristics; The hidden layer dimension is compressed to 1 / 4 of the teacher model; Add a temporal attention pooling layer to replace spatiotemporal pooling; Adversarial training module: Generator: multi-layer temporal convolutional network (TCN); Discriminator: The multi-scale structure includes: Original signal discrimination branch; Frequency domain feature discrimination branch; Statistics (Moments) discriminant branch.

[0097] Knowledge distillation process 1. Feature space alignment.

[0098] Cross-modal projection: Establish a mapping relationship between the teacher’s 3D convolutional features (C×T×H×W) and the student’s LSTM hidden state (D×T).

[0099] Loss function: SWD (Sliced ​​Wasserstein Distance) measures the difference in feature distribution.

[0100] 2. Dynamic knowledge transfer.

[0101] Graph Encoded Distillation: Embedding the teacher’s knowledge graph as a soft target:

[0102] Where τ is the temperature coefficient, which is adjusted dynamically:

[0103] Risk Communication Simulation: Construct a pseudo adjacency matrix to guide the student model:

[0104] Adversarial distillation: Generator objective function:

[0105] Feature matching loss:

[0106] Implement a two-stage distillation: Feature warm-up phase: Freeze the teacher model; Only optimize feature alignment loss ; Use Lookahead optimizer to accelerate convergence.

[0107] Against the Refining Stage: Jointly optimize the student model and the discriminator.

[0108] Loss function combination:

[0109] Introducing gradient penalty terms to stabilize training:

[0110] Compression acceleration strategy: Structured pruning: Implement block-level pruning on LSTM units to retain hidden connections at important time steps; Remove redundant gating units based on Hessian sensitivity analysis; Quantitative deployment: Using mixed precision quantization: Feature extractor: 8-bit dynamic quantization; Adversarial module: 16-bit floating point to preserve generation quality; Use the TVM graph optimizing compiler during deployment.

[0111] On the basis of the above embodiments, in order to further improve the accuracy of fault prediction, a feasible approach is to continuously fine-tune the model.

[0112] The prediction results and corresponding multimodal sensor data are pushed to the cloud via a message queue to fine-tune the teacher model in the cloud. The fine-tuned teacher model is then used to fine-tune the student model.

[0113] In some embodiments, the multimodal production line state inference system may include multiple functional modules composed of computer program segments. The computer program of each program segment in the multimodal production line state inference system may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Multimodal production line state reasoning capabilities.

[0114] In this embodiment, the multimodal production line state reasoning system can be divided into multiple functional modules according to the functions it performs, such as Figure 2 As shown. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0115] The data acquisition module is used to collect multimodal sensor data generated by multiple devices on the production line; The preprocessing module is used to align the timestamps of multimodal sensor data from different devices and fuse them to generate a multi-dimensional time series feature matrix; A prediction module, configured to input the multi-dimensional time series feature matrix into a lightweight prediction model pre-deployed in the edge processor and output a production line status prediction result; The lightweight prediction model is obtained by compressing the teacher model deployed in the cloud through the knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network.

[0116] Figure 3 The multimodal production line state reasoning method provided for the embodiment of the present application can be applied to equipment. Those skilled in the art will understand that the device structure involved in the embodiment of the present invention does not constitute a limitation on the device, and the device may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently. In the embodiment of the present invention, the device includes but is not limited to a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present application described and / or required herein.

[0117] The device 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will appreciate that the server structure shown in the figure does not limit the present invention. The server structure may be a bus structure or a star structure, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0118] The memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 can perform some or all of the steps in the above-described method embodiments.

[0119] The processor 310 is the control center of the storage device, which uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and / or processes data by running or executing software programs and / or modules stored in the memory 320, and calling data stored in the memory. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 310 can only include a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.

[0120] The communication unit 330 is configured to establish a communication channel so that the storage device can communicate with other devices, receive user data sent by other devices, or send user data to other devices.

[0121] The present invention also provides a computer medium, wherein the computer medium may store a program that, when executed, may include some or all of the steps of each embodiment provided by the present invention. The medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0122] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software and a necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a medium such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code, and includes instructions for causing a computer device (which can be a personal computer, a server, or a second device, a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0123] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

[0124] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or modules, and can be electrical, mechanical or other forms.

[0125] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected to achieve the purpose of the present embodiment according to actual needs.

[0126] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0127] Although the present invention has been described in detail with reference to the accompanying drawings and in conjunction with preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, persons of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and such modifications or substitutions shall be within the scope of the present invention. Any changes or substitutions that can be easily conceived by persons skilled in the art within the technical scope disclosed in the present invention shall be within the scope of protection of the present invention.

Claims

1. A multimodal production line state reasoning method, characterized in that: include: Collect multimodal sensor data generated by multiple devices on the production line; Align timestamps of multimodal sensor data from different devices and fuse them to generate a multi-dimensional time series feature matrix; Inputting the multi-dimensional time series feature matrix into a lightweight prediction model pre-deployed in the edge processor, and outputting a production line status prediction result; The lightweight prediction model is obtained by compressing the teacher model deployed in the cloud through the knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network.

2. The method according to claim 1, characterized in that Collect multimodal sensor data generated by multiple devices on the production line, including: Multimodal sensors are pre-deployed for multiple devices on the production line, and communication connections are established with the multimodal sensors of multiple devices; multimodal sensors include industrial cameras, acoustic sensor arrays, and infrared thermal imagers.

3. The method according to claim 1, characterized in that Align timestamps of multimodal sensor data from different devices and fuse them to generate a multi-dimensional time series feature matrix, including: For multimodal sensor data from a single device, significant event points are extracted for each modal data, and the time difference of the significant event points across modalities is added as a penalty term to the dynamic time warping loss function. The dynamic time warping loss function is used to align the timestamps of the multimodal sensor data of a single device; Map the timestamps of each device to a low-dimensional manifold space, use t-SNE to preserve the relative temporal relationship between devices, and align the timestamps between devices using a pre-built joint optimization loss function and the relative temporal relationship between devices; A 3D tensor is constructed using aligned multimodal sensor data from different devices, and Tucker is used to extract core features from the 3D tensor to obtain a multidimensional time series feature matrix.

4. The method according to claim 3, characterized in that For multimodal sensor data from a single device, significant event points are extracted from each modal data. The time difference between the significant event points across modalities is added as a penalty term to the dynamic time warping loss function. The dynamic time warping loss function is used to align the timestamps of the multimodal sensor data of a single device, including: Detect the instantaneous mutation of each modal data and mark the corresponding data point when the instantaneous mutation value exceeds the preset mutation threshold as a significant event point; Based on the significant event points in each modal data, cross-modal event anchor pairs are established between different modal data, and the alignment path is forced to pass through the cross-modal event anchor pairs ( ); Construct the dynamic time warping loss function: Among them, π is the alignment path, K is the number of matched event anchor points, is the data point corresponding to time point i in the first mode, is the data point corresponding to time point j in the second modality; λ is the parameter dynamically adjusted through adversarial learning; Determining the optimal cross-modal event anchor pair by minimizing the dynamic time warping loss function; Based on the optimal cross-modal event anchor point pair, timestamps of multimodal sensor data of a device are aligned using an interpolation method.

5. The method according to claim 3, characterized in that Map the timestamps of each device to a low-dimensional manifold space, use t-SNE to preserve the relative time relationship between devices, and align the timestamps between devices using a pre-built joint optimization loss function and the relative time relationship between devices, including: Map the device timestamp to the manifold space to obtain low-dimensional coordinates; Calculate the manifold time offset between low-dimensional coordinates of different devices; Adjust the manifold space through Procrustes analysis so that the reference event points of all devices coincide; The adjusted low-dimensional coordinates of each device are calibrated, and the calibrated low-dimensional coordinates are converted to a global time reference.

6. The method according to claim 1, characterized in that The teacher model includes: A three-dimensional multimodal feature extraction layer, comprising an input layer for receiving raw sensor data, a 3D convolution kernel group, and spatiotemporal pooling. The input layer employs grouped 3D convolution, with each group processing features of a different modality. The spatiotemporal pooling comprises dynamic temporal pooling and spatial maximum pooling. The dynamic knowledge graph embedding layer uses a graph embedding network to determine the topological embedding vector of each device based on the device's three-dimensional coordinates and the connection relationship between devices. It uses spatiotemporal graph convolution to determine the device status propagation risk code based on the historical fault diffusion path. It also uses a tree-shaped LSTM network to determine the compliance characteristics of the equipment process based on the process dependency tree. The multi-head cross attention fusion layer is used to associate the features extracted by the 3D multimodal feature extraction layer with the features extracted by the dynamic knowledge graph embedding layer to obtain the spatiotemporal-graph joint features; The spatiotemporal graph prediction layer is used to determine the production line status based on the spatiotemporal-graph-atlas joint features using the spatiotemporal gated graph network.

7. The method according to claim 1, characterized in that The method further comprises: The prediction results and corresponding multimodal sensor data are pushed to the cloud through a message queue to fine-tune the teacher model in the cloud.

8. A multimodal production line state reasoning system, characterized in that: include: The data acquisition module is used to collect multimodal sensor data generated by multiple devices on the production line; The preprocessing module is used to align the timestamps of multimodal sensor data from different devices and fuse them to generate a multi-dimensional time series feature matrix; A prediction module, configured to input the multi-dimensional time series feature matrix into a lightweight prediction model pre-deployed in the edge processor and output a production line status prediction result; The lightweight prediction model is obtained by compressing the teacher model deployed in the cloud through the knowledge distillation method. The lightweight prediction model includes a feature extractor composed of an LSTM network and an adversarial training module composed of a GAN network.

9. A device, characterized in that include: A memory for storing a multimodal production line state reasoning program; A processor is configured to implement the steps of the multimodal production line state reasoning method according to any one of claims 1 to 7 when executing the multimodal production line state reasoning program.

10. A computer-readable medium storing a computer program, characterized in that: The readable medium stores a multimodal production line state reasoning program, which, when executed by a processor, implements the steps of the multimodal production line state reasoning method according to any one of claims 1 to 7.

Images