Material batching and conveying process monitoring system based on digital twinning

Abstract

The invention discloses a material batching and conveying process monitoring system based on digital twinning, and belongs to the technical field of industrial automation and intelligent manufacturing. The system comprises a physical entity layer used for storing, proportioning and conveying materials; the data acquisition and sensing layer is used for acquiring multi-source heterogeneous data of materials and equipment in the physical entity layer in real time; the digital twinborn model layer is used for constructing and operating a digital twinborn model synchronously mapped with the physical entity layer; and the application service layer is internally provided with a multi-modal fusion fault diagnosis module, and the multi-modal fusion fault diagnosis module is used for diagnosing material batching and conveying faults. According to the method, the digital twinborn and multi-modal AI algorithms are deeply fused, the early warning and accurate root cause positioning of equipment faults are realized, the diagnosis confidence is improved through evidence fusion, the process decision is optimized in combination with a simulation sand table, the production continuity and quality stability are improved, the non-planned shutdown and maintenance cost is reduced, and the system is endowed with the continuous self-evolution capability.

Description

Technical Field

[0001] This invention relates to the field of industrial automation and intelligent manufacturing technology, specifically to a material batching and conveying process monitoring system based on digital twins. Background Technology

[0002] In the production of specialty materials such as rubber and plastics, precise and stable batching and continuous and reliable conveying are core elements to ensure final product quality, production efficiency, and operational safety. However, this process involves a wide variety of equipment, complex mechanisms, and variable material properties, posing significant challenges to traditional production process monitoring and fault diagnosis methods, primarily in the following aspects: 1. Limited Perception Dimensions and Delayed Diagnosis: Most existing monitoring systems rely on single-type sensors (such as monitoring only motor current or a single point temperature) for simple threshold alarms. This approach cannot comprehensively capture the mechanical vibration state of the equipment, electrical harmonic characteristics, and the surface quality of the material itself, creating "perception islands." When conveyor blockages, batching deviations, or equipment performance degradation occur, they are often only discovered after the fault has caused obvious consequences (such as downtime or scrap), resulting in severe diagnostic delays, long unplanned downtime, and high maintenance costs.

[0003] 2. Lack of correlation analysis and root cause localization capabilities: A production line is a complex system with tightly coupled equipment. Local failures often trigger chain reactions through material and energy flows. Existing methods lack the ability to model the topological relationships between equipment and the fault propagation path. When problems occur, it is difficult to quickly distinguish whether the cause is a fault in the equipment itself, improper matching of upstream and downstream processes, or changes in material characteristics. Root cause localization is difficult, and problems are prone to recurrence.

[0004] 3. Lack of predictive maintenance and reliance on experience for decision-making: Traditional maintenance models are mostly periodic preventative maintenance or reactive repairs, lacking the ability to predict based on the real-time health status of equipment. Adjustments to process parameters also heavily rely on operator experience, failing to adaptively optimize based on material characteristic fluctuations and equipment status changes. This can lead to wasted maintenance resources or an inability to effectively intervene before failures occur.

[0005] To address the aforementioned issues, there is an urgent need for a material batching and conveying process monitoring system based on digital twins to solve the problems associated with traditional methods. Summary of the Invention

[0006] The purpose of this invention is to provide a material batching and conveying process monitoring system based on digital twins, which deeply integrates digital twins and multimodal AI algorithms to achieve early warning and accurate root cause localization of equipment failures. By enhancing diagnostic confidence through evidence fusion and optimizing process decisions through simulation, it significantly improves production continuity and quality stability, reduces unplanned downtime and maintenance costs, and endows the system with continuous self-evolution capabilities.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A material batching and conveying process monitoring system based on digital twins, comprising: The physical layer is used for the storage, proportioning, and transportation of materials; The data acquisition and sensing layer is used to acquire multi-source heterogeneous data of materials and equipment in the physical entity layer in real time. The multi-source heterogeneous data includes time-series signals of vibration and current, as well as visual images of the material surface. The digital twin model layer is used to build and run digital twin models that are synchronously mapped to the physical entity layer; The application service layer contains a multimodal fusion fault diagnosis module. This module is used to diagnose faults in material batching and conveying. It includes a time-series prediction diagnosis module, a visual diagnosis module, a system-level topology analysis module, and an evidence fusion diagnostic tool. The time-series prediction diagnosis module uses a bidirectional long short-term memory network optimized based on an improved raccoon algorithm. The visual diagnosis module uses a convolutional neural network improved based on multi-scale receptive fields and adjustment optimization layers. The system-level topology analysis module uses a spatiotemporal graph convolutional network based on an improved output layer and activation function. The evidence fusion diagnostic tool is used to perform decision-level fusion of the module outputs.

[0008] Furthermore, the input to the time-series prediction and diagnosis module is a preprocessed time-series signal of vibration and current, and the output is a deep feature encoding vector and predicted values ​​for multiple future time steps.

[0009] Furthermore, the time-series prediction and diagnostic module includes: The BiLSTM network prediction unit is trained using the focus loss function on the basis of the original standard bidirectional long short-term memory network. An improved Raccoon algorithm optimization unit is used to adaptively search for the hyperparameters of the prediction unit in a BiLSTM network.

[0010] Furthermore, the hyperparameters of the BiLSTM network prediction unit optimized by the improved raccoon algorithm optimization unit include at least the number of hidden layer neurons, the learning rate, and the dropout rate.

[0011] Furthermore, the visual diagnostic module takes as input a visual image of the material surface after image enhancement, and outputs as a depth visual feature encoding vector, defect classification result, segmentation mask map, and severity index.

[0012] Furthermore, the improved convolutional neural network structure based on multi-scale receptive fields and adjustment optimization layers is specifically as follows: An adjustment and optimization layer is added to the original convolutional neural network structure to find the balance point between linearity and nonlinearity through continuous training, thereby reducing gradient loss. In the convolutional layers of the original convolutional neural network structure, the receptive field is adjusted by introducing convolutional kernels of a specific size, and the stride configuration is used to control the stride of the convolutional kernel as it slides across the image, thereby further adjusting the size of the receptive field and the sampling density of the features.

[0013] Furthermore, the system-level topology analysis module includes: The system spatiotemporal graph construction unit is used to abstract key equipment in the physical entity layer into graph nodes, and to abstract the material flow, energy flow and information flow relationships between equipment into edges, constructing a weighted adjacency matrix; at the same time, it integrates multi-source heterogeneous data to generate a time-series state sequence with multi-dimensional characteristics for each node, which together constitute the system spatiotemporal graph. An improved spatiotemporal graph convolutional network is used to perform spatiotemporal convolution and feature fusion processing on the input system spatiotemporal graph.

[0014] Furthermore, the improved spatiotemporal graph convolutional network is specifically as follows: Based on the original spatiotemporal graph convolutional network, the fully connected output layer at the end of the network is replaced with a temporally gated convolutional layer, and the ReLU activation function in the original spatiotemporal graph convolutional network is replaced with the LeakyReLU activation function.

[0015] In summary, the present invention has at least one of the following beneficial technical effects: 1. Achieved cross-modal deep perception and full-dimensional status monitoring: The system synchronously acquires vibration, current timing signals, and visual images of material surfaces through the data acquisition layer, constructing a full-dimensional perception system covering the mechanical and electrical health of equipment and the surface quality of materials. The digital twin model layer further integrates these data with the equipment's 3D model and process logic, providing a panoramic and contextualized monitoring view, completely changing the traditional one-sided and isolated situation of monitoring.

[0016] 2. Significantly improves the accuracy, foresight, and interpretability of fault diagnosis: Improved accuracy: The evidence fusion diagnostic tool adopts an improved weighted DS evidence theory to perform decision-level fusion of results from three heterogeneous diagnostic sources: time series, vision, and system topology. This effectively solves the uncertainty and conflict problems of single evidence, making the confidence of the comprehensive diagnostic conclusion much higher than that of any single module, and significantly reducing the false alarm and false negative rates.

[0017] Enhanced for Foresight: The time-series predictive diagnostic module utilizes an optimized BiLSTM network based on an improved Raccoon algorithm. It can predict the degradation trend of key performance indicators of equipment several hours in advance based on historical and current data, achieving true predictive maintenance and transforming fault handling from post-remediation to pre-intervention.

[0018] Improved interpretability: The system-level topology analysis module, through an improved spatiotemporal graph convolutional network, can clearly demonstrate the propagation path and root cause nodes of faults in the device network. The evidence fusion diagnostic tool's report clearly explains how different pieces of evidence collectively support the final diagnostic conclusion, providing an interpretable diagnostic chain that approximates expert reasoning.

[0019] 3. Possesses system-level root cause analysis and self-evolution capabilities: Root cause analysis: The system not only reports abnormal phenomena, but can also pinpoint the root cause equipment and specific components that cause a series of abnormalities, and explain the mechanism of the impact, providing direct guidance for precise maintenance.

[0020] Self-evolution capability: The evidence fusion diagnostic tool's built-in active learning trigger mechanism can guide experts to intervene and annotate complex cases with high conflict and low confidence, and use the newly annotated data to incrementally update each diagnostic model. Combined with the continuous accumulation of diagnostic cases from the domain knowledge graph, the entire system possesses the vitality to continuously learn from experience and adapt to new failure modes.

[0021] 4. A complete intelligent closed loop of perception, diagnosis, decision-making, and optimization has been formed: This invention goes beyond monitoring and diagnosis. The "What-If" simulation sandbox capability provided by the digital twin model layer allows for the safe testing of process optimization schemes or maintenance strategies in a virtual environment. The application service layer can automatically generate maintenance work orders and optimize process parameters based on diagnostic results, and feed the instructions back to the control system, thereby driving the production process to continuously evolve towards higher efficiency, reliability, and quality, realizing a complete closed loop from intelligent perception to intelligent decision-making and execution. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0024] like Figure 1 As shown, this invention provides a material batching and conveying process monitoring system based on digital twins, comprising: The physical layer is used for the storage, proportioning, and transportation of materials; The data acquisition and sensing layer is used to acquire multi-source heterogeneous data of materials and equipment in the physical entity layer in real time. The multi-source heterogeneous data includes time-series signals of vibration and current, as well as visual images of the material surface. The digital twin model layer is used to build and run digital twin models that are synchronously mapped to the physical entity layer; The application service layer contains a multimodal fusion fault diagnosis module. This module is used to diagnose faults in material batching and conveying. It includes a time-series prediction diagnosis module, a visual diagnosis module, a system-level topology analysis module, and an evidence fusion diagnostic tool. The time-series prediction diagnosis module uses a bidirectional long short-term memory network optimized based on an improved raccoon algorithm. The visual diagnosis module uses a convolutional neural network improved based on multi-scale receptive fields and adjustment optimization layers. The system-level topology analysis module uses a spatiotemporal graph convolutional network based on an improved output layer and activation function. The evidence fusion diagnostic tool is used to perform decision-level fusion of the module outputs.

[0025] Next, each module will be introduced in turn: I. Physical Entity Layer The specific composition of the physical entity layer can be adapted to different production processes, and typically includes: 1. Storage Unit: Primarily used for temporary storage and buffering of materials to ensure continuous material supply for subsequent processes. This unit specifically includes: (1) Raw material silo: usually a silo or square silo, used to store powdery or granular solid raw materials, such as PVC resin powder and rubber granules. The lower part of the silo is equipped with a discharge device, such as a gate valve or a rotary feeder.

[0026] (2) Liquid storage tank: used to store liquid additives or auxiliaries, such as plasticizers and stabilizers, and equipped with a level gauge, agitator and transfer pump.

[0027] (3) Intermediate buffer hopper: Located between two processes, it is used to temporarily store materials that have been proportioned or initially mixed to balance the production cycle.

[0028] 2. Proportioning Unit: Its core function is to accurately measure and add various materials according to the formula ratio, which is a crucial step in ensuring the quality of the final product. This unit specifically includes: (1) Weighing batching equipment: such as loss-in-weight scales, which control the feeding speed by monitoring the changes in the weight of the hopper in real time to achieve high-precision continuous metering; or weight-adding batching scales, which are used for batch proportioning.

[0029] (2) Volumetric batching equipment: such as screw feeders and vibrating feeders, which measure volumetric flow rate by controlling rotation speed or amplitude. They are suitable for occasions with relatively low accuracy requirements or as pre-feeding equipment.

[0030] (3) Liquid metering equipment: such as metering pumps, gear pumps, plunger pumps or mass flow meters, used to accurately control the amount of liquid materials added.

[0031] 3. Mixing and Conveying Unit: Responsible for the mixing, homogenization, reaction, and displacement of materials from one point to another. This unit consists of a variety of tightly coupled components, specifically including: (1) Mixing equipment: such as high-speed mixers and ribbon mixers, used to physically mix various solid or solid-liquid materials; twin-screw extruders have multiple functions such as mixing, melting, reaction and conveying in specific processes.

[0032] (2) Conveying equipment: Continuous conveying equipment: belt conveyors (for horizontal or inclined conveying), screw conveyors (suitable for powder and short-distance sealed conveying), bucket elevators (for vertical lifting), and pneumatic conveying systems (for long-distance pipeline conveying of powder). Power and transmission components: various drive motors, gearboxes, couplings, bearings, rollers, conveyor belts, etc., which are the main carriers of vibration and current signals; Fluid transport equipment: pumps and valves, used for transporting liquid materials or melts and controlling their flow.

[0033] These devices are physically connected through pipes, conveyor belts, mechanical connectors, etc., forming a production system.

[0034] II. Data Acquisition and Sensing Layer It consists of various sensors, data acquisition hardware, and preprocessing modules deployed at key nodes of the physical entity layer, ensuring that raw data is acquired accurately, completely, and synchronously. A detailed introduction follows: 1. Acquisition of vibration timing signals: Sensor type: Piezoelectric accelerometer or IEPE accelerometer is used. These sensors have a wide frequency response range, typically covering several Hz to tens of kHz, and can effectively capture broadband vibrations caused by equipment malfunctions such as imbalance, misalignment, bearing damage, and abnormal gear meshing.

[0035] Deployment Location: Based on the fault diagnosis mechanism, the sensor is fixed on the bearing housing or casing of critical rotating machinery, and the installation direction must cover both radial and axial directions. For example, it needs to be deployed on the bearing housing of the non-drive end of the drive motor, the bearing housing of the input and output shafts of the gearbox, and the pump body of a large pump.

[0036] Data Characteristics: The acquired raw vibration signal is a continuous time-series analog signal. After anti-aliasing filtering and analog-to-digital conversion by the data acquisition card, it becomes a high-sampling-rate discrete digital sequence, typically 2.56 to 4 times the highest frequency of interest, such as 12.8 kHz. This signal directly reflects the mechanical motion state of the equipment and is one of the core inputs of the timing prediction and diagnostic module.

[0037] 2. Acquisition of current timing signals: Sensor type: The three-phase current of the motor drive circuit is measured non-invasively using a current transformer or Hall effect current sensor.

[0038] Deployment location: It is usually installed in the motor control cabinet and connected to the main power supply line of the motor.

[0039] Data Characteristics: The acquired current signal is also a continuous time series. It not only contains fundamental frequency information but also rich harmonic components. Electrical faults in the motor windings, broken rotor bars, and load torque fluctuations will all generate characteristic harmonics in the current spectrum. The current signal and vibration signal are strictly synchronized in time, together forming the input pair for analyzing the overall electromechanical status of the equipment.

[0040] 3. Acquisition of visual images of material surfaces: Image acquisition equipment: Industrial area scan cameras or line scan cameras are used. Area scan cameras are used for fixed-point, intermittent shooting; line scan cameras achieve continuous, high-resolution surface scanning imaging by moving synchronously with the material being measured, and are particularly suitable for online detection of continuous materials on conveyor belts.

[0041] Deployment Location and Lighting: Cameras are installed at key quality monitoring points, such as the mixer outlet, conveyor belt turning points, or after cooling and setting. To ensure image quality, a dedicated industrial lighting system, such as LED strip lights, backlights, or coaxial lights, must be used to overcome ambient light interference and highlight surface textures, dents, scratches, impurities, and other features.

[0042] Data characteristics: The acquired visual images are two-dimensional spatial information. Images are transmitted in frames, and the frame rate must be matched with the material's running speed to ensure coverage. Image data is the sole input source for the visual diagnostic module, used to directly assess the physical state of the material itself.

[0043] 4. Data synchronization and preprocessing: Synchronization: All sensor data are synchronized at the millisecond level via the IEEE 1588 precision time protocol or hardware trigger signal, ensuring that multi-source data are aligned in time and space, laying the foundation for subsequent fusion analysis.

[0044] Preprocessing: Before being sent to the advanced diagnostic module, the raw data needs to undergo preliminary processing. For time-series signals, this includes noise reduction (such as wavelet thresholding) and missing value interpolation; for images, it includes geometric correction, contrast enhancement, and size normalization. These preprocessing steps are performed in the edge computing unit of the data acquisition layer or in the host computer, aiming to improve data quality without changing the basic modality of the data or the properties defined in the claims.

[0045] III. Digital Twin Model Layer The construction of the digital twin model layer is a systematic project, and its digital twin model includes the following multi-dimensional and multi-scale sub-models: 1. Geometric and Topological Models: Based on the actual size, layout, and connection relationships of the equipment in the physical entity layer, a precise three-dimensional geometric model is constructed using three-dimensional modeling technology. This includes the shape, size, and spatial location of all key equipment such as silos, conveyors, pipes, valves, and motors.

[0046] Based on the geometric model, the topological connections between devices are defined, clarifying the paths of material flow, energy flow, and signal flow. For example, "hopper A" is defined to be connected to "screw feeder B" through "discharge valve V1," forming a directed connection graph. This topological relationship serves as the static basis for subsequent system-level behavioral simulation and analysis.

[0047] 2. Physical and mechanistic models: Embed mechanistic models reflecting the inherent operating principles of key equipment and processes. For example: For conveyor belts, a dynamic model based on Newtonian mechanics can be established to simulate the changes in speed and tension under different loads; For batching scales, a weighing and feeding control model based on mass conservation can be established; For mixing processes, a simplified mixing uniformity model based on fluid dynamics or particle dynamics can be established; For rotating equipment such as motors and pumps, a common efficiency-load characteristic curve model can be integrated; These mechanistic models are described using differential equations, state equations, or empirical formulas, enabling digital twin models to simulate the fundamental responses of physical entities under specific input conditions, rather than simply replicating their appearance.

[0048] 3. Behavior and Rule Model: Define the operating logic and control rules of the equipment and system. For example, define the "automatic batching" work cycle of the batching scale: start-up - fast feeding - slow feeding - weighing stabilization - unloading.

[0049] Simulate the interlocking and safety logic of the production line. For example, when the downstream conveyor stops, the upstream feeder should automatically interlock and stop to prevent material blockage.

[0050] These behavioral rules are typically implemented using state machines, flowcharts, or scripting languages, enabling digital twin models to realistically replicate the automated operation processes of physical systems.

[0051] 4. Data-driven model interface: Although the core diagnostic algorithms reside in the application service layer, the digital twin model layer provides standardized data access and fusion interfaces for these algorithms. This layer performs unified temporal alignment, spatial registration, and formatting on raw, heterogeneous data from multiple sources, such as vibration, current, and images, from the data acquisition and sensing layer.

[0052] It provides two key data interfaces for upper-layer services: one is the real-time data stream interface, which continuously pushes the latest status data of each monitoring point; the other is the historical and contextual data query interface, which allows upper-layer services to query historical data and corresponding operating contexts (such as the production formula and ambient temperature at that time) by time, equipment, process section and other dimensions.

[0053] 5. Operation and Synchronization Mapping of Digital Twin Models (1) Model-driven and execution: The digital twin model runs in a dedicated simulation engine. The simulation engine receives real-time control commands from the physical entity layer control system and real-time sensor data from the data acquisition layer as inputs. The mechanistic model and behavioral rule model perform calculations and state deductions based on these inputs, driving the equipment components in the three-dimensional geometric model to produce corresponding motion animations.

[0054] (2) State synchronization mapping: This is the key to achieving synchronization mapping. The system ensures consistency between the virtual and physical worlds through the following mechanisms: Data-driven synchronization: The real-time state of key, measurable physical entities (such as the actual speed of a motor, the actual level of material in a silo, and the actual opening of a valve) is injected into the corresponding variables in the digital twin model, overriding the model's predicted values ​​and forcing the state of the digital twin model to closely match physical reality. This is the most basic synchronization.

[0055] Simulation Prediction and Calibration: For states that cannot be directly measured or whose measurement is delayed, the digital twin model uses its mechanistic model to perform real-time simulation prediction. Simultaneously, the system periodically compares and calibrates the model's predictions with subsequently arriving actual measurement data, dynamically adjusting model parameters (such as friction coefficient and efficiency coefficient) to maintain the model's long-term prediction accuracy.

[0056] (3) "What-If" Simulation Sandbox Capability: In addition to maintaining a master copy synchronized with the physical entity in real time, the digital twin model layer can quickly clone parallel simulation sandbox instances based on the current state snapshot. Within the sandbox, users can safely modify process parameters, inject simulated equipment faults, or adjust control logic, and observe the future evolution of the digital twin model under these assumptions. This function provides a zero-risk testing environment for advanced services such as process optimization and maintenance scheme verification in the application service layer.

[0057] IV. Application Service Layer 1. Time Series Prediction and Diagnostic Module The input to the time-series prediction and diagnosis module is the pre-processed time-series signals of vibration and current, and the output is a deep feature encoding vector and predicted values ​​for multiple future time steps.

[0058] The time-series prediction and diagnostic module includes: The BiLSTM network prediction unit is trained using the focus loss function on the basis of the original standard bidirectional long short-term memory network (BiLSTM network); An improved Raccoon algorithm optimization unit is used to adaptively search for the hyperparameters of the prediction unit in a BiLSTM network.

[0059] The specific structure of the prediction unit in the BiLSTM network will be explained in detail below: This invention improves upon the original BiLSTM network. First, it should be noted that since the original BiLSTM network structure is existing technology, it will not be described in detail. Instead, the improvements will be introduced, specifically: Introducing the focus loss function The mathematical expression for the focus loss function is: (1) In the formula, FL represents the focus loss function, γ represents the adjustment parameter, and α represents the adjustment parameter. ｋ p represents the weighting coefficient. ｋ This represents the probability of correctly predicting k classes of samples.

[0060] Next, the improved raccoon algorithm optimization unit will be introduced: This invention improves upon the original Raccoon algorithm. First, it should be noted that the original Raccoon algorithm is existing technology, and this invention will not describe it in detail. Instead, it will focus on the improvements made to the algorithm, which are as follows: (1) Initialize the population using the best point set During the raccoon initialization phase, the randomly generated population is unevenly distributed across the solution space, resulting in low population diversity. This invention employs a method based on optimal point sets to initialize the raccoon population.

[0061] (2) Convex lens imaging reverse learning During the raccoon exploration phase, the iguana's location corresponds to the global optimal solution, which is crucial for improving the algorithm's convergence speed. After the iguana falls, its location is randomly generated within the search space, but the probability of randomly generating a good solution is very small. This invention employs a back-learning strategy based on the lens imaging principle to determine the iguana's landing location.

[0062] (3) Cross-sectional strategy In the development phase of the raccoon algorithm, the position update method relies on the number of iterations and the boundary, which limits the algorithm's ability to explore the optimal solution and easily leads to local optima. Research shows that the cross-sectional search strategy can help the algorithm escape the local optimum trap. This invention introduces a cross-sectional search strategy based on the development search, including two dimensions: horizontal cross-section and vertical cross-section.

[0063] Lateral crossover is a crossover variation that occurs between two individuals along the same dimension. The specific formula is as follows: (2) (3) In the formula, r1 and r2 are random numbers within (0, 1); c1 and c2 are random numbers within (-1, 1); X1 and X2 are random individuals, X h1 and X h2 For the new individuals generated after horizontal crossover, a greedy selection is made between the old and new individuals; Longitudinal crossover is the crossover variation among all individuals across two different dimensions, as shown in the following formula: (4) In the formula, r3 is a random number within (0, 1); X3 and X4 are the values ​​of the same individual in two dimensions; X z For the new individuals generated after vertical crossover, the individuals with better results are retained between the old and new individuals.

[0064] The hyperparameters of the BiLSTM network prediction unit optimized by the improved raccoon algorithm optimization unit include at least the number of hidden layer neurons, the learning rate, and the dropout rate.

[0065] 2. Visual Diagnostic Module The visual diagnostic module takes an image-enhanced visual image of the material surface as input and outputs a depth visual feature encoding vector, defect classification results, segmentation mask, and severity index.

[0066] The improved convolutional neural network structure based on multi-scale receptive fields and adjustment optimization layers is specifically as follows: An adjustment and optimization layer is added to the original convolutional neural network structure to find the balance point between linearity and nonlinearity through continuous training, thereby reducing gradient loss. In the convolutional layers of the original convolutional neural network structure, the receptive field is adjusted by introducing convolutional kernels of a specific size, and the stride configuration is used to control the stride of the convolutional kernel as it slides across the image, thereby further adjusting the size of the receptive field and the sampling density of the features.

[0067] Specifically, it includes: The input layer performs filtering, cropping, and normalization on the input images. Convolutional layers are used to obtain feature sets. The number and size of output features are determined by the size of the convolution kernel. The output image after convolution is: (5) In the formula, S refers to the output size of the image after the convolution operation, A refers to the input size of the image before the convolution operation, B refers to the size of the convolution kernel, C refers to the size of the auxiliary pixels, and D refers to the stride. To ensure the accuracy of the monitoring and to better reflect real-world conditions, an activation function is added, with the following expression: (6) In the activation function β, if h is greater than 0, the variable value is h; if h is less than 0, the variable value is 0. Thus, the convolutional layer features are as follows: (7) In the formula, i = 1, 2, ..., M k-1 j=1,2,…,M k M k Refers to the feature quantity of layer k. Refers to the output of features; The term refers to the convolution of the feature weights of the k-th layer with the feature map of the (k-1)-th layer, and g refers to the bias.

[0068] Pooling layers are mainly used to address the problem of decreased computational efficiency during monitoring caused by the high-dimensional features of convolutional layers. Therefore, pooling layers are used to reduce the dimensionality of convolutional layers.

[0069] Fully connected layers. Because fully connected layers have many training parameters, overfitting can occur. To address this, the array string is concatenated with the feature vector to form a fully connected layer, and the ReLU function is used as the activation function to prevent overfitting.

[0070] The optimization layer is adjusted. In this layer, nonlinear features are enhanced by adding image scaling and translation, and continuous training is performed to find a balance between linearity and nonlinearity, thereby improving convergence speed and reducing gradient loss. The optimized features are as follows: (8) In the formula, V refers to the total number of samples, h v γ refers to the information of the input adjustment and optimization layer, φ refers to the scaling amount, and φ refers to the translation amount.

[0071] Output layer. The final features obtained from the output layer.

[0072] 3. System-level topology analysis module The system-level topology analysis module includes: The system spatiotemporal graph construction unit is used to abstract key equipment in the physical entity layer into graph nodes, and to abstract the material flow, energy flow and information flow relationships between equipment into edges, constructing a weighted adjacency matrix; at the same time, it integrates multi-source heterogeneous data to generate a time-series state sequence with multi-dimensional characteristics for each node, which together constitute the system spatiotemporal graph. An improved spatiotemporal graph convolutional network is used to perform spatiotemporal convolution and feature fusion processing on the input system spatiotemporal graph.

[0073] The output of the system-level topology analysis module is the system-level state coding vector and the influence analysis results between nodes.

[0074] The improved spatiotemporal graph convolutional network is specifically as follows: Based on the original spatiotemporal graph convolutional network, the fully connected output layer at the end of the network is replaced with a temporally gated convolutional layer, and the ReLU activation function in the original spatiotemporal graph convolutional network is replaced with the LeakyReLU activation function. Since the temporally gated convolutional layer and the LeakyReLU activation function are described in the prior art, they will not be described in detail here.

[0075] 4. Evidence fusion diagnostic device The evidence fusion diagnostic tool is used to perform decision-level fusion of the module's output; (1) Input: Deep feature encoding vector and predicted values ​​at multiple future time steps; Deep visual feature encoding vectors, defect classification results, segmentation mask images, and severity indices; Results of system-level state coding vector and influence analysis between nodes; (2) Output: Comprehensive Diagnostic Report: The final output of the evidence fusion diagnostic tool is a structured report that includes at least: Most likely fault conclusion: The proposition with the highest reliability in the fusion BPA, such as "Comprehensive diagnosis: partial blockage of feed hopper S3 discharge valve (fusion reliability: 0.89)".

[0076] Root Cause Analysis and Propagation Chain: Integrating the reasoning of the system topology modules, it clearly describes "how blockage leads to downstream feeder load fluctuations (temporal evidence), and ultimately causes uneven mixing, forming specific textures on the material surface (visual evidence)".

[0077] Secondary risk warning: Based on the predictive capabilities of spatiotemporal graph convolutional networks, a warning is given such as "It is expected that the mixer may be overloaded and shut down in 40 minutes".

[0078] Global fusion confidence level and uncertainty interval: clearly inform users of the credibility of the conclusion.

[0079] Recommended measures: such as "It is recommended to immediately check and clean the discharge valve of the S3 silo, and temporarily increase the downstream drying temperature by 5°C to mitigate the impact of material moisture."

[0080] Knowledge Graph Updates: For each successful fusion diagnosis (regardless of expert confirmation), its complete multimodal data flow, feature vectors, BPA evolution process, and final conclusions are structured and stored in the system's domain knowledge graph. This graph constructs multidimensional relationships between "fault phenomena - multimodal features - equipment components - process parameters," forming searchable, comparable, and reasonable enterprise knowledge assets, providing strong case support for future diagnoses.

[0081] (3) The specific process is as follows: The evidence fusion diagnostic tool adopts Dempster-Shafer (DS) evidence theory as the basic fusion framework and has made key improvements for industrial diagnostic scenarios, forming a conflict evidence fusion method based on adaptive weights.

[0082] Basic probability assignment construction: First, an identification framework Θ is constructed for each diagnostic module, which is the set of all possible fault propositions. For example, Θ = {bearing wear, gear damage, misalignment, material clumping, normal}.

[0083] Each module transforms its output into a basic probability assignment function on that framework. This function assigns a probability mass to each subset of the framework (a single proposition or a combination of propositions), and the sum of the probability masses for all subsets is 1. Masses not assigned to any explicit proposition are given an uncertainty term m(Θ) to quantify the total uncertainty of the source of evidence.

[0084] Example: The timing module may be assigned: m t (Bearing wear) = 0.70, m t (Θ) = 0.30.

[0085] Conflict detection and adaptive weighting: Traditional DS theory may produce counterintuitive results when dealing with highly conflicting evidence. This invention introduces a conflict coefficient K to measure the degree of inconsistency between different sources of evidence.

[0086] The key improvement lies in not treating all sources of evidence equally. The system dynamically calculates a credibility weight w for each source of evidence. This weight is based on: a) Historical diagnostic accuracy: The verification accuracy of the module's diagnostic results over a past period of time.

[0087] b) The inherent uncertainty of the current evidence: that is, the size of the evidence's own m(Θ). The higher the uncertainty, the lower the weight.

[0088] c) Consistency among evidence: Evidence that is highly consistent with other sources of evidence receives higher weight.

[0089] Before merging, each BPA is weighted and adjusted: m i '(A)=w i *m i (A), and for m i '(Θ) Make appropriate compensation to ensure that the sum is 1.

[0090] Evidence synthesis and decision-making: The Dempster combination rule is used to synthesize multiple weighted BPAs pairwise. This rule, through orthogonal summation, can effectively aggregate evidence supporting the same proposition and handle conflicting evidence in a compromise manner.

[0091] After multiple rounds of synthesis, the fused BPA is obtained. Finally, by calculating the confidence function and likelihood function of each proposition, a probability interval [confidence, likelihood] is obtained, rather than a single probability. This more scientifically expresses the uncertainty that still exists after fusion.

[0092] Active learning trigger mechanism: The evidence fusion diagnostic tool continuously monitors the fusion results. An active learning loop is triggered when the following conditions are met: a) High conflict: The conflict coefficient K exceeds the threshold, indicating that the existing model cannot explain the current situation.

[0093] b) Low confidence: The confidence of all propositions is lower than the preset threshold.

[0094] At this point, the system packages all the current multimodal raw data, the preliminary diagnostic results of each module, and the fusion process, marks them as "difficult cases," and pushes them to domain experts through the human-machine interface for final adjudication and annotation. The expert annotation results will be used to update the training set of each diagnostic module, enabling the system to self-evolve.

[0095] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] Contents not described in detail in this specification are prior art known to those skilled in the art. It is hereby indicated that the above description is intended to help those skilled in the art understand this invention, but does not limit the scope of protection of this invention. Any equivalent substitutions, modifications, improvements, or simplifications of the above descriptions that do not depart from the essential content of this invention fall within the scope of protection of this invention.

Claims

1. A material batching and conveying process monitoring system based on digital twins, characterized in that, include: The physical layer is used for the storage, proportioning, and transportation of materials; The data acquisition and sensing layer is used to acquire multi-source heterogeneous data of materials and equipment in the physical entity layer in real time. The multi-source heterogeneous data includes time-series signals of vibration and current, as well as visual images of the material surface. The digital twin model layer is used to build and run digital twin models that are synchronously mapped to the physical entity layer; The application service layer contains a multimodal fusion fault diagnosis module for diagnosing faults in material batching and conveying. This module includes a temporal prediction diagnosis module, a visual diagnosis module, a system-level topology analysis module, and an evidence fusion diagnostic tool. The temporal prediction diagnosis module employs a bidirectional long short-term memory network optimized based on an improved raccoon algorithm. The visual diagnosis module uses a convolutional neural network improved based on multi-scale receptive fields and an adjustment optimization layer. The adjustment optimization layer enhances nonlinear features by adding scaling and translation of images and continuously trains to find a balance between linearity and nonlinearity. The system-level topology analysis module is an improved spatiotemporal graph convolutional network. Specifically, the improvement process involves changing the output layer of the original spatiotemporal graph convolutional network to a time-gated convolutional layer and changing the activation function of the original spatiotemporal graph convolutional network to the LeakyReLU activation function. The evidence fusion diagnostic tool performs decision-level fusion of the module's outputs.

2. The material batching and conveying process monitoring system based on digital twins according to claim 1, characterized in that, The input to the time-series prediction and diagnosis module is the pre-processed time-series signals of vibration and current, and the output is a deep feature encoding vector and predicted values ​​for multiple future time steps.

3. The material batching and conveying process monitoring system based on digital twins according to claim 2, characterized in that, The time-series prediction and diagnostic module includes: The BiLSTM network prediction unit is trained using the focus loss function on the basis of the original standard bidirectional long short-term memory network. An improved Raccoon algorithm optimization unit is used to adaptively search for the hyperparameters of the prediction unit in a BiLSTM network.

4. The material batching and conveying process monitoring system based on digital twin according to claim 3, characterized in that, The hyperparameters of the BiLSTM network prediction unit optimized by the improved raccoon algorithm optimization unit include at least the number of hidden layer neurons, the learning rate, and the dropout rate.

5. A material batching and conveying process monitoring system based on digital twins according to claim 4, characterized in that, The visual diagnostic module takes an image-enhanced visual image of the material surface as input and outputs a depth visual feature encoding vector, defect classification results, segmentation mask, and severity index.

6. The material batching and conveying process monitoring system based on digital twins according to claim 5, characterized in that, The improved convolutional neural network structure based on multi-scale receptive fields and adjustment optimization layers is specifically as follows: An adjustment and optimization layer is added to the original convolutional neural network structure to find the balance point between linearity and nonlinearity through continuous training, thereby reducing gradient loss. In the convolutional layers of the original convolutional neural network structure, the receptive field is adjusted by introducing convolutional kernels of a specific size, and the stride configuration is used to control the stride of the convolutional kernel as it slides across the image, thereby further adjusting the size of the receptive field and the sampling density of the features.

7. A material batching and conveying process monitoring system based on digital twins according to claim 6, characterized in that, The system-level topology analysis module includes: The system spatiotemporal graph construction unit is used to abstract key equipment in the physical entity layer into graph nodes, and to abstract the material flow, energy flow and information flow relationships between equipment into edges, constructing a weighted adjacency matrix; at the same time, it integrates multi-source heterogeneous data to generate a time-series state sequence with multi-dimensional characteristics for each node, which together constitute the system spatiotemporal graph. An improved spatiotemporal graph convolutional network is used to perform spatiotemporal convolution and feature fusion processing on the input system spatiotemporal graph.

8. A material batching and conveying process monitoring system based on digital twins according to claim 7, characterized in that, The improved spatiotemporal graph convolutional network is specifically as follows: Based on the original spatiotemporal graph convolutional network, the fully connected output layer at the end of the network is replaced with a temporally gated convolutional layer, and the ReLU activation function in the original spatiotemporal graph convolutional network is replaced with the LeakyReLU activation function.

Images