A smart factory production visualization method and system based on digital twins

Abstract

The present invention relates to the field of factory visualization technology, and in particular to a method and system for intelligent factory production visualization based on digital twins. By fusing laser point clouds and image data, a three-dimensional spatial model of the factory production line layout and equipment structure is automatically constructed, replacing manual annotation to achieve digital reconstruction of the physical workshop. A deep learning model is used to analyze product paths and state changes, identify exposed nodes and infer associated equipment, complete intelligent division of equipment, and reduce manual modeling work. By extracting the behavioral rhythm characteristics of operating parameters and comparing them with the historical patterns of equipment sub-models, a rhythm similarity matrix is ​​automatically constructed, and parameter binding relationships are determined to avoid tedious manual configuration. The solution has good flexibility and adaptability, and can quickly update the model after process changes or equipment replacements, ensuring that the digital twin system is synchronized with the actual factory state, and achieving efficient deployment and visualization updates.

Description

Technical Field

[0001] The present invention relates to the field of factory visualization technology, and in particular to a smart factory production visualization method and system based on digital twins. Background Art

[0002] A digital twin factory should be able to faithfully replicate the production line layout and operational processes of a physical workshop, dynamically reflecting equipment behavior and product status, thereby providing visual support and decision-making for process optimization, maintenance scheduling, and anomaly diagnosis. However, existing technologies still rely heavily on manual labor to build production visualization systems, including manually annotating spatial locations, building equipment models, and configuring operating parameters and equipment relationships. This heavy reliance on manual intervention not only results in long modeling cycles but also low deployment efficiency.

[0003] At the same time, diverse production tasks and frequent adjustments to production lines are commonplace in modern manufacturing enterprises. Traditional modeling mechanisms that rely on manual updates struggle to respond promptly to process changes and equipment replacements. They lack the ability to adapt to dynamic operating conditions, often causing virtual models to lag behind actual factory operations, impacting the practical value of visualization systems. Summary of the Invention

[0004] To solve the above problems, the present invention provides a smart factory production visualization method and system based on digital twins.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A digital twin-based smart factory production visualization method includes the following steps:

[0007] S1. Obtain laser point cloud data and image data within the factory production area and construct a three-dimensional spatial model that includes the factory production line layout and equipment structure;

[0008] S2. Based on a deep learning model, the production path of the product in the 3D spatial model is obtained and the exposed nodes of the product on the production path are extracted. The product images at each exposed node are differentially analyzed to determine product status changes and match equipment types. The production equipment is divided in the 3D spatial model according to the equipment type and equipment connection relationship, and the production equipment sub-model is output.

[0009] S3. Collect operating parameters of each production equipment during the production process, and embed and map the operating parameters to generate corresponding operating parameter vectors;

[0010] S4. Based on the production equipment sub-model and the operating parameter vector, the candidate sub-model is screened; the behavioral rhythm characteristics of the operating parameters and the historical operating mode of the candidate sub-model are combined to calculate the rhythm similarity and determine the parameter binding relationship;

[0011] S5. Associate the operating parameters with the equipment sub-model according to the parameter binding relationship to obtain a visual production model.

[0012] Furthermore, the S1 includes the following steps:

[0013] By deploying LiDAR at several angles within the factory to collect point cloud data, the point cloud data is subjected to denoising and coordinate unification to obtain a 3D point cloud set.

[0014] By deploying cameras at several angles in the factory to collect image data, image enhancement and edge feature extraction are performed on the image data to obtain an image feature map containing texture, structure and contour information;

[0015] Based on the projection relationship between each spatial point in the three-dimensional point cloud set, the image feature map is mapped to the corresponding point cloud area, and point map fusion is performed to obtain a three-dimensional spatial model that includes the production line layout and equipment structure in the factory.

[0016] Furthermore, the deep learning model is constructed by the following steps:

[0017] S21. Based on a product image sequence collected during a historical production process and the corresponding spatiotemporal pose data, perform an inter-frame difference operation on the product image sequence to extract an image residual sequence representing changes in product appearance, thereby obtaining a residual feature atlas representing product state change characteristics.

[0018] S22. Based on the image residual feature atlas and the corresponding temporal pose data, construct a joint coding sequence including the image change pattern and the product motion path, and input the sequence into a pre-trained spatial perception path extraction model as a training sample to obtain a predicted path of the product in the three-dimensional space model;

[0019] S23. Based on the predicted path, perform residence time analysis and residual amplitude clustering on the continuously moving nodes on the path, extract a set of exposed nodes with state mutation characteristics, and obtain a key node dataset representing significant changes in product status;

[0020] S24. Based on the position of the key node data set in the three-dimensional space model, combined with the spatially adjacent equipment structure areas and the preset process type mapping rules, perform type judgment and regional attribution reasoning on the node state changes, and output a set of production equipment sub-models with spatial boundaries and type labels.

[0021] Furthermore, the S21 includes the following steps:

[0022] Based on the historically collected product image sequences and the corresponding timestamp data, the image sequences are paired in chronological order to construct adjacent frame image pairs;

[0023] Based on the adjacent frame image pairs, performing pixel-level difference calculation on each group of images to extract an initial difference map representing the image change area;

[0024] Based on the initial difference image, edge detection and regional filtering are performed on the difference area to remove background noise and non-target interference, thereby obtaining a difference image that only retains the area with significant product characterization changes;

[0025] The residual feature map is constructed through the difference image, and the spatial calibration of the image residual feature map is performed based on the product spatial pose information corresponding to the image acquisition moment to obtain the residual feature map set that represents the product state change characteristics.

[0026] Furthermore, the spatial perception path extraction model is trained by the following steps:

[0027] According to the residual feature atlas and the corresponding product spatial pose data, the image residual feature atlas is input into a convolutional neural network for two-dimensional feature extraction to obtain a time-ordered visual feature vector sequence;

[0028] Based on the product spatial pose data, the three-dimensional coordinates of each time step are position-encoded and feature-concatenated with the visual feature vector at the corresponding moment to obtain a joint feature sequence containing the product image state and spatial position information;

[0029] Based on the joint feature sequence, the input is extracted by the graph convolutional network to extract the local spatial adjacency graph structure features, and the multi-head attention mechanism is used to capture the spatiotemporal dependencies across time steps to obtain the global path representation vector;

[0030] Based on the global path representation vector, the vector is input into a gated recurrent unit decoder to perform a time series regression operation to predict the continuous motion node sequence of the product in the three-dimensional space model, thereby obtaining a product path prediction result.

[0031] A path regression loss function is constructed based on the Euclidean distance deviation between the path prediction result and the true trajectory label. The classification error of the residual amplitude mutation position is combined as an auxiliary supervision signal for training to obtain a spatial perception path extraction model.

[0032] Furthermore, the S23 includes the following steps:

[0033] Based on the node spatial coordinates and time series data in the path prediction results, the moving speed between adjacent nodes is calculated to obtain a node speed sequence. At the same time, based on the image residual feature map corresponding to each node, the image frame difference energy and regional change amplitude are calculated to obtain a node image representation difference sequence.

[0034] Based on the node velocity sequence and the image representation difference sequence, a state feature vector including velocity value, frame difference energy and regional change degree is constructed for each node, and nodes exceeding a set threshold in any dimension are screened out to obtain an initial state abnormal node set;

[0035] Inputting the state feature vector of the initial state abnormal node set into the K-means clustering model to perform feature pattern clustering, identifying the node subset belonging to the high-difference cluster center, and obtaining the initial exposed node set;

[0036] According to the time sequence of the initial exposed node set, it is determined whether the time interval between adjacent nodes is lower than a preset minimum interval threshold, and nodes with too small intervals are merged to output the final key node data set.

[0037] Furthermore, the operating parameters include equipment type, control instruction sequence, real-time operating condition variables, operating frequency and energy consumption characteristics.

[0038] Furthermore, the step S3 includes the following steps:

[0039] S31. Collect the operating parameters of each production device in real time, and perform timestamp alignment and missing value interpolation on the operating parameters to obtain a standardized operating parameter sequence;

[0040] S32. Based on each standardized operating parameter sequence, extract the mean, standard deviation, rate of change, and kurtosis, and perform time period feature encoding on the parameter sequence using a sliding window method to generate an operating status feature vector;

[0041] S33. Input the operating status feature vector of each device into principal component analysis, perform embedding mapping on the model, and output an operating parameter vector representing the operating behavior pattern of the device.

[0042] Furthermore, the S4 includes the following steps:

[0043] Based on the type labels of each production equipment sub-model, a type label index table is constructed;

[0044] According to the device type field in the operating parameters, a preliminary match is performed against the type tag index table to screen out a set of candidate sub-models with consistent device types, and an initial correspondence between the operating parameter vector and the candidate device sub-model is established;

[0045] Based on the spatial position of the candidate equipment sub-model in the three-dimensional space model, combined with the operation frequency and working condition change rhythm corresponding to the operating parameter vector, the difference value of its behavior rhythm with each candidate equipment sub-model is calculated;

[0046] A matching score matrix is ​​constructed based on the rhythm difference values, and the parameter binding relationship between the operating parameters and the device sub-model is confirmed according to the maximum score principle.

[0047] A digital twin-based smart factory production visualization system, applied to any of the aforementioned digital twin-based smart factory production visualization methods, comprising:

[0048] The model building module is used to obtain laser point cloud data and image data within the factory production area and build a three-dimensional spatial model that includes the production line layout and equipment structure within the factory;

[0049] The sub-model partitioning module is used to obtain the product's production path in the 3D spatial model based on a deep learning model, extract the product's exposed nodes on the production path, perform difference analysis on the product images at each exposed node, determine product status changes and match equipment types, divide production equipment in the 3D spatial model according to equipment type and equipment connection relationship, and output production equipment sub-models;

[0050] The parameter acquisition module is used to collect the operating parameters of each production equipment during the production process, embed and map the operating parameters, and generate corresponding operating parameter vectors;

[0051] The parameter binding module is used to screen candidate sub-models based on the production equipment sub-model and the operating parameter vector; it combines the behavioral rhythm characteristics of the operating parameters with the historical operating mode of the candidate sub-model, calculates the rhythm similarity, and determines the parameter binding relationship;

[0052] The association visualization module is used to associate the operating parameters with the equipment sub-model according to the parameter binding relationship to obtain a visual production model.

[0053] The beneficial effects of the present invention are as follows: the present invention constructs a high-precision three-dimensional spatial model that includes the production line layout and equipment structure within the factory through the fusion of laser point cloud and image data, replacing the traditional modeling method that relies on manual annotation, and realizing the automated digital reconstruction of the physical factory environment. Combined with the deep learning model, it automatically analyzes the movement path and state changes of the product in the three-dimensional model, identifies exposed nodes and infers the associated equipment types and process structures based on this, thereby completing the intelligent division of production equipment and significantly reducing the workload of manually building equipment models. In the process of binding the operating parameters and the equipment model, by extracting the behavioral rhythm characteristics in the operating parameters and dynamically comparing them with the historical operating modes of the equipment sub-model, a rhythm similarity matrix is ​​constructed, and the parameter attribution relationship is automatically confirmed, avoiding the tedious operations brought about by the traditional reliance on manual configuration of the binding relationship. In addition, this solution has high flexibility and adaptability, and can quickly update the parameter binding results after production line adjustments or equipment replacements, ensuring that the digital model is always consistent with the physical state, and effectively supporting rapid deployment and visualization updates under frequently changing working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a step flow chart of a smart factory production visualization method based on digital twins in the present invention.

[0055] Figure 2 It is a flow chart of the steps for constructing the deep learning model in the present invention. DETAILED DESCRIPTION

[0056] See also Figure 1-Figure 2 As shown, the present invention relates to a smart factory production visualization method based on digital twins, comprising the following steps:

[0057] S1. Obtain laser point cloud data and image data within the factory production area and construct a three-dimensional spatial model that includes the factory production line layout and equipment structure;

[0058] S2. Based on a deep learning model, the production path of the product in the 3D spatial model is obtained and the exposed nodes of the product on the production path are extracted. The product images at each exposed node are differentially analyzed to determine product status changes and match equipment types. The production equipment is divided in the 3D spatial model according to the equipment type and equipment connection relationship, and the production equipment sub-model is output.

[0059] S3. Collect operating parameters of each production equipment during the production process, and embed and map the operating parameters to generate corresponding operating parameter vectors;

[0060] S4. Based on the production equipment sub-model and the operating parameter vector, the candidate sub-model is screened; the behavioral rhythm characteristics of the operating parameters and the historical operating mode of the candidate sub-model are combined to calculate the rhythm similarity and determine the parameter binding relationship;

[0061] S5. Associate the operating parameters with the equipment sub-model according to the parameter binding relationship to obtain a visual production model.

[0062] In some embodiments, step S1 uses laser point cloud data to perform three-dimensional spatial reconstruction of physical structures such as equipment, production lines, and logistics channels within the factory, forming a point cloud scene with coordinate consistency and geometric accuracy. Through image enhancement and feature extraction, the image texture captured by the camera is mapped to the point cloud surface, completing the fusion of spatial structure and surface texture, and generating a three-dimensional spatial model with semantic information. This replaces the complex process of measuring the size, marking the position, and assembling 3D primitives of the equipment in traditional manual modeling, significantly improving modeling efficiency and accuracy. In step S2, for the automatic assembly task of a certain type of mobile phone motherboard in the actual process, the system constructs a spatial perception deep learning model by training the historically collected image sequences and pose trajectory data, and automatically identifies the path trajectory of the product. During the current task operation, the system can obtain the product's movement path in the three-dimensional model in real time, analyze its stay features at key locations such as assembly tables and inspection machines, extract exposed nodes, and further identify product status changes through image differential analysis, such as whether the parts are assembled and whether the labels are attached in place. By combining the spatial location and structural boundaries of the nodes, the equipment type and its process function are automatically inferred through the equipment process mapping table to complete the spatial division of the equipment sub-model. This process completely breaks away from the traditional method of engineers manually defining equipment boundaries and category labels based on experience, and reflects the significant advantages of the present invention in the intelligent deconstruction of equipment structure. In step S3, for equipment such as SMT placement machines, automatic screw machines, and test stations within the factory, the system collects control signals (such as instruction execution sequences), real-time operating variables (such as temperature, current, and speed), operating frequency, and energy consumption per unit time during their operation, and performs sliding window statistics and multidimensional feature encoding to form a high-dimensional operating parameter vector. PCA dimensionality reduction technology is used to further compress information redundancy and output an embedded representation for matching. In step S4, the system first filters out a set of candidate sub-models of the same type from the aforementioned equipment sub-model based on the equipment type field in the operating parameters. Furthermore, the system calculates the behavioral rhythm characteristics of the SMT machines' frequent start-stop and power fluctuation patterns over a specific period of time. These are then compared using Dynamic Time Warping (DTW) with the behavioral rhythms of the SMT equipment in the candidate sub-models over their historical operating cycles to construct a rhythm similarity matrix. The device sub-model with the highest matching score is determined as the target for the parameter vector, thereby establishing a binding relationship between the operating parameters and the device structure. This "dynamic matching mechanism driven by behavioral rhythm" differs from traditional static binding methods that rely on device IDs or manual logic configuration. It is not only suitable for scenarios with frequent equipment replacements, but also significantly enhances the system's adaptability to dynamic operating conditions. In step S5, based on the matching results, the system maps the operating data of each device type to its 3D model and renders it in real time, creating interactive, traceable, and diagnosable production visualizations.Through this system, users can quickly locate abnormal equipment status, causes of product defects, and process bottlenecks, thereby realizing production monitoring and intelligent decision-making based on digital twins, reflecting the practical value of this invention in automated modeling, intelligent parameter binding, and dynamic visualization.

[0063] Furthermore, the S1 includes the following steps:

[0064] By deploying LiDAR at several angles within the factory to collect point cloud data, the point cloud data is subjected to denoising and coordinate unification to obtain a 3D point cloud set.

[0065] By deploying cameras at several angles in the factory to collect image data, image enhancement and edge feature extraction are performed on the image data to obtain an image feature map containing texture, structure and contour information;

[0066] Based on the projection relationship between each spatial point in the three-dimensional point cloud set, the image feature map is mapped to the corresponding point cloud area, and point map fusion is performed to obtain a three-dimensional spatial model that includes the production line layout and equipment structure in the factory.

[0067] In some embodiments, first, in step S1, multiple LiDAR sensors (LiDARs) are deployed within the workshop space at different viewing angles and heights to acquire dense point cloud data reflecting spatial structure and object boundaries. The raw point cloud data contains a significant amount of noise and redundant points, so a denoising algorithm based on statistical outlier removal (Statistical Outlier Removal) is used to process it. Specifically, a statistical analysis of the Euclidean distance distribution within each point's neighborhood is performed to remove outliers with significantly low local density, thereby improving the overall geometric stability of the point cloud data. The denoised point cloud data may originate from multiple LiDAR devices, resulting in inconsistent coordinate systems. To address this, the ICP (Iterative Closest Point) registration algorithm is used to unify the coordinates of multiple LiDAR point clouds. In each iteration, the algorithm pairs each point in the source point cloud with the closest point in the target point cloud, minimizing the squared error between the two points. The rigid transformation matrix is ​​then iteratively optimized until convergence, merging the point cloud data from all viewpoints into a unified 3D coordinate system to form a complete 3D point cloud. For image processing, multiple cameras deployed throughout the workshop synchronously capture RGB image data. To enhance image usability, the original images are first subjected to histogram equalization and gamma correction to compensate for low-light areas and enhance overall image contrast. Subsequently, the Canny edge detection algorithm is used to extract edge structures within the image, and multi-scale keypoints and local texture descriptors are extracted based on the SIFT (Scale-Invariant Feature Transform) algorithm. This generates an image feature map that is geometrically stable and semantically significant. The fusion of point cloud and image data is achieved through a projection mapping mechanism. First, based on the camera's intrinsic and extrinsic parameter matrices, each spatial point in the 3D point cloud is projected onto the image plane, establishing a mapping relationship between 3D points and 2D image pixels. Bilinear interpolation is then used to map the texture, contour, and structure information in the image feature map back to the corresponding point cloud attributes, generating a fused point cloud containing multimodal features. Finally, spatial region segmentation and topological connectivity identification are performed on the fused 3D point cloud to construct a complete 3D spatial model of the factory.

[0068] Furthermore, the deep learning model is constructed by the following steps:

[0069] S21. Based on a product image sequence collected during a historical production process and the corresponding spatiotemporal pose data, perform an inter-frame difference operation on the product image sequence to extract an image residual sequence representing changes in product appearance, thereby obtaining a residual feature atlas representing product state change characteristics.

[0070] S22. Based on the image residual feature atlas and the corresponding temporal pose data, construct a joint coding sequence including the image change pattern and the product motion path, and input the sequence into a pre-trained spatial perception path extraction model as a training sample to obtain a predicted path of the product in the three-dimensional space model;

[0071] S23. Based on the predicted path, perform residence time analysis and residual amplitude clustering on the continuously moving nodes on the path, extract a set of exposed nodes with state mutation characteristics, and obtain a key node dataset representing significant changes in product status;

[0072] S24. Based on the position of the key node data set in the three-dimensional space model, combined with the spatially adjacent equipment structure areas and the preset process type mapping rules, perform type judgment and regional attribution reasoning on the node state changes, and output a set of production equipment sub-models with spatial boundaries and type labels.

[0073] In some embodiments, first, in S21, an image sequence and corresponding spatiotemporal pose data (e.g., track coordinates and timestamps) of historical batches of products from the workshop during production are collected. An inter-frame difference operation is performed on adjacent frames in the image sequence, namely, pixel-wise differences are calculated between each frame and the previous frame to generate a difference map representing changes in product appearance over that time period. To reduce the impact of background disturbances and lighting changes on the difference results, a mask filtering algorithm based on a regional stability threshold is employed to retain only regions with significant changes. Subsequently, the difference map is spatially registered with the product's pose data to generate a "residual feature atlas," which spatially demarcates the locations where product state changes occurred and their corresponding 3D poses. In S22, a joint encoding sequence is constructed based on each frame's residual feature map and its corresponding temporal pose data. Each entry in this sequence contains the image change pattern (deep features of the residual map extracted by a CNN) and the pose vector at the corresponding moment (after position encoding). This joint sequence serves as a training sample and is input into a pre-trained "spatial-aware path extraction model." This model utilizes a Transformer architecture. The encoder integrates image features and temporal position, utilizing a multi-head attention mechanism to model state transitions across time frames. The decoder predicts the node trajectory of the product as it continuously moves through the 3D spatial model, outputting a predicted path. In stage S23, dwell time analysis and residual amplitude clustering are performed on each node in the predicted path. Dwell time analysis measures the length of time the product spends at each node and combines image residual amplitudes (such as image frame difference energy and structural change rate) to construct a sequence of state feature vectors. Subsequently, a K-means clustering algorithm is used to cluster these node state feature vectors, identifying cluster centers with significant changes. Nodes with low-interval repetition are further filtered based on a time window to obtain an "exposed node set," representing key locations where significant changes in product state occur. In stage S24, the system maps the exposed node set to the 3D spatial model to determine its precise physical location. Based on the spatial coordinates of each exposed node, the system queries the adjacent equipment structure region and, in conjunction with pre-defined process type mapping rules (e.g., nodes corresponding to assembly, spraying, and crimping equipment types), performs region attribution inference. The reasoning process adopts the decision logic based on rule tree and spatial adjacency weight to classify the node state changes into the corresponding equipment structure modules, and generate a set of production equipment sub-models with clear spatial boundaries and type labels.

[0074] Furthermore, the S21 includes the following steps:

[0075] Based on the historically collected product image sequences and the corresponding timestamp data, the image sequences are paired in chronological order to construct adjacent frame image pairs;

[0076] Based on the adjacent frame image pairs, performing pixel-level difference calculation on each group of images to extract an initial difference map representing the image change area;

[0077] Based on the initial difference image, edge detection and regional filtering are performed on the difference area to remove background noise and non-target interference, thereby obtaining a difference image that only retains the area with significant product characterization changes;

[0078] The residual feature map is constructed through the difference image, and the spatial calibration of the image residual feature map is performed based on the product spatial pose information corresponding to the image acquisition moment to obtain the residual feature map set that represents the product state change characteristics.

[0079] In some embodiments, a sequence of images of the same motherboard captured continuously throughout the entire assembly process is first retrieved from a historical acquisition library and automatically sorted by timestamp. To capture subtle changes between adjacent frames, the platform pairs these sorted frames together to form temporally aligned image pairs. The algorithm module then performs a pixel-by-pixel difference operation on each image pair to generate an initial difference image. This difference image records the magnitude of the motherboard's appearance change between two time points in grayscale intensity, but also includes interference from background lighting fluctuations and laser reflections. To identify valid product change regions, Canny edge detection is first applied to the initial difference image, retaining only high-gradient boundaries. Morphological opening and closing operations are then used to erode and dilate noisy pixels to further remove discrete speckles and surround noise. This combined "edge enhancement + area filtering" strategy removes irrelevant factors such as background jitter and lighting variations, leaving only the difference regions directly related to the motherboard assembly progress, and outputs a clean difference image. The system then constructs a residual feature map based on the difference image. Unlike traditional single-channel differencing, the residual feature map internally stores multidimensional information in a multi-channel structure: one layer records contour morphology, another records the intensity of texture changes, and a third layer records regional connectivity. Through this multi-channel encoding, the residual feature map can simultaneously present multiple state factors such as "where the change occurred," "how strong the change is," and "whether the changed area is continuous," providing high-value input for subsequent deep network learning. To accurately map the purely two-dimensional residual information into three-dimensional space, the system uses the camera's calibrated extrinsic parameters and the spatial anchor points in the factory laser point cloud to perform a projective backcalculation of the image: that is, for each changed pixel in the image, the actual three-dimensional coordinates in the workshop coordinate system are calculated, and then the coordinates are aligned based on the product's pose data at the same time. After alignment, each residual feature map not only describes the visual state changes of the motherboard, but also carries the motherboard's absolute spatial position within the production line, achieving a fusion of two-dimensional visual changes and three-dimensional spatial semantics.

[0080] Furthermore, the spatial perception path extraction model is trained by the following steps:

[0081] According to the residual feature atlas and the corresponding product spatial pose data, the image residual feature atlas is input into a convolutional neural network for two-dimensional feature extraction to obtain a time-ordered visual feature vector sequence;

[0082] Based on the product spatial pose data, the three-dimensional coordinates of each time step are position-encoded and feature-concatenated with the visual feature vector at the corresponding moment to obtain a joint feature sequence containing the product image state and spatial position information;

[0083] Based on the joint feature sequence, the input is extracted by the graph convolutional network to extract the local spatial adjacency graph structure features, and the multi-head attention mechanism is used to capture the spatiotemporal dependencies across time steps to obtain the global path representation vector;

[0084] Based on the global path representation vector, the vector is input into a gated recurrent unit decoder to perform a time series regression operation to predict the continuous motion node sequence of the product in the three-dimensional space model, thereby obtaining a product path prediction result.

[0085] A path regression loss function is constructed based on the Euclidean distance deviation between the path prediction result and the true trajectory label. The classification error of the residual amplitude mutation position is combined as an auxiliary supervision signal for training to obtain a spatial perception path extraction model.

[0086] In some embodiments, a set of image residual feature atlases containing typical state changes is first selected from historical assembly records. The spatial pose data (including position and attitude in six degrees of freedom) of the product in the production line's three-dimensional coordinate system is simultaneously extracted for each image frame. This image residual information is then fed into a structured convolutional neural network (CNN), which leverages its local perception capabilities in image coding to extract two-dimensional features from each frame. The resulting output is a time-ordered sequence of visual feature vectors, each representing a summary of the appearance change for a frame. Next, to unify the modeling of visual changes and the product's spatial behavior, the system employs a positional encoding strategy to encode the three-dimensional coordinates of each time step. This encoding scheme draws on the position embedding method used in the Transformer architecture, introducing sine / cosine functions to perform multi-frequency component expansion of the X, Y, and Z three-dimensional positions, enabling the network to perceive the relative positional structure of the product's path. This positional encoding is then concatenated with the visual feature vectors of the corresponding time step to generate a joint feature sequence representing the complex evolutionary trajectory of the product's state across time, space, and appearance. To model the complex spatial topological relationships within the path, the system constructs a graph convolutional network (GCN), where nodes represent joint features across time steps, and edges represent a graph structure based on spatial adjacency and process order. The GCN extracts spatial connectivity features between local workstations and further feeds into a multi-head attention unit to model long-term dependencies across time, such as the implicit correspondence between sudden changes in product state before and after welding and inspection points. This results in a full path representation vector that integrates local and global perspectives. This path representation vector is then fed into a decoder composed of gated recurrent units (GRUs), which performs regression and gradually generates predicted nodes along the product's motion path within a three-dimensional spatial model. Unlike conventional path planning models, this approach does not rely on rules or navigation graphs for reasoning, but instead learns path patterns entirely from visual and spatial states, resulting in significant adaptability. To optimize path prediction accuracy, the system defines a composite loss function. The primary loss term is the Euclidean distance error between the predicted path nodes and the ground-truth trajectory; the auxiliary loss term is the classification error at the locations of sudden changes in amplitude in the residual feature graph, which is used to enhance learning of key turning points (such as equipment handover nodes). This auxiliary supervision mechanism helps the model improve its response sensitivity to key events and prevents the path prediction from being too smooth and missing state transition points.

[0087] Furthermore, the S23 includes the following steps:

[0088] Based on the node spatial coordinates and time series data in the path prediction results, the moving speed between adjacent nodes is calculated to obtain a node speed sequence. At the same time, based on the image residual feature map corresponding to each node, the image frame difference energy and regional change amplitude are calculated to obtain a node image representation difference sequence.

[0089] Based on the node velocity sequence and the image representation difference sequence, a state feature vector including velocity value, frame difference energy and regional change degree is constructed for each node, and nodes exceeding a set threshold in any dimension are screened out to obtain an initial state abnormal node set;

[0090] Inputting the state feature vector of the initial state abnormal node set into the K-means clustering model to perform feature pattern clustering, identifying the node subset belonging to the high-difference cluster center, and obtaining the initial exposed node set;

[0091] According to the time sequence of the initial exposed node set, it is determined whether the time interval between adjacent nodes is lower than a preset minimum interval threshold, and nodes with too small intervals are merged to output the final key node data set.

[0092] In some embodiments, the system first extracts the three-dimensional spatial coordinates and corresponding timestamp data for each node based on the predicted product path node sequence. By calculating the spatial distance and time difference between adjacent nodes, the instantaneous motion speed between nodes is obtained, thereby constructing a node velocity sequence to characterize the changes in the product's movement rhythm along the path. Simultaneously, the system extracts the image residual feature map corresponding to each node. By calculating the pixel-level frame difference total energy (e.g., the L2 norm) and the magnitude of significant regional change (e.g., based on the area change of the bounding box), an image representation difference sequence is generated to measure the intensity of visual changes in the product's state over time. Next, the motion speed value, frame difference energy value, and regional change degree of each node are combined to form a state feature vector containing three-dimensional indicators. The system sets a set of thresholds based on statistical distributions (e.g., based on quantiles or standard deviation ranges) and filters out nodes exceeding the threshold in each dimension, thereby identifying an initial set of abnormal nodes with significant state changes. This process ensures that both pauses and accelerations in motion behavior and changes in appearance, such as those during processing, assembly, and inspection, are captured. In order to further improve the reliability of abnormal node identification and eliminate occasional disturbances, the system introduces the K-means clustering algorithm to cluster the characteristic patterns of the initial abnormal node set. By setting the number of cluster centers and evaluating the Euclidean distance similarity between nodes, a subset of nodes concentrated in the high-difference feature space is identified. This subset is the initial exposed node set, representing those key positions with significant state transitions on the path that may correspond to specific process events. In order to avoid node redundancy in dense areas due to excessively high frame rates, the initial exposed node set is sorted in chronological order, and adjacent nodes with a time interval lower than the set minimum threshold (for example, 1 second or 1 frame) are merged to retain representative nodes. This time deduplication strategy avoids the introduction of noise nodes due to short-term high-frequency vibrations, while ensuring the uniqueness and clarity of key event nodes.

[0093] Furthermore, the operating parameters include equipment type, control instruction sequence, real-time operating condition variables, operating frequency and energy consumption characteristics.

[0094] Specifically, the device type field identifies the structural category and functional positioning of each device, such as "injection molding," "labeling machine," or "high-voltage testing," providing basic semantic labels for the initial matching between parameters and sub-models; the control instruction sequence records the flow of action instructions executed by the device within a certain period of time, such as start, pause, mold change, temperature increase, etc. This part of the data usually comes from the operation records of the device controller or the process scheduling instructions of the MES system, which can reflect the state change mode of the equipment at the task execution level. In addition, real-time operating condition variables include key physical quantity data of the equipment during operation, such as temperature, current, voltage, pressure, torque, vibration amplitude, etc. These data are continuously monitored and reported in real time by PLCs, sensors, or embedded acquisition terminals, which can highly characterize the current operating environment and process status of the equipment. The operating frequency, as a core indicator reflecting the rhythmic behavior of the equipment, is usually quantified by the number of operations per unit time, the number of products passing through, etc. Its fluctuations can be used to identify rhythmic anomalies and load adjustments; while the energy consumption characteristics refer to the energy consumption pattern of the equipment in a certain time window, including average power consumption, peak power, periodic fluctuations in energy consumption, etc., to assist in identifying high-load operating conditions or energy efficiency deviation behaviors.

[0095] Furthermore, the step S3 includes the following steps:

[0096] S31. Collect the operating parameters of each production device in real time, and perform timestamp alignment and missing value interpolation on the operating parameters to obtain a standardized operating parameter sequence;

[0097] S32. Based on each standardized operating parameter sequence, extract the mean, standard deviation, rate of change, and kurtosis, and perform time period feature encoding on the parameter sequence using a sliding window method to generate an operating status feature vector;

[0098] S33. Input the operating status feature vector of each device into principal component analysis, perform embedding mapping on the model, and output an operating parameter vector representing the operating behavior pattern of the device.

[0099] In some embodiments, various equipment operating parameters are first collected in real time from the factory floor, including control command logs, sensor sampling values ​​(such as temperature, voltage, current, torque, vibration, etc.), and energy consumption information. Due to the varying acquisition frequencies of different equipment, the parameter data must be timestamp aligned at a uniform time granularity. Missing segment values ​​are then inserted through interpolation or based on process synchronization points to fill in discontinuous sampling, resulting in a complete, standardized operating parameter sequence. Furthermore, the system segments each parameter sequence according to a fixed sliding window length and extracts statistical features for each window segment, such as mean, standard deviation, rate of change (such as slope and first-order difference), and kurtosis. These features accurately reflect the equipment's operating status fluctuations and abnormal operating conditions within that time period. Next, the embodiment employs principal component analysis (PCA) to perform embedding mapping on these high-dimensional statistical feature vectors. Specifically, this involves constructing a parameter feature matrix for all equipment samples in the training data, performing covariance matrix calculation and eigendecomposition, and selecting the top principal components with a cumulative contribution exceeding 95% to form the embedding space. For the new operating parameter sequence, its feature vector is mapped to the principal component space through linear projection transformation to obtain the operating parameter vector. This vector not only maintains the representativeness of the original features, but also significantly compresses the data dimension, thereby improving the subsequent processing efficiency and generalization ability of the system. For example, in a certain CNC processing area, three days of operating data were collected from five lathe equipment with slightly different models, and the mean, change rate and power consumption kurtosis of each segment were extracted with a window of 15 seconds, and finally the dimension was reduced to a six-dimensional vector through PCA. Subsequent cluster analysis shows that the operating parameter vectors form a clear distinction between different devices and are highly consistent with the actual processing task rhythm, reflecting the high adaptability and expressiveness of this embodiment in behavior modeling and equipment profiling.

[0100] Furthermore, the S4 includes the following steps:

[0101] Based on the type labels of each production equipment sub-model, a type label index table is constructed;

[0102] According to the device type field in the operating parameters, a preliminary match is performed against the type tag index table to screen out a set of candidate sub-models with consistent device types, and an initial correspondence between the operating parameter vector and the candidate device sub-model is established;

[0103] Based on the spatial position of the candidate equipment sub-model in the three-dimensional space model, combined with the operation frequency and working condition change rhythm corresponding to the operating parameter vector, the difference value of its behavior rhythm with each candidate equipment sub-model is calculated;

[0104] A matching score matrix is ​​constructed based on the rhythm difference values, and the parameter binding relationship between the operating parameters and the device sub-model is confirmed according to the maximum score principle.

[0105] In some embodiments, based on the structural information of each production equipment submodel output in step S2, the system first extracts its equipment type label (e.g., "injection molding machine," "stamping machine," "conveyor belt," "spray arm," etc.) and constructs a type label index table. This index table uses the equipment type as the primary key and maps the corresponding submodel identifier, three-dimensional spatial location, structural dimensions, and adjacency relationship, forming a data structure that supports fast retrieval. Subsequently, the system extracts the equipment type field from each operating parameter vector generated in step S3 and uses it as a query keyword to search the type label index table for corresponding entries. This filter selects a set of submodels with consistent types, forming a set of candidate equipment submodels. For example, if the operating parameters are from a conveyor motor with a rated power of 3kW, a preliminary matching process may yield five "conveyor motor" submodels spatially distributed across different production line segments as candidate binding targets. This process significantly narrows the matching space, eliminates submodels that do not match the equipment type, and effectively improves binding efficiency and accuracy. For this candidate set, embodiments further implement refined screening using a behavioral rhythm feature matching algorithm. Specifically, the system first extracts the operating frequency and periodic indicators of operating condition variables (such as temperature variation period and current surge frequency) from the operating parameter vector to construct a time spectrum vector of the equipment's operating behavior. Simultaneously, recent operating monitoring data is extracted from the candidate sub-model's historical records to calculate the equipment's historical rhythm vector. Using the Dynamic Time Warping (DTW) algorithm, the difference in rhythm vectors between the operating parameters and the candidate sub-models is calculated as the rhythm difference value. Finally, a matching score matrix is ​​constructed based on the rhythm difference values. Each score represents the rhythm consistency between an operating parameter vector and a candidate equipment sub-model, with higher scores indicating a more likely match. The final parameter binding relationship is determined by selecting the matching pair with the highest score in the score matrix. For example, if a set of operating parameters has scores of 0.72, 0.65, and 0.91 for the three candidate "conveyor motor" sub-models, respectively, the system automatically binds the operating parameters to the sub-model with a score of 0.91, forming an accurate parameter-model mapping.

[0106] The present invention also includes a smart factory production visualization system based on digital twins, comprising:

[0107] The model building module is used to obtain laser point cloud data and image data within the factory production area and build a three-dimensional spatial model that includes the production line layout and equipment structure within the factory;

[0108] The sub-model partitioning module is used to obtain the product's production path in the 3D spatial model based on a deep learning model, extract the product's exposed nodes on the production path, perform difference analysis on the product images at each exposed node, determine product status changes and match equipment types, divide production equipment in the 3D spatial model according to equipment type and equipment connection relationship, and output production equipment sub-models;

[0109] The parameter acquisition module is used to collect the operating parameters of each production equipment during the production process, embed and map the operating parameters, and generate corresponding operating parameter vectors;

[0110] The parameter binding module is used to screen candidate sub-models based on the production equipment sub-model and the operating parameter vector; it combines the behavioral rhythm characteristics of the operating parameters with the historical operating mode of the candidate sub-model, calculates the rhythm similarity, and determines the parameter binding relationship;

[0111] The association visualization module is used to associate the operating parameters with the equipment sub-model according to the parameter binding relationship to obtain a visual production model.

[0112] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary engineering technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A digital twin-based smart factory production visualization method, characterized in that: The following steps are involved: S1. Obtain laser point cloud data and image data within the factory production area and construct a three-dimensional spatial model that includes the factory production line layout and equipment structure; S2. Based on a deep learning model, the production path of the product in the 3D spatial model is obtained and the exposed nodes of the product on the production path are extracted. The product images at each exposed node are differentially analyzed to determine product status changes and match equipment types. The production equipment is divided in the 3D spatial model according to the equipment type and equipment connection relationship, and the production equipment sub-model is output. S3. Collect operating parameters of each production equipment during the production process, and embed and map the operating parameters to generate corresponding operating parameter vectors; S4. Based on the production equipment sub-model and the operating parameter vector, the candidate sub-model is screened; the behavioral rhythm characteristics of the operating parameters and the historical operating mode of the candidate sub-model are combined to calculate the rhythm similarity and determine the parameter binding relationship; S5. Associate the operating parameters with the equipment sub-model according to the parameter binding relationship to obtain a visual production model; The deep learning model is constructed by the following steps: S21. Based on a product image sequence collected during a historical production process and the corresponding spatiotemporal pose data, perform an inter-frame difference operation on the product image sequence to extract an image residual sequence representing changes in product appearance, thereby obtaining a residual feature atlas representing product state change characteristics. S22. Based on the residual feature atlas and the corresponding time-space pose data, construct a joint coding sequence containing the image change pattern and the product motion path, and input the sequence into a pre-trained spatial perception path extraction model as a training sample to obtain a predicted path of the product in the three-dimensional space model; S23. Based on the predicted path, perform residence time analysis and residual amplitude clustering on the continuously moving nodes on the path, extract a set of exposed nodes with state mutation characteristics, and obtain a key node dataset representing significant changes in product status; S24. Based on the position of the key node data set in the three-dimensional space model, combined with the spatially adjacent equipment structure areas and the preset process type mapping rules, perform type judgment and regional attribution reasoning on the node state changes, and output a set of production equipment sub-models with spatial boundaries and type labels.

2. The method for visualizing production of a smart factory based on digital twins according to claim 1, characterized in that: Said S1 comprises the following steps: By deploying LiDAR at several angles within the factory to collect point cloud data, the point cloud data is subjected to denoising and coordinate unification to obtain a 3D point cloud set. By deploying cameras at several angles in the factory to collect image data, image enhancement and edge feature extraction are performed on the image data to obtain an image feature map containing texture, structure and contour information; Based on the projection relationship between each spatial point in the three-dimensional point cloud set, the image feature map is mapped to the corresponding point cloud area, and point map fusion is performed to obtain a three-dimensional spatial model that includes the production line layout and equipment structure in the factory.

3. The method for visualization of smart factory production based on digital twin according to claim 1, characterized in that: The S21 includes the following steps: Based on the historically collected product image sequences and the corresponding timestamp data, the image sequences are paired in chronological order to construct adjacent frame image pairs; Based on the adjacent frame image pairs, performing pixel-level difference calculation on each group of images to extract an initial difference map representing the image change area; Based on the initial difference image, edge detection and regional filtering are performed on the difference area to remove background noise and non-target interference, thereby obtaining a difference image that only retains the area with significant product characterization changes; The residual feature map is constructed through the difference image, and the spatial calibration of the image residual feature map is performed based on the product spatial pose information corresponding to the image acquisition moment to obtain the residual feature map set that represents the product state change characteristics.

4. The method for visualization of smart factory production based on digital twin according to claim 1, characterized in that: The spatial perception path extraction model is trained by the following steps: According to the residual feature atlas and the corresponding product spatial pose data, the image residual feature atlas is input into a convolutional neural network for two-dimensional feature extraction to obtain a time-ordered visual feature vector sequence; Based on the product spatial pose data, the three-dimensional coordinates of each time step are position-encoded and feature-concatenated with the visual feature vector at the corresponding moment to obtain a joint feature sequence containing the product image state and spatial position information; Based on the joint feature sequence, the input is extracted by the graph convolutional network to extract the local spatial adjacency graph structure features, and the multi-head attention mechanism is used to capture the spatiotemporal dependencies across time steps to obtain the global path representation vector; Based on the global path representation vector, the vector is input into a gated recurrent unit decoder to perform a time series regression operation to predict the continuous motion node sequence of the product in the three-dimensional space model, thereby obtaining a product path prediction result. A path regression loss function is constructed based on the Euclidean distance deviation between the path prediction result and the true trajectory label. The classification error of the residual amplitude mutation position is combined as an auxiliary supervision signal for training to obtain a spatial perception path extraction model.

5. The method for visualization of smart factory production based on digital twin according to claim 1, characterized in that: The S23 includes the following steps: Based on the node spatial coordinates and time series data in the path prediction results, the moving speed between adjacent nodes is calculated to obtain a node speed sequence. At the same time, based on the image residual feature map corresponding to each node, the image frame difference energy and regional change amplitude are calculated to obtain a node image representation difference sequence. Based on the node velocity sequence and the image representation difference sequence, a state feature vector including velocity value, frame difference energy and regional change degree is constructed for each node, and nodes exceeding a set threshold in any dimension are screened out to obtain an initial state abnormal node set; Inputting the state feature vector of the initial state abnormal node set into the K-means clustering model to perform feature pattern clustering, identifying the node subset belonging to the high-difference cluster center, and obtaining the initial exposed node set; According to the time sequence of the initial exposed node set, it is determined whether the time interval between adjacent nodes is lower than a preset minimum interval threshold, and nodes with too small intervals are merged to output the final key node data set.

6. The method for visualization of smart factory production based on digital twin according to claim 1, characterized in that: The operating parameters include equipment type, control instruction sequence, real-time operating condition variables, operating frequency and energy consumption characteristics.

7. The method for visualization of smart factory production based on digital twin according to claim 6, characterized in that: The S3 includes the following steps: S31. Collect the operating parameters of each production device in real time, and perform timestamp alignment and missing value interpolation on the operating parameters to obtain a standardized operating parameter sequence; S32. Based on each standardized operating parameter sequence, extract the mean, standard deviation, rate of change, and kurtosis, and perform time period feature encoding on the parameter sequence using a sliding window method to generate an operating status feature vector; The operating status feature vector of each device is input into principal component analysis, the model performs embedding mapping, and outputs the operating parameter vector representing the operating behavior mode of the device.

8. The method for visualization of smart factory production based on digital twin according to claim 6, characterized in that: The S4 comprises the following steps: Based on the type labels of each production equipment sub-model, a type label index table is constructed; According to the device type field in the operating parameters, a preliminary match is performed against the type tag index table to screen out a set of candidate sub-models with consistent device types, and an initial correspondence between the operating parameter vector and the candidate device sub-model is established; Based on the spatial position of the candidate equipment sub-model in the three-dimensional space model, combined with the operation frequency and working condition change rhythm corresponding to the operating parameter vector, the difference value of its behavior rhythm with each candidate equipment sub-model is calculated; A matching score matrix is ​​constructed based on the rhythm difference values, and the parameter binding relationship between the operating parameters and the device sub-model is confirmed according to the maximum score principle.

9. A digital twin-based smart factory production visualization system, applied to a digital twin-based smart factory production visualization method according to any one of claims 1 to 8, characterized in that: include: The model building module is used to obtain laser point cloud data and image data within the factory production area and build a three-dimensional spatial model that includes the production line layout and equipment structure within the factory; The sub-model partitioning module is used to obtain the product's production path in the 3D spatial model based on a deep learning model, extract the product's exposed nodes on the production path, perform difference analysis on the product images at each exposed node, determine product status changes and match equipment types, divide production equipment in the 3D spatial model according to equipment type and equipment connection relationship, and output production equipment sub-models; The parameter acquisition module is used to collect the operating parameters of each production equipment during the production process, embed and map the operating parameters, and generate corresponding operating parameter vectors; The parameter binding module is used to screen candidate sub-models based on the production equipment sub-model and the operating parameter vector; it combines the behavioral rhythm characteristics of the operating parameters with the historical operating mode of the candidate sub-model, calculates the rhythm similarity, and determines the parameter binding relationship; The association visualization module is used to associate the operating parameters with the equipment sub-model according to the parameter binding relationship to obtain a visual production model.

Images