Unsupervised process monitoring for repetitive production system

Abstract

An unsupervised process monitoring method and apparatus for a repetitive production system, and a device. The method comprises: collecting a plurality of videos of a target production process, and obtaining an image sequence of each video (210); using a CVAE model to perform feature extraction on the image sequences, so as to obtain latent variable sequences (220); performing adaptive change-point detection on the latent variable sequence of each video, so as to obtain an initial change-point set (230); using an IDTW algorithm to process the initial change-point sets of the plurality of videos, so as to obtain a universal change-point set (240); on the basis of the universal change-point set, updating the CVAE model until the CVAE model converges (250); using the converged CVAE model to process an image sequence of the target production process that is collected in real time, so as to predict a production stage (260); and on the basis of the predicted production stage, detecting a progress deviation (270). Thus, the present disclosure significantly reduces reliance on manually labeled data, lowers implementation costs, can adapt to different production scenarios, and realizes high-precision unsupervised process monitoring, thereby improving production efficiency.

Description

Unsupervised process monitoring for repetitive production systems Technical Field

[0001] This disclosure relates to the field of advanced manufacturing, and more particularly to unsupervised process monitoring for repetitive production systems. Background Technology

[0002] In modern manufacturing systems, especially advanced production systems involving multi-step assembly processes, process monitoring and real-time schedule management are crucial. Currently, the industry has developed various process monitoring and management methods. Traditional methods, such as RFID or QR code implementations, monitor production progress by installing reading devices on the production line to track the flow of parts / assemblies. Another common method is to use wearable sensors to infer production activities by recognizing operator actions. With advancements in computer vision technology, vision-based methods are also increasingly being applied, identifying production stages and progress by analyzing images or videos of the production site. Summary of the Invention

[0003] This disclosure proposes an unsupervised process monitoring scheme for repetitive production systems, which can automatically identify and segment production stages without extensive manual intervention and can adapt to different production environments.

[0004] According to one embodiment of this disclosure, an unsupervised process monitoring method for repetitive production systems is proposed, comprising: acquiring multiple videos of a target production process; for each video, extracting features from the image sequence included in the video using a contrastive variational autoencoder (CVAE) model to obtain a latent variable sequence; performing adaptive change point detection on the latent variable sequence of the video to obtain an initial change point set of the video; processing the initial change point set of the multiple videos using an iterative dynamic time warping (IDTW) algorithm to obtain a general change point set; updating the CVAE model according to the general change point set until the CVAE model converges; processing the image sequence of the target production process acquired in real time using the converged CVAE model to predict the production stage; and detecting progress deviations based on the predicted production stage.

[0005] In some implementations, the CVAE model is configured to: use a ResNet50 architecture as an encoder to generate latent variables; and reconstruct the image using a convolutional decoder.

[0006] In some implementations, the CVAE model uses the following total loss function, Loss, during training: Loss = λ * Loss trip +L CVAE Where λ is the adjustment parameter, and Loss trip For triplet loss, LCVAE This is the CVAE loss function.

[0007] In some implementations, based on the prior distribution assumption that the latent variables follow a Gaussian mixture model (GMM) distribution, the CVAE loss function L is calculated according to the following formula. CVAE Each production stage corresponds to a Gaussian component in the GMM: in, To reconstruct the image, x is the original input image of the CVAE model, and σ is the standard deviation of the pixels. Let y be the potential standard deviation representing the production stage. Let μ be the latent mean of the production stage y. y Let be the mean parameter of the Gaussian component of y corresponding to the production stage in the prior distribution.

[0008] In some implementations, the triplet loss (Loss) is calculated according to the following formula. trip Loss trip =min(|μ anchor -μ positive | 2 -|μ anchor -μ negative | 2 +m 2 ,0), where μ anchor μ is the latent representation mean of the anchor point image. positive μ is the latent representation mean of the positive sample images. negative The latent mean of the negative sample images.

[0009] In some implementations, the adaptive change point detection is performed using the following objective function: Where i is the video ID, t is the time, y is the production stage ID, and C i The number of identified variable points, z i,t μ is the latent variable of the image of video i at time t. i,y Let T be the mean of the latent variables of video i during the production stage y. i,y and T i,y+1 For the time points corresponding to adjacent change points, pen(C) i ) is related to the number of variable points C i Related penalties.

[0010] In some implementations, the initial set of variable points in the plurality of videos is processed using the IDTW algorithm to obtain a general variable point set, including: pairing the plurality of videos in pairs to obtain a plurality of video pairs; for each video pair, iteratively matching the variable points in the video pair, wherein the iterative matching includes: processing the variable point sequences of the two videos in the video pair using the Dynamic Time Warping algorithm to obtain a DTW pair sequence, wherein the j-th DTW pair is represented as... i1 and i2 are the video numbers of the two videos in the video pair, respectively; the DTW pair sequence of the video pair is traversed forward, and for each DTW pair in the DTW sequence, when the DTW pair satisfies and When the DTW pair is identified as a common variable point for the video pair, the DTW pair is retained; however, when the DTW pair does not satisfy the condition... and At that time, the DTW pair is deleted from the DTW pair sequence, the time periods before and after the DTW pair are merged to obtain the merged segment, and the mean of the latent variables in the merged segment is recalculated, as well as the DTW pair sequence of the video pair is recalculated; the above process is repeated until a one-to-one correspondence is established between the change points of the two videos in the video pair.

[0011] After performing the iterative matching on the multiple video pairs, the remaining DTW pairs are used as the final universal variable points to form the universal variable point set.

[0012] In some implementations, updating the CVAE model based on the universal variable set includes: for each of the plurality of videos, determining the time point corresponding to each universal variable point in the universal variable set in that video to obtain multiple time periods; calculating the mean of the latent variables within each of the multiple time periods; using the mean of the latent variables within each time period as the mean parameter of the corresponding Gaussian component in the Gaussian Mixture Model (GMM); and updating the CVAE model using the time point corresponding to each universal variable point in the universal variable set and the GMM mean parameter.

[0013] In some implementations, detecting schedule deviations based on the predicted current production stage includes: determining that a schedule deviation exists and / or the predicted production stage differs from the planned production stage when the following schedule detection conditions are not met, and generating corresponding alarm information: or in, It refers to the production stage at the predicted time point t. It is the production stage at the predicted time point t+1.

[0014] According to one embodiment of this disclosure, an unsupervised process monitoring device for a repetitive production system is proposed, comprising: an image acquisition unit for acquiring multiple videos of a target production process; a feature extraction unit for extracting features from the image sequence included in each video using a contrastive variational autoencoder (CVAE) model to obtain a latent variable sequence; an adaptive change point detection unit for adaptively detecting change points in the latent variable sequence of each video to obtain an initial change point set for the video; a general change point filtering unit for processing the initial change point set of the multiple videos using an iterative dynamic time warping (IDTW) algorithm to obtain a general change point set; a training cycle control unit for updating the CVAE model according to the general change point set until the CVAE model converges; a real-time prediction unit for processing the image sequence of the target production process acquired in real time using the converged CVAE model to predict the production stage; and a progress deviation detection unit for detecting progress deviations based on the predicted production stage.

[0015] According to one embodiment of this disclosure, an electronic device is provided, the device including a memory and a processor, the memory being used to store computer instructions executable on the processor, the processor being used to implement the method as described in any of the preceding claims when executing the computer instructions.

[0016] According to one embodiment of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method as described in any of the preceding claims.

[0017] The end-to-end unsupervised embedding technology proposed in this disclosure brings significant innovation and advantages to process monitoring in repetitive production systems. By intelligently separating and labeling production stages, this disclosure achieves accurate stage identification and process monitoring, providing a highly efficient and automated solution for the manufacturing industry.

[0018] First, this disclosure proposes an innovative Contrastive Variational Autoencoder (CVAE) that enables unsupervised embedding of repetitive production processes into Gaussian Mixture Models (GMMs). By considering the separability and temporal information of features at each production stage (also known as a process class), CVAE provides a more comprehensive and accurate representation of the production process. This approach allows the system to automatically capture and learn key features and patterns in complex production processes, adapting to different production scenarios without human intervention.

[0019] Secondly, the adaptive change point detection method proposed in this disclosure, combined with the Iterative Dynamic Time Warping (IDTW) algorithm, provides a powerful tool for the automatic division and labeling of production stages. This innovative method can not only accurately identify key transition points in the production process, but also effectively prevent human bias and reduce reliance on manual annotation. In this way, the objectivity and efficiency of process monitoring are greatly improved, enabling more accurate capture of subtle changes and important turning points in the production process.

[0020] Furthermore, this disclosure includes in-depth theoretical analysis and research, particularly regarding the asymptotic optimality of change-point detection. This theoretical foundation ensures high-precision progress identification and process monitoring even when using unsupervised methods. This characteristic provides strong support for the reliability and stability of the solution, enhancing its persuasiveness and credibility in practical applications.

[0021] The end-to-end unsupervised process monitoring technology proposed in this disclosure achieves efficient integration of production stage identification and process monitoring by learning the temporal representation and key features of repetitive processes. Its inherent unsupervised nature not only eliminates the need for large amounts of manually annotated data but also significantly improves the efficiency of process monitoring and production management. It can automatically adapt to different production environments and product variations without frequent manual intervention and adjustments.

[0022] The effectiveness and adaptability of the unsupervised process monitoring scheme for repetitive production systems presented in this disclosure have been fully validated through multiple case studies, including the production of complex commercial aircraft spoilers. These diverse application scenarios demonstrate the versatility and robustness of the scheme, proving that it is suitable not only for standardized production lines but also for highly complex and customized production processes. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0024] Figure 1 illustrates an unsupervised contrastive learning method framework according to an embodiment of the present disclosure.

[0025] Figure 2 shows a flowchart of an unsupervised process monitoring method for a repetitive production system according to an embodiment of the present disclosure.

[0026] Figure 3 illustrates a schematic diagram of a CVAE model with triplet loss according to an exemplary embodiment of the present disclosure.

[0027] Figure 4 shows a schematic diagram of the margin of the triplet loss according to an exemplary embodiment of the present disclosure.

[0028] Figure 5 shows a schematic diagram of the IDTW algorithm for identifying general variable points according to an exemplary embodiment of the present disclosure.

[0029] Figure 6 shows a pseudocode illustration of an iterative matching algorithm according to an exemplary embodiment of the present disclosure.

[0030] Figure 7 illustrates a schematic framework for offline training, real-time prediction, and progress detection according to an exemplary embodiment of the present disclosure.

[0031] Figure 8 shows a visual display of various production stages in the water valve production process tested according to an exemplary embodiment of the present disclosure.

[0032] Figure 9 shows a visualization of features extracted from water valve production according to an exemplary embodiment of the present disclosure.

[0033] Figure 10 shows a visualization of the various production stages of the HA4M dataset detected according to an exemplary embodiment of the present disclosure.

[0034] Figure 11 shows a visualization of feature extraction from the HA4M dataset according to an exemplary embodiment of the present disclosure.

[0035] Figure 12 illustrates the classification results of the actual production progress of commercial aircraft spoilers in the validation and test sets according to an exemplary embodiment of this disclosure.

[0036] Figure 13 shows a visualization of features extracted from aircraft spoiler production according to an exemplary embodiment of the present disclosure.

[0037] Figure 14 shows a schematic visualization of the various production stages of garment folding detected according to an exemplary embodiment of the present disclosure.

[0038] Figure 15 shows a visualization of extracting features from clothing folds according to an exemplary embodiment of the present disclosure.

[0039] Figure 16 is a schematic diagram of the structure of an electronic device shown in at least one embodiment of the present disclosure. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0041] In manufacturing systems, especially those involving multi-step assembly processes, process monitoring and real-time progress management are crucial. The industry has developed various process monitoring and management methods. Traditional methods, such as RFID or QR code implementations, monitor production progress by installing reading devices on the production line to track the flow of parts / assemblies. Another common method is to use wearable sensors to infer production activities by recognizing operator actions. With advancements in computer vision technology, vision-based methods are also increasingly being applied, identifying production stages and progress by analyzing images or videos of the production site.

[0042] However, these related technologies face numerous challenges in practical applications. While RFID and QR code technologies can effectively track the flow of parts, they often increase scanning time, potentially extending circulation time and reducing productivity. Furthermore, these technologies have high initial implementation costs and may present compatibility issues across different production sites. The application of wearable sensor technology in production environments is also limited, primarily due to its lower accuracy and the potential impact of wearing requirements on operator comfort.

[0043] While vision-based methods have become more efficient at acquiring production progress information involving human-computer interaction and repetitive manual operations, several challenges remain. Many conventional vision methods require substantial amounts of manually labeled data, which is costly and time-consuming to implement in complex and dynamic production environments. Furthermore, these methods typically require predefined process stages, making it difficult to automatically adapt to different production flows and lacking sufficient flexibility and versatility. When faced with new production scenarios or product variations, these methods often require retraining or significant adjustments, exhibiting limited generalization capabilities.

[0044] Furthermore, the relevant technologies still have shortcomings in providing real-time and accurate progress information and deviation detection. Especially in complex, multi-stage production processes, accurately locating critical progress checkpoints remains a challenge. These methods often struggle to adapt to dynamically changing production environments, particularly when there is a wide variety of products or frequent adjustments to production processes.

[0045] Given these limitations, there is an urgent need to develop a process monitoring method that can automatically, accurately, and in real time monitor production progress and adapt to different production environments.

[0046] This disclosure introduces an unsupervised embedding technique that simultaneously divides a target production process into different stages and predicts real-time progress information through an end-to-end process. To achieve this goal, this disclosure proposes a Comparative Variational Auto Encoder (CVAE) for image feature embedding, combined with a proposed adaptive variable point detection method and Iterative Dynamic Time Warping (IDTW) technique to automatically divide production stages. These methods enable the embedded feature space to effectively capture key information related to production progress, which is helpful for process monitoring in repetitive production systems. This disclosure envisions deploying suitable camera systems at strategic locations in the production environment to continuously capture data. The application scenarios and data assumptions of this disclosure are as follows: 1. The production process is standardized, meaning that the sequence of each production stage and the operations before and after it are fixed unless anomalies occur; 2. One or more actual production videos can be acquired, each video i covering a completed production process; 3. Parts and workpieces are located within a fixed area and are within the camera's field of view; 4. Workpieces in production stage y are visually separable and can be represented as a set. With sets There is no overlap. Among them, z i,t The image features obtained from the training production video i at time t are the latent variables in the CVAE model; F(k) is a deterministic function that maps the continuous production progress k = F(f(t,i)) to discrete production stages.

[0047] In repetitive production processes, some points of instability undergo significant feature changes within a short period. For example, additional parts may be added, or specific operations may be affected by random feature changes, often caused by manual operations or background variations. More detailed examples can be found in subsequent application examples. Therefore, images associated with a production process are suitable for modeling using a Gaussian Mixture Model (GMM) distribution, with each production stage corresponding to a Gaussian distribution.

[0048] The framework of the unsupervised contrastive learning method proposed in this disclosure is shown in Figure 1. This disclosure utilizes the Gaussian Mixture Model (GMM) distribution as a prior hypothesis and uses variational inference to map the image to the optimal GMM distribution. This process enables the disclosure to detect change points where image feature values ​​actually change, employing a contrastive learning method.

[0049] Suppose the image from video i at time t is represented as (Width is w, height is h). This disclosure uses a multivariate GMM to represent x. i,t Transform into a vector with features The latent space of the feature vector includes key features for accurate identification related to process monitoring. Specifically, this disclosure proposes a method for mapping images to GMMs based on a contrastive variational autoencoder (CVAE), and then applying change point detection to the features obtained by the CVAE to identify visual features. The changes in progress are significant. Due to operational variations or different angles, each video may exhibit multiple inconsistent change points. To address this issue, this disclosure further proposes a method based on the IDTW algorithm to refine the detected change points in each video and identify common change points that consistently represent production progress as the boundaries of each production stage. In a real-time prediction environment, images are processed using CVAE to obtain feature values, and then the most recent progress cluster is predicted as the progress stage, which can then be used for subsequent detection of progress deviations.

[0050] This disclosure can be applied to computer systems / servers that can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, etc.

[0051] Computer systems / servers can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0052] Figure 2 shows a flowchart of an unsupervised process monitoring method for a repetitive production system according to an embodiment of the present disclosure. As shown, the method includes steps 210 to 270.

[0053] In step 210, multiple videos of the target production process are acquired to obtain an image sequence for each video.

[0054] For example, existing surveillance camera systems within the workshop can be used to collect video data. During the acquisition process, each video corresponds to a complete manufacturing cycle, covering all production stages from start to finish, in order to capture key information throughout the production process and learn about subtle differences that may exist between different production batches, thereby improving the robustness and adaptability of the method.

[0055] To ensure data quality, parts and workpieces can be placed within a fixed area and kept within the camera's field of view. These videos may include scenes of multiple workers operating simultaneously and frequent appearances of unrelated personnel, reflecting the characteristics of a real production environment. Furthermore, the acquired videos can be converted into discrete image sequences, and the images obtained from the video conversion can be preprocessed as needed.

[0056] In step 220, the image sequence is used to extract features to obtain a latent variable sequence.

[0057] In some implementations, the ResNet50 architecture is used as the encoder to generate latent variables, and a convolutional decoder is used to reconstruct these latent variables back into the original image space. Specifically, ResNet50 is used to transform the input image into latent variables for feature representation, capturing key features and abstract representations from the original image. In some examples, ImageNet pre-trained weights can be used to initialize the CVAE model.

[0058] The CVAE model proposed in this embodiment utilizes powerful function approximators provided by the variational inference domain to learn how to embed the image into the latent variable z. i,t A function in . This embodiment assumes the point of change T. i,y and T i,y+1 The latent variable z between i,t Follows an independent multivariable Gaussian distribution That is, the entire distribution exhibits heterogeneity. Furthermore, for each video i, μ y It is the mean parameter of y in the production stage of video i, which can also be called the common feature center, as shown in equation (1):

[0059] The latent variables of the entire production process should follow a GMM distribution, as shown in equation (2). Here, cluster p(z) corresponds to its respective production stage y, α y This represents the weight of the Gaussian component of y in the production stage.

[0060] In fact, some studies have used Gaussian distributional assumptions as a basis for the latent variables of VAEs (Variational Autoencoders). In these studies, the distribution p(y|x) of image class y is independent. However, in the application scenario disclosed here, the Gaussian distribution clusters of each image data are determined by the percentage of progress k in the video. Note that k is different from time t in video i, because the speed of each production process may be different, i.e.: k = f(t, i).

[0061] In other words, after determining the time point of the change point according to this embodiment, the image data between two adjacent change points belong to the same category, as shown in equation (3). y=F(k)=F(f(t,i)). (3)

[0062] This embodiment proposes a CVAE model, as shown in Figure 3, to approximate the actual distribution. This CVAE model outputs the predicted distribution parameters. and reconstructed image Similar to some well-known VAE models, this embodiment assumes that the image pixel values ​​follow a Gaussian distribution, as shown in equation (4). It is a hyperparameter:

[0063] The training objective of CVAE is to maximize the likelihood function for a given data point. Let q(z|y) denote the posterior probability obtained from CVAE. According to Jensen's inequality theorem, equation (5) is derived:

[0064] The right-hand side of equation (5) is the Evidence Lower Bound (ELBO) in CVAE, which can be used as the loss function for the CVAE network. ELBO is used in CVAE because it can effectively approximate complex data distributions, facilitating the learning of useful and compact latent representations.

[0065] Since the production stage y can be obtained through change point detection, assuming the production stage is known, i.e., p(y) = q(y|x) = 1, we get:

[0066] In some implementations, the mathematical form of the CVAE loss function is derived based on the above derivation. That is, under the prior assumption of the GMM distribution and the production stage y obtained through change point detection, the CVAE loss function L... CVAE It can be represented as:

[0067] in, To reconstruct the image, x is the original input image of the CVAE model, and σ is the standard deviation of the pixels. Let y be the potential standard deviation representing the production stage. Let μ be the latent mean of the production stage y. y Let be the mean parameter of the Gaussian component of y corresponding to the production stage in the prior distribution. In the above formula, the first term is the reconstruction loss, and the second term comes from the Kullback-Leibler (KL) loss between the predicted distribution and the assumed distribution.

[0068] In some implementations, a triplet loss is introduced to ensure that production stages are distinguishable, so that samples of the same class (i.e., the same production stage) should be grouped together, i.e., have a smaller feature distance, while samples of different classes (i.e., different production stages) have a larger feature distance.

[0069] According to some implementation methods, the triplet loss (Loss) can be calculated according to the following equation (6). trip Loss trip =min(|μ anchor -μ positive | 2 -|μ anchor -μ negative | 2 +m 2 ,0), (6)

[0070] Where, μ anchor μ is the latent representation mean of the anchor point image. positive μ is the latent representation mean of the positive sample images. negative Let m be the latent representation mean of the negative sample images, and m be the margin parameter. It is the expected margin.

[0071] During training, two images belonging to the same production stage can be randomly extracted as anchor images and positive sample images, and images belonging to another production stage can be selected as negative sample images, as shown in Figure 4.

[0072] To optimize the hyperparameter m, the following parameter setting guidelines are introduced to improve the statistical properties of the CVAE model proposed in this embodiment: assuming the GMM distribution covariance matrix is ​​the identity matrix and given the 3-σ rule, the margin m of the triplet loss needs to be greater than 6 to ensure that the features generated in each production stage do not overlap. The proof of these setting guidelines will not be elaborated in this disclosure.

[0073] When the CVAE encoder outputs a perfect GMM distribution, i.e. μ anchor =μ positive At that time, the triplet loss term Loss trip It will become: Loss trip =m 2 -|μanchor -μ negative | 2 .

[0074] According to some implementation methods, the following total loss function Loss can be used during the training process of the CVAE model: Loss = λ * Loss trip +L CVAE (7)

[0075] Where λ is the adjustment parameter, and Loss trip For triplet loss, L CVAE This is the CVAE loss function.

[0076] Returning to Figure 2, in step 230, adaptive change point detection is performed on the latent variable sequence of each video to obtain an initial change point set.

[0077] After obtaining the low-dimensional features (i.e., latent variables) of the image, important changes related to the production progress can be identified, which can be distinguished by visual algorithms. Specifically, at these time points, the external features of the workpiece undergo significant changes.

[0078] In related technologies, there has been much research on change point detection algorithms. Change point detection can be divided into two main categories: detection with a known number of changes and detection with an unknown number of changes. For the former, the number of changes is known, so the location of each change point can be determined by solving a deterministic optimization problem. For the latter, the number of changes is uncertain.

[0079] To obtain a monitoring solution that can flexibly adapt to different production environments, this embodiment adopts detection based on the number of unknown variables. After in-depth research, it is assumed that the output feature vector z of the latent variables in each production stage... i,t If each production stage follows an independent GMM distribution and corresponds to a single cluster in the GMM model, then change point detection can be modeled as Gaussian distribution mean change detection with constant and known variance, as shown in equations (8) and (9): v i,t =μ i,y ,t∈(T i,y ,T i,y+1 ),and

[0080] Where μ i,yLet y be the mean of the latent variables within the production stage y of video i. Under Maximum Likelihood Estimation (MLE), the cost function for parameter learning includes an L2 norm loss function and a penalty term. Notably, the L2 norm loss function is the same as the cost function for change point detection using a linear kernel function. The L2 norm loss function can be used to fully utilize the consistency properties of MLE and kernel change point detection.

[0081] Based on the above research, in some implementations, the following objective function is used for change point detection:

[0082] Where i is the video ID, t is the time, y is the production stage ID, and c is the video ID. i To determine the number of identified variable points, z i,t μ is the latent variable of the image of video i at time t. i,y Let T be the mean of the latent variables of video i during the production stage y. i,y and T i,y+1 For the time points of adjacent change points, pen(c) i ) is the number of variable points c i Related penalties.

[0083] According to this embodiment, the objective function can automatically balance the magnitude of feature vector changes and the number of variable points by combining a squared error term and a penalty term, thereby achieving automatic optimization of the number and location of variable points. Specifically, the squared error term ensures that variable points are set when features change significantly, while the penalty term effectively prevents over-segmentation and overfitting by increasing the number of variable points, thus meaning that the optimal balanced solution set contains the Pareto optimal solution for the number of variable points. This embodiment can adaptively adjust the number and location of variable points according to the characteristics of the data, without pre-specifying a fixed number of variable points or pre-setting variable point conditions, thus flexibly adapting to production processes with different complexities.

[0084] The following is a theoretical demonstration of the performance of this embodiment, proving that the adaptive variable point detection performed according to this embodiment satisfies the asymptotic optimality theorem.

[0085] Since the production stage is automatically divided according to the design concept of this embodiment, it is necessary to ensure that the segmentation method used is optimal. "Optimality" includes the algorithm's ability to identify all real change points in the mean of image features and accurately locate these points in their actual positions. First, a standard metric d is used. max (T i ,T′ i To evaluate the segmented sequence T i and T′ iThe similarity between them is defined by equation (11).

[0086] This metric captures the maximum temporal deviation between two sets of change points and can be used as a boundary for detecting delay. In any given production video i, let c... i N represents the number of detected variables. i τ represents the number of feature values ​​in video i. min This represents the minimum time ratio between adjacent change points. This reflects the shortest production stage. It is a time series z i,t The number of actual change points (separable points) in the process. The upper limit c of the distinguishable production stages. max It is inferred from the complexity of the process, while δ min =min y |μ y -μ y+1 This describes the minimum mean difference between adjacent production stages. The following is an asymptotic optimality theorem to demonstrate the asymptotic optimality property of the point of change.

[0087] Asymptotic Optimality Theorem: Let M i If inequality (12) is satisfied, and a bound b∈R satisfies inequality (13), then the linear penalty function... Ensure variable point number The probability is at least 1-e -a And the position of the point changes with O p (1) velocity convergence:

[0088] In this embodiment, the midpoint of the interval can be chosen as the value of b. The proof of the asymptotic optimality theorem described above will not be elaborated here. max The value can be derived from production operation data. δ min The value of m can be adjusted using the triplet loss margin. It's worth noting that the upper limit varies with N. i The growth rate increases, while the lower bound increases with log(N). i Linear growth. Therefore, for a sufficiently large N, i The upper limit always exceeds the lower limit. Furthermore, due to z... i,t Reparameterization from the CVAE model, if N i If the sample size is insufficient, it can be increased by drawing more samples from the CVAE model. The asymptotic optimality theorem guarantees that the number of detected production stages and the locations of change points converge to the actual production stages as the sample size increases; these stages are distinguishable by a given feature extractor and dataset. The locations of change points converge to the same percentage of progress as the actual image feature changes.

[0089] In step 240, the initial variable point set of the multiple videos is processed using the Iterative Dynamic Time Warping (IDTW) algorithm to obtain a general variable point set.

[0090] Based on the adaptive change point detection in step 230, initial change points can be extracted from each video. However, change points in different production videos may originate from anomalies such as operational errors. Therefore, it is worth considering extracting common change points that consistently appear across various production processes. Optimizing the posterior probability of common change points is a reasonable choice, since the model is based on the GMM distribution. Furthermore, common change points should be a subset of the initial change points in each training video i.

[0091] Let l i,y μ represents the number of latent variables between point y-1 and y in video i. i,y and μ′ i,y =μ y E represents the mean (i.e., feature mean) of the latent variables before and after merging different video segments. i,y It is the merged production stage index, used as a decision variable, while c i This represents the number of change points detected in video i. The general change point detection model can be represented as:

[0092] Equation (14) is used as the objective function of the general change point detection model to maximize the posterior probability of the general change point. Constraint (15) stipulates that the first and last change points of each video belong to the same stage. Constraint (16) indicates that the process between two adjacent change points can only belong to one production stage. Constraint (17) defines the mean of the merged latent variables, which is also the same as the MLE estimate.

[0093] Considering the complexity introduced by constraint (17), significant computational challenges remain. To overcome this challenge, the Iterative Dynamic Time Warping (IDTW) algorithm is proposed to effectively address the computational cost issue.

[0094] It was observed that constraints (15) and (16) are consistent with those in the IDTW algorithm. To determine the optimal stopping point for merging, the following lemma was derived and verified, namely, merging two consecutive datasets y and y+1 in video i will produce a loss, which can be expressed by the following formula:

[0095] Among them, l i,y The length T is the distance between point y and y+1 in video i. i,y+1 -T i,y .

[0096] The lemma above shows that if the current variable already satisfies these constraints, there is no need to merge more variables, as this would negatively impact the results. Based on this insight, an Iterative Dynamic Time Warping (IDTW) algorithm was developed to identify universal variables, as shown in Figure 5.

[0097] In some implementations, step 240 may include: pairing the videos in pairs to obtain multiple video pairs; performing iterative matching on the variable points in each video pair to establish a one-to-one correspondence between the variable points of the two videos in the video pair; and after performing the iterative matching on the multiple video pairs, using the remaining DTW pairs as the final universal variable points to constitute the universal variable point set.

[0098] Specifically, the variable points in each video pair can be iteratively matched according to the algorithm described in Figure 6. For example, for each video pair, the iterative matching includes: processing the variable point sequences of the two videos in the video pair using the Dynamic Time Warping (DTW) algorithm to obtain the corresponding DTW pair sequence, where the j-th DTW pair can be represented as... i1 and i2 are the video numbers of the two videos in the video pair, respectively; perform a forward traversal of the DTW pair sequence of the video pair, and when the DTW pair satisfies the judgment condition... and When the DTW pair is identified as a common variable point of the video pair, the DTW pair is retained. When the DTW pair does not meet the judgment condition, the DTW pair is deleted from the DTW pair sequence, the time periods before and after the DTW pair are merged (refer to line 10 of the algorithm in Figure 6), and the mean of the latent variables in the merged segment is recalculated, as well as the DTW pair sequence of the video pair is recalculated. The above process is repeated until a one-to-one correspondence is established between the variable points of the two videos in the video pair.

[0099] After performing the iterative matching on the multiple video pairs, the remaining DTW pairs are used as the final universal variable points to form the universal variable point set, as shown in Figure 5.

[0100] After identifying the common variable points in all videos, the precise time T′ of each variable point in each video i can be determined by backtracking. i,y Then, the mean of the corresponding change point is calculated according to equation (17) as a component of the GMM, and the CVAE model is updated. During training, the model can determine the value of the aforementioned deterministic function F(k) through the production stage y, that is: F(k)=F(f(t,i))=argmax y (T′ i,y ≤t).

[0101] Returning to Figure 2, in step 250, the CVAE model is updated according to the general variable point set until the CVAE model converges.

[0102] At the end of each training cycle, it can be determined whether the CVAE model has converged. If the model has not converged, the CVAE model is updated based on the set of general change points obtained in this training cycle, and the process returns to step 220 to start the next training cycle; if the model has converged, the training process ends and the trained CVAE model is obtained.

[0103] The CVAE model can be trained offline. In some examples, learning can begin from class zero. Given the diversity of videos, feature distributions across different videos can be aligned. To this end, additional maximum mean difference (MMD) loss can be considered for the initial 20 training epochs; for example, the total loss function used for training can combine the CVAE loss function and the MMD loss function, similar to an augmented VAE. After initial training, adaptive change point detection and IDTW are performed as described above to update the class (i.e., production stage) information and the corresponding CVAE class centers (i.e., the mean of all latent variables within the production stage, also known as feature means). In subsequent iterations, as the model covers multiple classes, triplet losses can be integrated, i.e., training is performed using the following total loss function Loss: Loss = λ * Loss trip +L CVAE .

[0104] In each training cycle, the number of classes and the centers of each class can be dynamically updated through adaptive variable point detection and IDTW.

[0105] In some implementations, updating the CVAE model based on the set of common variable points includes: determining the time point of each common variable point in each video; calculating the mean of the latent variables in each time period according to the time periods divided by the common variable points, which can be calculated according to the above equation (17); using the calculated mean of the latent variables as the mean parameter of the corresponding Gaussian component in the Gaussian mixture model (GMM); and updating the CVAE model using the obtained time points of the common variable points and the mean parameter of the GMM.

[0106] When the loss function value decreases below a certain threshold or no longer decreases significantly, or when the model's performance on the validation set reaches the expected standard, the CVAE model can be judged to have converged. For example, when the stability of the total loss function reaches the expected level, the model's accuracy in predicting production stages on the validation set reaches the expected level, the area under the receiver operating characteristic curve (AUC) of the model's progress deviations in different production environments reaches the expected level, and / or the mean absolute percentage error (MAPE) reaches the expected level, the CVAE model can be judged to have converged based on one of the above or by considering multiple factors.

[0107] After model convergence, CVAE can extract a multivariate Gaussian mixture distribution from production site images, capturing both temporal and visual data. The mean μ of the latent variables... y These progress points are saved for future predictions. To capture the stage division points learned by the model, predictions can be made on the validation set, and the transition points during this prediction process can be used as the final division results. When testing the model, these progress nodes (i.e., transition points) can be used as the standard for labeling the test set. Furthermore, if it is necessary to identify any specific point in the process, simply add that point to the results of IDTW for semi-supervised training.

[0108] Furthermore, according to this embodiment, the time cost l of each process stage y in each video i can be provided. i,y By calculating the average value It can automatically determine the time standards for each production stage. This information is of great importance for developing schedules and detecting schedule anomalies.

[0109] Step 260: Use the converged CVAE model to process the image sequence of the target production process acquired in real time to predict the production stage.

[0110] In real-time prediction, the real-time image is processed using a CVAE model to estimate the distribution, and then the latent representation mean of the image predicted by the model is compared with the feature mean μ corresponding to each production stage. y The process involves matching and identifying the production stage corresponding to the closest mean feature as the predicted production stage. The schedule prediction results can then be further used for real-time scheduling, material flow control, and performance management.

[0111] Step 270: Detect progress deviations based on the predicted production stage.

[0112] In some implementations, a schedule deviation is determined and a corresponding alarm is generated when the following schedule detection conditions are not met: Whether the production stages at the predicted adjacent time points t and t+1 meet the conditions. or in It refers to the production stage at the predicted time point t. It refers to the production stage at the predicted time point t+1, and / or the predicted production stage differs from the planned production stage.

[0113] Managers who receive alarm information (such as operations managers) can check it in a timely manner.

[0114] Figure 7 illustrates a schematic framework for offline training, real-time prediction, and progress detection according to an exemplary embodiment of the present disclosure.

[0115] The unsupervised process monitoring method proposed in this embodiment brings significant improvements to repetitive production systems. This embodiment innovatively combines CVAE, adaptive change point detection, and the IDTW algorithm to achieve automated, real-time monitoring and precise analysis of the production process. The unsupervised learning characteristic greatly reduces reliance on manually labeled data, lowers implementation costs, and improves the method's adaptability and scalability. Adaptive change point detection technology can accurately identify key stage transition points in the production process, while the IDTW algorithm ensures the identification of highly consistent common change points across multiple production cycles. The end-to-end unsupervised process monitoring method proposed in this embodiment not only adapts to subtle differences between different production batches but also provides more robust and accurate progress monitoring, enabling production managers to respond quickly to anomalies and significantly reducing the risk of production delays and quality problems. Furthermore, the generalization capability of this embodiment allows it to adapt to different types of manufacturing processes, providing strong support for the intelligent transformation of the manufacturing industry.

[0116] The following uses four different datasets to evaluate the implementation effectiveness of the unsupervised process monitoring scheme for repetitive production systems proposed in this embodiment. The first dataset focuses on the water valve production process, where assembly parts are the main focus of the image data. The second dataset focuses on HA4M, where parts occupy the smallest part of the image and are often difficult to identify, mainly showing operator actions. The third dataset represents a real-world production scenario of a commercial aircraft spoiler, including assembly operations as well as tasks such as lamination, coating, and vacuum extraction, involving mechanical parts and flexible composite materials. The last dataset is a general motion recognition dataset. The diversity of these four datasets aims to test the robustness and versatility of the end-to-end unsupervised process monitoring scheme proposed in this embodiment in different environments. (I) Water Valve Production

[0117] A dataset containing 8 videos was used as the training set, 1 video as the validation set, and 25 videos as the test set. Each video captures the assembly process of a water valve, comprising approximately 2000 image frames, which were produced on a single workstation. The production process involves a worker inserting a rubber plug into the middle valve body and securing it with screws. The worker then removes the lower part of the valve, places a gasket on top, and assembles the middle and upper valve bodies and related parts. Next comes the installation of the rocker arm, attaching screws and a plastic cap. Finally, the upper and lower clamps are assembled.

[0118] The CVAE model presented in this disclosure successfully identifies eight distinct production stages. These stages are visualized in Figure 8 and include: 1) production of the intermediate valve body; 2) fixing screws; 3) assembling the lower part of the valve; 4) assembling the upper part of the valve; 5) assembling the rocker arm; 6) assembling parts on the rocker arm; 7) completing the production of the lower clamp of the water valve; and 8) assembling the upper clamp. These production stage divisions are logical because each production stage corresponds to the beginning or end of each observed part addition during the production cycle.

[0119] Figure 9 shows a visualization of features extracted from water valve production. The horizontal and vertical axes represent the first and second dimensions of t-SNE, respectively. Blue represents training set features, red represents test set features, and green represents validation set features. The gradient from light to dark corresponds to the temporal proportions of images in the video sequence, with darker colors representing later stages of the production process. Figure 9 clearly shows eight time-related clusters. (II) HA4M Dataset

[0120] The HA4M dataset comprises 217 videos demonstrating a repetitive manual production process. Forty-one subjects in the dataset perform a single production task. Test analysis focuses on 106 videos from a consistent environment, with eight progress states visualized as shown in Figure 10. The remaining videos depict the same production process but in an informal environment and are considered outside the test parameters. Twelve videos were used as the training set, one as the validation set, and 82 as the test set. The remaining 11 videos assemble different procedures solely for progress deviation detection and evaluation purposes. The assembly process involves placing several parts onto a fixture, installing three gears and their outer rails, and securing them. The workpiece is then moved to another fixture to install parts on the other side.

[0121] This embodiment successfully identified eight states in the production process, as shown in Figure 10. These production stage divisions are logical because each stage corresponds to the start of part assembly or the completion of an operation. Furthermore, the visualization of features extracted from the HA4M dataset is shown in Figure 11, using the same interpretation as Figure 9. (III) Actual Production of Commercial Aircraft Spoilers

[0122] This embodiment aims to evaluate the performance of an unsupervised process monitoring method in the actual production of spoilers for commercial aircraft. The spoiler production workstation was chosen because it involves not only mechanical parts but also flexible materials. During production, operations are performed on or beside the tool. The production process includes workers performing some pre-assembly work beside the tool (checkpoint 1), waiting for the primer to dry and inspecting auxiliary materials (checkpoint 2), placing a layer of yellow composite material at a designated location on the tool, and then applying composite films of various colors (checkpoint 3). Then, the assembled rigid structure is inserted into a grinder, the mold is assembled, and covered with a red transparent plastic film (checkpoint 4). Excess parts of the mold are trimmed, then covered with a white cloth and trimmed to the shape of the mold (checkpoint 5). A layer of transparent green plastic film is covered and vacuum-treated (checkpoint 6). After completion, it is transferred to an autoclave for curing. This embodiment focuses on the spoiler production stage performed on the tool before curing, typically spanning approximately 2000-3000 image frames. Since the tool needs to be pushed into the autoclave after completion and brought back in the next cycle, the position of the tool is not fixed each time. Using surveillance cameras in the workshop to collect video footage will not affect production or increase hardware costs. This process is a hybrid of manual and human-machine integrated manufacturing. Due to the simultaneous operation of multiple workers and the frequent presence of irrelevant personnel and actions, it is difficult to segment based on part retrieval or worker movements. Furthermore, given the long time span, manually identifying separable points using visual algorithms is even more challenging. There is often a delay in recording information for the few checkpoints a worker needs to record, and they may accumulate information for multiple checkpoints or simply forget to record them altogether.

[0123] A dataset of approximately 46,000 images was created from three weeks of factory surveillance video. Six units were divided into a training set, one unit into a validation set, and six units into a test set. The training set contained videos from the production of three different types of spoilers, while the test and validation sets included spoilers from the training set and one untrained spoiler type. The variations in spoilers were reflected in their shape (from rectangular to trapezoidal) and size, as well as different mounting structures designed for wing attachment. Despite these variations, the production process remained consistent across the different spoiler types, consistent with the assumptions of this disclosure regarding repetitive production scenarios. The dataset is based on factory operating time, excluding production data from periods of factory downtime. The dataset includes instances where tools were temporarily removed from the camera's field of view halfway through processing, workers assembled parts and then disassembled and reassembled them, workshop lights were temporarily turned off, or construction vehicles entered and obstructed the camera's view.

[0124] During the manufacturing process, the CVAE model proposed in this embodiment can accurately and automatically divide these production processes into up to seven production stages. These production stages include all checkpoints involving visual changes in the process document, with each checkpoint further subdivided into two parts. Figure 12 shows the progress classification results for the validation and test sets, with the x-axis representing the time progression of the production video and the y-axis representing the predicted production stage. The visualization of features extracted from aircraft spoiler production is shown in Figure 13, using the same interpretation as Figure 9.

[0125] To evaluate schedule deviation detection, the test dataset was annotated using the collective expertise of multiple factory personnel and management. A total of 16 different schedule deviation scenarios and 1892 normal scenarios were annotated. Deviations included premature movement of tools and equipment, reoperation, etc. (IV) Garment Folding Dataset

[0126] This dataset contains 970 videos demonstrating the process of folding clothing, identifying six unique short-sleeved shirts and various folding methods. In this embodiment, the analysis focuses on one main technique: folding the clothing after placing it vertically, representing 270 videos in the dataset. To maintain consistency, instances of folding two garments simultaneously or refolding them were excluded. Therefore, the dataset for analysis was reduced to 246 folding events, divided into 12 videos as the training set, 1 video as the validation set, and 233 videos as the test set.

[0127] Since there are multiple garments in the scene, data preprocessing can be performed by background subtraction, taking the absolute difference from the first frame, and retaining pixels with differences exceeding a certain threshold. The CVAE model proposed in this embodiment consistently identifies four action classes: laying the garment on the table, folding horizontally, folding vertically, and placing it aside, achieving an accuracy of 84.81%. The accuracy is lower than the aforementioned production progress recognition, mainly due to non-standard folding operations and the almost identical image features of garment placement and unfolding states in some videos. These are the four common steps observed in all training videos. The detected garment folding stages are shown in Figure 14. The visualization of the extracted features is shown in Figure 15, where the four clusters are clearly distinguished, using the same interpretation method as Figure 9.

[0128] To illustrate the superiority of this embodiment, the unsupervised process monitoring method proposed in this embodiment is compared with monitoring methods based on existing benchmark models. Specifically, the CVAE model proposed in this embodiment is compared with the Time Constraint Network (TCN) and two variant models: TCN-C (similar to the enhanced TCN model) without VAE components and VaDE-C (similar to the enhanced VaDE model) without triple components. This comparison comprehensively demonstrates the advantages of the CVAE model according to this embodiment.

[0129] Three metrics are used to quantitatively compare schedule prediction performance. The first is schedule prediction accuracy; the second is the number of production stages, reflecting the accuracy of a given production stage division; and the third is the MAPE of the time calculated according to equation (19) to take into account the variability in the number of process stages that different models may identify.

[0130] in Indicates the predicted progress status in video i The change point time estimate, T i,-1 =0, reflecting the conceptualization of schedule forecasting as a continuous task; after forecasting the production stage of the current schedule state, the midpoint of the production state is determined. The predicted values ​​are compared with the actual occurrence times to comprehensively evaluate the accuracy of the process stage division and the accuracy of the prediction.

[0131] In the field of schedule deviation detection, the problem is framed as a binary classification task—identifying deviations or normal instances. Its effectiveness can be evaluated using two widely accepted metrics: F1 score and area under the receiver operating characteristic (AUC). Table 1 shows the quantitative results of schedule prediction. Table 2 Quantitative results of schedule deviation detection

[0132] Table 1 compares the performance of the proposed method in this embodiment with that of three benchmark models on four different datasets in terms of schedule class (i.e., the number of production stages), classification accuracy, and schedule MAPE.

[0133] As can be seen from the quantitative performance analysis results presented in Table 1, the method proposed in this embodiment significantly outperforms the baseline model in terms of progress status identification accuracy across all datasets, achieving higher progress prediction accuracy and lower MAPE, even when identifying more progress states. It is worth noting that the TCN model shows significantly lower accuracy and MAPE in Table 1, primarily because its design is suited for multi-view camera data, while this embodiment collects single-view camera data. In the three production cases (water valve production, HA4M, and aircraft spoiler production), the average accuracy of production progress prediction reached 98.14%.

[0134] As can be seen from the quantitative performance analysis results presented in Table 2, the method proposed in this embodiment also shows higher performance in terms of AUC and F1 score in schedule deviation detection.

[0135] Analysis suggests that the main reasons for the superior performance of this embodiment are as follows: First, the adaptive variable point detection, IDTW, and CVAE adopted in this embodiment not only capture time information and locate the precise process stage cutting points, but also enable the latent variables to effectively extract pixel-level information. This key information enables the unsupervised embedding method according to this embodiment to learn distinguishable points and build a classifier in an end-to-end manner. In addition, the implementation of triplet loss further ensures that each progress class has a unique feature distribution.

[0136] The architecture and monitoring mechanism of the CVAE model proposed in this embodiment are based on a theoretically supported loss function and an adaptive change point detection strategy that conforms to the statistical characteristics of a specific dataset, eliminating the need for parameter adjustment across datasets. Furthermore, research has confirmed that this CVAE model can accurately distinguish process states. Moreover, the proposed IDTW algorithm can establish a general state in the training video, effectively filtering out spurious changes.

[0137] This disclosure also proposes an unsupervised process monitoring device for repetitive production systems, comprising an image acquisition unit, a feature extraction unit, an adaptive change point detection unit, a general change point screening unit, a training cycle control unit, a real-time prediction unit, and a schedule deviation detection unit.

[0138] The image acquisition unit is used to acquire multiple videos of the target production process.

[0139] The feature extraction unit is used to extract features from the image sequence included in each video using a contrastive variational autoencoder (CVAE) model to obtain a sequence of latent variables.

[0140] The adaptive change point detection unit is used to perform adaptive change point detection on the latent variable sequence of each video to obtain an initial change point set of the video.

[0141] The general variable point filtering unit is used to process the initial variable point set of the multiple videos using the Iterative Dynamic Time Warping (IDTW) algorithm to obtain a general variable point set.

[0142] The training cycle control unit is used to update the CVAE model according to the general variable point set until the CVAE model converges.

[0143] The real-time prediction unit is used to process the image sequence of the target production process acquired in real time using the converged CVAE model to predict the production stage.

[0144] The schedule deviation detection unit is used to detect schedule deviations based on the predicted production stage.

[0145] For other details and beneficial effects of this embodiment, please refer to the relevant description above, which will not be repeated here.

[0146] Figure 16 shows an electronic device provided in at least one embodiment of the present disclosure. The device includes a memory and a processor. The memory is used to store computer instructions that can be executed on the processor. The processor is used to implement the unsupervised process monitoring method for repetitive production systems as described in any embodiment or implementation of the present disclosure when executing the computer instructions.

[0147] At least one embodiment of this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the unsupervised process monitoring method for repetitive production systems as described in any embodiment or implementation of this disclosure.

[0148] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the data processing device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0150] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0151] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0152] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0153] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0154] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0155] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0156] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0157] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0158] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. An unsupervised process monitoring method for repetitive production systems, characterized in that, include: Collect multiple videos of the target production process; For each of the aforementioned videos, The contrastive variational autoencoder (CVAE) model is used to extract features from the image sequences included in the video to obtain a sequence of latent variables; Adaptive change point detection is performed on the latent variable sequence of the video to obtain an initial change point set for the video; The initial set of variable points in the multiple videos is processed using the Iterative Dynamic Time Warping (IDTW) algorithm to obtain a general set of variable points; Update the CVAE model according to the general set of variable points until the CVAE model converges; The converged CVAE model is used to process the image sequence of the target production process acquired in real time to predict the production stage; Based on the predicted deviations in the production phase inspection schedule.

2. The method according to claim 1, characterized in that, The CVAE model is set as follows: The ResNet50 architecture is used as the encoder, which is used to generate latent variables; The image is reconstructed using a convolutional decoder.

3. The method according to claim 2, characterized in that, The CVAE model uses the following total loss function, Loss, during training: Loss = λ * Loss trip +L CVAE , Where λ is the adjustment parameter. Loss trip For triple loss, L CVAE This is the CVAE loss function.

4. The method according to claim 3, characterized in that, Based on the prior distribution assumption that the latent variables follow a Gaussian mixture model (GMM) distribution, the CVAE loss function L is calculated according to the following formula. CVAE Each production stage corresponds to a Gaussian component in the GMM: in, To reconstruct the image, x is the original input image of the CVAE model. σ is the standard deviation of the pixels. Let y be the potential standard deviation for the production stage. Let y be the latent mean of the production stage. μ y Let be the mean parameter of the Gaussian component of y corresponding to the production stage in the prior distribution.

5. The method according to claim 3, characterized in that, The triplet loss Loss is calculated using the following formula. trip Loss trip =min(|μ anchor -μ positive | 2 -|μ anchor -μ negative | 2 +m 2 ,0), Where, μ anchor The latent representation mean of the anchor point image. μ positive The latent representation mean of the positive sample images. μ negative The latent representation mean of the negative sample images.

6. The method according to claim 1, characterized in that, The adaptive change point detection is performed using the following objective function: Where i is the video number. t is time, y represents the production stage number. C i The number of identified variable points z i,t Let i be the latent variable of the image of video i at time t. μ i,y Let be the mean of the latent variables of video i during the production stage y. T i,y and T i,y+1 These are the time points corresponding to adjacent change points. pen(C i ) is related to the number of variable points C i Related penalties.

7. The method according to claim 1, characterized in that, The initial set of variable points in the multiple videos is processed using the IDTW algorithm to obtain a universal set of variable points, including: The multiple videos are paired up to obtain multiple video pairs; For each of the plurality of video pairs, iterative matching is performed on the variable points in that video pair, the iterative matching including: The dynamic time warping (DTW) algorithm is used to process the variable point sequences of the two videos in this video pair, resulting in a DTW pair sequence, where the j-th DTW pair is represented as... i1 and i2 are the video numbers of the two videos in the video pair, respectively; Perform a forward traversal of the DTW pair sequence for the video pair, and for each DTW pair in the DTW sequence, When the DTW pair satisfies and At that time, the DTW pair is identified as a common variable point for the video pair, and the DTW pair is retained. When the DTW pair does not satisfy and At that time, delete the DTW pair from the DTW pair sequence. The time periods before and after merging the DTW pair are used to obtain the merged segment. The mean of the latent variables within the merged segment is recalculated, and the DTW pair sequence of the video pair is recalculated. Repeat the above process until a one-to-one correspondence is established between the variable points of the two videos in the video pair; After performing the iterative matching on the multiple video pairs, the remaining DTW pairs are used as the final universal variable points to form the universal variable point set.

8. The method according to claim 4, characterized in that, Updating the CVAE model based on the general set of change points includes: For each of the plurality of videos, Determine the time point corresponding to each universal variable point in the universal variable point set described in the video to obtain multiple time periods; Calculate the mean of the latent variables within each of the multiple time periods; The mean of the latent variable in each time period is used as the mean parameter of the corresponding Gaussian component in the Gaussian mixture model (GMM); The CVAE model is updated using the time point and GMM mean parameter corresponding to each universal variable point in the universal variable set.

9. The method according to claim 1, characterized in that, Based on the predicted deviations in the current production phase inspection progress, including: If the following progress detection conditions are not met, a progress deviation is determined to exist and / or the predicted production stage differs from the planned production stage, and corresponding alarm information is generated: or in, It refers to the production stage at the predicted time point t. It is the production stage at the predicted time point t+1.

10. An unsupervised process monitoring device for repetitive production systems, characterized in that, include: The image acquisition unit is used to acquire multiple videos of the target production process; The feature extraction unit is used to extract features from the image sequence included in each video using a contrastive variational autoencoder (CVAE) model to obtain a sequence of latent variables. An adaptive change point detection unit is used to perform adaptive change point detection on the latent variable sequence of each video to obtain an initial change point set of the video. A general variable point filtering unit is used to process the initial variable point set of the multiple videos using the Iterative Dynamic Time Warping (IDTW) algorithm to obtain a general variable point set; A training cycle control unit is used to update the CVAE model according to the general variable point set until the CVAE model converges. The real-time prediction unit is used to process the image sequence of the target production process acquired in real time using the converged CVAE model to predict the production stage. The schedule deviation detection unit is used to detect schedule deviations based on the predicted production stage.

11. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the method of any one of claims 1 to 9 when executing the computer instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method described in any one of claims 1 to 9.

Images