Unsupervised process monitoring method, device and equipment for repeated production system

By comparing the combination of variational autoencoder and iterative dynamic time regularization algorithm, automatic and accurate process monitoring in complex production environments is achieved, and the problem of insufficient flexibility and universality of progress monitoring in the existing technology is solved, and the problem of improvement of the Monitoring efficiency and adaptability.

CN119065333AActive Publication Date: 2024-12-03TSINGHUA UNIVERSITY

Patent Information

Application Number
CN202411196529.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-12-03
Estimated Expiration
2044-08-28

AI Technical Summary

Technical Problem

The existing process monitoring methods are difficult to achieve automatic, accurate and real-time progress monitoring in complex and multi-stage production processes, and lack flexibility and versatility, making it difficult to adapt to dynamically changing production environments and product variants.

Method used

Using the contrasting variational autoencoder (CVAE) model combined with iterative dynamic time regularization (IDTW) algorithm, it automatically recognizes and segments the production stage through unsupervised learning, and realizes efficient monitoring of the production process, and uses adaptive variable point detection and real-time prediction. Detect progress deviation.

Benefits of technology

Automatic and accurate process monitoring in complex production environments is achieved, the dependence on manual intervention is reduced, the efficiency and adaptability of monitoring is improved, and it can adapt to different production environments and product variants without frequent adjustments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065333A_ABST
    Figure CN119065333A_ABST
Patent Text Reader

Abstract

The invention discloses an unsupervised process monitoring method, device and equipment for a repeated production system. The method comprises the following steps: acquiring a plurality of videos of a target manufacturing process to obtain an image sequence; performing feature extraction on the image sequence by using a CVAE model to obtain a potential variable sequence; performing adaptive change point detection on the potential variable sequence to obtain an initial change point set; using an IDTW algorithm to screen out a general change point set; updating the CVAE model according to the general change point set until convergence; processing an image sequence acquired in real time by using a convergent CVAE model, and predicting a production stage; and detecting the progress deviation according to the predicted production stage. According to the method and the device, the dependence on manual annotation data is remarkably reduced, the implementation cost is reduced, the method and the device can adapt to different production scenes, high-precision unsupervised process monitoring is realized, and the production efficiency is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of advanced manufacturing, and in particular to an unsupervised process monitoring method, device and equipment for a repetitive production system. Background Art

[0002] In modern manufacturing systems, especially advanced production systems involving multi-step assembly processes, process monitoring and real-time progress management are crucial. At present, the industry has developed a variety of process monitoring and management methods. Traditional methods such as RFID or QR code implementation track the flow of parts / components by installing reading devices on the production line to monitor production progress. Another common method is to use wearable sensors to infer production activities by identifying the operator's actions. With the advancement of computer vision technology, vision-based methods have also gradually been applied. These methods identify production stages and progress by analyzing images or videos of the production site.

[0003] However, these existing technology approaches face many challenges in practical application. Although RFID and QR code technologies can effectively track the flow of parts, they often increase scanning time, which may extend circulation time and reduce productivity. In addition, these technologies have high initial implementation costs and may have compatibility issues between different partners. The application of wearable sensor technology in production environments is also limited, mainly due to its low accuracy and wearing requirements that may affect the operator's working comfort.

[0004] Although vision-based methods have become more efficient in obtaining production progress information involving human-computer interaction and repetitive manual operations, some problems still exist. Many existing vision methods require a large amount of manually annotated data, which is costly and time-consuming to implement in complex and changing production environments. At the same time, these methods usually require predefined process stages, making it difficult to automatically adapt to different production processes, and lack sufficient flexibility and versatility. When faced with new production scenarios or product variants, these methods often need to be retrained or significantly adjusted, and their generalization capabilities are limited.

[0005] In addition, existing methods still have shortcomings in providing real-time, accurate progress information and deviation detection. Especially in complex multi-stage production processes, accurately locating key progress checkpoints remains a challenge. These methods are usually difficult to adapt to dynamically changing production environments, especially when there are a variety of product types or frequent adjustments to the production process.

[0006] In view of these limitations, there is an urgent need to develop a process monitoring method that can automatically, accurately, and in real time monitor production progress and can adapt to different production environments. Summary of the invention

[0007] The present disclosure proposes an unsupervised process monitoring solution for a repetitive production system, which can automatically identify and segment production stages without a large amount of manual intervention and can adapt to different production environments.

[0008] According to one embodiment of the present disclosure, an unsupervised process monitoring method for a repetitive production system is proposed, comprising:

[0009] Collecting multiple videos of the target production process to obtain an image sequence of each of the videos;

[0010] Using a contrastive variational autoencoder (CVAE) model to extract features from the image sequence to obtain a latent variable sequence;

[0011] Performing adaptive change point detection on the latent variable sequence of each video to obtain an initial change point set;

[0012] Processing the initial change point sets of the multiple videos using an iterative dynamic time warping (IDTW) algorithm to obtain a universal change point set;

[0013] Updating the CVAE model according to the universal change point set until the CVAE model converges;

[0014] The converged CVAE model is used to process the image sequences of the target production process acquired in real time to predict the production stage;

[0015] Detect schedule deviations based on forecasted production phases.

[0016] In some embodiments, the CVAE model is set up as:

[0017] Use the ResNet50 architecture as an encoder to generate latent variables;

[0018] Reconstruct the image using a convolutional decoder.

[0019] In some embodiments, the following total loss function Loss is used during the training process of the CVAE model:

[0020] Loss = λ*Loss trip +L CVAE ,

[0021] Among them, λ is the adjustment parameter, Loss trip is the triplet loss, L CVAE is the CVAE loss function.

[0022] In some embodiments, based on the prior distribution assumption that the latent variables obey the Gaussian mixture model (GMM) distribution, the CVAE loss function L is calculated according to the following formula: CVAE, where each production stage corresponds to a Gaussian component in GMM:

[0023]

[0024] in, is the reconstructed image, X is the original input image of the CVAE model, σ is the standard deviation of the pixel, is the potential standard deviation of the production stage y, is the potential representation mean under the production stage y, μ y is the mean parameter of the Gaussian component corresponding to the production stage y in the prior distribution.

[0025] In some embodiments, the triplet loss Loss is calculated according to the following formula: trip :

[0026] Loss trip =min(|μ anchor -μ positive | 2 -|μ anchor -μ negative | 2 +m 2 ,0),

[0027] Among them, μ anchor is the potential representation mean of the anchor image, μ positive is the potential representation mean of the positive sample image, μ negative is the mean of the latent representations of negative sample images.

[0028] In some implementations, the adaptive change point detection includes:

[0029] The following objective function is used for change point detection:

[0030]

[0031] Where i is the video number, t is the time, y is the number of the production stage, C i The number of change points identified, z i,t is the latent variable of the image of video i at time t, μ i,y is the mean of the latent variable of video i in production stage y, T i,y and T i,y+1 is the time point of the adjacent change point, pen(C i ) is the number of change points C i Related penalties.

[0032] In some implementations, the initial change point sets of the multiple videos are processed using an IDTW algorithm to obtain a universal change point set, including:

[0033] Pair the videos together;

[0034] The change points in each video pair are iteratively matched. The iterative matching includes: using the dynamic time warping algorithm to process the change point sequences of the two videos in the video pair to obtain a DTW pair sequence. The jth DTW pair is represented as i1 and i2 are the numbers of the two videos in the video pair respectively; the DTW pair sequence of the video pair is traversed forward, and when the DTW pair meets the judgment condition and When the DTW pair meets the judgment condition, the DTW pair is identified as the common change point of the video pair and the DTW pair is retained. When the DTW pair does not meet the judgment condition, the DTW pair is deleted from the DTW pair sequence, the time periods before and after the DTW pair are merged, and the means of the latent variables in the merged segments are recalculated, and the DTW pair sequence of the video pair is recalculated; the above process is repeated until a one-to-one correspondence is established between the change points of the two videos in the video pair;

[0035] After repeatedly performing the iterative matching on all video pairs, the remaining DTW pairs are used as final universal change points to form the universal change point set.

[0036] In some implementations, updating the CVAE model according to the universal change point set includes:

[0037] Determine the timing of each universal change point in each video;

[0038] According to the time periods divided by the common change point, the mean of the latent variable in each time period is calculated;

[0039] The calculated mean of the latent variable is used as the mean parameter of the corresponding Gaussian component in the Gaussian mixture model (GMM);

[0040] The CVAE model is updated using the obtained time points of the universal change point and the GMM mean parameters.

[0041] In some embodiments, detecting a schedule deviation based on a predicted current production stage includes:

[0042] When the following progress detection conditions are not met, it is judged that there is a progress deviation and a corresponding alarm message is generated:

[0043] Whether the production stages at the adjacent predicted time points t and t+1 meet or in is the production stage at the predicted time point t, It is the predicted production phase at time point t+1, and / or the predicted production phase is different from the planned production phase.

[0044] According to one embodiment of the present disclosure, an unsupervised process monitoring device for a repetitive production system is proposed, comprising:

[0045] An image acquisition unit, used to collect multiple videos of the target production process to obtain an image sequence of each of the videos;

[0046] A feature extraction unit, used to extract features from the image sequence using a contrastive variational autoencoder (CVAE) model to obtain a latent variable sequence;

[0047] An adaptive change point detection unit, used to perform adaptive change point detection on the latent variable sequence of each video to obtain an initial change point set;

[0048] a universal change point screening unit, configured to process the initial change point sets of the plurality of videos using an iterative dynamic time warping (IDTW) algorithm to obtain a universal change point set;

[0049] A training cycle control unit, used for updating the CVAE model according to the universal change point set until the CVAE model converges;

[0050] A real-time prediction unit is used to process the image sequence of the target production process collected in real time using a converged CVAE model to predict the production stage;

[0051] The schedule deviation detection unit is used to detect the schedule deviation according to the predicted production stage.

[0052] According to one embodiment of the present disclosure, an electronic device is provided, the device comprising a memory and a processor, the memory being used to store computer instructions executable on the processor, the processor being used to implement any of the above methods when executing the computer instructions.

[0053] According to an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, any of the above methods is implemented.

[0054] The end-to-end unsupervised embedding technology proposed in this disclosure brings significant innovation and advantages to process monitoring of repetitive production systems. This disclosure achieves accurate stage identification and process monitoring by intelligently separating and marking production stages, providing an efficient and automated solution for the manufacturing industry.

[0055] First, this paper proposes an innovative contrastive variational autoencoder (CVAE) that can embed repetitive production processes into a Gaussian mixture model (GMM) in an unsupervised manner. By considering the separability and temporal information of the features of each production stage (also known as process class), CVAE provides a more comprehensive and accurate representation of the production process. This approach enables the system to automatically capture and learn key features and patterns in complex production processes, and adapt to different production scenarios without human intervention.

[0056] Secondly, the adaptive change point detection method proposed in this disclosure, combined with the iterative dynamic time warping (IDTW) algorithm, provides a powerful tool for the automatic division and labeling of production stages. This innovative method can not only accurately identify key transition points in the production process, but also effectively prevent human bias and reduce reliance on manual annotations. In this way, the objectivity and efficiency of process monitoring are greatly improved, and subtle changes and important turning points in the production process can be captured more accurately.

[0057] In addition, the present disclosure has carried out in-depth analysis and research in theory, especially on the asymptotic optimality of change point detection. This theoretical basis ensures that high-precision progress identification and process monitoring can be achieved even when unsupervised methods are used. This feature provides strong support for the reliability and stability of the scheme, and enhances its persuasiveness and credibility in practical applications.

[0058] The end-to-end unsupervised process monitoring technology proposed in this disclosure achieves efficient integration of production stage identification and process monitoring by learning the temporal representation and key features of repeated processes. Its inherent unsupervised nature not only eliminates the need for a large amount of manually annotated data, but also significantly improves the efficiency of process monitoring and production management, and can automatically adapt to different production environments and product variants without frequent manual intervention and adjustment.

[0059] The effectiveness and adaptability of the unsupervised process monitoring scheme for repetitive production systems proposed in this disclosure have been fully verified through multiple different case studies, including the complex production of commercial aircraft spoilers. These diverse application scenarios fully demonstrate the versatility and robustness of the scheme, proving that it is not only suitable for standardized production lines, but also can effectively cope with highly complex and customized production processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0061] Figure 1 The framework of an unsupervised contrastive learning method according to an embodiment of the present disclosure is shown;

[0062] Figure 2 A flow chart of an unsupervised process monitoring method for a repetitive production system according to an embodiment of the present disclosure is shown;

[0063] Figure 3 A schematic diagram of a CVAE model with triplet loss according to an exemplary embodiment of the present disclosure is shown;

[0064] Figure 4 A schematic diagram showing a margin of triplet loss proposed according to an exemplary embodiment of the present disclosure is shown;

[0065] Figure 5 A schematic diagram showing the identification of universal change points by an IDTW algorithm according to an exemplary embodiment of the present disclosure is shown;

[0066] Figure 6 A pseudo code diagram of an iterative matching algorithm according to an exemplary embodiment of the present disclosure is shown;

[0067] Figure 7 A schematic framework of offline training, real-time prediction and progress detection according to an exemplary embodiment of the present disclosure is shown;

[0068] Figure 8 A schematic diagram showing visualization of various production stages in a water valve production process detected according to an exemplary embodiment of the present disclosure is shown;

[0069] Fig. 9 shows a visualization display of features extracted from water valve production according to an exemplary embodiment of the present disclosure;

[0070] Fig.10 A schematic diagram showing visualization of various production stages of a HA4M data set detected according to an exemplary embodiment of the present disclosure is shown;

[0071] Fig.11 shows a visualization display of features extracted from a HA4M dataset according to an exemplary embodiment of the present disclosure;

[0072] Fig.12 The progress classification results of monitoring the actual production of commercial aircraft spoilers in the validation set and the test set according to an exemplary embodiment of the present disclosure are shown;

[0073] Fig.13 A visualization display of features extracted from aircraft spoiler production according to an exemplary embodiment of the present disclosure is shown;

[0074] Fig.14A schematic diagram showing a visual display of various production stages of clothing folding detected according to an exemplary embodiment of the present disclosure is shown;

[0075] Fig.15 FIG. 1 shows a visualization display of extracting features from clothing folding according to an exemplary embodiment of the present disclosure;

[0076] Fig.16 It is a schematic diagram of the structure of an electronic device shown in at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0077] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0078] The present disclosure introduces an unsupervised embedding technology that simultaneously divides the target production process into different stages and predicts real-time progress information through an end-to-end process. To achieve this goal, the present disclosure proposes a contrastive variational autoencoder (CVAE) for image feature embedding, and combines the proposed adaptive change point detection method and iterative dynamic time registration (IDTW) technology to automatically divide the production stages. These means enable the embedded feature space to effectively capture key information related to production progress, which is helpful for process monitoring of repetitive production systems. The present disclosure envisions deploying suitable camera systems at strategic locations in the production environment to continuously capture data. The application scenarios and data assumptions of the present disclosure are as follows:

[0079] 1. The production process is standardized, which means that the order of each production stage and the operations before and after are fixed unless there are abnormal circumstances;

[0080] 2. One or more actual production videos can be obtained, each video covering a completed production process;

[0081] 3. Parts and workpieces are located in a fixed area and within the camera's field of view;

[0082] 4. Artifacts are visually separable at production stage y. Collection and There is no intersection, where z i,t It is the image feature obtained by the present invention from the training production video i at time t, that is, the latent variable in the VAE model; F(k) is a deterministic function that maps the continuous production progress k=F(f(t,i)) to the discrete production stage.

[0083] In a repetitive production process, some change points have significant feature changes in a short period of time. For example, some additional parts are added or a specific operation is affected by random feature changes, usually caused by manual operation or background. More detailed examples can be found in the subsequent application examples. Therefore, images related to a production process are suitable for modeling with GMM distribution, with each production stage corresponding to a Gaussian distribution.

[0084] The framework of the unsupervised contrastive learning method proposed in this disclosure is as follows: Figure 1 As shown. The present disclosure uses the GMM distribution as a priori assumption and uses variational inference to map the image to the optimal GMM distribution. This process enables the present disclosure to detect the change points where the image feature values ​​actually change, using a contrastive learning method.

[0085] Assume that the image from video i at time t is represented as (width is w and height is h). The present disclosure uses multivariate GMM to transform x i,t Convert to a feature vector The latent space allows accurate identification of key features related to process monitoring. Specifically, the present disclosure proposes a novel contrastive variational autoencoder (CVAE) for mapping images to GMMs, and then change point detection is applied to CVAE features to identify visual features. Due to operational variations or different angles, each video may have multiple inconsistent change points. To address this problem, the IDTW algorithm proposed in the present disclosure is used to refine the change points detected in each video and identify universal change points that consistently represent the production progress. These refined universal change points can be regarded as the boundaries of each production stage. In a real-time prediction environment, the image is processed by CVAE to obtain feature values, and then the nearest progress cluster is predicted as the progress stage. Post-processing is then used to detect progress deviations.

[0086] The disclosed embodiments may be applied to a computer system / server that may operate with numerous other general or special computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with a computer system / server include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, and the like.

[0087] Computer systems / servers may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. In general, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

[0088] Figure 2 The flowchart of the unsupervised process monitoring method for a repetitive production system according to an embodiment of the present disclosure is shown. As shown in the figure, the method includes steps 1 to 7.

[0089] Step 1: collect multiple videos of the target production process to obtain an image sequence of each video.

[0090] The existing surveillance camera system in the workshop can be used to collect video data.

[0091] During the acquisition process, each video corresponds to a complete manufacturing cycle, covering all production stages from start to finish, so as to capture key information of the entire production process and learn the subtle differences that may exist between different production batches, thereby improving the robustness and adaptability of the method.

[0092] To ensure data quality, parts and workpieces are placed in a fixed area and kept within the camera's field of view. These videos may contain multiple workers operating simultaneously and scenes with frequent appearances of unrelated personnel, which are complex situations that reflect the characteristics of a real production environment.

[0093] The acquired video can be converted into a discrete image sequence, and the images can be preprocessed as needed.

[0094] Step 2: Use a contrastive variational autoencoder (CVAE) model to extract features from the image sequence to obtain a latent variable sequence.

[0095] In some embodiments, a ResNet50 architecture is used as an encoder to generate latent variables; a convolutional decoder is used to reconstruct the image.

[0096] According to this embodiment, ResNet50 is used to convert the input image into a latent variable for feature representation to capture the key features and abstract representations in the original image. In some examples, ImageNet pre-trained weights can be used to initialize the CVAE model. The convolutional decoder is used to reconstruct the latent variables back to the original image space.

[0097] The CVAE model proposed in this embodiment uses a powerful function approximator provided by the variational inference field to learn a function to embed the image into the latent variable z i,t In this embodiment, it is assumed that the change point T i,y and T i,y+1 The latent variable z between i,t Independent multivariate Gaussian distribution For each video i. y is the mean parameter of the production stage y of video i, also known as the common feature center, that is, there is heterogeneity in the entire distribution, as shown in Equation 1:

[0098]

[0099] The latent variables of the entire production process should follow the GMM distribution, which is expressed in Equation 2, where the clusters correspond to their respective production stages, α y Represents the weight of the Gaussian component of the production stage y.

[0100]

[0101] In fact, there have been some studies that use GMM as a distribution hypothesis for VAE (Variational Autoencoder) latent variables. In these studies, the distribution p(y|x) of image class y is independent. However, in the scenario disclosed herein, the Gaussian distribution cluster of each image data is determined by the progress percentage k in the video. Note that k is different from the time t in video i, because the speed of each production process may be different, that is, k = f(t,i). That is, after the time point of the change point is determined according to this embodiment, the image data between two adjacent change points belongs to the same class, and the formula is shown in Equation 3.

[0102] y=F(k)=F(f(t,i)). (3)

[0103] This embodiment proposes a model such as Figure 3 The CVAE model shown is used to approximate the actual distribution. The CVAE model outputs the predicted distribution parameters and reconstructed image Similar to some well-known VAE models, this embodiment assumes that the image pixel values ​​follow a Gaussian distribution, as shown in Equation 4, is a hyperparameter:

[0104]

[0105] The training objective of CVAE is to maximize the likelihood function of a given data point. Let q(z|y) represent the posterior probability obtained by CVAE. According to Jensen's inequality theorem, the inventors infer Equation 5:

[0106]

[0107] The right side of Equation 5 is the Evidence Lower Bound (ELBO) in CVAE, which can be used as the loss function of the CVAE network. ELBO is used in VAE because it can effectively approximate complex data distributions and promote useful and compact latent representation learning.

[0108] Since the production stage y can be obtained through change point detection, assuming that the production stage is known, p(y) = q(y|x) = 1, we get:

[0109]

[0110] In some embodiments, based on the above derivation, the inventors obtain the following mathematical form of the CVAE loss function according to this embodiment, that is, under the prior assumption of GMM distribution and the production stage y obtained by change point detection, the CVAE loss function L CVAE It can be expressed as:

[0111]

[0112] in, is the reconstructed image, x is the original input image of the CVAE model, σ is the standard deviation of the pixel, is the potential standard deviation of the production stage y, is the potential representation mean under the production stage y, μ y is the mean parameter of the Gaussian component corresponding to the production stage y in the prior distribution. The first term in the above formula is the reconstruction loss, and the second term comes from the Kullback-Leibler (KL) loss between the predicted distribution and the hypothesis distribution.

[0113] In some embodiments, to ensure that the production stages are distinguishable, a triplet loss is introduced so that samples of the same class (ie, the same production stage) should be grouped together, while the feature distances between samples of different classes (ie, different production stages) should be far apart.

[0114] According to some embodiments, the triplet loss Loss may be calculated according to the following formula: trip :

[0115] Loss trip =min(|μ anchor -μ positive | 2 -|μ anchor -μ negative | 2 +m 2 ,0), (6)

[0117] Among them, μ anchor is the potential representation mean of the anchor image, μ positive is the potential representation mean of the positive sample image, μ negative is the potential representation mean of the negative sample image, and m is the margin parameter.

[0118] During training, two images belonging to the same production stage can be randomly extracted as anchor images and positive sample images, and an image belonging to another production stage can be selected as a negative sample image, such as Figure 4 As shown. In the above formula is the desired margin.

[0119] In order to optimize the hyperparameter m, the inventor introduces the following parameter setting guidance to improve the statistical properties of the CVAE model proposed in this embodiment: Under the assumption that the GMM distribution covariance matrix is ​​the unit matrix and given the 3-σ rule, the margin m of the triple loss needs to be greater than 6 to ensure that the features generated in each production stage do not overlap. The proof process of the setting guidance is not repeated in this disclosure.

[0120] When the CVAE encoder outputs a perfect GMM distribution, i.e., μ anchor =μ positive When the triplet loss term becomes m 2 -|μ anchor -μ negative | 2 .

[0121] According to some implementations, the following total loss function Loss is used in the training process of the CVAE model proposed in this embodiment:

[0122] Loss = λ*Loss trip +L CVAE , (7)

[0123] Among them, λ is the adjustment parameter, Loss trip is the triplet loss, L CVAE is the CVAE loss function.

[0124] Back to Figure 2 , step 3, performing adaptive change point detection on the latent variable sequence of each of the videos to obtain an initial change point set.

[0125] After obtaining the low-dimensional features (i.e., latent variables) of the image, we can look for important change points related to the production progress, which can be distinguished by visual algorithms. Specifically, at these time points, the external features of the workpiece change significantly.

[0126] In the prior art, there have been many studies on change point detection algorithms. Change point detection can be divided into two categories: detection with a known number of change points and detection with an unknown number of change points. For the former, the number of change points is known, so the position of each change point can be determined by solving a deterministic optimization problem. For the latter, the number of change points is uncertain.

[0127] In order to obtain a monitoring solution that can flexibly adapt to different production environments, this embodiment adopts detection based on the number of unknown change points. After in-depth research, the inventors assume that the output feature vector z of the potential variables in each production stage is i,t Following an independent GMM distribution, each production stage corresponds to a single cluster in the GMM model, and the change point detection can be modeled as a Gaussian distribution mean change detection with a constant and known variance, as shown in Equations 8 and 9:

[0128]

[0129] where μ i,y is the mean of the latent variables within the production stage y of video i. Under maximum likelihood estimation (MLE), the cost function for parameter learning includes the L2 norm loss function and a penalty term. It is worth noting that the L2 norm loss function is the same as the cost function for change point detection using a linear kernel function. The L2 norm loss function is used so that the consistency properties of MLE and kernel change point detection can be fully exploited.

[0130] According to the above research of the inventors, in some implementations, the following objective function is used for change point detection:

[0131]

[0132]

[0133] Where i is the video number, t is the time, y is the number of the production stage, c i The number of change points identified, z i,t is the latent variable of the image of video i at time t, μ i,y is the mean of the latent variable of video i in production stage y, T i,y and T i,y+1 is the time point of the adjacent change point, pen(c i ) is the number of change points c i Related penalties.

[0134] According to the objective function of this embodiment, by combining the squared error term and the penalty term, the change amplitude of the feature vector and the number of change points are automatically balanced, thereby achieving automatic optimization of the number and position of change points. The squared error term ensures that the change point is set when the feature changes significantly, while the penalty term effectively prevents over-segmentation and overfitting by increasing the number of change points, which means that the optimal balanced solution set contains the Pareto optimal solution for the number of change points. According to this embodiment, the number and position of change points can be adaptively adjusted according to the characteristics of the data, without pre-specifying a fixed number of change points or pre-setting change point conditions, so as to flexibly adapt to production processes with different complexities.

[0135] The performance of this embodiment is theoretically demonstrated below, proving that the adaptive change point detection performed according to this embodiment satisfies the asymptotic optimality theorem.

[0136] Since the production stages are automatically divided according to the design concept of this embodiment, it is necessary to ensure that the segmentation method used is optimal. "Optimality" includes the ability of the algorithm to identify all true change points in the mean of image features and accurately locate these points at their actual locations. First, use the standard metric d max (T i ,T′ i ) to evaluate the segment sequence T i and T′ i The similarity between , as defined in Equation 11.

[0137]

[0138] This measure captures the maximum temporal deviation between two sets of change points and can be used as a bound on the detection delay. In any given production video i, let c i represents the number of detected change points, N i Represents the number of eigenvalues ​​of video i. τ min represents the minimum time ratio between adjacent change points, Reflects the shortest production phase. is the time series z i,t The number of actual change points (separable points) in the equation. The upper limit of the distinguishable production stages c max is inferred from the complexity of the process, while δ min =min y |μ y -μ y+1 | describes the minimum mean difference between adjacent production stages. The following is an asymptotic optimality theorem to demonstrate the asymptotic optimality properties of the change point.

[0139] Asymptotic Optimality Theorem Let M isatisfies inequality 12, and a bound b∈R satisfies inequality 13, then the linear penalty function Ensure the number of change points The probability is at least 1-e -a , and the change point position is O p (1) The speed convergence:

[0140] In this embodiment, the midpoint of the interval can be selected as the value of b. The proof process of the above asymptotic optimality theorem is not repeated here. max The value of can be derived from data from production operations. min =m can be adjusted by the triplet loss margin. It is worth noting that the upper limit varies with N i grows, while the lower limit increases with log(N i ) grows linearly. Therefore, for sufficiently large N i , the upper limit always exceeds the lower limit. In addition, due to z i,t From the reparameterization of the CVAE model, if N i If is not large enough, the sample size can be increased by drawing more samples from the CVAE. The asymptotic optimality theorem guarantees that the number of detected production phases and the locations of change points converge to the actual production phases that are distinguishable by the given feature extractor and dataset as the sample size increases. The locations of change points converge to the same percentage of progress as the actual image features change.

[0141] Step 4: Use an iterative dynamic time warping (IDTW) algorithm to process the initial change point sets of the multiple videos to obtain a universal change point set.

[0142] Based on the adaptive change point detection in step 3, the initial change points can be extracted from each video. However, the change points in different production videos may originate from abnormal situations such as operating errors. Therefore, it is possible to consider extracting universal change points that appear consistently in various production processes. Optimizing the posterior probability of universal change points is a reasonable choice because the model is based on the GMM distribution. In addition, the universal change points should be a subset of the initial change points in each training video i.

[0143] Let l i,y μ represents the number of latent variables between the change points y-1 and y in video i. i,y and μ′ i,y =μ y They represent the means of the latent variables (i.e., feature means) before and after merging different video segments. i,y is the merged production stage index, which serves as a decision variable, and c i represents the number of change points detected in video i. The general change point detection model can be expressed as:

[0144]

[0145]

[0146] Equation 14 is used as the objective function of the universal change point detection model in order to maximize the posterior probability of the refined universal change point. Constraint 15 stipulates that the first and last change points of each video belong to the same stage. Constraint 16 indicates that the process between two adjacent change points can only belong to one production stage. Constraint 17 defines the mean of the merged latent variable, which is also the same as the MLE estimate.

[0147] Considering the complexity introduced by constraint 17, the problem still has significant computational challenges. To overcome this challenge, the inventors proposed an iterative dynamic time registration (IDTW) algorithm to effectively solve the computational cost problem.

[0148] It is observed that constraints 15 and 16 are consistent with the constraints in the DTW algorithm. To determine the optimal time to stop merging change points, the inventors derived and verified the following lemma. The proof of this lemma is not repeated here.

[0149] Lemma: Merging two consecutive datasets y and y+1 in video i results in a loss, which is given by:

[0150]

[0151] Among them l i,y is the length between the change point y and y+1 in video i: T i,y+1 -T i,y .

[0152] The above lemma shows that if the current change point already satisfies these constraints, there is no need to merge further change points, as this will have an adverse effect on the results. Based on this revelation, the inventors developed an iterative dynamic time registration (IDTW) algorithm to identify universal change points, such as Figure 5 shown.

[0153] In some embodiments, step 4 may include:

[0154] Pair the videos together;

[0155] You can follow Figure 6 The described algorithm performs iterative matching on the change points in each video pair. The iterative matching includes: using the dynamic time warping algorithm to process the change point sequences of the two videos in the video pair to obtain the corresponding DTW pair sequence. The jth DTW pair is represented as i1 and i2 are the numbers of the two videos in the video pair respectively; the DTW pair sequence of the video pair is traversed forward, and when the DTW pair meets the judgment condition and When the DTW pair does not meet the judgment condition, the DTW pair is deleted from the DTW pair sequence, and the time periods before and after the DTW pair are merged (refer to Figure 6 Algorithm line 10), and recalculate the mean of the latent variables in the merged segment, and recalculate the DTW pair sequence of the video pair; repeat the above process until a one-to-one correspondence is established between the change points of the two videos in the video pair;

[0156] After repeating the iterative matching for all video pairs, the remaining DTW pairs are used as the final universal change points to form the universal change point set, such as Figure 5 shown.

[0157] After identifying the common change points in all videos, we can trace back to determine the exact time T′ of each change point in each video i i,y Then, the mean of the corresponding change points is calculated as the component of GMM according to Equation 17, and the CVAE model is updated. During the training process, the model can determine F(k) = F(f(t,i)) = argmax through y. y (T′ i,y ≤t).

[0158] Back to Figure 2 , Step 5, updating the CVAE model according to the universal change point set until the CVAE model converges.

[0159] At the end of each training cycle, it can be determined whether the CVAE model has converged. If the model has not converged, the CVAE model is updated according to the universal change point set obtained in this training cycle, and the process returns to step 2 to start the next training cycle; if the model converges, the training process ends and a trained CVAE model is obtained.

[0160] The CVAE model can be trained offline. In some examples, learning can start from class zero. Given the diversity of videos, the feature distributions of different videos can be aligned. To this end, one can consider adding an additional maximum mean difference (MMD) loss during the initial 20 training cycles. For example, the total loss function used for training can comprehensively consider the CVAE loss function and the MMD loss function, which is similar to an enhanced VAE. After the initial training, adaptive change point detection and IDTW are performed as described above to update the class (ie, production stage) information and the corresponding CVAE class center (ie, the mean of all latent variables in the production stage, also called the feature mean). In subsequent iterations, as the model covers multiple classes, the triplet loss can be integrated, that is, the total loss function Loss = λ*Loss is used. trip +L CVAE Conduct training.

[0161] In each training cycle, the number of classes and the centers of each class can be dynamically updated through adaptive change point detection and IDTW.

[0162] In some implementations, updating the CVAE model according to the universal change point set includes:

[0163] Determine the timing of each universal change point in each video;

[0164] According to the time periods divided by the universal change point, the mean of the latent variable in each time period is calculated. The mean of the latent variable in each time period can be calculated according to the above equation 17;

[0165] The calculated mean of the latent variable is used as the mean parameter of the corresponding Gaussian component in the Gaussian mixture model (GMM);

[0166] The CVAE model is updated using the obtained time points of the universal change point and the GMM mean parameters.

[0167] When the loss function value drops below a certain threshold or no longer drops significantly, or the performance of the model on the validation set reaches the expected standard, the CVAE model can be judged to have converged. For example, when the stability of the total loss function reaches the expected standard, the accuracy of the model in predicting the production stage on the validation set reaches the expected standard, the area under the receiver operating characteristic curve (AUC) of the progress deviation of the model in different production environments reaches the expected standard, and / or the mean absolute percentage error (MAPE) reaches the expected standard, the CVAE model can be judged to have converged based on one of the above or a combination of the above.

[0168] After the model converges, CVAE is able to extract multivariate Gaussian mixture distributions from production site images, capturing both temporal and visual data. The mean μ of the latent variables yis saved for future predictions. In order to capture the stage division points learned by the model, predictions can be made on the validation set, and the change points of the model in this prediction process can be used as the final division results. When testing the model, these progress nodes (i.e., change points) can be used as criteria for annotating the test set. In addition, if any specific point of the identification process needs to be identified, just add that point to the results of IDTW and train in a semi-supervised manner.

[0169] In addition, according to this embodiment, the time cost l of each process stage y in each video i can be provided. i,y By calculating the average The time standard for each production stage can be automatically obtained. This information is very important for making a plan and detecting abnormal progress.

[0170] Step 6: Use the converged CVAE model to process the image sequence of the target production process collected in real time to predict the production stage.

[0171] In real-time prediction, the real-time image is processed by the CVAE model to estimate the distribution, and then the latent representation mean of the image predicted by the model is compared with the feature mean μ corresponding to each production stage. y Matching is performed to identify the production stage corresponding to the closest feature mean as the predicted production stage. The progress prediction results can be further used for real-time scheduling, material flow control and performance management.

[0172] Step 7, detect progress deviations based on the predicted production stages.

[0173] In some implementations, when the following progress detection conditions are not met, it is determined that there is a progress deviation and a corresponding alarm message is generated:

[0174] Whether the production stages at the adjacent predicted time points t and t+1 meet or in is the production stage at the predicted time point t, It is the predicted production phase at time point t+1, and / or the predicted production phase is different from the planned production phase.

[0175] A manager (eg, an operations manager) who receives the alarm information can conduct a check in a timely manner.

[0176] Figure 7 A schematic framework of offline training, real-time prediction and progress detection according to an exemplary embodiment of the present disclosure is shown.

[0177] The unsupervised process monitoring method proposed in this embodiment brings significant improvements to the repetitive production system. This embodiment realizes the automation, real-time monitoring and precise analysis of the production process by innovatively combining contrastive variational autoencoders (CVAE), adaptive change point detection and iterative dynamic time warping (IDTW) algorithms. Its unsupervised learning characteristics greatly reduce the dependence on manually labeled data, reduce the implementation cost, and improve the adaptability and scalability of the method. The adaptive change point detection technology can accurately identify the key stage transition points in the production process, and the IDTW algorithm ensures that the universal change points with high consistency are identified in multiple production cycles. The end-to-end unsupervised process monitoring method proposed in this embodiment can not only adapt to the subtle differences between different production batches, but also provide more robust and accurate progress monitoring, so that production managers can respond quickly to abnormal situations, greatly reducing the risk of production delays and quality problems. In addition, the generalization ability of this embodiment enables it to adapt to different types of production and manufacturing processes, providing strong support for the transformation of the manufacturing industry to intelligence.

[0178] The implementation effect of the unsupervised process monitoring scheme for repetitive production systems proposed in this embodiment is evaluated through four different datasets. The first dataset targets the water valve production process, where the assembly parts are the main focus of the image data. The second dataset targets HA4M, where the parts occupy the smallest part of the image, are usually difficult to recognize, and mainly show the operator's actions. The third dataset represents an actual production scenario of a realistic commercial aircraft spoiler, including assembly operations as well as tasks such as lamination, coating, and vacuum extraction, involving mechanical parts and flexible composite materials. The last dataset is a general action recognition dataset. The diversity of these four datasets is intended to test the robustness and versatility of the end-to-end unsupervised process monitoring scheme proposed in this embodiment in different environments.

[0179] (1) Water valve production

[0180] A dataset containing 8 videos as training set, 1 video as validation set, and 25 videos as test set is used. Each video captures the assembly process of a water valve. The production process of about 2000 time frames is carried out at a single workstation. The production process involves workers inserting the rubber plug into the middle valve body and fixing it with screws. Subsequently, the worker removes the lower part of the valve, puts the gasket on it, and then assembles the middle and upper valve bodies and related parts. Next comes the installation of the rocker arm, installing screws and plastic covers. Finally, the assembly of the upper and lower clamps is completed.

[0181] The CVAE model proposed in this disclosure successfully identified eight different production stages. These stages are Figure 8The visualization in Figure 1 shows the production of the intermediate valve body to the final upper clamp. The identified steps are: 1) production of the intermediate valve body; 2) fixing screws; 3) assembly of the lower part of the valve; 4) assembly of the upper part of the valve; 5) assembly of the rocker arm; 6) assembly of the parts on the rocker arm; 7) completion of the production of the lower clamp of the water valve; 8) assembly of the upper clamp. These production phases are logical because each production phase corresponds to the beginning or end of each observed addition of parts during the production cycle.

[0182] Visualization of features extracted from water valve production Fig. 9 As shown, the horizontal and vertical axes represent the first and second dimensions of t-SNE, respectively. Blue represents the training set features, red represents the test set features, and green represents the validation set features. The gradient from light to dark corresponds to the time proportion of the images in the video sequence, and the dark color represents the later stage of the production process. Fig. 9 It can be seen that the eight time-related clusters are clearly separated.

[0183] (II) HA4M Dataset

[0184] The HA4M dataset consists of 217 videos showing repetitive manual production processes. There are 41 subjects in the dataset performing a recorded production task. The test analysis focuses on 106 videos from a consistent environment, visualizing eight progress states such as Fig.10 The remaining videos show the same production process, but in an informal setting and are considered outside the test parameters. 12 videos were used as a training set, 1 video as a validation set, and 82 videos as a test set. The remaining 11 videos assembled different procedures for the purpose of progress deviation detection evaluation only. The assembly process involves placing several parts onto a fixture, installing three gears and their outer rails, and securing them. The workpiece is then moved to another fixture to install the parts on the other side.

[0185] This embodiment successfully identifies 8 states in the production process, such as Fig.10 These production phases are logical because each phase corresponds to the beginning of part assembly or the completion of an operation. In addition, the visualization of features extracted from the HA4M dataset is shown in Fig.11 As shown, use Fig. 9 Same explanation.

[0186] (III) Actual production of commercial aircraft spoilers

[0187] The purpose of this study is to evaluate the performance of the unsupervised process monitoring method proposed in this embodiment in the actual production of commercial aircraft spoilers. The workstation for spoiler production was selected because it involves not only mechanical parts but also flexible materials. During the production process, operations are performed on or next to the tool. The production process includes workers doing some pre-installation work next to the tool (checkpoint 1), waiting for the primer to dry and checking the auxiliary materials (checkpoint 2), placing a layer of yellow composite material at the designated position of the tool, and then applying composite films of various colors (checkpoint 3). The combined rigid structure is then inserted into the grinder, the mold is spliced ​​and covered with a red transparent plastic film (checkpoint 4). Trim off the excess part of the mold, then cover it with a layer of white cloth and trim it to the shape of the mold (checkpoint 5). Cover it with a layer of transparent green plastic film and vacuum it (checkpoint 6). After completion, it is moved into an autoclave for curing. This study focuses on the spoiler production stage performed on the tool before curing, which usually spans about 2000-3000 time frames. Since the tool needs to be pushed into the autoclave after completion and brought back in the next cycle, the position of the tool is not fixed each time it appears. Video is collected using surveillance cameras on the shop floor without any impact on production or additional hardware costs. The process is a hybrid operation of manual and human-machine integrated manufacturing. It is difficult to segment by part retrieval or worker action due to multiple workers operating simultaneously and the frequent presence of unrelated personnel and actions. In addition, manually identifying separable points for visual algorithms is even more difficult given the long time span. There is often a delay in the information of several checkpoints that workers need to record, and they may accumulate information for multiple checkpoints to record, or forget to record it altogether.

[0188] The dataset was created from three weeks of data from factory surveillance videos, totaling approximately 46,000 images. Six units were divided into training sets, one unit as a validation set, and the other six units as test sets. The training set contains videos from the production of three different types of spoilers, while the test and validation sets include spoilers from the training set and one type of untrained spoiler. The changes in spoilers are reflected in the shape (from rectangular to trapezoidal) and size, as well as different fixing structures designed for wing fixation. Despite these changes, the production process remains consistent across different spoiler types, consistent with the assumptions of the present disclosure about repeated production scenarios. The dataset is based on the operating hours of the factory, excluding production data during factory shutdowns. In the dataset, some tools are temporarily moved out of the camera's field of view halfway through processing, workers assemble halfway and then dismantle and reassemble, or the lights in the workshop are temporarily turned off, and engineering vehicles enter and block the camera's field of view.

[0189] In the manufacturing process studied, the CVAE model proposed in this embodiment can accurately and automatically divide these production processes into up to seven production stages. These production stages include all the inspection points involving visual changes in the process file, and one inspection point is further subdivided into two parts. Fig.12 The progress classification results of the validation set and the test set are shown, with the x-axis representing the time course of the production video and the y-axis representing the predicted production stage. Fig.13 As shown, use Fig. 9 Same explanation.

[0190] To evaluate schedule deviation detection, the test dataset was annotated with the collective expertise of multiple factory personnel and management. A total of 16 different schedule deviation situations and 1,892 normal situations were annotated. Deviations included moving tools and equipment ahead of schedule, re-operation, etc.

[0191] (IV) Clothing Folding Dataset

[0192] The dataset contains 970 videos showing the process of folding clothes, identifying six unique short sleeves and a variety of different folding methods. The analysis for this study focuses on one main technique: the method of folding after placing the clothes vertically, represented by 270 videos in the dataset. For consistency, instances in the videos where two clothes are folded at the same time or refolded are excluded. Therefore, the dataset for analysis is reduced to 246 folding events, divided into 12 videos as training set, 1 video as validation set, and 233 videos as test set.

[0193] Since there are multiple pieces of clothing in the scene, data preprocessing can be performed through background subtraction, taking the absolute difference from the first frame and retaining pixels whose difference exceeds a certain threshold. The CVAE model proposed in this embodiment consistently identified four action classes: spreading clothing on the table, folding horizontally, folding vertically, and putting aside, achieving an accuracy rate of 84.81%. The accuracy rate is lower than the aforementioned production progress recognition. The main reason for this is the non-standard folding operation and the fact that the image features of the placed and unfolded states of clothing in some videos are almost identical. These are the four common steps observed in all training videos. The detected clothing folding stages are as follows: Fig.14 The extracted features are visualized as shown in Fig.15 As shown, four clusters are clearly distinguished using Fig. 9 Same explanation.

[0194] In order to illustrate the superiority of this embodiment, the unsupervised process monitoring method proposed in this embodiment is compared with the monitoring method based on the existing benchmark model. Specifically, the CVAE model proposed in this embodiment is compared with the time constraint network (TCN) and two variant models, which are TCN-C without VAE components (similar to the enhanced TCN model) and VaDE-C without triple components (similar to the enhanced VaDE model). Through this comparison, the advantages of the CVAE model according to this embodiment are fully demonstrated.

[0195] When comparing the schedule prediction performance quantitatively, three metrics are used. The first is the schedule prediction accuracy; the second is the number of production stages divided, which reflects the accuracy of the given production stage division; the third is the mean absolute percentage error (MAPE) of time calculated according to Equation 19 to take into account the variability in the number of process stages that different models may identify:

[0196]

[0197] in Indicates the prediction progress status in video i The estimated change point time, T i,-1 = 0, reflecting the conceptualization of the schedule forecast as a continuous task; after forecasting the production stage of the current schedule state, the time midpoint of the production state is determined The predicted value is used for comparison with the actual occurrence time, so as to comprehensively evaluate the accuracy of process stage division and prediction accuracy.

[0198] In the field of progress deviation detection, the problem is framed as a binary classification task - identifying deviant or normal instances. Its effectiveness can be evaluated using two widely recognized metrics: F1 score and area under the receiver operating characteristic curve (AUC).

[0199] Table 1 Quantitative results of progress prediction

[0200]

[0201] Table 2 Quantitative results of schedule deviation detection

[0202]

[0203] Table 1 compares the performance of the method proposed in this embodiment with respect to three benchmark models on four different data sets in terms of progress class (i.e., the number of divided production stages), classification accuracy, and progress MAPE.

[0204] From the quantitative performance analysis results presented in Table 1, it can be seen that the proposed method in this embodiment has significantly better progress state recognition accuracy than the baseline model on all data sets, achieving higher progress prediction accuracy and lower MAPE, even when identifying more progress states. It is worth noting that in Table 1, the accuracy and MAPE of the TCN model are significantly lower, mainly because it is designed for multi-view camera data, while this study collects single-view camera data. In the three production cases (water valve production, HA4M, and aircraft spoiler production), the average accuracy of production progress prediction reached 98.14.

[0205] It can be seen from the quantitative performance analysis results presented in Table 2 that the method proposed in this embodiment also shows higher performance in terms of AUC and F1 score indicators in progress deviation detection.

[0206] It is analyzed that the reasons for the excellent performance of this embodiment mainly include: First, the adaptive change point detection, IDTW and CVAE adopted in this embodiment not only capture time information and locate the precise process stage cutting points, but also enable the latent variables to effectively extract pixel-level information. These key information enable the unsupervised embedding method according to this embodiment to learn distinguishable points and build a classifier in an end-to-end manner. In addition, the implementation of triplet loss further ensures that each progress class has a unique feature distribution.

[0207] The architecture and related monitoring mechanism of the CVAE model proposed in this embodiment are derived from the loss function supported by theory and the adaptive change point detection strategy that conforms to the statistical characteristics of a specific data set, eliminating the need for cross-dataset parameter adjustment. In addition, the study also confirmed that the CVAE model can accurately distinguish process states. Moreover, through the proposed IDTW algorithm, a common state can be established in the training video, effectively filtering out false change points.

[0208] The present disclosure also proposes an unsupervised process monitoring device for a repetitive production system, including an image acquisition unit, a feature extraction unit, an adaptive change point detection unit, a universal change point screening unit, a training cycle control unit, a real-time prediction unit and a progress deviation detection unit.

[0209] The image acquisition unit is used to collect multiple videos of the target production process to obtain an image sequence of each video.

[0210] The feature extraction unit is used to extract features from the image sequence using a contrastive variational autoencoder (CVAE) model to obtain a latent variable sequence.

[0211] The adaptive change point detection unit is used to perform adaptive change point detection on the potential variable sequence of each video to obtain an initial change point set.

[0212] The universal change point screening unit is used to process the initial change point sets of the multiple videos using an iterative dynamic time warping (IDTW) algorithm to obtain a universal change point set.

[0213] The training cycle control unit is used to update the CVAE model according to the universal change point set until the CVAE model converges.

[0214] The real-time prediction unit is used to process the image sequence of the target production process collected in real time using the converged CVAE model to predict the production stage.

[0215] The schedule deviation detection unit is used to detect the schedule deviation according to the predicted production stage.

[0216] For other details and beneficial effects of this embodiment, please refer to the above related introduction, which will not be repeated here.

[0217] Fig.16 An electronic device provided for at least one embodiment of the present disclosure comprises a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the unsupervised process monitoring method for a repetitive production system described in any embodiment or implementation of the present disclosure when executing the computer instructions.

[0218] At least one embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the unsupervised process monitoring method for a repetitive production system described in any embodiment or implementation of the present disclosure.

[0219] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0220] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0221] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0222] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0223] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0224] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0225] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0226] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0227] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0228] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0229] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. An unsupervised process monitoring method for a repetitive production system, characterized in that: include: Collecting multiple videos of the target production process to obtain an image sequence of each of the videos; Using a contrastive variational autoencoder (CVAE) model to extract features from the image sequence to obtain a latent variable sequence; Performing adaptive change point detection on the latent variable sequence of each video to obtain an initial change point set; Processing the initial change point sets of the multiple videos using an iterative dynamic time warping (IDTW) algorithm to obtain a universal change point set; Updating the CVAE model according to the universal change point set until the CVAE model converges; The converged CVAE model is used to process the image sequences of the target production process acquired in real time to predict the production stage; Detect schedule deviations based on forecasted production phases.

2. The method according to claim 1, characterized in that The CVAE model is set up as: Use the ResNet50 architecture as an encoder to generate latent variables; Reconstruct the image using a convolutional decoder.

3. The method according to claim 2, characterized in that The following total loss function Loss is used during the training of the CVAE model: Loss=λ*Loss trip +L CVAE , Among them, λ is the adjustment parameter, Loss trip is the triplet loss, L CVAE is the CVAE loss function.

4. The method according to claim 3, characterized in that Based on the prior distribution assumption that the latent variables obey the Gaussian mixture model (GMM) distribution, the CVAE loss function L is calculated according to the following formula: CVAE , where each production stage corresponds to a Gaussian component in GMM: in, is the reconstructed image, x is the original input image of the CVAE model, σ is the standard deviation of the pixel, is the potential standard deviation of the production stage y, is the potential representation mean under the production stage y, μ y is the mean parameter of the Gaussian component corresponding to the production stage y in the prior distribution.

5. The method according to claim 3, characterized in that: The triplet loss Loss is calculated according to the following formula: trip : Loss trip =min(|μ anchor -m positive | 2 -|m anchor -m negatice | 2 +m 2 ,0), Among them, μ anchor is the potential representation mean of the anchor image, μ positive is the potential representation mean of the positive sample image, μ negative is the mean of the latent representations of negative sample images.

6. The method according to claim 1, characterized in that The adaptive change point detection comprises: The following objective function is used for change point detection: Where i is the video number, t is the time, y is the number of the production stage, C i The number of change points identified, z i,t is the latent variable of the image of video i at time t, μ i,y is the mean of the latent variable of video i in production stage y, T i,y and T i,y+1 is the time point of the adjacent change point, pen(C i ) is the number of change points C i Related penalties.

7. The method according to claim 1, characterized in that The initial change point sets of the multiple videos are processed using the IDTW algorithm to obtain a universal change point set, including: Pair the videos together; The change points in each video pair are iteratively matched. The iterative matching includes: using the dynamic time warping algorithm to process the change point sequences of the two videos in the video pair to obtain a DTW pair sequence. The jth DTW pair is represented as i1 and i2 are the numbers of the two videos in the video pair respectively; the DTW pair sequence of the video pair is traversed forward, and when the DTW pair meets the judgment condition and When the DTW pair meets the judgment condition, the DTW pair is identified as the common change point of the video pair and the DTW pair is retained. When the DTW pair does not meet the judgment condition, the DTW pair is deleted from the DTW pair sequence, the time periods before and after the DTW pair are merged, and the means of the latent variables in the merged segments are recalculated, and the DTW pair sequence of the video pair is recalculated; the above process is repeated until a one-to-one correspondence is established between the change points of the two videos in the video pair; After repeatedly performing the iterative matching on all video pairs, the remaining DTW pairs are used as final universal change points to form the universal change point set.

8. The method according to claim 4, characterized in that Updating the CVAE model according to the universal change point set includes: Determine the timing of each universal change point in each video; According to the time periods divided by the common change point, the mean of the latent variable in each time period is calculated; The calculated mean of the latent variable is used as the mean parameter of the corresponding Gaussian component in the Gaussian mixture model (GMM); The CVAE model is updated using the obtained time points of the universal change point and the GMM mean parameters.

9. The method according to claim 1, characterized in that: Detect schedule deviations based on the predicted current production stage, including: When the following progress detection conditions are not met, it is judged that there is a progress deviation and a corresponding alarm message is generated: Whether the production stages at the adjacent predicted time points t and t+1 meet or in is the production stage at the predicted time point t, It is the predicted production phase at time point t+1, and / or the predicted production phase is different from the planned production phase.

10. An unsupervised process monitoring device for a repetitive production system, characterized in that: include: An image acquisition unit, used to collect multiple videos of the target production process to obtain an image sequence of each of the videos; A feature extraction unit, used to extract features from the image sequence using a contrastive variational autoencoder (CVAE) model to obtain a latent variable sequence; An adaptive change point detection unit, used to perform adaptive change point detection on the latent variable sequence of each video to obtain an initial change point set; a universal change point screening unit, configured to process the initial change point sets of the plurality of videos using an iterative dynamic time warping (IDTW) algorithm to obtain a universal change point set; A training cycle control unit, used for updating the CVAE model according to the universal change point set until the CVAE model converges; A real-time prediction unit is used to process the image sequence of the target production process collected in real time using the converged CVAE model to predict the production stage; The schedule deviation detection unit is used to detect the schedule deviation according to the predicted production stage.

11. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the method according to any one of claims 1 to 9 when executing the computer instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • Sensitive image recognition method based on feature learning

    CN108460390A

  • Defect detection method and model training method for defect detection

    CN111833306A

  • Laser point cloud processing method, electronic device and storage medium

    CN112417965A

  • Boundary box distribution model training method and device and target detection method and device

    CN117132754A

  • VAE and deep neural network mixed intelligent manufacturing factory process quality monitoring method

    CN117193184A

Cited By

  • Unsupervised process monitoring for repetitive production system

    WO2026045085A1