Unsupervised process monitoring method, apparatus and device for repetitive production systems

Abstract

Disclosed are an unsupervised process monitoring method, device and equipment for a repetitive production system. The method comprises: collecting multiple videos of a target manufacturing process to obtain an image sequence; performing feature extraction on the image sequence by using a CVAE model to obtain a latent variable sequence; performing adaptive change point detection on the latent variable sequence to obtain an initial change point set; screening out a general change point set by using an IDTW algorithm; updating the CVAE model according to the general change point set until convergence; processing a real-time collected image sequence by using the converged CVAE model to predict a production stage; and detecting a progress deviation according to the predicted production stage. The disclosure significantly reduces the dependence on artificial labeled data, reduces the implementation cost, can adapt to different production scenarios, realizes high-precision unsupervised process monitoring, and improves the production efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of advanced manufacturing, and in particular to an unsupervised process monitoring method, apparatus and equipment for repetitive production systems. Background Technology

[0002] In modern manufacturing systems, especially advanced production systems involving multi-step assembly processes, process monitoring and real-time schedule management are crucial. Currently, the industry has developed various process monitoring and management methods. Traditional methods, such as RFID or QR code implementations, monitor production progress by installing reading devices on the production line to track the flow of parts / assemblies. Another common method is to use wearable sensors to infer production activities by recognizing operator actions. With advancements in computer vision technology, vision-based methods are also increasingly being applied, identifying production stages and progress by analyzing images or videos of the production site.

[0003] However, these existing technological approaches face numerous challenges in practical applications. While RFID and QR code technologies can effectively track the flow of parts, they often increase scanning time, potentially extending circulation time and reducing productivity. Furthermore, these technologies have high initial implementation costs and may present compatibility issues between different partners. The application of wearable sensor technology in production environments is also limited, primarily due to its lower accuracy and the potential impact of wearing requirements on operator comfort.

[0004] While vision-based methods have become more efficient at acquiring production progress information involving human-computer interaction and repetitive manual operations, several challenges remain. Many existing vision methods require substantial amounts of manually labeled data, which is costly and time-consuming to implement in complex and ever-changing production environments. Furthermore, these methods typically require predefined process stages, making it difficult to automatically adapt to different production flows and lacking sufficient flexibility and versatility. When faced with new production scenarios or product variations, these methods often require retraining or significant adjustments, exhibiting limited generalization capabilities.

[0005] Furthermore, existing methods still fall short in providing real-time, accurate progress information and deviation detection. Accurately locating critical progress checkpoints remains a challenge, particularly in complex, multi-stage production processes. These methods often struggle to adapt to dynamically changing production environments, especially when product variety is diverse or production processes are frequently adjusted.

[0006] Given these limitations, there is an urgent need to develop a process monitoring method that can automatically, accurately, and in real time monitor production progress and adapt to different production environments. Summary of the Invention

[0007] This disclosure proposes an unsupervised process monitoring scheme for repetitive production systems, which can automatically identify and segment production stages without extensive manual intervention and can adapt to different production environments.

[0008] According to one embodiment of this disclosure, an unsupervised process monitoring method for repetitive production systems is proposed, comprising:

[0009] Multiple videos of the target production process are acquired, and an image sequence of each video is obtained;

[0010] The image sequence is used to extract features to obtain a sequence of latent variables.

[0011] Adaptive change point detection is performed on the latent variable sequence of each video to obtain an initial change point set;

[0012] The initial set of variable points in the multiple videos is processed using the Iterative Dynamic Time Warping (IDTW) algorithm to obtain a general set of variable points;

[0013] Update the CVAE model according to the general set of variable points until the CVAE model converges;

[0014] A convergent CVAE model is used to process image sequences of the target production process acquired in real time to predict the production stage;

[0015] Based on the predicted deviations in the production phase inspection schedule.

[0016] In some implementations, the CVAE model is set as follows:

[0017] The ResNet50 architecture is used as the encoder to generate latent variables;

[0018] The image is reconstructed using a convolutional decoder.

[0019] In some implementations, the CVAE model uses the following total loss function, Loss, during training:

[0020] Loss = λ * Loss trip +L CVAE ,

[0021] Where λ is the adjustment parameter, and Loss trip For triplet loss, L CVAE This is the CVAE loss function.

[0022] In some implementations, based on the prior distribution assumption that the latent variables follow a Gaussian mixture model (GMM) distribution, the CVAE loss function L is calculated according to the following formula. CVAEEach production stage corresponds to a Gaussian component in the GMM:

[0023]

[0024] in, To reconstruct the image, X is the original input image of the CVAE model, and σ is the standard deviation of the pixels. Let y be the potential standard deviation representing the production stage. Let μ be the latent mean of the production stage y. y Let be the mean parameter of the Gaussian component of y corresponding to the production stage in the prior distribution.

[0025] In some implementations, the triplet loss (Loss) is calculated according to the following formula. trip :

[0026] Loss trip =min(|μ anchor -μ positive | 2 -|μ anchor -μ negative | 2 +m 2 ,0),

[0027] Where, μ anchor μ is the latent representation mean of the anchor point image. positive μ is the latent representation mean of the positive sample images. negative The latent mean of the negative sample images.

[0028] In some implementations, the adaptive change point detection includes:

[0029] Change point detection is performed using the following objective function:

[0030]

[0031] Where i is the video ID, t is the time, y is the production stage ID, and C i The number of identified variable points, z i,t μ is the latent variable of the image of video i at time t. i,y Let T be the mean of the latent variables of video i during the production stage y. i,y and T i,y+1 For the time points of adjacent change points, pen(C) i ) is related to the number of variable points C i Related penalties.

[0032] In some implementations, the initial set of variable points in the plurality of videos is processed using the IDTW algorithm to obtain a universal set of variable points, including:

[0033] Pair the videos together;

[0034] For each video pair, iterative matching is performed on the variable points. This iterative matching includes: processing the variable point sequences of the two videos in the video pair using a dynamic time warping algorithm to obtain a DTW pair sequence, where the j-th DTW pair is represented as... i1 and i2 are the video numbers of the two videos in the video pair, respectively; perform a forward traversal of the DTW pair sequence of the video pair, and when the DTW pair satisfies the judgment condition... and When the DTW pair is identified as a common variable point of the video pair, the DTW pair is retained. When the DTW pair does not meet the judgment condition, the DTW pair is deleted from the DTW pair sequence, the time periods before and after the DTW pair are merged, the mean of the latent variables in the merged segment is recalculated, and the DTW pair sequence of the video pair is recalculated. The above process is repeated until a one-to-one correspondence is established between the variable points of the two videos in the video pair.

[0035] After repeating the iterative matching for all video pairs, the remaining DTW pairs are used as the final universal variable points to form the universal variable point set.

[0036] In some implementations, updating the CVAE model based on the general variable point set includes:

[0037] Determine the time point of each common variable in each video;

[0038] Calculate the mean of the latent variables within each time period based on the time period divided by the common change point;

[0039] The mean of the calculated latent variables is used as the mean parameter of the corresponding Gaussian component in the Gaussian mixture model (GMM);

[0040] Update the CVAE model using the obtained universal change point time points and GMM mean parameters.

[0041] In some implementations, the detection progress deviation is based on the predicted current production stage, including:

[0042] If the following progress detection conditions are not met, a progress deviation is determined to exist, and a corresponding alarm message is generated:

[0043] Do the predicted production stages at adjacent time points t and t+1 meet the requirements? or in It refers to the production stage at the predicted time point t. It refers to the production stage at the predicted time point t+1, and / or the predicted production stage differs from the planned production stage.

[0044] According to one embodiment of this disclosure, an unsupervised process monitoring device for repetitive production systems is proposed, comprising:

[0045] An image acquisition unit is used to acquire multiple videos of the target production process and obtain an image sequence for each video;

[0046] The feature extraction unit is used to extract features from the image sequence using a contrastive variational autoencoder (CVAE) model to obtain a sequence of latent variables;

[0047] An adaptive change point detection unit is used to perform adaptive change point detection on the latent variable sequence of each video to obtain an initial change point set;

[0048] A general variable point filtering unit is used to process the initial variable point set of the multiple videos using the Iterative Dynamic Time Warping (IDTW) algorithm to obtain a general variable point set;

[0049] The training cycle control unit is used to update the CVAE model according to the general variable point set until the CVAE model converges.

[0050] The real-time prediction unit is used to process the image sequence of the target production process acquired in real time using a converged CVAE model to predict the production stage.

[0051] The schedule deviation detection unit is used to detect schedule deviations based on the predicted production stage.

[0052] According to one embodiment of this disclosure, an electronic device is provided, the device including a memory and a processor, the memory being used to store computer instructions executable on the processor, the processor being used to implement the method as described in any of the preceding claims when executing the computer instructions.

[0053] According to one embodiment of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method as described in any of the preceding claims.

[0054] The end-to-end unsupervised embedding technology proposed in this disclosure brings significant innovation and advantages to process monitoring in repetitive production systems. By intelligently separating and labeling production stages, this disclosure achieves accurate stage identification and process monitoring, providing a highly efficient and automated solution for the manufacturing industry.

[0055] First, this disclosure proposes an innovative Contrastive Variational Autoencoder (CVAE) that can unsupervisedly embed repetitive production processes into Gaussian Mixture Models (GMMs). By considering the separability and temporal information of features at each production stage (also known as a process class), CVAE provides a more comprehensive and accurate representation of the production process. This approach enables the system to automatically capture and learn key features and patterns in complex production processes, adapting to different production scenarios without human intervention.

[0056] Secondly, the adaptive change point detection method proposed in this disclosure, combined with the Iterative Dynamic Time Warping (IDTW) algorithm, provides a powerful tool for the automatic segmentation and labeling of production stages. This innovative method can not only accurately identify key transition points in the production process, but also effectively prevent human bias and reduce reliance on manual annotation. In this way, the objectivity and efficiency of process monitoring are greatly improved, enabling more accurate capture of subtle changes and important turning points in the production process.

[0057] Furthermore, this disclosure includes in-depth theoretical analysis and research, particularly regarding the asymptotic optimality of change-point detection. This theoretical foundation ensures high-precision progress identification and process monitoring even when using unsupervised methods. This characteristic provides strong support for the reliability and stability of the solution, enhancing its persuasiveness and credibility in practical applications.

[0058] The end-to-end unsupervised process monitoring technology proposed in this disclosure achieves efficient integration of production stage identification and process monitoring by learning the temporal representation and key features of repetitive processes. Its inherent unsupervised nature not only eliminates the need for large amounts of manually annotated data but also significantly improves the efficiency of process monitoring and production management. It can automatically adapt to different production environments and product variations without frequent manual intervention and adjustments.

[0059] The effectiveness and adaptability of the unsupervised process monitoring scheme for repetitive production systems presented in this disclosure have been fully validated through multiple case studies, including the production of complex commercial aircraft spoilers. These diverse application scenarios demonstrate the versatility and robustness of the scheme, proving that it is suitable not only for standardized production lines but also for highly complex and customized production processes. Attached Figure Description

[0060] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0061] Figure 1 An unsupervised contrastive learning method framework according to an embodiment of the present disclosure is shown;

[0062] Figure 2 A flowchart of an unsupervised process monitoring method for a repetitive production system according to an embodiment of the present disclosure is shown;

[0063] Figure 3 A schematic diagram of a CVAE model with triplet loss according to an exemplary embodiment of the present disclosure is shown;

[0064] Figure 4 A schematic diagram of the margin of triplet loss according to an exemplary embodiment of the present disclosure is shown;

[0065] Figure 5 A schematic diagram illustrating the identification of general variable points using the IDTW algorithm according to an exemplary embodiment of the present disclosure is shown.

[0066] Figure 6 The pseudocode illustration shows an iterative matching algorithm according to an exemplary embodiment of the present disclosure;

[0067] Figure 7 An exemplary framework for offline training, real-time prediction, and progress detection according to an exemplary embodiment of the present disclosure is shown.

[0068] Figure 8 A visual display diagram of each stage of the water valve manufacturing process, as tested according to an exemplary embodiment of the present disclosure, is shown.

[0069] Figure 9 A visualization showing the extraction of features from water valve production according to an exemplary embodiment of the present disclosure is illustrated;

[0070] Figure 10 A schematic visualization of the various production stages of an HA4M dataset detected according to an exemplary embodiment of the present disclosure is shown.

[0071] Figure 11 A visualization of feature extraction from the HA4M dataset according to an exemplary embodiment of the present disclosure is shown;

[0072] Figure 12 The results of the progress classification of actual production of commercial aircraft spoilers in the validation and test sets, according to an exemplary embodiment of this disclosure, are shown.

[0073] Figure 13 A visualization showing the extraction of features from aircraft spoiler production according to an exemplary embodiment of the present disclosure is provided;

[0074] Figure 14A visual display diagram showing the various production stages of garment folding detected according to an exemplary embodiment of the present disclosure is shown;

[0075] Figure 15 A visualization of features extracted from clothing folds according to an exemplary embodiment of the present disclosure is shown;

[0076] Figure 16 This is a schematic diagram of the structure of an electronic device shown in at least one embodiment of the present disclosure. Detailed Implementation

[0077] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0078] This disclosure introduces an unsupervised embedding technique that simultaneously divides a target production process into different stages and predicts real-time progress information through an end-to-end process. To achieve this goal, this disclosure proposes a contrastive variational autoencoder (CVAE) for image feature embedding, combined with a proposed adaptive change-point detection method and iterative dynamic time registration (IDTW) technique to automatically divide production stages. These methods enable the embedded feature space to effectively capture key information related to production progress, which is helpful for process monitoring in repetitive production systems. This disclosure envisions deploying suitable camera systems in strategic locations within the production environment to continuously capture data. The application scenarios and data assumptions of this disclosure are as follows:

[0079] 1. The production process is standardized, which means that the sequence of each production stage and the operations before and after it are fixed, unless an abnormal situation occurs;

[0080] 2. It can acquire one or more actual production videos, each video covering a complete production process;

[0081] 3. The parts and workpieces are located within a fixed area and are within the camera's field of view;

[0082] 4. During the production stage, the workpiece y is visually separable. (Set) and There is no intersection, where z i,t The image features obtained from the training production video i at time t are the latent variables in the VAE model; F(k) is a deterministic function that maps the continuous production progress k = F(f(t,i)) to discrete production stages.

[0083] In repetitive production processes, some points of inflection undergo significant characteristic changes within a short period. For example, additional parts may be added, or specific operations may be affected by random characteristic changes, often caused by manual operation or background. More detailed examples can be found in subsequent application examples. Therefore, images associated with a production process are suitable for modeling using a Gaussian distribution, with each production stage corresponding to a Gaussian distribution.

[0084] The framework of the unsupervised contrastive learning method proposed in this disclosure is as follows: Figure 1 As shown. This disclosure utilizes the Gaussian Mixture Model (GMM) distribution as a prior hypothesis and uses variational inference to map the image to the optimal GMM distribution. This process enables the disclosure to detect change points where image feature values ​​actually change, employing a contrastive learning method.

[0085] Suppose the image from video i at time t is represented as (Width is w, height is h). This disclosure uses a multivariate GMM to measure x i,t Transform into a vector with features The potential space allows for the precise identification of key features relevant to process monitoring. Specifically, this disclosure proposes a novel contrastive variational autoencoder (CVAE) for image-to-GMM mapping, followed by change point detection applied to the CVAE features to identify visual features. The changes in progress are significant. Due to operational variations or different angles, each video may exhibit multiple inconsistent change points. To address this issue, the IDTW algorithm proposed in this disclosure is used to refine the detected change points in each video and identify common change points that consistently represent production progress. These refined common change points can be considered as the boundaries of each production stage. In a real-time prediction environment, images are processed via CVAE to obtain feature values, and then the most recent progress cluster is predicted as the progress stage. Progress deviations are then detected through post-processing.

[0086] This disclosure can be applied to computer systems / servers that can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, etc.

[0087] Computer systems / servers can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0088] Figure 2 A flowchart of an unsupervised process monitoring method for a repetitive production system according to an embodiment of the present disclosure is shown. As shown, the method includes steps 1 to 7.

[0089] Step 1: Collect multiple videos of the target production process to obtain an image sequence for each video.

[0090] The existing surveillance camera system in the workshop can be used to collect video data.

[0091] During the acquisition process, each video corresponds to a complete manufacturing cycle, covering all production stages from start to finish. This allows for the capture of key information throughout the entire production process and the learning of subtle differences that may exist between different production batches, thereby improving the robustness and adaptability of the method.

[0092] To ensure data quality, parts and workpieces can be placed within a fixed area and kept within the camera's field of view. These videos may include scenes of multiple workers operating simultaneously and frequent appearances of unrelated personnel; these complexities accurately reflect the characteristics of a real production environment.

[0093] The captured video can be converted into a discrete image sequence, and the images can be preprocessed as needed.

[0094] Step 2: Use the contrastive variational autoencoder (CVAE) model to extract features from the image sequence to obtain the latent variable sequence.

[0095] In some implementations, the ResNet50 architecture is used as the encoder to generate latent variables; a convolutional decoder is used to reconstruct the image.

[0096] In this implementation, ResNet50 is used to transform the input image into latent variables for feature representation, capturing key features and abstract representations from the original image. In some examples, ImageNet pre-trained weights can be used to initialize the CVAE model. A convolutional decoder is used to reconstruct the latent variables back into the original image space.

[0097] The CVAE model proposed in this embodiment utilizes powerful function approximators provided by the variational inference domain to learn a function that embeds the image into the latent variable z. i,t In this embodiment, we assume the variable point T is... i,y and T i,y+1 The latent variable z between i,t Follows an independent multivariable Gaussian distribution For each video i. μ y y is the mean parameter of the production stage y of video i, also known as the common feature center, that is, the entire distribution is heterogeneous, as shown in Equation 1:

[0098]

[0099] The latent variables of the entire production process should follow a GMM distribution, as shown in Equation 2, where clusters correspond to their respective production stages, α y This represents the weight of the Gaussian component of y in the production stage.

[0100]

[0101] In fact, some studies have used Gaussian distributional assumptions (GMMs) as latent variables in VAEs (Variational Autoencoders). In these studies, the distribution p(y|x) of image class y is independent. However, in the scenario of this disclosure, the Gaussian distribution cluster of each image data is determined by the percentage of progress k in the video. Note that k is different from time t in video i, because the speed of each production process may be different, i.e., k = f(t, i). That is, after determining the time point of the change point according to this embodiment, the image data between two adjacent change points belong to the same class, as shown in Equation 3.

[0102] y = F(k) = F(f(t,i)). (3)

[0103] This embodiment proposes a model, such as Figure 3 The CVAE model shown is used to approximate the actual distribution. This CVAE model outputs the predicted distribution parameters. and reconstructed image Similar to some well-known VAE models, this embodiment assumes that the image pixel values ​​follow a Gaussian distribution, as shown in Equation 4. It is a hyperparameter:

[0104]

[0105] The training objective of CVAE is to maximize the likelihood function for a given data point. Let q(z|y) denote the posterior probability obtained from CVAE. Based on Jensen's inequality theorem, the inventors derived equation 5:

[0106]

[0107] The right-hand side of Equation 5 represents the Evidence Lower Bound (ELBO) in CVAE, which can be used as the loss function for CVAE networks. ELBO is used in VAEs because it can effectively approximate complex data distributions, facilitating the learning of useful and compact latent representations.

[0108] Since the production stage y can be obtained through change point detection, assuming the production stage is known, p(y) = q(y|x) = 1, we get:

[0109]

[0110] In some implementations, based on the above derivation, the inventors obtained the following mathematical form of the CVAE loss function according to this embodiment, namely, under the prior assumption of the GMM distribution and the production stage y obtained through change point detection, the CVAE loss function L CVAE It can be represented as:

[0111]

[0112] in, To reconstruct the image, x is the original input image of the CVAE model, and σ is the standard deviation of the pixels. Let y be the potential standard deviation representing the production stage. Let μ be the latent mean of the production stage y. y Let be the mean parameter of the Gaussian component of y corresponding to the production stage in the prior distribution. In the above formula, the first term is the reconstruction loss, and the second term comes from the Kullback-Leibler (KL) loss between the predicted distribution and the assumed distribution.

[0113] In some implementations, a triplet loss is introduced to ensure that production stages are distinguishable, so that samples of the same class (i.e., the same production stage) should be grouped together, while samples of different classes (i.e., different production stages) should be far apart in terms of feature distance.

[0114] According to some implementation methods, the triplet loss (Loss) can be calculated using the following formula. trip :

[0115] Loss trip =min(|μ anchor -μ positive | 2 -|μ anchor -μ negative | 2 +m 2 ,0), (6)

[0117] Where, μ anchor μ is the latent representation mean of the anchor point image. positive μ is the latent representation mean of the positive sample images. negative is the latent representation mean of the negative sample images, and m is the margin parameter.

[0118] During training, two images belonging to the same production stage can be randomly extracted as anchor images and positive sample images, while images belonging to another production stage can be selected as negative sample images, such as... Figure 4 As shown in the above formula. It is the expected margin.

[0119] To optimize the hyperparameter m, the inventors introduce the following parameter setting guidelines to improve the statistical properties of the CVAE model proposed in this embodiment: assuming the GMM distribution covariance matrix is ​​the identity matrix and given the 3-σ rule, the margin m of the triplet loss needs to be greater than 6 to ensure that the features generated in each production stage do not overlap. The proof of these setting guidelines will not be elaborated in this disclosure.

[0120] When the CVAE encoder outputs a perfect GMM distribution, i.e. μ anchor =μ positive At that time, the triplet loss term becomes m 2 -|μ anchor -μ negative | 2 .

[0121] According to some implementation methods, the CVAE model proposed in this embodiment uses the following total loss function, Loss, during training:

[0122] Loss = λ * Loss trip +L CVAE (7)

[0123] Where λ is the adjustment parameter, and Loss trip For triplet loss, L CVAE This is the CVAE loss function.

[0124] Back Figure 2 Step 3: Perform adaptive change point detection on the latent variable sequence of each video to obtain an initial change point set.

[0125] After obtaining the low-dimensional features (i.e., latent variables) of the image, important changes related to the production progress can be identified, which can be distinguished by visual algorithms. Specifically, at these time points, the external features of the workpiece undergo significant changes.

[0126] Existing technologies have yielded numerous studies on change point detection algorithms. Change point detection can be broadly categorized into two types: detection with a known number of changes and detection with an unknown number of changes. In the former, the number of changes is known, thus the location of each change point can be determined by solving a deterministic optimization problem. In the latter, the number of changes is uncertain.

[0127] To obtain a monitoring solution that can flexibly adapt to different production environments, this embodiment adopts detection based on the number of unknown variable points. Through in-depth research, the inventors assumed that the output feature vector z of the latent variables in each production stage... i,t If each production stage follows an independent Gaussian Mixture Model (GMM) distribution, and each production stage corresponds to a single cluster in the GMM model, then change point detection can be modeled as Gaussian distribution mean change detection with constant and known variance, as shown in Equations 8 and 9:

[0128]

[0129] Where μ i,y Let y be the mean of the latent variables within the production stage y of video i. Under maximum likelihood estimation (MLE), the cost function for parameter learning includes an L2 norm loss function and a penalty term. Notably, the L2 norm loss function is the same as the cost function for change point detection using a linear kernel function. The L2 norm loss function can be used to fully utilize the consistency properties of MLE and kernel change point detection.

[0130] Based on the inventors' research above, in some implementations, the following objective function is used for change point detection:

[0131]

[0132]

[0133] Where i is the video ID, t is the time, y is the production stage ID, and c is the video ID. i The number of identified variable points, z i,t μ is the latent variable of the image of video i at time t. i,y Let T be the mean of the latent variables of video i during the production stage y. i,y and T i,y+1 For the time points of adjacent change points, pen(c) i ) is the number of variable points c i Related penalties.

[0134] According to this embodiment, the objective function automatically balances the magnitude of feature vector changes and the number of variable points by combining a squared error term and a penalty term, thus achieving automatic optimization of the number and location of variable points. The squared error term ensures that variable points are set when features change significantly, while the penalty term effectively prevents over-segmentation and overfitting by increasing the number of variable points. This means that the optimal balanced solution set contains the Pareto optimal solution for the number of variable points. This embodiment can adaptively adjust the number and location of variable points according to the characteristics of the data, without pre-specifying a fixed number of variable points or pre-setting variable point conditions, thus flexibly adapting to production processes with varying complexities.

[0135] The following is a theoretical demonstration of the performance of this embodiment, proving that the adaptive variable point detection performed according to this embodiment satisfies the asymptotic optimality theorem.

[0136] Since the production stage is automatically divided according to the design concept of this embodiment, it is necessary to ensure that the segmentation method used is optimal. "Optimality" includes the algorithm's ability to identify all real change points in the mean of image features and accurately locate these points in their actual positions. First, a standard metric d is used. max (T i ,T′ i To evaluate the segmented sequence T i and T′ i The similarity between them is defined as in Equation 11.

[0137]

[0138] This metric captures the maximum temporal deviation between two sets of change points and can be used as a boundary for detecting delay. In any given production video i, let c... i N represents the number of detected variables. i τ represents the number of feature values ​​in video i. min This represents the minimum time ratio between adjacent change points. This reflects the shortest production stage. It is a time series z i,t The number of actual change points (separable points) in the process. The upper limit c of the distinguishable production stages. max It is inferred from the complexity of the process, while δ min =min y |μ y -μ y+1 This describes the minimum mean difference between adjacent production stages. The following is an asymptotic optimality theorem to demonstrate the asymptotic optimality property of the point of change.

[0139] Asymptotic Optimality Theorem Let M iIf inequality 12 is satisfied, and a bound b∈R satisfies inequality 13, then the linear penalty function... Ensure variable points The probability is at least 1-e -a And the position of the point changes with O p (1) velocity convergence:

[0140] In this embodiment, the midpoint of the interval can be chosen as the value of b. The proof of the asymptotic optimality theorem described above will not be elaborated here. max The value can be derived from production operation data. δ min The value of m can be adjusted using the triplet loss margin. It's worth noting that the upper limit varies with N. i The growth rate increases, while the lower bound increases with log(N). i Linear growth. Therefore, for a sufficiently large N, i The upper limit always exceeds the lower limit. Furthermore, due to z... i,t Reparameterization from the CVAE model, if N i If the sample size is insufficient, it can be increased by drawing more samples from CVAE. The asymptotic optimality theorem guarantees that the number of detected production stages and the location of change points converge to the actual production stages as the sample size increases; these stages are distinguishable by a given feature extractor and dataset. The location of change points converges to the same percentage of progress as the actual image feature changes.

[0141] Step 4: Use the Iterative Dynamic Time Warping (IDTW) algorithm to process the initial set of variable points in the multiple videos to obtain a general set of variable points.

[0142] Based on the adaptive change point detection in step 3, initial change points can be extracted from each video. However, change points in different production videos may originate from anomalies such as operational errors. Therefore, it is worth considering extracting common change points that consistently appear across all production processes. Optimizing the posterior probability of common change points is a reasonable choice, since the model is based on the GMM distribution. Furthermore, common change points should be a subset of the initial change points in each training video i.

[0143] Let l i,y μ represents the number of latent variables between point y-1 and y in video i. i,y and μ′ i,y =μ y E represents the mean (i.e., feature mean) of the latent variables before and after merging different video segments. i,y It is the merged production stage index, used as a decision variable, while c i This represents the number of change points detected in video i. The general change point detection model can be represented as:

[0144]

[0145]

[0146] Equation 14 is used as the objective function of the general change point detection model to maximize the posterior probability of the refined general change points. Constraint 15 stipulates that the first and last change points of each video belong to the same stage. Constraint 16 indicates that the process between two adjacent change points can only belong to one production stage. Constraint 17 defines the mean of the merged latent variables, which is also the same as the MLE estimate.

[0147] Given the complexity introduced by constraint 17, the problem still presents significant computational challenges. To overcome this challenge, the inventors propose an Iterative Dynamic Time Registration (IDTW) algorithm to effectively address the computational cost issue.

[0148] It was observed that constraints 15 and 16 are consistent with the constraints in the DTW algorithm. To determine the optimal stopping merge change point time, the inventors derived and verified the following lemma. The proof of this lemma will not be elaborated here.

[0149] The lemma states that merging two consecutive datasets y and y+1 in video i will produce a loss, as shown in the formula:

[0150]

[0151] Among them l i,y The length T is the distance between point y and y+1 in video i. i,y+1 -T i,y .

[0152] The above lemma shows that if the current change point already satisfies these constraints, further merging of change points is unnecessary, as this would adversely affect the results. Based on this inspiration, the inventors developed an Iterative Dynamic Time Registration (IDTW) algorithm to identify general change points, such as... Figure 5 As shown.

[0153] In some implementations, step 4 may include:

[0154] Pair the videos together;

[0155] You can follow as follows Figure 6 The described algorithm iteratively matches the variable points in each video pair. This iterative matching includes: processing the variable point sequences of the two videos in the video pair using a dynamic time warping algorithm to obtain the corresponding DTW pair sequence, where the j-th DTW pair is represented as... i1 and i2 are the video numbers of the two videos in the video pair, respectively; perform a forward traversal of the DTW pair sequence of the video pair, and when the DTW pair satisfies the judgment condition... and When a DTW pair is identified as a common variable point for the video pair, it is retained. However, when a DTW pair does not meet the aforementioned criteria, it is deleted from the DTW pair sequence, and the time periods before and after the DTW pair are merged (refer to...). Figure 6 (Algorithm line 10), and recalculate the mean of the latent variables within the merged segment, and recalculate the DTW pair sequence of the video pair; repeat the above process until a one-to-one correspondence is established between the change points of the two videos in the video pair;

[0156] After repeating the iterative matching for all video pairs, the remaining DTW pairs are used as the final universal variable points, forming the universal variable point set, as follows: Figure 5 As shown.

[0157] After identifying the common variable points in all videos, the precise time T′ of each variable point in each video i can be determined by backtracking. i,y Then, the mean of the corresponding change points is calculated according to Equation 17 as a component of the GMM, and the CVAE model is updated. During training, the model can determine F(k) = F(f(t,i)) = argmax through y. y (T′ i,y ≤t).

[0158] Back Figure 2 Step 5: Update the CVAE model according to the general variable point set until the CVAE model converges.

[0159] At the end of each training cycle, it can be determined whether the CVAE model has converged. If the model has not converged, the CVAE model is updated based on the set of general change points obtained in this training cycle, and the process returns to step 2 to start the next training cycle; if the model has converged, the training process ends and the trained CVAE model is obtained.

[0160] The CVAE model can be trained offline. In some examples, learning can begin from class zero. Given the diversity of videos, feature distributions across different videos can be aligned. To this end, additional maximum mean difference (MMD) loss can be considered for the initial 20 training epochs; for example, the total loss function used for training can combine the CVAE loss function and the MMD loss function, similar to an augmented VAE. After initial training, adaptive change point detection and IDTW are performed as described above, thereby updating the class (i.e., production stage) information and the corresponding CVAE class centers (i.e., the mean of all latent variables within the production stage, also known as feature means). In subsequent iterations, as the model covers multiple classes, triplet losses can be integrated, i.e., using the total loss function Loss = λ * Loss. trip +L CVAE Conduct training.

[0161] In each training cycle, the number of classes and the centers of each class can be dynamically updated through adaptive variable point detection and IDTW.

[0162] In some implementations, updating the CVAE model based on the general variable point set includes:

[0163] Determine the time point of each common variable in each video;

[0164] Based on the time periods divided by the general change point, calculate the mean of the latent variables in each time period. The mean of the latent variables in each time period can be calculated according to Equation 17 above.

[0165] The mean of the calculated latent variables is used as the mean parameter of the corresponding Gaussian component in the Gaussian mixture model (GMM);

[0166] Update the CVAE model using the obtained universal change point time points and GMM mean parameters.

[0167] When the loss function value decreases below a certain threshold or no longer decreases significantly, or when the model's performance on the validation set reaches the expected standard, the CVAE model can be judged to have converged. For example, when the stability of the total loss function reaches the expected level, the model's accuracy in predicting production stages on the validation set reaches the expected level, the area under the receiver operating characteristic (AUC) of the model's progress deviations in different production environments reaches the expected level, and / or the mean absolute percentage error (MAPE) reaches the expected level, the CVAE model can be judged to have converged based on one of the above or by considering multiple factors.

[0168] After model convergence, CVAE can extract a multivariate Gaussian mixture distribution from production site images, capturing both temporal and visual data. The mean μ of the latent variables... yThese progress points are saved for future predictions. To capture the stage division points learned by the model, predictions can be made on the validation set, and the transition points during this prediction process can be used as the final division results. When testing the model, these progress nodes (i.e., transition points) can be used as the standard for labeling the test set. Furthermore, if it is necessary to identify any specific point in the process, simply add that point to the results of IDTW for semi-supervised training.

[0169] Furthermore, according to this embodiment, the time cost l of each process stage y in each video i can be provided. i,y By calculating the average value It can automatically determine the time standards for each production stage. This information is of great importance for developing schedules and detecting schedule anomalies.

[0170] Step 6: Use the converged CVAE model to process the image sequence of the target production process acquired in real time to predict the production stage.

[0171] In real-time prediction, the real-time image is processed using a CVAE model to estimate the distribution, and then the latent representation mean of the image predicted by the model is compared with the feature mean μ corresponding to each production stage. y The process involves matching and identifying the production stage corresponding to the closest mean feature as the predicted production stage. The schedule prediction results can then be further used for real-time scheduling, material flow control, and performance management.

[0172] Step 7: Check the progress deviation based on the predicted production stage.

[0173] In some implementations, a schedule deviation is determined to exist and a corresponding alarm message is generated when the following schedule detection conditions are not met:

[0174] Do the predicted production stages at adjacent time points t and t+1 meet the requirements? or in It refers to the production stage at the predicted time point t. It refers to the production stage at the predicted time point t+1, and / or the predicted production stage differs from the planned production stage.

[0175] Managers who receive alarm information (such as operations managers) can check it in a timely manner.

[0176] Figure 7 An exemplary framework for offline training, real-time prediction, and progress detection according to an exemplary embodiment of this disclosure is shown.

[0177] The unsupervised process monitoring method proposed in this embodiment brings significant improvements to repetitive production systems. This embodiment innovatively combines a contrastive variational autoencoder (CVAE), adaptive change point detection, and iterative dynamic time warping (IDTW) algorithm to achieve automated, real-time monitoring and precise analysis of the production process. Its unsupervised learning characteristics greatly reduce reliance on manually labeled data, lower implementation costs, and improve the method's adaptability and scalability. The adaptive change point detection technology can accurately identify key stage transition points in the production process, while the IDTW algorithm ensures the identification of highly consistent common change points across multiple production cycles. The end-to-end unsupervised process monitoring method proposed in this embodiment can not only adapt to subtle differences between different production batches but also provides more robust and accurate progress monitoring, enabling production managers to respond quickly to anomalies and significantly reducing the risk of production delays and quality problems. Furthermore, the generalization capability of this embodiment allows it to adapt to different types of manufacturing processes, providing strong support for the intelligent transformation of the manufacturing industry.

[0178] The following evaluation uses four different datasets to assess the effectiveness of the proposed unsupervised process monitoring scheme for repetitive production systems in this embodiment. The first dataset focuses on a water valve production process, where assembly parts are the primary focus of the image data. The second dataset targets HA4M, where parts occupy a minimal portion of the images and are often difficult to identify, primarily showcasing operator actions. The third dataset represents a real-world production scenario for commercial aircraft spoilers, including assembly operations as well as tasks such as lamination, coating, and vacuum extraction, involving mechanical parts and flexible composite materials. The final dataset is a general motion recognition dataset. The diversity of these four datasets aims to test the robustness and versatility of the proposed end-to-end unsupervised process monitoring scheme in different environments.

[0179] (I) Water Valve Production

[0180] A dataset containing 8 videos was used as the training set, 1 video as the validation set, and 25 videos as the test set. Each video captured the assembly process of one water valve, with approximately 2000 timeframes of the production process performed on a single workstation. The production process involves workers inserting rubber plugs into the middle valve body and securing them with screws. Next, workers remove the lower part of the valve, place a gasket on top, and then assemble the middle and upper valve bodies and related parts. Next comes the installation of the rocker arm, installing screws and a plastic cap. Finally, the upper and lower clamps are assembled.

[0181] The CVAE model presented in this disclosure successfully identifies eight distinct production stages. These stages are... Figure 8The visualization shows the production process from the intermediate valve body to the final clamp. The identified steps include: 1) intermediate valve body production; 2) fixing screws; 3) assembling the lower valve; 4) assembling the upper valve; 5) assembling the rocker arm; 6) assembling parts on the rocker arm; 7) completing the production of the lower valve clamp; and 8) assembling the upper clamp. These production stage divisions are logical because each stage corresponds to the beginning or end of each observed part addition during the production cycle.

[0182] Visualization of features extracted from water valve production, such as Figure 9 As shown, the horizontal and vertical axes represent the first and second dimensions of t-SNE, respectively. Blue represents training set features, red represents test set features, and green represents validation set features. The gradient from light to dark corresponds to the temporal proportions of images in the video sequence, with darker colors representing later stages of the production process. Figure 9 As can be seen, the eight time-related clusters are clearly separated.

[0183] (II) HA4M Dataset

[0184] The HA4M dataset contains 217 videos demonstrating a repetitive manual production process. There are 41 subjects in the dataset performing a single production task. Test analysis focuses on 106 videos from a consistent environment, visualizing eight progress states as follows: Figure 10 As shown. The remaining videos demonstrate the same production process, but in an informal setting, they are considered outside the scope of the test parameters. Twelve videos were used as the training set, one as the validation set, and 82 as the test set. The remaining eleven videos assembled different procedures and were used solely for schedule deviation detection and evaluation purposes. The assembly process involves placing several parts onto a fixture, installing three gears and their outer rails, and securing them. The workpiece is then moved to another fixture to install parts on the other side.

[0185] This embodiment successfully identified eight states in the production process, such as... Figure 10 As shown. These production stage divisions are logical because each stage corresponds to the start of part assembly or the completion of an operation. Furthermore, a visualization of features extracted from the HA4M dataset is shown below. Figure 11 As shown, using and Figure 9 The same explanation.

[0186] (III) Actual Production of Commercial Aircraft Spoilers

[0187] The purpose of this study is to evaluate the performance of the unsupervised process monitoring method proposed in this embodiment in the actual production of spoilers for commercial aircraft. The spoiler production workstation was chosen because it involves not only mechanical parts but also flexible materials. During production, operations are performed on or beside the tool. The production process includes workers performing some pre-assembly work beside the tool (checkpoint 1), waiting for the primer to dry and checking auxiliary materials (checkpoint 2), placing a layer of yellow composite material at a designated location on the tool, and then applying composite films of various colors (checkpoint 3). Then, the assembled rigid structure is inserted into a grinder, the mold is assembled, and covered with a red transparent plastic film (checkpoint 4). Excess parts of the mold are trimmed, then covered with a white cloth and trimmed to the shape of the mold (checkpoint 5). A layer of transparent green plastic film is covered and vacuum-treated (checkpoint 6). After completion, it is transferred to an autoclave for curing. This study focuses on the spoiler production stage performed on the tool before curing, typically spanning approximately 2000-3000 time frames. Since the tool needs to be pushed into the autoclave after completion and brought back in the next cycle, the location of the tool is not fixed each time. Using surveillance cameras in the workshop to collect video footage will not affect production or increase hardware costs. This process is a hybrid of manual and human-machine integrated manufacturing. Due to the simultaneous operation of multiple workers and the frequent presence of irrelevant personnel and actions, it is difficult to segment based on part retrieval or worker movements. Furthermore, given the long time span, manually identifying separable points using visual algorithms is even more challenging. There is often a delay in recording information for the few checkpoints a worker needs to record, and they may accumulate information for multiple checkpoints or simply forget to record them altogether.

[0188] A dataset of approximately 46,000 images was created from three weeks of factory surveillance video. Six units were divided into a training set, one unit into a validation set, and six units into a test set. The training set contained videos from the production of three different types of spoilers, while the test and validation sets included spoilers from the training set and one untrained spoiler type. The variations in spoilers were reflected in their shape (from rectangular to trapezoidal) and size, as well as different mounting structures designed for wing attachment. Despite these variations, the production process remained consistent across the different spoiler types, consistent with the assumptions of this disclosure regarding repetitive production scenarios. The dataset is based on factory operating time, excluding production data from periods of factory downtime. The dataset includes instances where tools were temporarily removed from the camera's field of view halfway through processing, workers assembled parts and then disassembled and reassembled them, workshop lights were temporarily turned off, or construction vehicles entered and obstructed the camera's view.

[0189] In the manufacturing process studied, the CVAE model proposed in this embodiment can accurately and automatically divide these production processes into up to seven production stages. These production stages include all checkpoints involving visual changes in the process documentation, with each checkpoint further subdivided into two parts. Figure 12 The progress classification results for the validation and test sets are shown, with the x-axis representing the time progression of the production video and the y-axis representing the predicted production stage. Visualization of features extracted from aircraft spoiler production is also shown. Figure 13 As shown, using and Figure 9 The same explanation.

[0190] To evaluate schedule deviation detection, the test dataset was annotated using the collective expertise of multiple plant personnel and management. A total of 16 different schedule deviation scenarios and 1892 normal scenarios were annotated. Deviations included situations such as moving tools and equipment ahead of schedule and reoperation.

[0191] (iv) Clothing Folding Dataset

[0192] This dataset contains 970 videos demonstrating the process of folding clothing, identifying six unique short-sleeved shirts and various folding methods. The analysis in this study focuses on one primary technique: folding the clothing after it is placed vertically, representing 270 videos in the dataset. To maintain consistency, instances of folding two garments simultaneously or refolding were excluded. Therefore, the analyzed dataset was reduced to 246 folding events, divided into 12 videos for the training set, 1 video for the validation set, and 233 videos for the test set.

[0193] Since there are multiple garments in the scene, background subtraction can be used for data preprocessing. The absolute difference from the first frame is taken, and pixels with differences exceeding a certain threshold are retained. The CVAE model proposed in this embodiment consistently identifies four action classes: laying the garment on the table, folding horizontally, folding vertically, and placing it aside, achieving an accuracy of 84.81%. The accuracy is lower than the aforementioned production progress recognition. The main reason for this is the non-standard folding operation and the fact that the image features of the garment placement and unfolding states are almost identical in some videos. These are the four common steps observed in all training videos. The detected garment folding stages are as follows: Figure 14 As shown. The visualization of the extracted features is as follows. Figure 15 As shown, the four clusters are clearly distinguished and used with Figure 9 The same explanation.

[0194] To illustrate the superiority of this embodiment, the unsupervised process monitoring method proposed in this embodiment is compared with monitoring methods based on existing benchmark models. Specifically, the CVAE model proposed in this embodiment is compared with the Time Constraint Network (TCN) and two variant models: TCN-C (similar to the enhanced TCN model) without VAE components and VaDE-C (similar to the enhanced VaDE model) without triple components. This comparison comprehensively demonstrates the advantages of the CVAE model according to this embodiment.

[0195] Three metrics are used to quantitatively compare schedule prediction performance. The first is schedule prediction accuracy; the second is the number of production stages, reflecting the precision of a given production stage division; and the third is the mean absolute percentage error (MAPE) of time calculated according to Equation 19, to take into account the variability in the number of process stages that different models may identify.

[0196]

[0197] in Indicates the predicted progress status in video i The change point time estimate, T i,-1 =0, reflecting the conceptualization of schedule forecasting as a continuous task; after forecasting the production stage of the current schedule state, the midpoint of the production state is determined. The predicted values ​​are compared with the actual occurrence times to comprehensively evaluate the accuracy of the process stage division and the accuracy of the prediction.

[0198] In the field of schedule deviation detection, the problem is framed as a binary classification task—identifying deviations or normal instances. Its effectiveness can be evaluated using two widely accepted metrics: the F1 score and the area under the receiver operating characteristic (AUC).

[0199] Table 1 Quantitative Results of Schedule Forecasting

[0200]

[0201] Table 2 Quantitative results of schedule deviation detection

[0202]

[0203] Table 1 compares the performance of the proposed method in this embodiment with that of three benchmark models on four different datasets in terms of schedule class (i.e., the number of production stages), classification accuracy, and schedule MAPE.

[0204] As can be seen from the quantitative performance analysis results presented in Table 1, the method proposed in this embodiment significantly outperforms the baseline model in terms of progress status identification accuracy across all datasets, achieving higher progress prediction accuracy and lower MAPE, even when identifying more progress states. It is worth noting that the TCN model shows significantly lower accuracy and MAPE in Table 1, primarily because its design is suited for multi-view camera data, while this study collected single-view camera data. In the three production cases (water valve production, HA4M, and aircraft spoiler production), the average accuracy of production progress prediction reached 98.14%.

[0205] As can be seen from the quantitative performance analysis results presented in Table 2, the method proposed in this embodiment also shows higher performance in terms of AUC and F1 score in schedule deviation detection.

[0206] Analysis suggests that the main reasons for the superior performance of this embodiment are as follows: First, the adaptive variable point detection, IDTW, and CVAE adopted in this embodiment not only capture time information and locate the precise process stage cutting points, but also enable the latent variables to effectively extract pixel-level information. This key information enables the unsupervised embedding method according to this embodiment to learn distinguishable points and build a classifier in an end-to-end manner. In addition, the implementation of triplet loss further ensures that each progress class has a unique feature distribution.

[0207] The architecture and monitoring mechanism of the CVAE model proposed in this embodiment are based on a theoretically supported loss function and an adaptive change point detection strategy that conforms to the statistical characteristics of a specific dataset, eliminating the need for parameter adjustment across datasets. Furthermore, research has confirmed that this CVAE model can accurately distinguish process states. Moreover, the proposed IDTW algorithm can establish a general state in the training video, effectively filtering out spurious changes.

[0208] This disclosure also proposes an unsupervised process monitoring device for repetitive production systems, comprising an image acquisition unit, a feature extraction unit, an adaptive change point detection unit, a general change point screening unit, a training cycle control unit, a real-time prediction unit, and a schedule deviation detection unit.

[0209] The image acquisition unit is used to acquire multiple videos of the target production process and obtain an image sequence for each video.

[0210] The feature extraction unit is used to extract features from the image sequence using a contrastive variational autoencoder (CVAE) model to obtain a sequence of latent variables.

[0211] The adaptive change point detection unit is used to perform adaptive change point detection on the latent variable sequence of each video to obtain an initial change point set.

[0212] The general variable point filtering unit is used to process the initial variable point set of the multiple videos using the Iterative Dynamic Time Warping (IDTW) algorithm to obtain a general variable point set.

[0213] The training cycle control unit is used to update the CVAE model according to the general variable point set until the CVAE model converges.

[0214] The real-time prediction unit is used to process the image sequence of the target production process acquired in real time using a converged CVAE model to predict the production stage.

[0215] The schedule deviation detection unit is used to detect schedule deviations based on the predicted production stage.

[0216] For other details and beneficial effects of this embodiment, please refer to the relevant description above, which will not be repeated here.

[0217] Figure 16 An electronic device provided in at least one embodiment of the present disclosure includes a memory and a processor. The memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement the unsupervised process monitoring method for repetitive production systems as described in any embodiment or implementation of the present disclosure when executing the computer instructions.

[0218] At least one embodiment of this disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the unsupervised process monitoring method for repetitive production systems as described in any embodiment or implementation of this disclosure.

[0219] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0220] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the data processing device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0221] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0222] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0223] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0224] Suitable computers for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0225] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0226] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0227] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0228] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0229] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. An unsupervised process monitoring method for repetitive production systems, characterized in that, include: Multiple videos of the target production process are acquired, and an image sequence of each video is obtained; The image sequence is used to extract features to obtain a sequence of latent variables. Adaptive change point detection is performed on the latent variable sequence of each video to obtain an initial change point set; The initial set of variable points in the multiple videos is processed using the Iterative Dynamic Time Warping (IDTW) algorithm to obtain a general set of variable points; Update the CVAE model according to the general set of variable points until the CVAE model converges; A convergent CVAE model is used to process image sequences of the target production process acquired in real time to predict the production stage; Based on the predicted deviation in the production phase inspection schedule; The CVAE model uses the following total loss function during training. Loss : ， in, To adjust the parameters, For triple loss, The CVAE loss function; Based on the prior distribution assumption that the latent variables follow a Gaussian mixture model (GMM) distribution, the CVAE loss function is calculated according to the following formula. Each production stage corresponds to a Gaussian component in the GMM: ， in, To reconstruct the image, The original input image for the CVAE model. The standard deviation of pixels, For the production stage The underlying standard deviation is below. For the production stage The underlying mean of the representation, For the corresponding production stage in the prior distribution The mean parameter of the Gaussian components; The triplet loss is calculated according to the following formula. : , in, The latent representation mean of the anchor point image. The latent representation mean of the positive sample images. The latent representation mean of the negative sample images. m This refers to the margin parameter.

2. The method according to claim 1, characterized in that, The CVAE model is set as follows: The ResNet50 architecture is used as the encoder to generate latent variables; The image is reconstructed using a convolutional decoder.

3. The method according to claim 1, characterized in that, The adaptive change point detection includes: Change point detection is performed using the following objective function: ， in, For video numbering, For time, This is a number indicating the production stage. The number of identified variable points For video In time The latent variables of the image, For video During the production stage The mean of the latent variables within, and For adjacent change points, To the number of variable points Related penalties.

4. The method according to claim 1, characterized in that, The initial set of variable points in the multiple videos is processed using the IDTW algorithm to obtain a universal set of variable points, including: Pair the videos together; For each video pair, iterative matching is performed on the variable points. This iterative matching includes: processing the variable point sequences of the two videos in the video pair using a dynamic time warping algorithm to obtain a DTW pair sequence. j Each DTW pair is represented as , and These are the video numbers of the two videos in the video pair; perform a forward traversal of the DTW pair sequence of the video pair, and when the DTW pair satisfies the judgment condition... and When the DTW pair is identified as a common variable point of the video pair, the DTW pair is retained. When the DTW pair does not meet the judgment condition, the DTW pair is deleted from the DTW pair sequence, the time periods before and after the DTW pair are merged, the mean of the latent variables in the merged segment is recalculated, and the DTW pair sequence of the video pair is recalculated. The above process is repeated until a one-to-one correspondence is established between the variable points of the two videos in the video pair. After repeating the iterative matching for all video pairs, the remaining DTW pairs are used as the final universal variable points to form the universal variable point set.

5. The method according to claim 1, characterized in that, Updating the CVAE model based on the general set of change points includes: Determine the time point of each common variable in each video; Calculate the mean of the latent variables within each time period based on the time period divided by the common change point; The mean of the calculated latent variables is used as the mean parameter of the corresponding Gaussian component in the Gaussian mixture model (GMM); Update the CVAE model using the obtained universal change point time points and GMM mean parameters.

6. The method according to claim 1, characterized in that, Based on the predicted deviations in the current production phase inspection progress, including: If the following progress detection conditions are not met, a progress deviation is determined to exist, and a corresponding alarm message is generated: Predicted adjacent time points and Does the production stage meet the requirements? or ,in It is the predicted time point. The production stage, It is the predicted time point. The production stage, and / or the predicted production stage differs from the planned production stage.

7. An unsupervised process monitoring device for repetitive production systems, characterized in that, include: An image acquisition unit is used to acquire multiple videos of the target production process and obtain an image sequence for each video; The feature extraction unit is used to extract features from the image sequence using a contrastive variational autoencoder (CVAE) model to obtain a sequence of latent variables; An adaptive change point detection unit is used to perform adaptive change point detection on the latent variable sequence of each video to obtain an initial change point set; A general variable point filtering unit is used to process the initial variable point set of the multiple videos using the Iterative Dynamic Time Warping (IDTW) algorithm to obtain a general variable point set; The training cycle control unit is used to update the CVAE model according to the general variable point set until the CVAE model converges. The real-time prediction unit is used to process the image sequence of the target production process acquired in real time using a converged CVAE model to predict the production stage. The schedule deviation detection unit is used to detect schedule deviations based on the predicted production stage. The CVAE model uses the following total loss function during training. Loss : ， in, To adjust the parameters, For triple loss, The CVAE loss function; Based on the prior distribution assumption that the latent variables follow a Gaussian mixture model (GMM) distribution, the CVAE loss function is calculated according to the following formula. Each production stage corresponds to a Gaussian component in the GMM: ， in, To reconstruct the image, The original input image for the CVAE model. The standard deviation of pixels, For the production stage The underlying standard deviation is below. For the production stage The underlying mean of the representation, For the corresponding production stage in the prior distribution The mean parameter of the Gaussian components; The triplet loss is calculated according to the following formula. : , in, The latent representation mean of the anchor point image. The latent representation mean of the positive sample images. The latent representation mean of the negative sample images. m This refers to the margin parameter.

8. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store computer instructions executable on the processor, and the processor being used to implement the method of any one of claims 1 to 6 when executing the computer instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method described in any one of claims 1 to 6.

Images