Video-based self-supervised production anomaly detection and progress prediction

Abstract

Disclosed are a video-based self-supervised production anomaly detection and progress prediction method and apparatus, and a device. The method comprises: using an encoder to map image features of a production process video to a latent space of a Gaussian mixture model; generating a spatial activation map, so as to extract production-related attention features; using a decoder to reconstruct an image and compute a spatial activation reconstruction error; training a model on the basis of a comprehensive loss function; and using the trained model to perform anomaly detection and production stage prediction.

Description

Video-based self-supervised production anomaly detection and progress prediction TECHNICAL FIELD

[0001] The present disclosure relates to the field of advanced manufacturing, and in particular, to video-based self-supervised production anomaly detection and progress prediction. BACKGROUND

[0002] With the rapid development of Industry 4.0 and smart manufacturing, real-time production monitoring plays an increasingly important role in modern production management. Efficient and accurate anomaly detection and progress prediction are crucial for improving production quality, optimizing production efficiency, and preventing safety accidents. However, in complex manufacturing environments, especially in scenarios involving human-machine collaboration or high-precision manual assembly, traditional monitoring methods face many challenges. SUMMARY

[0003] The present disclosure proposes a video-based self-supervised production anomaly detection and progress prediction method. The present disclosure also proposes corresponding devices, electronic equipment, and computer-readable storage media.

[0004] According to one embodiment of the present disclosure, a video-based self-supervised production anomaly detection and progress prediction method is proposed, which comprises: acquiring video data of a production process, the video data comprising multiple frames of original images; using an encoder to map image features in the video data to a latent space of a Gaussian Mixture Model (GMM), to obtain latent variables for representing latent features in the original images, the Gaussian components in the GMM corresponding one-to-one to pre-divided production stages; using a feature map generated before the last convolution block of the encoder to generate a spatial activation map, the spatial activation map being used to extract attention features related to different production stages from the original images; using a decoder to reconstruct the latent variables output by the encoder into reconstructed images, and using the spatial activation map to weight reconstruction errors of the reconstructed images compared to corresponding original images, to obtain corresponding spatial activation reconstruction errors; performing model training based on a comprehensive loss function, wherein the comprehensive loss function comprises a cross-entropy loss of production progress prediction, a triple loss of feature embedding of the latent space for feature representation of different production stages, a negative log-likelihood loss of the latent variables in the GMM, and the spatial activation reconstruction errors; using the encoder of the trained model to process real-time collected video data, to perform production anomaly detection and predict production stages based on the obtained latent variables.

[0005] In some embodiments, the mean μ y and the covariance Σ y of each Gaussian component in the GMM are associated with the continuous production progress k by the following piecewise functions F μ (k) and F Σ (k): F μ (k) = μy if k y <k≤k y+1 ,y∈{0,…,c-1}, F Σ (k)=Σ y if k y <k≤k y+1 y∈{0,…,c-1}, where k∈[0,1] represents the production progress in percentage form, c is the number of pre-divided production stages, and k y Let μ be the starting point of the production schedule for the y-th production stage. y Let Σ be the mean of the latent variables in the y-th production stage. y Let be the covariance of the latent variables in the y-th production stage; after each training cycle, based on the maximum likelihood estimate of the latent variable samples output by the current encoder, update the mean μ of each Gaussian component in the GMM. y Covariance Σ y .

[0006] In some implementations, generating a spatial activation map using a feature map generated before the last convolutional block of the encoder includes: processing the feature map Q generated before the last convolutional block of the encoder using a multilayer perceptron (MLP) to obtain a feature map Q′; calculating a progress activation map on the feature map Q′ using a weight matrix f of a fully connected network without bias, where each column of f corresponds to a production stage; and averaging the progress activation map across the production stages to generate the spatial activation map.

[0007] In some implementations, calculating the progress activation graph includes: calculating the progress activation graph according to the following formula: Among them, PAM y,i,j Let represent the progress score at spatial location (i,j) in the progress activation graph, and y represent the y-th production stage. Let Q' be the transpose of the y-th column of the weight matrix f, corresponding to the y-th production stage. :,i,j Let Q' represent the feature vector at position (i,j). express and Q′ :,i,j Take the absolute value after dot product.

[0008] In some implementations, the spatial activation map is used to weight the reconstruction error of the reconstructed image compared to the corresponding original image to obtain the corresponding spatial activation reconstruction error, including: calculating the spatial activation reconstruction error SARE according to the following formula: SARE = ​​∑ c ∑ i ∑ j (X′ c,i,j -X c,i,j ) 2SAM i,j , where X' denotes the reconstructed image, X denotes the original image, c, i, j denote the channel, height, width index of the reconstructed image and the original image respectively, (X' c,i,j -X c,i,j ) 2 denotes the square of the pixel-level reconstruction error of the reconstructed image and the original image at position (c, i, j), SAM i,j denotes the value of the spatial activation map at position (i, j).

[0009] In some embodiments, the comprehensive loss function is calculated according to the following formula: Loss = Loss CE + λ1Loss trip + λ2NLL + λ3SARE, where Loss CE is the cross-entropy loss of the production progress prediction, used to optimize the progress prediction task; Loss trip is the triple loss of the feature embedding of the latent space, used to optimize the identification of different production stages by the model; NLL is the negative log-likelihood loss of the latent variable in the GMM, used to optimize the feature distribution of the latent space; SARE is the spatial activation reconstruction error, used to guide the identification of the image area related to production in the model training process; λ1, λ2, λ3 are weight coefficients used to balance different loss terms.

[0010] In some embodiments, the triple loss Loss trip is calculated according to the following formula: Loss trip = max(|z anchor -z positive | 2 -|z anchor -z negative | 2 +m 2 , 0), where z anchor denotes the latent variable of the current image sample, z positive is the latent variable of the image sample belonging to the same production stage as z anchor , z negative is the latent variable of the image sample belonging to a different production stage from z anchor , and m is a predefined margin.

[0011] In some embodiments, the method further comprises: randomly selecting an input image with a predetermined probability; randomly selecting a local area on the selected image; setting the pixel values in the local area to a predetermined value to form a partially erased image; using the partially erased image for model training.

[0012] In some embodiments, the local area is a rectangular area, and the position (r xr y ) and dimension (r w , h ) satisfy: r x ~ U(0, w), r y ~ U(0, h), a ~ U(a l , a u ), b ~ U(b l , b u ), wherein the dimension of the input image is w x h, U() represents a uniform distribution, a l , a u , b l , b u are predefined parameters.

[0013] In some embodiments, the production anomaly detection comprises: mapping an image in the real-time collected video data to the latent space of the GMM by using the encoder of the trained model to obtain a latent variable z; calculating the probability density p(z) of the latent variable z in the Gaussian mixture model based on the following formula: wherein c is the number of production stages, a y is the weight of the yth production stage, is a multivariate Gaussian distribution probability density function with mean μ y and covariance matrix Σ y ; taking p(z) as an anomaly score index, and determining that the corresponding image is an abnormal image when p(z) is lower than a preset anomaly detection threshold.

[0014] In some embodiments, the predicted production stage comprises: mapping an image in the real-time collected video data to the latent space of the GMM by using the encoder of the trained model to obtain a latent variable z; predicting the current production stage y based on the latent variable z. calculating a reliability index of the predicted production stage y When y is less than a preset reliability detection threshold, determining that the predicted production stage y is unreliable, taking the prediction result y t-1 of the last moment as the current prediction result, i.e., y t = y t-1 ; when y is greater than the preset reliability detection threshold, taking the current predicted production stage y as the current prediction result, i.e.

[0015] According to one embodiment of the present disclosure, a video-based self-supervised production anomaly detection and progress prediction apparatus is provided, the apparatus comprising: a video acquisition unit configured to acquire video data of a production process, the video data comprising a plurality of original images; a latent feature extraction unit configured to map image features in the video data to a latent space of a Gaussian Mixture Model (GMM) using an encoder, to obtain latent variables representing latent features in the original images, the Gaussian components in the GMM corresponding one-to-one to pre-divided production stages; a SAM generation unit configured to generate a spatial activation map using feature maps generated before a last convolution block of the encoder, the spatial activation map being used to extract attention features related to different production stages from the original images; an image reconstruction unit configured to reconstruct the latent variables output by the encoder into reconstructed images using a decoder, and to weight reconstruction errors of the reconstructed images compared to the corresponding original images using the spatial activation map, to obtain corresponding spatial activation reconstruction errors; model training based on a comprehensive loss function, the comprehensive loss function comprising a cross-entropy loss of production progress prediction, a triple loss of feature embedding of latent spaces representing features of different production stages, a negative log-likelihood loss of latent variables in the GMM, and the spatial activation reconstruction errors; an anomaly detection and progress prediction unit configured to process real-time acquired video data using the encoder of the trained model, to perform production anomaly detection and predict production stages based on obtained latent variables.

[0016] According to one embodiment of the present disclosure, an electronic device is provided, the device comprising a memory configured to store computer instructions executable on a processor, and the processor configured to implement the method of any one of the above when executing the computer instructions.

[0017] According to one embodiment of the present disclosure, a computer-readable storage medium is provided, having stored thereon a computer program, the program being executed by a processor to implement the method of any one of the above.

[0018] The video-based self-supervised production anomaly detection and progress prediction scheme proposed by the present disclosure provides an innovative solution for production monitoring in complex manufacturing environments. The present disclosure uses autoencoder process probability embedding (APPE) technology to map the image distribution of normal production to a Gaussian mixture model (GMM) related to progress, enabling the model to effectively capture the spatio-temporal heterogeneity in the production process and more accurately identify production-related features. By introducing spatial activation maps (SAMs), the present disclosure minimizes the impact of background interference, enabling the model to focus more on image areas directly related to production. The self-supervised learning feature of the present disclosure enables training using only normal production data, greatly addressing the problem of scarce abnormal data and enhancing the model's ability to anticipate unknown abnormalities. In addition, by integrating anomaly detection into the prediction model, the present disclosure significantly improves the accuracy and reliability of overall monitoring.

[0019] The scheme proposed by the present disclosure also improves the accuracy of anomaly detection and the accuracy of progress prediction by considering the characteristics of different production stages, making the model better adapt to complex production environments and diverse abnormal situations.

[0020] The scheme proposed by the present disclosure has shown obvious advantages in practical application verification. Through verification on real data sets such as water valve assembly and commercial aircraft spoiler production, the scheme of the present disclosure has achieved significant improvement in the detection of various types of anomalies, especially in identifying foreign object anomalies, worker operation anomalies, part interaction anomalies, and progress-related anomalies. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present specification and serve to explain the principles of the present specification.

[0022] FIGS. 1a and 1b show a schematic diagram of a certain production scenario.

[0023] FIG. 2 shows a schematic diagram of comparative analysis of computer vision and industrial production anomalies.

[0024] FIG. 3 shows a flowchart of a video-based self-supervised production anomaly detection and progress prediction method according to an embodiment of the present disclosure.

[0025] FIG. 4 shows a schematic diagram of spatial heterogeneity of image features according to an exemplary embodiment of the present disclosure.

[0026] FIG. 5 shows a visualization process schematic diagram of generating SAM according to an exemplary embodiment of the present disclosure.

[0027] FIG. 6 illustrates a structural diagram of a model, according to an example embodiment of the present disclosure.

[0028] FIG. 7 illustrates a flow diagram of combining anomaly detection and progress prediction, according to an example embodiment of the present disclosure.

[0029] FIGS. 8a-8i illustrate visualization diagrams of production stages of water valve production, according to an example embodiment of the present disclosure.

[0030] FIGS. 9a-9e illustrate diagrams of an abnormal assembly process in water valve production, according to an example embodiment of the present disclosure.

[0031] FIG. 10 illustrates a feature visualization diagram of latent variables output by a model in water valve production, according to an example embodiment of the present disclosure.

[0032] FIG. 11 illustrates a visualization diagram of generating a SAM and a reconstructed image X' in water valve production, according to an example embodiment of the present disclosure.

[0033] FIG. 12 illustrates a finished aircraft spoiler, according to an example embodiment of the present disclosure.

[0034] FIG. 13 illustrates a feature visualization diagram of latent variables output by a model in aircraft spoiler production, according to an example embodiment of the present disclosure.

[0035] FIG. 14 is a structural diagram of an electronic device, according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The example embodiments will be described in detail herein with reference to the attached drawings. The description of the example embodiments is intended to apply to various embodiments and adaptations of the example embodiments. For example, although the example embodiments are described in the context of a water valve production, the example embodiments can be applied to other production scenarios, such as aircraft spoiler production. The example embodiments are not intended to be limited to the embodiments described herein, but are to be accorded the full scope that the appended claims are entitled to.

[0037] Currently, the monitoring methods based on sensor data in automated production lines have been greatly improved compared to traditional techniques, but it is still difficult to meet the needs in industrial production scenarios involving continuous manual operations. The main reason is the lack of targeted sensors, making it difficult to achieve real-time progress tracking and process monitoring. With high-resolution video monitoring systems becoming more popular and affordable, production monitoring methods based on video data have gradually attracted attention.

[0038] However, some vision-based production monitoring methods can have several limitations. First, these methods can struggle to effectively distinguish production-related features from background noise. In real production environments, background variations can be much larger than subtle changes related to the production process, making it difficult to accurately identify abnormal situations. Second, these methods do not adequately consider the spatiotemporal heterogeneity of production processes. The appearance of workpieces can undergo significant dynamic changes throughout the production process, which makes anomaly detection extremely complex.

[0039] Another challenge lies in the lack of sufficient abnormal data for model training. Since production lines operate under normal conditions most of the time, abnormal data is scarce, which poses a significant challenge for model development and parameter learning. Moreover, some vision-based production monitoring methods treat production anomaly detection and progress prediction as independent tasks, ignoring the potential connections between the two.

[0040] Some vision-based production monitoring methods also commonly rely on pre-defined abnormal classes or rules. However, in a dynamically changing production environment, it is impossible to pre-define and label all possible abnormal situations. This reliance limits the ability to adapt to new or unknown abnormalities.

[0041] Production monitoring is crucial for improving manufacturing efficiency and product quality, reducing injury risk, and preventing safety incidents. Although sensor-based methods excel in automated process control, anomaly detection, production monitoring, and diagnosis, they still fall short in industrial production scenarios involving continuous manual operations, such as human-robot collaboration tasks or high-precision manual assembly in aerospace manufacturing. Specifically, due to the lack of targeted sensors, sensor-based methods are insufficient for real-time progress tracking and process monitoring. High-flow video data has become ubiquitous in smart factories, especially as high-resolution video monitoring systems have become more affordable. There is an urgent need to develop production monitoring based on real-time high-flow video data and extract value from the collected data.

[0042] In production monitoring, there are two main problems: (i) progress prediction: predicting the ongoing process step; (ii) anomaly detection: detecting abnormal activities in the production process. Vision-based production monitoring has received increasing research interest in recent years. Despite some research progress, vision-based industrial production process monitoring still faces unique challenges and complexities.

[0043] 1) The changes of irrelevant pixels can significantly exceed the changes of the region of interest. In the left large rectangular frame in FIG. 1a and FIG. 1b, the background region is in the middle rectangular frame, and the production region related to production is in the middle rectangular frame. FIG. 1a is an image of normal production, and FIG. 1b is an image in which part of the production region appears abnormal. It can be seen that, compared with FIG. 1a, the change of the background region in FIG. 1b is much greater than the change of the abnormal pixels in the production region. This requires the trained model to pay more attention to the pixels related to abnormalities.

[0044] 2) The appearance of the workpiece undergoes considerable dynamic changes throughout the production process, resulting in spatio-temporal heterogeneity. The huge dynamic changes and heterogeneity make the production anomaly detection extremely complex.

[0045] 3) The inherent scarcity of abnormal data in the production process, as the production line operates under normal conditions most of the time. The scarcity and imbalance of data pose significant challenges to model development and parameter learning.

[0046] Due to these challenges, typical video monitoring methods in the field of computer vision (CV) may not work well in industrial production monitoring. In addition, anomaly classification in CV is completely different from anomaly classification in production monitoring. The production anomalies focused on by the present disclosure include the following four categories: foreign object anomaly, which may interfere with the manufacturing process due to the presence of irrelevant or redundant objects in the production environment; worker operation anomaly, which includes worker actions or posture anomalies that can be observed without reference to the workpiece, such as inappropriate assembly actions or unsafe behavior; part interaction anomaly, which can be identified without observing worker actions, and which involves interactions between parts, such as incorrect assembly order or position, indicating deviation from the expected assembly process; progress-related anomaly, which is specific to different assembly stages. They can involve the misuse of tools or deviation from the prescribed process at certain production stages. The inventors explain these anomalies in detail and their mapping relationship in the field of industrial production and computer vision, as shown in FIG. 2.

[0047] Currently, research on abnormal production activities mainly focuses on pre-defined anomaly categories or rules. However, in a dynamic production environment, it is impractical to predict and label all possible future abnormal activities. Therefore, there is an urgent need for a self-supervised anomaly detection scheme that is trained only using normal image data. This scheme does not require abnormal data and pre-defined rules, greatly improving the model's ability to foresee unknown anomalies.

[0048] In addition, traditional research usually treats progress prediction and anomaly detection as independent tasks. However, the inventors believe that anomaly detection can be integrated into the prediction model to stop stage prediction in abnormal cases. Since anomaly detection criteria can vary by production stage, the model must integrate real-time progress information to achieve accurate and context-dependent anomaly recognition.

[0049] The present disclosure aims to provide a technical solution that can identify an impending anomaly using only normal production image data; use the inherent characteristics of the production process as a self-supervised learning signal to enable the model to focus on production-related features in a wide range of background element variations; and combine anomaly detection with progress prediction tasks to not only improve the model's accuracy in predicting production stages but also provide progress-related anomaly predictions.

[0050] FIG. 3 shows a flowchart of a video-based self-supervised production anomaly detection and progress prediction method according to an embodiment of the present disclosure. As shown in the figure, the method includes steps 110-160.

[0051] Step 110: Obtain video data of a production process.

[0052] The video data can be collected by existing monitoring cameras in the workshop, without the need for additional installation of specialized equipment, thereby ensuring that the normal production process is not disturbed and additional hardware costs are not increased. For example, a high-resolution monitoring camera system already installed in the factory can be used for continuous recording. These cameras usually cover the entire production area and can fully capture all aspects of the production process. The video data of the production process includes multiple frames of original images.

[0053] The data collection process can take into account the actual situation of the production. For example, data during factory downtime can be excluded to ensure that the collected data truly reflects the production status, thereby improving the quality and relevance of the data.

[0054] In some examples, the data collection can cover different types of production processes. For example, in the case of commercial aircraft spoiler production, the collected data includes multiple types of spoiler production processes, covering products of different shapes and sizes such as rectangles and trapezoids. This diverse data collection helps improve the model's generalization ability, enabling it to adapt to different production scenarios.

[0055] Step 120: Use an encoder to map image features in the video data to a latent space of a Gaussian Mixture Model (GMM), obtaining latent variables for representing latent features in the original images, wherein the Gaussian components in the GMM correspond one-to-one to pre-divided production stages.

[0056] Key frame images can be extracted from the video. For example, one image can be extracted from the video stream every three seconds, which can ensure that key changes in the production process are captured while effectively controlling the amount of data to balance information integrity and computational efficiency.

[0057] Unlike traditional vision-based anomaly detection, the image features of a production process exhibit heterogeneity across spatial and temporal dimensions, as shown in Figure 4. Regarding spatial heterogeneity, some locations in the image are relevant to production, while others are background features with varying degrees of importance. Regarding temporal heterogeneity, image features differ across different production steps. Due to this complexity, the inventors believe that a more nuanced approach can be employed to analyze these features.

[0058] This embodiment assumes that the image features zk generated at a specific progress percentage k∈[0,1] follow a Gaussian distribution and exhibit functional heterogeneity along the process timeline (see Equations 1, 2 and 3). μ k =F μ (k) (2) Σ k =F Σ (k) (3)

[0059] The inventors conducted a comprehensive study of the manufacturing industry and found that certain stages of the production process (such as inspection or polishing tasks) have a relatively small impact on the visual characteristics of components. Based on these observations, the inventors proposed F... μ (k) is not a continuously varying function, but a piecewise constant function represented by the separation points y∈{0,…,c}, where k0=0 and k c =0. Therefore, the inventors will determine the separable time point k in an unsupervised manner. y The method is defined as a change point detection problem. In some implementations, it is further solved by Equation 4 and defining a piecewise function F. μ (k) and F Σ (k), with the mean μ of each Gaussian component in the GMM as the correlation. y Covariance Σ y With production schedule k: F μ (k)=μ y if k y <k≤k y+1 , y∈{0,…,n-1} (4) F Σ (k)=Σ y if k y <k≤k y+1 , y∈{0,…,n-1} (5)

[0060] Where k∈[0,1] represents the production progress in percentage form, c is the number of pre-divided production stages, and k y Let μ be the starting point of the production schedule for the y-th production stage. y Let Σ be the mean of the latent variables in the y-th production stage. yCovariance of latent variables for the yth production phase.

[0061] Thus, it can be considered that the distribution of z is consistent with a multivariate Gaussian mixture distribution, as shown in equation 6:

[0062] The inventors comprehensively consider various factors, especially the sensitivity to outliers, and select to incorporate the negative log-likelihood loss NLL of latent variables in GMM into the training objective. The cluster y in the embodiment is predefined by the production progress k, which is known a priori in the training phase. Therefore, the log-likelihood function in the embodiment is represented as follows:

[0063] In this formula, F μ (k) and F Σ (k) are derived by maximum likelihood estimation (MLE), which provides a statistical basis for the model.

[0064] In some embodiments, the training objective negative log-likelihood NLL can be calculated according to equation 8:

[0065] In some embodiments, after each training cycle, the mean μ y and the covariance Σ y of each Gaussian component in the GMM can be updated based on the maximum likelihood estimation of the latent variable samples output by the current encoder, so as to update F μ (k) and F Σ (k). For example, an iterative algorithm based on expectation maximization (EM) can be used to find the parameter estimation that maximizes the likelihood of the observed data, updating the GMM model.

[0066] Step 130, generating a spatial activation map using the feature map generated before the last convolution block of the encoder, the spatial activation map being used to extract attention features related to different production phases from the original image.

[0067] As shown in FIG. 1b, the change of abnormal pixels can be much smaller than the normal background change. Therefore, it is crucial to let the model focus on the pixels related to production. With object classes as weak supervision signals, CAM is proposed to extract object-related pixels. However, in the actual production process studied in the present disclosure, it is difficult or even impossible to label object classes, because each image can present hundreds of parts in a complex production process.

[0068] Therefore, the inventors propose a spatial activation map The spatial activation map can be generated using the feature map Q generated before the last convolutional block of the encoder, which is used to extract attention features related to different production stages from the original image, in other words, using the progress state y as a self-supervised signal to extract the production-related regions in the image. The underlying idea is that when trying to identify the ongoing process state, the weight of a pixel is only large if it is highly related to the process.

[0069] In some embodiments, the feature map Q can be processed using a multi-layer perceptron (MLP) to obtain a feature map Q'; a progress activation map is calculated using a weight matrix f of a fully connected network without bias on the feature map Q', wherein each column of f corresponds to a production stage; and the spatial activation map is generated by averaging each production stage of the progress activation map.

[0070] A specific example is given in FIG. 5. The feature map Q is passed to a convolutional network with parameters θ to generate a map with the same shape conv(Q, θ) = Q'. A fully connected network (FCN) without bias is then used to classify the process state of a given feature map Q'. The predicted class vector is denoted by the weight matrix of the FCN. The predicted class vector The predicted class is then calculated and the weights of the MLP are trained by cross-entropy loss. In addition, in order to emphasize the features related to the progress and distinguish the embedding features between classes, the inventors further introduce a triple loss function as shown in equation 9. This function aims to maintain the margin Loss trip = max (|z anchor -z positive | 2 -|z anchor -z negative | 2 +m 2 ,0) (9)

[0071] where z anchor denotes the latent variable of the current image sample, z positive is the latent variable of the image sample belonging to the same production stage as z anchor , z negative is the latent variable of the image sample belonging to a different production stage as z anchor , and m is a predefined margin.

[0072] In some embodiments, the progress activation map is calculated according to the following formula:

[0073] where PAM y,i,j denotes the progress score at the spatial position (i, j) in the progress activation map, y denotes the y-th production stage, for the y-th column of the weight matrix f, corresponding to the y-th production stage, Q' :,i,j denotes the feature vector of the feature map Q' at position (i, j), denotes and Q' :,i,j after point multiplication and taking the absolute value. The progress activation map maps the importance of each progress class.

[0074] In the present embodiment, PAM y,i,j may represent the progress score of each spatial position, providing a quantification of the contribution of a particular position to the prediction result. This actually reflects the importance of a particular spatial feature to the overall result. If the absolute value of this score is small, it means that removing the image feature at this position has little effect on the progress information of the current stage. Thus, in some embodiments, a spatial activation map can be calculated according to equation 10, which represents the importance of spatial features in the production process.

[0075] This explanatory formula adopted in the present embodiment enhances the explainability of the neural network, making its operation more transparent.

[0076] Step 140, using the decoder to reconstruct the latent variable output by the encoder into a reconstructed image, and using SAM to weight the reconstruction error of the reconstructed image compared to the corresponding original image to obtain the corresponding spatial activation reconstruction error.

[0077] The spatial activation map can be used as attention weight in both the encoder and the decoder. Specifically, in the encoder, the spatial activation map can be multiplied with Q, and then the optimized features are passed to the subsequent network. In the decoder, in the calculation of the pixel-level reconstruction error (X' 2 After that, the spatial activation map is used as weight to generate the spatial activation reconstruction error (SARE). In some embodiments, the spatial activation reconstruction error SARE can be calculated according to equation 11: SARE =∑ c ∑ i ∑ j (X' c,i,j - X c,i,j ) 2 SAM i,j , (11)

[0078] where X' represents the reconstructed image, X represents the original image corresponding to the reconstructed image, c, i, j represent the channel, height, and width index of the reconstructed image and the original image, respectively, (X' c,i,j - X c,i,j ) 2 represents the square of the pixel-level reconstruction error of the reconstructed image and the original image at position (c, i, j), SAM i,jrepresents a value of the spatial activation map at position (i, j).

[0079] The inventors use SARE as the reconstruction loss to train the decoder. As shown in FIG. 6, the reconstruction error caused by background or irrelevant pixels is subtracted in SARE, making the model pay more attention to the pixels related to production. In addition, the correlation of SAM with the progress category enhances the interpretability of the network, explaining the features that affect the model's decision.

[0080] Some studies have shown that class activation map (CAM) usually only activates the most discriminative parts of the object, rather than the entire object, causing the model to ignore some key but non-significant areas in the production process. In some embodiments, random erasing can be introduced during the training process to avoid this phenomenon from appearing in the spatial activation map proposed in the present embodiment. Random erasing can include: randomly selecting an input image with a predetermined probability; randomly selecting a local area on the selected image; setting the pixel values in the local area to a predetermined value to form a partially erased image; using the partially erased image to train the model.

[0081] In some examples, the local area is a rectangular area, and the position (r x ,r y ) and dimension (r w ,r h ) of the rectangle satisfy: r x ~U(0, w), r y ~U(0, h), a ~ U(a l ,a u ), b ~ U(b l ,b u ),

[0082] wherein the dimension of the input image is w x h, U() represents a uniform distribution, a l , a u , b l , b u are predefined parameters.

[0083] In some example application examples, a l = 0.02, a u = 0.33, b l = 0.3, and b u = 3.3 can be set. The erased area can be filled with black pixels, as shown in FIG. 6.

[0084] During the training process, the most discriminative parts can be erased, and according to the present embodiment, by randomly erasing the image, the model proposed in the present embodiment not only learns the most discriminative parts, but also learns all features that can contribute to determining the process stage.

[0085] At step 150, model training is performed based on a comprehensive loss function including cross-entropy loss of production progress prediction, triplet loss of feature embedding of latent space for different production stage feature representations, negative log-likelihood loss of latent variable in GMM, and spatial activation reconstruction error.

[0086] As described above, the model proposed in the embodiment utilizes an encoder-decoder architecture, and further includes a spatial activation map branch, constituting a multi-task learning framework. The encoder maps the image to the latent space of the GMM; the decoder reconstructs the image from the latent variable output by the encoder; the SAM branch generates a spatial activation map using the feature map generated before the last convolutional block of the encoder, including MLP, CNN, and FCN processing steps, as a spatial attention guide for feature extraction by the encoder. The training process of the model combines multiple loss functions. In some embodiments, the comprehensive loss function is calculated according to the following formula: Loss = Loss CE + λ1Loss trip + λ2NLL + λ3SARE,

[0087] wherein Loss CE is the cross-entropy loss of production progress prediction, used to optimize the progress prediction task; Loss trip is the triplet loss of feature embedding of the latent space, used to optimize the identification of different production stages by the model; NLL is the negative log-likelihood loss of the latent variable in the GMM, used to optimize the feature distribution of the latent space; SARE is the spatial activation reconstruction error, used to guide the model regarding the production-related image regions; λ1, λ2, and λ3 are weight coefficients used to balance different loss terms.

[0088] FIG. 6 shows a model framework according to an exemplary embodiment. The model receives two inputs, one of which is the original image X, and the other of which is the image after random erasing processing. Both input images are processed through the encoder. The encoder is composed of multiple convolutional layers, and finally outputs a feature map Q and a latent variable z.

[0089] The feature map Q is first processed through the MLP, then further processed through the CNN, and finally generates a spatial activation map SAM through the FCN. The generation process of SAM also involves the calculation of cross-entropy loss (Loss CE ).

[0090] The decoder receives the latent variable z as input and reconstructs the image X'.

[0091] NLL Loss (NLL loss) and Triplet Loss (Triplet loss Loss trip) based on the latent variable z. The spatial activation reconstruction error (SARE) is computed by comparing the reconstructed image X' and the original image X and multiplying with the spatial activation map.

[0092] The output of the encoder is used for multiple tasks: generating the spatial activation map, computing the negative log-likelihood loss of the latent variable in the GMM and the triple loss, and image reconstruction.

[0093] The SAM branch is used both to guide the attentional feature extraction of the encoder and to compute the spatial activation reconstruction error.

[0094] Dark arrows in the figure represent paths with backpropagation, and light arrows represent paths without backpropagation.

[0095] Regarding the parameter update of the Gaussian Mixture Model (GMM) distribution, specifically the F μ (k) and F Σ (k), some studies suggest jointly training these parameters to improve performance, but the inventors have conducted extensive research and a large number of verifications on the production process data set and believe that the large variability of the normal data distribution of the production process will lead to instability in the joint training process. Therefore, as mentioned above, in some embodiments, F μ (k) and F Σ (k) can be updated based on the maximum likelihood estimation of the latent variable sample output by the current encoder, for example, using an iterative algorithm based on expectation maximization (EM). In each training cycle, the maximum likelihood estimation derived from the latent variable z is used to optimize F μ (k) and F Σ (k), thereby improving the stability and accuracy of the model by adapting to various changes observed in the data, ensuring more robust and reliable anomaly detection and progress prediction.

[0096] Step 160, using the trained encoder of the model to process the real-time collected video data to perform production anomaly detection and predict the production stage based on the obtained latent variable.

[0097] Using the trained encoder to map the image features Xt generated at time t in the production process to the Gaussian mixture latent variable z, which follows the GMM distribution: where α y = k y+1 -k yThe Gaussian component weight representing production stage y is inferred from the average time cost of stage y in the training videos. p(z) calculated represents the likelihood of observing the feature value under normal assembly conditions. Therefore, in some embodiments, the inventors use p(z) calculated according to equation 16 as an anomaly score indicator, and when p(z) is lower than a preset anomaly detection threshold, the corresponding image is determined as an abnormal image. In some studies, reconstruction error is used as an anomaly score indicator, but the inventors found that the existence of dynamic background in the production process may cause difficulties in the reconstruction of even normal process images, and in addition, the marginal difference between abnormal images and normal images means that abnormal images may only show minimal reconstruction loss. In subsequent application examples, the performance of AE+SARE further illustrates this phenomenon.

[0098] According to the present embodiment, the introduction of the spatial activation map SAM and the progress-related GMM can significantly improve the anomaly detection performance by guiding the model to focus on the features related to production.

[0099] In some embodiments, the anomaly detection result can also be used to optimize the progress prediction, including: using the trained encoder to map the images in the real-time collected video data to the latent space of the GMM, to obtain latent variables z; predicting the current production stage y based on the latent variables z The reliability index of the predicted production stage is calculated When , where ε is a preset reliability detection threshold, it is determined that the predicted production stage is unreliable, and the prediction result y t-1 of the last time is taken as the current prediction result, i.e., y t =y t-1 ; when , the current predicted production stage is taken as the current prediction result, i.e.

[0100] According to the present embodiment, given the progress prediction The reliability of this prediction is quantified by the posterior probability of p(z) generated at production stage y . Predictions related to a low confidence threshold are discarded and wait for a reliable prediction later.

[0101] FIG. 7 shows a flowchart of combining anomaly detection and progress prediction according to one exemplary embodiment of the present disclosure.

[0102] In summary, the self-supervised production anomaly detection and progress prediction method based on high-flow video proposed in this embodiment combines the encoder-decoder structure with Gaussian mixture model and spatial activation map, realizing comprehensive monitoring of the production process. By introducing the self-encoder process probability embedding technology, the spatio-temporal heterogeneity of the production process is effectively captured, and the background interference is minimized using the spatial activation map, so that the model can be trained using only normal production data, solving the problem of lack of abnormal data. In addition, the method integrates anomaly detection and progress prediction tasks into a unified framework, not only improving the accuracy and reliability of monitoring, but also enhancing the adaptability of the model to complex production environments.

[0103] The self-supervised production anomaly detection and progress prediction method based on video proposed in this embodiment is evaluated using two different datasets. The first dataset covers the water valve production process, where the assembly components are the main focus of the image data; the second dataset represents the actual production scene, covering assembly operations, lamination, coating and vacuum extraction tasks, involving mechanical parts and flexible composite materials.

[0104] The project test environment is configured with Python 3.8 and CentOS 7.9. To ensure the performance and efficiency of the training process, single A100 GPU or dual A100 GPU settings are used. To ensure the fairness of the evaluation, all models use the ResNet50 architecture as their backbone network.

[0105] A. Water valve production

[0106] This example demonstrates the performance of the proposed model on the single-station water valve production process dataset. The visualization of the production stages is shown in Figures 8a-8i, including: Figure 8a assembling the middle valve body; Figure 8b tightening the screw; Figure 8c installing the sealing ring; Figure 8d assembling the middle part of the valve; Figure 8e assembling the upper part of the valve; Figure 8f assembling the rocker arm; Figure 8g assembling the parts to the rocker arm; Figure 8h completing the lower clamp assembly of the water stop valve; and finally Figure 8i assembling the upper clamp. The entire process lasts about 3000 time frames per cycle.

[0107] The training set contains 11 assembly videos, with 29529 images extracted for training. The test set contains 31 assembly videos, with 700 abnormal data points and 5790 normal data points labeled every ten time frames. The anomalies in the dataset are shown in Figures 9a-9e, including foreign object anomaly shown in Figure 9a, worker operation anomaly shown in Figures 9d and 9e, progress-related anomaly shown in Figure 9b, and part interaction anomaly shown in Figures 9c and 9d. It should be noted that some abnormal images may exhibit multiple abnormal conditions simultaneously, and this example focuses on detecting whether an anomaly exists, rather than its type.

[0108] The visualization of the model features is shown in Figure 10. The color goes from light to dark corresponding to the production stage of the images in the video sequence, with darker colors indicating later stages of the assembly process. It can be seen in Figure 10 that each process class is clustered together, both for the training and test sets. The visualization of the SAM and the reconstructed image X' is shown in Figure 11. The visualization results in Figure 11 clearly show that the SAM is mainly focused on the position of the parts being assembled, the hands of the workers performing production-related operations, or the position of the hands that will assemble.

[0109] B. Commercial aircraft spoiler production

[0110] The performance of the model in the actual production of commercial aircraft spoilers was evaluated in this example. This workstation was chosen because it involves mechanical parts and flexible materials. The finished spoiler is shown in Figure 12. The production process is complex and varied, including preliminary pre-assembly work at the tool, primer drying, inspection of auxiliary materials, lamination of composite materials, and application of various color films. The subsequent stages include assembly of the rigid structure, covering the release film, and bagging and leak testing, followed by curing in an autoclave. This example focuses on the production stages of the spoiler before the autoclave process, which lasts approximately 3 hours and is characterized by mixed lines and completely manual manufacturing. One of the stages involves waiting and inspection, with no significant changes in the appearance of the workpiece. In addition, the position of the tool changes within the range of the cameras during each cycle. The visual image data was captured by the workshop monitoring cameras, ensuring that it did not interfere with production and had no additional hardware costs.

[0111] The dataset was carefully prepared based on three weeks of monitoring videos from the factory. By extracting a frame every three seconds, this example collected approximately 46,000 images. This dataset was systematically divided into different units: six for training and seven for testing, as described in Table 1. The training set included three different types of spoilers, while the test and validation sets included spoilers from the training set and another untrained spoiler type. The spoilers differed in shape (from rectangular to trapezoidal), size, and specific structure for wing mounting. Despite these differences, the production process remained consistent across different spoiler types. Data during factory downtime was excluded based on the operating time of the facility. Table 1. Description of the commercial aircraft spoiler production dataset

[0112] To evaluate the effectiveness of anomaly detection, the test data was labeled by the collective expertise of several factory personnel and managers. A total of 23 different anomalies and 1847 normal cases were labeled. The anomalies included pre-assembled parts getting stuck, workers sitting at the tool station during active assembly, and moving the tool and equipment prematurely before the assembly cycle was complete. The visualization of the features is shown in Figure 13, with the same explanation as in Figure 10.

[0113] C. Benchmark comparison

[0114] The video-based self-supervised production anomaly detection and progress prediction method proposed in this disclosure is compared with three well-known anomaly detection algorithms: autoencoder (AE) using reconstruction error as anomaly score; variational autoencoder (VAE) where reconstruction probability is used as anomaly indicator; and DaGMM. In addition, the inventors also test the use of SARE instead of reconstruction loss of AE in equation 11 (AE + SARE column in the table) to demonstrate the effectiveness of SAM in focusing on production-related features. Table 2 Quantitative results of anomaly detection in water valve production Table 3 Quantitative results of anomaly detection in commercial aircraft spoiler production

[0115] In the examples, the inventors adopt two widely recognized metrics: area under the receiver operating characteristic curve (AUC) and equal error rate (EER) to evaluate the effectiveness of various methods. The quantitative results shown in Tables 2 and 3 show that the method proposed in this disclosure outperforms all baseline models in overall anomaly detection performance. In particular, it can be observed that AE enhanced with SARE in equation 11 (AE + SARE) performs better than the other three baseline models. This indicates that SAM is very effective in directing the model to focus on production-related features. However, its detection performance is still significantly inferior to the technical solution proposed in this disclosure. The main reason is that although SAM itself has highlighted production-related features, the reconstruction of normal images is still challenging. This is mainly because in a typical production environment, the number of available training samples is usually limited and may not cover all normal conditions present in the test set. Therefore, the decoder part of the model is difficult to learn effectively, resulting in a large reconstruction loss for normal images (as shown in the last row of Figure 11). However, the encoder part only needs to extract relevant information from the image to reconstruct production-related features, and the task is relatively simple. Therefore, analyzing the probability of feature values instead of reconstruction loss can circumvent the problem related to the poorly trained decoder.

[0116] In addition, the method proposed in this disclosure is significantly superior to all baseline models in detecting specific anomaly subclasses, including foreign objects, worker operations, part interactions, and progress-related anomalies, as shown in Table 2.

[0117] D. Ablation Study

[0118] To rigorously evaluate the indispensability of each component in the model framework proposed by the present disclosure (as shown in FIG. 6), the inventors also conducted a series of ablation studies. These studies include variants of the model shown in FIG. 6, including replacing SAM with CAM, denoted as APPECAM; removing SAM, denoted as APPE-noSAM; removing NLL, denoted as APPE-noNLL; removing random erasing, denoted as APPE-noErase. Through these modifications, the contribution of each component to the overall performance of the model proposed by the present disclosure can be indicated. Table 4 Quantitative results of ablation studies

[0119] The results of the ablation studies are shown in Table 4. The results in the second and third columns show that the SAM proposed by the present disclosure is superior to the scheme using CAM as an alternative. This superiority is due to the fact that CAM only focuses on determining the features of the current stage, while SAM can capture all features related to production. The SAM proposed by the present disclosure not only avoids the loss of important information, but also filters out features unrelated to production. In addition, the overall performance of APPE-noNLL and APPE-noSAM is lower than the model proposed by the present disclosure as shown in FIG. 6, further proving the importance of the SAM and the progress-related GMM embedding proposed by the present disclosure. In addition, it can also be observed that the impact of removing NLL on performance is much smaller than removing SAM. This is because after removing NLL, the model becomes a metric learning without considering the correlation matrix of the feature vector. The progress-related attention SAM plays a key role in anomaly detection, and the loss of information is relatively small. In addition, the model proposed by the present disclosure is superior to APPE-noErase, proving that the random erasing process proposed in some embodiments has a beneficial effect in improving the overall performance of the model.

[0120] E. Combining anomaly detection with progress prediction

[0121] In addition to anomaly detection, the model proposed by the present disclosure also simultaneously outputs the prediction of the current production stage of the production process, denoted as y. An abnormal operation can adversely affect the accuracy of production stage identification. Therefore, as discussed above, according to some embodiments of the present disclosure, the probability of the production stage y The identification results are post-processed. A video describing normal operation can be selected as a validation set, and the 99th percentile of is calculated to determine the threshold.

[0122] To evaluate the effectiveness of the model in task progress prediction, the inventors used two key indicators. The main indicator is the classification accuracy, which directly measures the ability of the model to correctly identify the current stage of the production process; the second indicator is the mean absolute percentage error (MAPE) of time, as shown in equation 17:

[0123] in, Indicates the stage in video i The formula calculates the start time of the predicted category. As a predicted value, schedule forecasting is treated as a continuous process. This approach not only assesses the likelihood of model prediction accuracy but also quantitatively measures the model's prediction bias by comparing the predicted value with the actual occurrence time. This dual evaluation strategy effectively captures both the accuracy of precision and time forecasting across different production stages. Table 5 shows the schedule forecast accuracy before and after anomaly detection.

[0124] The comparison results before and after the combination are shown in Table 5. After the combination, the accuracy and mean absolute percentage error (MAPE) of both datasets decreased.

[0125] In summary, the comparative analysis of the two different datasets and benchmark models in Tables 2 and 3 clearly demonstrates the superiority of the proposed autoencoder process probabilistic embedding in detecting all anomalies summarized in Figure 2. The results of the ablation study, shown in Table 4, further reinforce the effectiveness of this disclosure, highlighting the key contributions of each component in the model. The video-based self-supervised production anomaly detection and schedule prediction method proposed in this disclosure cleverly addresses the dual challenges of anomaly detection and schedule prediction within a single, unified framework. This combination not only halves the computational complexity of these tasks, which are traditionally handled independently, but also synergistically enhances the performance of both. Furthermore, the superior visual insights provided by the Spatial Activation Graph (SAM), as shown in Figure 11, are also highlighted.

[0126] According to one embodiment of this disclosure, a video-based self-supervised production anomaly detection and progress prediction device is also proposed, the device comprising:

[0127] The video acquisition unit is used to acquire video data of the production process, which includes multiple frames of raw images.

[0128] The latent feature extraction unit is used to map the image features in the video data to the latent space of the Gaussian mixture model (GMM) using the encoder, and obtain latent variables to represent the latent features in the original image. The Gaussian components in the GMM correspond one-to-one with the pre-divided production stages.

[0129] The SAM generation unit is used to generate a spatial activation map using the feature map generated before the last convolutional block of the encoder. The spatial activation map is used to extract attention features related to different production stages from the original image.

[0130] an image reconstruction unit configured to reconstruct the latent variable output by the encoder into a reconstructed image using a decoder, and to weight a reconstruction error of the reconstructed image compared to the corresponding original image using the spatial activation map to obtain a corresponding spatial activation reconstruction error;

[0131] train the model based on a comprehensive loss function including a cross-entropy loss of the production progress prediction, a triple loss of the feature embedding of the latent space for different production stage feature representations, a negative log-likelihood loss of the latent variable in the GMM, and the spatial activation reconstruction error:

[0132] an anomaly detection and progress prediction unit configured to process real-time collected video data using the trained encoder of the model to perform production anomaly detection and production stage prediction based on the obtained latent variable.

[0133] Other details and benefits of the embodiments are described above, and will not be repeated here.

[0134] The embodiments of the present disclosure can be applied to a computer system / server, which can operate with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that can be suitable for use with computer system / server include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or the like.

[0135] The computer system / server can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like, which perform particular tasks or implement particular abstract data types. The computer system / server can be practiced in distributed cloud computing environments with other computer systems coupled via communication networks. In a distributed cloud computing environment, program modules can be located in both local and remote computer system storage media including memory storage devices.

[0136] FIG. 14 is an electronic device provided by at least one embodiment of the present disclosure, which includes a memory for storing computer instructions executable on a processor, and a processor for implementing the video-based self-supervised production anomaly detection and progress prediction method according to any embodiment or implementation of the present disclosure when executing the computer instructions.

[0137] The at least one embodiment of the present disclosure also provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the video-based self-supervised production anomaly detection and progress prediction method according to any one of the embodiments or implementation manners of the present disclosure.

[0138] Those skilled in the art will appreciate that one or more embodiments of the disclosure can be provided as a method, a system, or a computer program product. Accordingly, one or more embodiments of the disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) containing computer usable program code.

[0139] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0140] The above describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in an order other than that in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or possible.

[0141] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier, for execution by, or to control the operation of, data processing apparatus. Alternatively, or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0142] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit), and / or by programmable data processing apparatuses, which can be portion of hardware processing circuitry that executes specific tasks as described. Apparatuses can also be implemented as a combination of special purpose logic circuitry, e.g., an FPGA or an ASIC, and / or one or more programmed computers.

[0143] Computers suitable for the execution of a computer program include, by way of example, general and / or special purpose microprocessors, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0144] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0145] While this specification contains many specifics, these should not be construed as limitations on the scope of any invention or on the required scope of patent protection. Certain features outside the scope of the specific embodiments described herein can be applied to some embodiments and vice versa. Additionally, the described features can be applied in any combination, even if such a combination is not expressly described in the specification. Furthermore, while features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0146] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order nor limiting of all illustrations to that order, nor requiring that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0147] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures need not be performed in the particular order described or in sequential order at all. In certain implementations, multitasking and parallel processing can be advantageous.

[0148] The above descriptions are only preferred embodiments of one or more embodiments of the present specification, and are not intended to limit one or more embodiments of the present specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the present specification should be included in the protection scope of one or more embodiments of the present specification.

Claims

1. A video-based self-supervised production anomaly detection and progress prediction method, comprising: obtaining video data of a production process, the video data comprising a plurality of original images; mapping image features in the video data to a latent space of a Gaussian Mixture Model (GMM) using an encoder, to obtain latent variables representing latent features in the original images, Gaussian components in the GMM corresponding to pre-divided production stages one by one; generating a spatial activation map using feature maps generated before a last convolutional block of the encoder, the spatial activation map being used to extract attention features related to different production stages from the original images; reconstructing the latent variables output by the encoder into reconstructed images using a decoder; weighting reconstruction errors of the reconstructed images compared to corresponding original images using the spatial activation map, to obtain corresponding spatial activation reconstruction errors; training a model based on a comprehensive loss function, wherein the comprehensive loss function comprises a cross-entropy loss of production progress prediction, a triple loss of feature embedding of the latent space for feature representation of different production stages, a negative log-likelihood loss of the latent variables in the GMM, and the spatial activation reconstruction errors; and processing real-time collected video data using the encoder of the trained model to perform production anomaly detection and production stage prediction based on obtained latent variables.

2. The method of claim 1, wherein, the mean μ of each Gaussian component in the GMM y and the covariance Σ y by the piecewise function F μ (k) and F Σ (k) associated with the successive production progress k: F μ (k) = μ y if k y <k≤k y+1 , y e {0,..., c - 1}, F Σ (k) = Σ y if k y <k≤k y+1 , y e {0,..., c - 1}, wherein k∈[0,1] represents production progress in percentage form, c is the number of pre-divided production stages, k y start of production progress for the yth production phase, μ y the mean of the potential variables for the yth production phase, ∑ y Covariance of potential variables for the yth production phase; After each training cycle, the means μ of the individual Gaussian components in the GMM are updated based on a maximum likelihood estimate of the latent variable samples of the current encoder output y and the covariances∑ y .

3. The method of claim 1, wherein, generating a spatial activation map using feature maps generated before a last convolutional block of the encoder, comprises: processing the feature maps Q generated before the last convolutional block of the encoder using a multi-layer perceptron (MLP) to obtain feature maps Q′; calculating a progress activation map for the feature maps Q′ using a weight matrix f of a bias-free fully connected network, wherein each column of f corresponds to a production stage; averaging each production stage of the progress activation map to generate the spatial activation map.

4. The method of claim 3, wherein, calculating the progress activation map comprises: The progress activation map is calculated according to the following formula: where PAM y,i,j denotes the progress fraction at spatial position (i,j) in the progress activation map, y represents the yth production stage, is the transpose of the yth column of the weight matrix f, corresponding to the yth production stage, Q′ :,i,j denotes the feature vector of the feature map Q' at position (i,j), denotes and Q' :,i,j Point multiplication followed by absolute value.

5. The method of claim 1, wherein, weighting reconstruction errors of the reconstructed images compared to corresponding original images using the spatial activation map to obtain corresponding spatial activation reconstruction errors, comprises: calculating a spatial activation reconstruction error SARE according to the following formula: SARE =∑ c ∑ i ∑ j (X' c,i,j -X c,i,j ) 2 SAM i,j , wherein X′ represents the reconstructed image, X represents the original image corresponding to the reconstructed image, c, i, j represent the channel, height, and width index of the reconstructed image and the original image, respectively, (X′ c,i,j -X c,i,j ) 2 denotes the squared pixel-level reconstruction error of the reconstructed image and the original image at position (c, i, j), SAM i,j denotes the value of the spatial activation map at position (i,j).

6. The method of claim 1, wherein, The overall loss function is calculated according to the following formula: Loss = Loss CE + λ1Loss trip + λ2NLL + λ3SARE, wherein Loss CE is a cross-entropy loss for the production progress prediction, used to optimize the progress prediction task; Loss trip Triple loss for the feature embedding of the latent space, for optimizing the model's recognition of different production stages; NLL is a negative log-likelihood loss of the latent variables in the GMM, used to optimize the feature distribution of the latent space; SARE is the spatial activation reconstruction error, used to guide the identification of production-related image regions in the model training process; λ1, λ2, λ3 are weight coefficients used to balance different loss terms.

7. The method according to claim 1 or 6, characterized in that, The triple loss Loss is calculated according to the formula trip : Loss trip = max(|z anchor -z positive | 2 -|z anchor -z negative | 2 +m 2 , 0), where z anchor represents the latent variable of the current image sample, z positive to z anchor latent variables of image samples belonging to the same production phase, z negative to z anchor latent variables of image samples belonging to different production phases, m is a predefined margin.

8. The method of claim 1, wherein, The method further comprises: randomly selecting an input image with a predetermined probability; randomly selecting a local region on the selected image; setting pixel values within the local region to a predetermined value to form a partially erased image; using the partially erased image for model training.

9. The method of claim 8, wherein, The local region is a rectangular region, the position (r x , r y ) and dimension (r w , r h ) of the rectangle satisfy: r x ~U(0,w), r y ~U(0,h), a~U(a l , a u ), b~U(b l , b u ), wherein the input image has a dimension of w x h, U() represents a uniform distribution, a l , a u , b l , b u are predefined parameters.

10. The method of claim 1, wherein, The production anomaly detection comprises: mapping an image in real-time collected video data to a latent space of a GMM using an encoder of the trained model to obtain a latent variable z; The probability density p(z) of the latent variable z in the Gaussian mixture model is calculated based on the following equation: wherein c is the number of production stages, a y weight for the yth production phase, For a multivariate Gaussian distribution with mean μ y and covariance matrix Σ y the probability density function is Taking p(z) as an anomaly score index, when p(z) is lower than a preset anomaly detection threshold, the corresponding image is determined as an abnormal image.

11. The method of claim 1, wherein, The predicted production stage comprises: mapping an image in real-time collected video data to a latent space of a GMM using an encoder of the trained model to obtain a latent variable z; predicting a current production phase based on the latent variable z computing the predicted production phase reliability indicator When at a time, ε is a preset reliability detection threshold value, and the predicted production stage Unreliable, the prediction result y of the last time t-1 As the current prediction result, i.e. y t = y t-1 ; When at a time, the predicted production phase As the current prediction result, i.e.

12. A video-based self-supervised production anomaly detection and progress prediction apparatus, comprising: a video acquisition unit configured to acquire video data of a production process, the video data comprising a plurality of original images; a latent feature extraction unit configured to map image features in the video data to a latent space of a Gaussian Mixture Model (GMM) using an encoder to obtain a latent variable representing latent features in the original images, the Gaussian components in the GMM corresponding one-to-one to pre-divided production stages; a SAM generation unit configured to generate a spatial activation map using feature maps generated before the last convolution block of the encoder, the spatial activation map being used to extract attention features related to different production stages from the original images; an image reconstruction unit configured to reconstruct the latent variable output by the encoder into a reconstructed image using a decoder, and to weight reconstruction errors of the reconstructed image compared to the corresponding original image using the spatial activation map to obtain a corresponding spatial activation reconstruction error; model training based on a comprehensive loss function, the comprehensive loss function comprising a cross-entropy loss of production progress prediction, a triple loss of feature embedding of the latent space for feature representation of different production stages, a negative log-likelihood loss of the latent variable in the GMM, and the spatial activation reconstruction error; and an anomaly detection and progress prediction unit configured to process real-time collected video data using an encoder of the trained model to perform production anomaly detection and predict production stages based on the obtained latent variable.

13. An electronic device, comprising: The device comprises a memory for storing computer instructions executable on a processor, and a processor for implementing the method of any one of claims 1 to 11 when executing the computer instructions.

14. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by a processor, implements the method of any one of claims 1 to 11.

Images