Video-based self-supervised production anomaly detection and progress prediction method, device and equipment

Abstract

The invention discloses a video-based self-supervised production anomaly detection and progress prediction method, device and equipment. The method comprises the following steps: mapping image features of a production process video to a potential space of a Gaussian mixture model by using an encoder; generating a space activation graph and extracting attention features related to production; reconstructing the image by using a decoder and calculating a space activation reconstruction error; training a model based on a comprehensive loss function; and performing anomaly detection and production stage prediction by using the trained model. According to the method, an auto-encoder process probability embedding technology and a space activation graph are combined, the space-time heterogeneity of the production process is effectively captured, and meanwhile background interference is minimized; abnormality detection and progress prediction tasks are integrated into a unified framework, so that the monitoring accuracy and reliability are improved; and training can be carried out under the condition that only normal production data is used, so that the problem of scarcity of abnormal data is solved, and the adaptability of the model to a complex production environment is enhanced.

Description

Technical Field

[0001] The present disclosure relates to the field of advanced manufacturing, and in particular to a method, device and equipment for self-supervised production anomaly detection and progress prediction based on video. Background Art

[0002] With the rapid development of Industry 4.0 and smart manufacturing, real-time production monitoring plays an increasingly important role in modern production management. Efficient and accurate anomaly detection and progress prediction are essential to improving production quality, optimizing production efficiency, and preventing safety accidents. However, in complex manufacturing environments, especially those involving human-machine collaboration or high-precision manual assembly, traditional monitoring methods face many challenges.

[0003] Currently, the monitoring methods based on sensor data used in automated production lines have been greatly improved compared to traditional technologies, but they are still difficult to meet the needs in industrial production scenarios involving continuous manual operations. The main reason is that there is a lack of targeted sensors, which makes it difficult to achieve real-time progress tracking and process monitoring. As high-resolution video surveillance systems become more popular and affordable, production monitoring methods based on video data are gradually gaining attention.

[0004] However, existing vision-based production monitoring methods have many limitations. First, these methods have difficulty distinguishing between production-related features and background noise. In actual production environments, background changes may be much larger than subtle changes related to the production process, making it difficult for traditional methods to accurately identify anomalies. Second, existing methods do not fully consider the spatiotemporal heterogeneity of the production process. The appearance of the workpiece may undergo significant dynamic changes throughout the production process, and this heterogeneity makes anomaly detection extremely complicated.

[0005] Another challenge is the lack of sufficient abnormal data for model training. Since production lines operate under normal conditions most of the time, abnormal data is scarce, which poses a huge challenge to model development and parameter learning. In addition, most existing methods treat production anomaly detection and progress prediction as independent tasks, ignoring the potential correlation between the two.

[0006] Existing methods also generally rely on predefined exception categories or rules. However, in a dynamically changing production environment, it is impossible to predefine and annotate all possible exceptions. This reliance limits the adaptability of the method when facing new or unknown exceptions. Summary of the invention

[0007] The present disclosure proposes a video-based self-supervised production anomaly detection and progress prediction method, which can effectively distinguish between background changes and production-related features, adapt to the spatiotemporal heterogeneity of the production process, and achieve effective monitoring in the absence of abnormal data. The present disclosure also proposes a corresponding device, electronic device, and computer-readable storage medium.

[0008] According to an embodiment of the present disclosure, a video-based self-supervised production anomaly detection and progress prediction method is proposed, the method comprising:

[0009] Acquire video data of the production process;

[0010] Mapping the image features in the video data to a latent space of a Gaussian mixture model (GMM) using an encoder to obtain latent variables for representing the latent features in the image, wherein the Gaussian components in the GMM correspond one-to-one to the pre-divided production stages;

[0011] Generate a spatial activation map using the feature map generated before the last convolutional block of the encoder, wherein the spatial activation map is used to extract and represent attention features related to different production stages in the image;

[0012] Reconstruct the latent variables output by the encoder into an image using the decoder, and calculate the spatial activation reconstruction error of the reconstructed image weighted by the spatial activation map;

[0013] The model is trained based on a comprehensive loss function that includes the cross entropy loss for production schedule prediction, the triple loss for latent space feature embedding for different production stage feature representations, the negative log-likelihood loss of latent variables in GMM, and the spatial activation reconstruction error:

[0014] The encoder of the trained model is used to process the real-time collected video data, and production anomalies are detected and the production stage is predicted based on the obtained latent variables.

[0015] In some embodiments, the mean μ of each Gaussian component in the GMM is y and covariance∑ y By the following piecewise function F μ (k) and F ∑ (k) is associated with the continuous production progress k:

[0016] F μ (k) = μ y if k y <k≤k y+1 ,y∈{0,…,c-1},

[0017] F ∑ (k)=∑ y if ky <k≤k y+1 ,y∈{0,…,c-1},

[0018] Among them, k∈[0,1] represents the production progress in percentage form, c is the number of pre-divided production stages, and k y is the starting point of the production schedule of the yth production stage, μ y is the mean of the latent variable in the y-th production stage, ∑ y is the covariance of the latent variables in the y-th production stage;

[0019] After each training cycle, the mean μ of each Gaussian component in the GMM is updated based on the maximum likelihood estimate of the latent variable samples output by the current encoder. y and covariance Σ y .

[0020] In some implementations, generating a spatial activation map using a feature map generated before the last convolutional block of the encoder includes:

[0021] Use a multi-layer perceptron (MLP) to process the feature map Q generated before the last convolutional block of the encoder to obtain the feature map Q′;

[0022] Using the weight matrix f of the fully connected network without bias, calculate the progress activation map for the feature map Q′, where each column of f corresponds to a production stage;

[0023] The spatial activation map is generated by averaging each production stage of the progress activation map.

[0024] In some implementations, calculating the progress activation map includes:

[0025] The progress activation graph is calculated according to the following formula:

[0026]

[0027] Among them, PAMy ,i,j represents the progress score at the spatial position (i, j) in the progress activation diagram, y represents the y-th production stage, is the transpose of the y-th column of the weight matrix f, corresponding to the y-th production stage, Q′ :,i,j represents the feature vector of feature map Q′ at position (i, j), express and Q′ :,i,j Take the absolute value after dot multiplication.

[0028] In some embodiments, calculating a spatial activation reconstruction error of a reconstructed image weighted by a spatial activation map includes:

[0029] The spatial activation reconstruction error SARE is calculated according to the following formula:

[0030] SARE=∑ c ∑ i ∑ j (X′ c,i,j -X c,j,j ) 2 SAM i,j ,

[0031] Among them, X′ represents the reconstructed image, X represents the original image, c, i, j represent the channel, height, and width index of the image respectively, (X′ c,i,j -X c,i,j ) 2 Represents the square of the pixel-level reconstruction error between the reconstructed image and the original image at position (c,i,j), SAM i,j Represents the value of the spatial activation map at position (i, j).

[0032] In some embodiments, the comprehensive loss function is calculated according to the following formula:

[0033] Loss=Loss CE +λ 1 Loss trip +λ 2 NLL+λ 3 SARE,

[0034] Among them, Loss CE The cross entropy loss for production progress prediction is used to optimize the progress prediction task; Loss trip is the triple loss of latent space feature embedding, which is used to optimize the model's recognition of different production stages; NLL is the negative log-likelihood loss of the latent variable in GMM, which is used to optimize the feature distribution of the latent space; SARE is the spatial activation reconstruction error, which is used to guide the model about the production-related image area; λ 1 , 2 , 3 is the weight coefficient used to balance different loss terms.

[0035] In some embodiments, triplet loss Loss is calculated according to the following formula: trip :

[0036] Loss trip =max(|z anchor -z positive | 2 -|z anchor -Z negative | 2 +m 2 ,0),

[0037] Among them, z anchor represents the latent variable of the current image sample, z positive For z anchor The latent variable of image samples belonging to the same production stage, z negative For z anchor Latent variables of image samples belonging to different production stages, m is a predefined margin.

[0038] In some embodiments, the method further includes introducing random erasures during the training process, including:

[0039] Randomly select an input image with a predetermined probability;

[0040] Randomly select a local area on the selected image;

[0041] Setting the pixel values ​​in the local area to predetermined values ​​to form a partially erased image;

[0042] Model training using partially erased images.

[0043] In some embodiments, the local area is a rectangular area, and the position of the rectangle (r x ,r y ) and dimension (r w ,r h )satisfy:

[0044] r x ~U(0,w),r y ~U(0,h),

[0045] a~U(a l ,a u ),b~U(b l ,b u ),

[0046]

[0047] Among them, the dimension of the input image is w×h, U() represents uniform distribution, and a l 、a u 、b l 、b u are predefined parameters.

[0048] In some embodiments, the local area for the production anomaly detection includes:

[0049] Use the trained encoder to map the image in the real-time collected video data to the latent space of GMM to obtain the latent variable z;

[0050] The probability density p(z) of the latent variable z in the Gaussian mixture model is calculated based on the following formula:

[0051]

[0052] Where c is the number of production stages, α y is the weight of the y-th production stage, The mean is μ y , the covariance matrix is ​​Σ y The probability density function of the multivariate Gaussian distribution;

[0053] p(z) is used as the anomaly score indicator. When p(z) is lower than the preset anomaly detection threshold, the corresponding image is judged to be an abnormal image.

[0054] In some implementations, the step of predicting the progress of the local area includes:

[0055] Use the trained encoder to map the image in the real-time collected video data to the latent space of GMM to obtain the latent variable z;

[0056] Predict the current production stage based on latent variable z

[0057] Production phase of calculation forecast Reliability index

[0058] when When ∈ is the preset reliability detection threshold, the predicted production stage is determined Unreliable, the prediction result y of the previous moment t-1 As the current prediction result, y t =y t-1 ;

[0059] when When the current predicted production stage As the current prediction result,

[0060] According to one embodiment of the present disclosure, a video-based self-supervised production anomaly detection and progress prediction device is proposed, the device comprising:

[0061] A video acquisition unit, used to acquire video data of the production process;

[0062] A latent feature extraction unit, used for mapping the image features in the video data to the latent space of a Gaussian mixture model (GMM) by using an encoder, to obtain latent variables for representing the latent features in the image, wherein the Gaussian components in the GMM correspond one-to-one to the pre-divided production stages;

[0063] A SAM generation unit, for generating a spatial activation map using a feature map generated before the last convolutional block of the encoder, wherein the spatial activation map is used to extract and represent attention features related to different production stages in the image;

[0064] An image reconstruction unit, configured to reconstruct the latent variables output by the encoder into an image using the decoder, and calculate a spatial activation reconstruction error of the reconstructed image weighted by the spatial activation map;

[0065] The model is trained based on a comprehensive loss function that includes the cross entropy loss for production schedule prediction, the triple loss for latent space feature embedding for different production stage feature representations, the negative log-likelihood loss of latent variables in GMM, and the spatial activation reconstruction error:

[0066] The anomaly detection and progress prediction unit is used to process the real-time collected video data using the encoder of the trained model, perform production anomaly detection and predict the production stage based on the obtained latent variables.

[0067] According to one embodiment of the present disclosure, an electronic device is provided, the device comprising a memory and a processor, the memory being used to store computer instructions executable on the processor, the processor being used to implement any of the above methods when executing the computer instructions.

[0068] According to an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, any of the above methods is implemented.

[0069] The video-based self-supervised production anomaly detection and progress prediction scheme proposed in the present disclosure provides an innovative solution for production monitoring in complex manufacturing environments. The present disclosure adopts the autoencoder process probability embedding (APPE) technology to map the image distribution of normal production to the Gaussian mixture model (GMM) related to the progress, so that the model can effectively capture the spatiotemporal heterogeneity in the production process, thereby more accurately identifying production-related features; the present disclosure minimizes the impact of background interference by introducing the spatial activation map (SAM, Spatial Attention Map), so that the model can focus more on the image areas directly related to production; the self-supervised learning characteristics of the present disclosure enable it to be trained using only normal production data, which greatly solves the problem of scarcity of abnormal data and enhances the model's ability to foresee unknown anomalies. In addition, by integrating anomaly detection into the prediction model, the present disclosure also significantly improves the accuracy and reliability of overall monitoring.

[0070] The solution proposed in the present disclosure also greatly improves the accuracy of anomaly detection and enhances the accuracy of progress prediction by comprehensively considering the characteristics of different production stages, so that the model can better adapt to complex production environments and diverse abnormal situations.

[0071] The solution proposed in this disclosure has shown obvious advantages in practical application verification. Through verification on real data sets such as water valve assembly and commercial aircraft spoiler production, the solution proposed in this disclosure has achieved significant improvement in the detection of various types of anomalies, especially in identifying foreign body anomalies, worker operation anomalies, part interaction anomalies, and progress-related anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0073] Figure 1a and Figure 1b A schematic diagram of a production scenario is shown.

[0074] Figure 2 A schematic diagram of comparative analysis of anomalies in computer vision and industrial production is shown.

[0075] Figure 3 A flowchart of a video-based self-supervised production anomaly detection and progress prediction method according to an embodiment of the present disclosure is shown.

[0076] Figure 4 A schematic diagram showing the spatial heterogeneity of image features according to an exemplary embodiment of the present disclosure is shown.

[0077] Figure 5 A schematic diagram of a visualization process of generating a SAM according to an exemplary embodiment of the present disclosure is shown.

[0078] Figure 6 A structural schematic diagram of a model according to an exemplary embodiment of the present disclosure is shown.

[0079] Figure 7 A schematic diagram of a flow chart for combining anomaly detection and progress prediction according to an exemplary embodiment of the present disclosure is shown.

[0080] Figure 8a to Figure 8i A schematic diagram showing a visualization of production stages of water valve production according to an exemplary embodiment of the present disclosure is shown.

[0081] Figure 9a to Figure 9e A schematic diagram showing an abnormal assembly process in water valve production according to an exemplary embodiment of the present disclosure is shown.

[0082] Fig.10A schematic diagram illustrating the characteristic visualization of potential variables of a model output in water valve production according to an exemplary embodiment of the present disclosure.

[0083] Fig.11 FIG. 4 is a schematic diagram showing a visualization of generating a SAM and reconstructing an image X′ in water valve production according to an exemplary embodiment of the present disclosure.

[0084] Fig.12 A finished aircraft spoiler according to an exemplary embodiment of the present disclosure is shown.

[0085] Fig.13 A schematic diagram showing the characteristic visualization of potential variables of a model output in aircraft spoiler production according to an exemplary embodiment of the present disclosure is shown.

[0086] Fig.14 It is a schematic diagram of the structure of an electronic device shown in at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0087] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0088] Production monitoring is critical to improving manufacturing efficiency and product quality, reducing the risk of injury, and preventing safety incidents. Although existing approaches based on sensor data perform well in automated process control, anomaly detection, production monitoring, and diagnosis, they are still insufficient in industrial production scenarios involving continuous manual operations, such as human-machine collaborative tasks or aerospace manufacturing with high-precision manual assembly. Specifically, due to the lack of targeted sensors, existing methods are insufficient for real-time progress tracking and process monitoring. High-stream video data has become ubiquitous in smart factories, especially as high-resolution video surveillance systems become more affordable. There is an urgent need to develop production monitoring based on real-time high-stream video data and to mine value from the collected data.

[0089] In production monitoring, there are two main problems: (i) progress prediction: predicting the ongoing process steps; (ii) anomaly detection: detecting abnormal activities in the production process. Vision-based production monitoring has been increasingly studied in recent years. Despite some research progress, vision-based industrial production process monitoring still faces unique challenges and complexities:

[0090] 1) The changes in irrelevant pixels may significantly exceed the changes in the region of interest. Figure 1aand Figure 1b The green frame is the background area, and the red frame is the production-related area. Figure 1a For normal production images, Figure 1b The image in the red box shows an abnormality. It can be seen that Figure 1b Relative to Figure 1a , the background changes in the green box are much greater than the abnormal pixel changes in the red box. This requires the training model to pay more attention to the pixels related to the abnormality.

[0091] 2) The appearance of workpieces undergoes considerable dynamic changes throughout the production process, resulting in spatiotemporal heterogeneity. The huge dynamic changes and heterogeneity make production anomaly detection extremely complex.

[0092] 3) The inherent scarcity of abnormal data in the production process, since the production line operates under normal conditions most of the time. The scarcity and imbalance of data bring significant challenges to model development and parameter learning.

[0093] Due to these challenges, typical video surveillance methods in the field of computer vision (CV) may not work well in industrial production monitoring. In addition, and more importantly, the classification of anomalies in CV is completely different from that in production monitoring. The production anomalies that this disclosure focuses on include the following four categories: Foreign object anomalies: These anomalies are due to the presence of irrelevant or redundant objects in the production environment, which may interfere with the manufacturing process; Worker operation anomalies: This category includes worker movement or posture anomalies that can be observed without reference to the workpiece, such as inappropriate assembly movements or unsafe behaviors; Part interaction anomalies: Anomalies that can be identified without observing worker movements, they involve interactions between parts, such as incorrect assembly order or incorrect position, indicating a deviation from the expected assembly process; Progress-related anomalies: These anomalies are specific to different assembly stages. They may involve misuse of tools or operations that deviate from the prescribed process at certain production stages. The inventors explain these anomalies and their mapping relationship between industrial production and computer vision (CV) fields in detail, such as Figure 2 shown.

[0094] Existing research on abnormal production activities mainly focuses on predefined abnormal categories or rules. However, in a dynamic production environment, it becomes impractical to predict and label all possible future abnormal activities. Therefore, there is an urgent need for a self-supervised anomaly detection scheme trained only with normal image data. This scheme does not require abnormal data and predefined rules, greatly improving the model's ability to foresee unknown anomalies.

[0095] In addition, traditional research usually treats progress prediction and anomaly detection as independent tasks. However, the inventors believe that anomaly detection can be integrated into the prediction model to stop the stage prediction under abnormal circumstances. Since the anomaly detection criteria may vary depending on the production stage, the model must integrate real-time progress information to achieve accurate and context-related anomaly identification.

[0096] The present disclosure aims to provide a technical solution that can identify impending anomalies using only normal production image data; utilize the inherent characteristics of the production process as self-supervised learning signals so that the model focuses on production-related features amidst a wide range of background element changes; and combine anomaly detection with progress prediction tasks, which not only improves the accuracy of the model in predicting the production stage, but also provides progress-related anomaly predictions.

[0097] Figure 3 A flowchart of a video-based self-supervised production anomaly detection and progress prediction method according to an embodiment of the present disclosure is shown. As shown in the figure, the method includes steps 1 to 6.

[0098] Step 1: Obtain video data of the production process.

[0099] Video data can be collected through existing surveillance cameras in the workshop, without the need to install additional specialized equipment, thus ensuring that normal production processes are not disrupted and no additional hardware costs are added. For example, high-resolution surveillance camera systems already installed in the factory can be used for continuous recording. These cameras usually cover the entire production area and can fully capture every aspect of the production process.

[0100] The data collection process can take into account the actual production situation. For example, data during factory downtime will be excluded to ensure that the collected data truly reflects the production status, thereby improving the quality and relevance of the data.

[0101] In some examples, data collection can cover different types of production processes. For example, in the case of commercial aircraft spoiler production, the collected data includes multiple types of spoiler production processes, covering products of different shapes and sizes, from rectangular to trapezoidal. This diversity of data collection helps improve the generalization ability of the model, enabling it to adapt to different production scenarios.

[0102] Step 2: Use an encoder to map the image features in the video data to the latent space of a Gaussian mixture model (GMM) to obtain latent variables for representing the latent features in the image, wherein the Gaussian components in the GMM correspond one-to-one to the pre-divided production stages.

[0103] Key frame images can be extracted from the video. For example, one frame of image can be extracted from the video stream every three seconds, which can not only ensure that key changes in the production process are captured, but also effectively control the amount of data to balance information integrity and computing efficiency.

[0104] Different from traditional vision-based anomaly detection, the image features of the production process in spatial and temporal dimensions are heterogeneous, such as Figure 4 As shown in Figure 2. For spatial heterogeneity, some locations of the image are relevant to production, while other locations are background features with different importance. For temporal heterogeneity, image features vary in different production steps. Because of this complexity, the inventors believe that a more detailed approach can be used to analyze these features.

[0105] This embodiment assumes that the image feature z generated at a specific progress percentage k∈[0,1] k follows a Gaussian distribution, and there is functional heterogeneity along the process timeline (see Equations 1, 2, and 3).

[0106]

[0107] μ k =F μ (k) (2)

[0108] ∑ k =F Σ (k) (3)

[0109] The inventors conducted a comprehensive study of the manufacturing industry and found that certain stages of the production process (such as inspection or polishing tasks) have little effect on the visual characteristics of components. Based on these observations, the inventors proposed F μ (k) is not a continuously changing function, but a piecewise constant function represented by the separation points y∈{0,…,c}, where k 0 = 0 and k c = 0. Thus, the inventors will determine the separable time point k in an unsupervised manner y The method is defined as a change point detection problem. In some embodiments, further through equation 4 and define the piecewise function F μ (k) and F Σ (k), to associate the mean μ of each Gaussian component in the GMM y and covariance∑ y And production progress k:

[0110] F μ (k) = μ y if k y <k≤k y+1 ,y∈{0,…,n-1} (4)

[0111] FΣ (k)=Σ y if k y <k≤k y+1 ,y∈{0,…,n-1} (5)

[0112] Among them, k∈[0,1] represents the production progress in percentage form, c is the number of pre-divided production stages, and k y is the starting point of the production schedule of the yth production stage, μ y is the mean of the latent variable in the y-th production stage, Σ y is the covariance of the latent variables in the y-th production stage.

[0113] Therefore, it can be considered that the distribution of z is consistent with the multivariate Gaussian mixture distribution, as shown in Equation 6:

[0114]

[0115] The inventors comprehensively considered various factors, especially the sensitivity to outliers, and chose to include the negative log-likelihood loss NLL of latent variables in GMM into the training objective. Unlike the existing methods, the cluster y in this embodiment is predefined by the production schedule k, which is known a priori during the training phase. Therefore, the log-likelihood function in this embodiment is expressed as follows:

[0116]

[0117] In this formula, F μ (k) and F Σ (k) Provide a statistical basis for the model through maximum likelihood estimation (MLE) derivation.

[0118] In some embodiments, the training target negative log-likelihood NLL may be calculated according to Equation 8:

[0119]

[0120] In some embodiments, after each training cycle, the mean μ of each Gaussian component in the GMM may be updated based on the maximum likelihood estimate of the latent variable sample output by the current encoder. y and covariance Σ y , thereby updating F μ (k) and F Σ (k) For example, an iterative algorithm based on expectation maximization (EM) can be used to find the parameter estimates that maximize the likelihood of the observed data and update the GMM model.

[0121] Step 3, the feature map generated before the last convolutional block of the encoder is used to generate a spatial activation map, which is used to extract and represent the attention features related to different production stages in the image.

[0122] like Figure 1b As shown, the variation of abnormal pixels may be much smaller than the normal background variation. Therefore, it is crucial to let the model focus on pixels related to production. Using object classes as weak supervision signals, CAM is proposed to extract pixels related to objects. However, in the actual production process studied in this disclosure, it is difficult or even impossible to annotate object classes because each image may present hundreds of parts in a complex production process.

[0123] Therefore, the inventors proposed a spatial activation map The feature map q generated before the last convolutional block of the encoder can be used to generate a spatial activation map, which is used to extract and represent the attention features related to different production stages in the image. In other words, the progress state y is used as a self-supervisory signal to extract the production-related areas in the image. The idea is that when you want to identify the ongoing process state, the weight of the pixel is large only when it is highly relevant to the process.

[0124] In some embodiments, a multilayer perceptron (MLP) may be used to process a feature map Q to obtain a feature map Q′; a progress activation map may be calculated for the feature map Q′ using a weight matrix f of a fully connected network without bias, where each column of f corresponds to a production stage; and the production stages of the progress activation map may be averaged to generate the spatial activation map.

[0125] Figure 5 A specific example is given. The feature map Q is passed to a convolutional network with parameters θ, generating a map conv(Q,θ) = Q′ with the same shape. A fully connected network (FCN) without bias is then used to classify the process state given the feature map Q′. Represents the weight matrix of FCN. Predicted category vector The predicted category is then calculated and the weights of the MLP are trained by cross entropy loss. In addition, in order to emphasize features related to progress and distinguish embedded features between categories, the inventors further introduced a triple loss function, as described in Equation 9. This function aims to maintain the margin

[0126] Loss trip =max(|z anchor -z positive | 2 -|z anchor -z negative | 2 +m 2 ,0) (9)

[0128] Among them, z anchorrepresents the latent variable of the current image sample, z positive For z anchor The latent variable of image samples belonging to the same production stage, z negative For z anchor Latent variables of image samples belonging to different production stages, m is a predefined margin.

[0129] In some implementations, the progress activation map is calculated according to the following formula:

[0130]

[0131] Among them, PAM y,i,j represents the progress score at the spatial position (i, j) in the progress activation diagram, y represents the y-th production stage, is the transpose of the y-th column of the weight matrix f, corresponding to the y-th production stage, Q′ :,i,j represents the feature vector of feature map Q′ at position (i, j), express and Q′ :,i,j The absolute value is taken after the dot product. The progress activation map maps the importance of each progress category.

[0132] In this embodiment, PAM y,i,j The progress score of each spatial position can be represented, which provides a quantification of the contribution of a specific position to the prediction result. This actually reflects the importance of a specific spatial feature to the overall result. If the absolute value of this score is small, it means that deleting the image feature at this position has little effect on the progress information of the current stage. Therefore, in some embodiments, a spatial activation map can be calculated according to Equation 10, which represents the importance of spatial features in the production process.

[0133]

[0134] This explanatory formula adopted in this embodiment enhances the interpretability of the neural network and makes its operation more transparent.

[0135] Step 4: Use the decoder to reconstruct the latent variables output by the encoder into an image and calculate the spatial activation reconstruction error of the reconstructed image weighted by SAM.

[0136] The spatial activation map can be used as attention weights in both the encoder and the decoder. Specifically, in the encoder, the spatial activation map can be multiplied with Q, and then the optimized features are passed to the subsequent network. In the decoder, the pixel-level reconstruction error (X′-X) is calculated. 2 The spatial activation map is then used as a weight to generate a spatial activation reconstruction error (SARE). In some embodiments, the spatial activation reconstruction error SARE can be calculated according to Equation 11:

[0137] SARE=∑ c ∑ i ∑ j (X′ c,i,j -X c,i,j ) 2 SAM i,j , (11)

[0138] Among them, X′ represents the reconstructed image, X represents the original image, c, i, j represent the channel, height, and width index of the image respectively, (X′ c,i,j -X c,i,j ) 2 Represents the square of the pixel-level reconstruction error between the reconstructed image and the original image at position (c,i,j), SAM i,j Represents the value of the spatial activation map at position (i, j).

[0139] The inventors used SARE as the reconstruction loss for training the decoder. Figure 6 As shown, the reconstruction error caused by background or irrelevant pixels is subtracted in SARE, making the model pay more attention to pixels related to production. In addition, the correlation of SAM with progress categories enhances the interpretability of the network and explains the features that contribute to the model's decision.

[0140] As pointed out by some studies, class activation maps (CAMs) usually only activate the most discriminative parts of an object, rather than the entire object, causing the model to ignore certain critical but insignificant areas in the production process. In some embodiments, random erasure can be introduced during the training process to avoid this phenomenon from appearing in the spatial activation map proposed in this embodiment. Random erasure may include:

[0141] Randomly select an input image with a predetermined probability;

[0142] Randomly select a local area on the selected image;

[0143] Setting the pixel values ​​in the local area to predetermined values ​​to form a partially erased image;

[0144] Model training using partially erased images.

[0145] In some examples, the local area is a rectangular area, and the position of the rectangle (r x ,r y ) and dimension (r w ,r h )satisfy:

[0146] r x ~U(0,w),r y ~U(0,h),

[0147] a~U(a l ,a u ),b~U(b l ,b u ),

[0148]

[0149] Among them, the dimension of the input image is w×h, U() represents uniform distribution, and a l 、a u 、b l 、b u are predefined parameters.

[0150] In some exemplary application examples, a l =0.02,a u =0.33,b l =0.3,b u = 3.3. The erased area can be filled with black pixels, such as Figure 6 shown.

[0151] During the training process, the most discriminative parts may be erased. According to this embodiment, by randomly erasing the image, the model proposed in this embodiment not only learns the most discriminative parts, but also learns all the features that may contribute to the determination process stage.

[0152] Step 5: Train the model based on a comprehensive loss function, which includes the cross entropy loss for production schedule prediction, the triple loss for latent space feature embedding for different production stage feature representations, the negative log-likelihood loss of latent variables in GMM, and the spatial activation reconstruction error.

[0153] As described above, the model proposed in this embodiment uses an encoder-decoder architecture and also includes a spatial activation map branch to form a multi-task learning framework. The encoder maps the image to the latent space of the GMM; the decoder reconstructs the image using the latent variables output by the encoder; the SAM branch uses the feature map generated before the last convolution block of the encoder to generate a spatial activation map, which includes MLP, CNN and FCN processing steps as spatial attention to guide the feature extraction of the encoder. The training process of the model combines multiple loss functions. In some embodiments, the comprehensive loss function is calculated according to the following formula:

[0154] Loss=Loss CE +λ 1 Loss trip +λ 2 NLL+λ 3 SARE,

[0155] Among them, LossCE The cross entropy loss for production progress prediction is used to optimize the progress prediction task; Loss trip is the triple loss of latent space feature embedding, which is used to optimize the model's recognition of different production stages; NLL is the negative log-likelihood loss of the latent variable in GMM, which is used to optimize the feature distribution of the latent space; SARE is the spatial activation reconstruction error, which is used to guide the model about the production-related image area; λ 1 , 2 , 3 is the weight coefficient used to balance different loss terms.

[0156] Figure 6 The model framework according to an exemplary embodiment is shown. The model receives two inputs, one is the original image X, and the other is the image after random erasure. Both input images are processed by the encoder. The encoder consists of multiple convolutional layers and finally outputs a feature map Q and a latent variable z.

[0157] The feature map Q is first processed by MLP, then further processed by CNN, and finally generated by FCN to generate the spatial activation map SAM. The generation process of SAM also involves cross entropy loss (Loss CE ) calculation.

[0158] The decoder receives the latent variable z as input and reconstructs the image X′.

[0159] NLL Loss and Triplet Loss trip ) is calculated based on the latent variable z. The spatial activation reconstruction error (SARE) is obtained by comparing the reconstructed image X′ with the original image X and multiplying it with the spatial activation map.

[0160] The output of the encoder is used for multiple tasks: generating spatial activation maps, computing the negative log-likelihood loss and triplet loss of latent variables in GMM, and image reconstruction.

[0161] The SAM branch is used both to guide the encoder’s attention feature extraction and to compute the spatial activation reconstruction error.

[0162] In the figure, the black arrows indicate the paths with backpropagation, and the yellow arrows indicate the paths without backpropagation.

[0163] Regarding the parameter update of the Gaussian mixture model (GMM) distribution, especially F μ (k) and F ∑(k), some studies suggest jointly training these parameters to improve performance, but the inventors have conducted in-depth research and extensive verification on production process data sets and believe that the huge variability of normal data distribution in the production process will cause instability in the joint training process. Therefore, as described above, in some embodiments, F can be updated based on the maximum likelihood estimate of the latent variable samples output by the current encoder. μ (k) and F Σ (k), for example, using an iterative algorithm based on expectation maximization (EM). In each training cycle, F is optimized using the maximum likelihood estimate derived from the latent variable z. μ (k) and F Σ (k), thereby improving the stability and accuracy of the model by adapting to various changes observed in the data, ensuring more robust and reliable anomaly detection and progress prediction.

[0164] Step 6: Use the encoder of the trained model to process the real-time collected video data, detect production anomalies based on the obtained latent variables, and predict the production stage.

[0165] The image features X generated at time t in the production process are transformed using the trained encoder t Mapped to a Gaussian mixture latent variable z, which follows the GMM distribution:

[0166]

[0167] where α y =k y+1 -k y Represents the Gaussian component weight of production stage y, which is inferred from the average time cost of stage y in the training video. The calculated p(z) represents the probability of observing the feature value under normal assembly conditions. Therefore, in some embodiments, the inventor uses p(z) calculated according to equation 16 as an abnormality score indicator, and when p(z) is lower than a preset abnormality detection threshold, the corresponding image is judged to be an abnormal image. In some studies, reconstruction error is used as an abnormality score indicator, but the inventors have found that the presence of dynamic background in the production process may make it difficult to reconstruct even normal process images. In addition, the marginal difference between abnormal images and normal images means that abnormal images may only show minimal reconstruction losses. In subsequent application examples, the performance of AE+SARE further illustrates this phenomenon.

[0168] According to this embodiment, by introducing the spatial activation map SAM and the progress-related GMM to guide the model to focus on production-related features, the anomaly detection performance can be significantly improved.

[0169] In some embodiments, the progress prediction may be optimized by utilizing the abnormal detection results, including:

[0170] Use the trained encoder to map the image in the real-time collected video data to the latent space of GMM to obtain the latent variable z;

[0171] Predict the current production stage based on latent variable z

[0172] Production phase of calculation forecast Reliability index

[0173] when When ∈ is the preset reliability detection threshold, the predicted production stage is determined Unreliable, the prediction result y of the previous moment t-1 As the current prediction result, y t =y t-1 ;

[0174] when When the current predicted production stage As the current prediction result,

[0175] According to this embodiment, given a schedule forecast The posterior probability of p(z) generated by the production stage y to quantify the reliability of the prediction. The relevant predictions are discarded and wait for a later reliable prediction.

[0176] Figure 7 A schematic diagram of a flow chart for combining anomaly detection and progress prediction according to an exemplary embodiment of the present disclosure is shown.

[0177] In summary, the self-supervised production anomaly detection and progress prediction method based on high-flow video proposed in this embodiment combines the encoder-decoder structure with the Gaussian mixture model and the spatial activation map to achieve comprehensive monitoring of the production process; by introducing the autoencoder process probability embedding technology, the spatiotemporal heterogeneity of the production process is effectively captured, and the background interference is minimized by using the spatial activation map, so that the model can be trained using only normal production data, solving the problem of scarce abnormal data; in addition, the method integrates the anomaly detection and progress prediction tasks into a unified framework, which not only improves the accuracy and reliability of monitoring, but also enhances the model's adaptability to complex production environments.

[0178] The video-based self-supervised production anomaly detection and progress prediction method proposed in this embodiment is evaluated on two different datasets. The first dataset covers the water valve production process, where assembly parts are the main focus of the image data; the second dataset represents the actual production scenario, covering tasks such as assembly operations, lamination, coating, and vacuum extraction, involving mechanical parts and flexible composite materials.

[0179] The project test environment was configured with Python 3.8 and CentOS 7.9. To ensure the performance and efficiency of the training process, a single A100 GPU or dual A100 GPU setup was used. To ensure fairness in the evaluation, all models used the ResNet50 architecture as their backbone network.

[0180] A. Water valve production

[0181] This example demonstrates the performance of the proposed model on a single-station water valve production process dataset. The production phase is visualized as follows: Figure 8a to Figure 8i As shown, including: Figure 8a Assemble the intermediate valve body; Figure 8b Tighten the screws; Figure 8c Install the sealing ring; Figure 8d Assemble the middle part of the valve; Figure 8e Assemble the upper part of the valve; Figure 8f Assemble the rocker arm; Figure 8g Assemble the parts onto the rocker arm; Figure 8h Complete the lower clamp assembly of the water shut-off valve; finally Figure 8i The clamp is assembled. The whole process lasts about 3000 time frames per cycle.

[0182] The training set contains 11 assembly videos, and 29,529 images are extracted for training. The test set contains 31 assembly videos, which are annotated every ten time frames, with a total of 700 abnormal data points and 5,790 normal data points. Figure 9a to Figure 9e Shown, including Figure 9a The foreign body abnormality shown, Figure 9d and Fig.9e The worker shown is operating abnormally. Figure 9b Progress-related exceptions shown and Fig.9c and Figure 9d The part interaction is abnormal. It should be noted that some abnormal images may show multiple abnormalities at the same time. This example focuses on detecting whether there is an abnormality, not its type.

[0183] Visualization of model features such as Fig.10 As shown in Figure 2. Light to dark colors correspond to the production stages of the images in the video sequence, with darker colors indicating later stages of the assembly process. Fig.10As can be seen in Fig. 1, each process category is clustered together, whether in the training set or the test set. The visualization of SAM and reconstruction X′ is shown in Fig.11 shown. Fig.11 The visualization results clearly show that SAM mainly focuses on the position of parts being assembled, to be assembled, or the position of hands performing production-related operations.

[0184] B. Commercial aircraft spoiler production

[0185] In this example, the performance of the model in the actual production of a commercial aircraft spoiler is evaluated. This workstation was chosen because it involves mechanical parts and flexible materials. The finished spoiler is shown in Figure 1. Fig.12 As shown. The production process is complex and diverse, including preliminary pre-assembly work next to the tool, primer drying, inspection of auxiliary materials, lamination of composite materials and application of various color films. Subsequent stages include assembly of rigid structures, covering with isolation films, as well as bagging and leak testing, and finally curing in the autoclave. This example focuses on the spoiler production stage before the autoclave process, which lasts about 3 hours and is characterized by mixed lines and complete manual manufacturing. A certain stage involves waiting and inspection, and there is no obvious change in the appearance of the workpiece. In addition, the position of the tool changes within the camera's shooting range within each cycle. Visual image data is collected by workshop monitoring cameras to ensure no interruption of production and no additional hardware costs.

[0186] The dataset was carefully compiled based on three weeks of surveillance footage from the factory. By extracting one frame every three seconds, a total of approximately 46,000 images were collected for this example. This dataset was systematically divided into different units: six for training and seven for testing, as described in Table 1. The training set includes three different types of spoilers, while the testing and validation sets include spoilers from the training set and one other untrained spoiler type. The spoilers vary in shape (from rectangular to trapezoidal), size, and specific structure used for wing mounting. Despite these differences, the production process remains consistent between different spoiler types. Data during factory downtime was excluded based on the facility's operating hours.

[0187] Table 1 Description of the commercial aircraft spoiler production dataset

[0188]

[0189] To evaluate the effectiveness of anomaly detection, the test data was annotated using the collective expertise of multiple factory personnel and managers. A total of 23 different anomaly conditions and 1,847 normal conditions were annotated. Anomalies included pre-assembly parts getting stuck, workers sitting at tool stations during the active assembly phase, and tools and equipment being moved prematurely before the assembly cycle is complete. Features are visualized as Fig.13 As shown, the explanation and Fig.10 same.

[0190] C. Benchmark comparison

[0191] Here, the video-based self-supervised production anomaly detection and progress prediction method proposed in the present disclosure is compared with three well-known anomaly detection algorithms: autoencoder (AE), which uses reconstruction error as anomaly score; variational autoencoder (VAE), in which reconstruction probability is used as anomaly indicator; and DaGMM. In addition, the inventors also tested the use of SARE instead of AE's reconstruction loss in equation 11 (the AE+SARE column in the table) to demonstrate the effectiveness of SAM in focusing on production-related features.

[0192] Table 2 Quantitative results of anomaly detection in water valve production

[0193]

[0194] Table 3 Quantitative results of anomaly detection in commercial aircraft spoiler production

[0195] The Disclosure AE AE+SARE VAE DaGMM AUC 80.45±6.39 40.22±7.93 50.37±2.95 41.79±0.94 52.51±10.93 EER 27.24±4.99 57.05±8.27 51.63±5.46 57.24±1.78 48.41±5.97

[0196] In the example, the inventors used two widely recognized indicators: the area under the receiver operating characteristic curve (AUC) and the equal error rate (EER) to evaluate the effectiveness of various methods. The quantitative results shown in Tables 2 and 3 show that the method proposed in the present disclosure surpasses all benchmark models in overall anomaly detection performance. In particular, it can be observed that AE enhanced with SARE in Equation 11 (AE+SARE) performs better than the other three benchmark models. This shows that SAM is very effective in guiding the model to focus on production-related features. However, its detection performance is still significantly inferior to the technical solution proposed in the present disclosure. The main reason is that although SAM itself has highlighted production-related features, the reconstruction of normal images is still challenging. This is mainly because in a typical production environment, the number of available training samples is usually limited and may not cover all normal conditions present in the test set. Therefore, the decoder part of the model is difficult to learn effectively, resulting in a large reconstruction loss of normal images (such as Fig.11 The encoder part, however, only needs to extract relevant information from the image to reconstruct production-related features, which is a relatively simple task. Therefore, analyzing the probability of feature values ​​rather than reconstruction loss can circumvent the problems associated with poorly trained decoders.

[0197] In addition, the method proposed in this disclosure significantly outperforms all baseline models in detecting specific anomaly subclasses (including foreign objects, worker operations, part interactions, and progress-related anomalies), as shown in Table 2.

[0198] D. Ablation studies

[0199] In order to rigorously evaluate the model framework constructed by the method proposed in this disclosure (such as Figure 6 The inventors also conducted a series of ablation studies to demonstrate the indispensability of each component in the Figure 6 A variation of the model shown, including the replacement of SAM with CAM, is denoted APPE CAM ; Remove SAM, denoted as APPE-noSAM; Remove NLL, denoted as APPE-noNLL; Remove random erasure, denoted as APPE-noErase. Through these modifications, the contribution of each component to the overall performance of the model proposed in the present disclosure can be shown.

[0200] Table 4 Quantitative results of ablation study

[0201]

[0202]

[0203] The results of the ablation study are shown in Table 4. The results in the second and third columns show that the SAM proposed in the present disclosure is superior to the alternative solution of using CAM. This superiority is due to the fact that CAM only focuses on determining the features of the current stage, while SAM can capture all production-related features. The SAM proposed in the present disclosure not only avoids the loss of important information, but also filters out features that are not related to production. In addition, the overall performance of APPE-noNLL and APPE-noSAM is lower than that of the proposed method Figure 6 The model shown further demonstrates the importance of the SAM and progress-related GMM embeddings proposed in the present disclosure. In addition, it can be observed that the impact of removing NLL on the performance is much smaller than removing SAM. This is because after removing NLL, the model becomes metric learning without considering the correlation matrix of the feature vector. The progress-related attention SAM plays a key role in anomaly detection with relatively small information loss. In addition, the model proposed in the present disclosure outperforms APPE-noErase, proving that the random erasure process proposed in some embodiments has a beneficial effect in improving the overall performance of the model.

[0204] E. Combining anomaly detection with progress prediction

[0205] In addition to anomaly detection, the model proposed in the present disclosure also outputs a prediction of the current production stage of the production process, denoted as y. Abnormal operation may have an adverse effect on the accuracy of production stage identification. Therefore, as discussed above, according to some embodiments of the present disclosure, the probability of the production stage y may be based on the probability of the production stage y. Post-process the recognition results. You can select a video describing normal operation as a validation set and calculate The 99% quantile is used to determine the threshold.

[0206] To evaluate the effectiveness of the model in predicting task progress, the inventors used two key metrics. The primary metric is classification accuracy, which directly measures the model’s ability to correctly identify the current stage of the production process; the second metric is the mean absolute percentage error (MAPE) over time, as shown in Equation 17:

[0207]

[0208] in Indicates the stage in the video The formula calculates the midpoint of the predicted category As a prediction value, schedule forecasting is considered as a continuous process. This approach not only evaluates the likelihood of model prediction accuracy, but also quantitatively measures the model's prediction deviation by comparing the prediction value with the actual occurrence time. This dual evaluation strategy effectively captures the accuracy of the precision and time forecasts that distinguish different production stages.

[0209] Table 5 Progress prediction accuracy before and after anomaly detection

[0210]

[0211] The comparison results before and after the combination are shown in Table 5. After the combination, the accuracy and mean absolute percentage error (MAPE) of the two datasets are reduced.

[0212] In summary, through the comparative analysis of two different datasets and the benchmark model in Table 2 and Table 3, it is clearly shown that the autoencoder process probability embedding proposed in this disclosure is effective in detecting Figure 2 The results of the ablation study are shown in Table 4, which further reinforces the effectiveness of this disclosure and highlights the key contributions of each component in the model. The video-based self-supervised production anomaly detection and progress prediction method proposed in this disclosure is able to cleverly address the dual challenges of anomaly detection and progress prediction within a single, unified framework. This combination not only halves the computational complexity of traditionally handling these tasks independently, but also synergistically enhances the performance of both. In addition, the spatial activation map (SAM) also provides excellent visual insights, such as Fig.11 shown.

[0213] According to an embodiment of the present disclosure, a video-based self-supervised production anomaly detection and progress prediction device is also proposed, the device comprising:

[0214] A video acquisition unit, used to acquire video data of the production process;

[0215] A latent feature extraction unit, used for mapping the image features in the video data to the latent space of a Gaussian mixture model (GMM) by using an encoder, to obtain latent variables for representing the latent features in the image, wherein the Gaussian components in the GMM correspond one-to-one to the pre-divided production stages;

[0216] A SAM generation unit, for generating a spatial activation map using a feature map generated before the last convolutional block of the encoder, wherein the spatial activation map is used to extract and represent attention features related to different production stages in the image;

[0217] An image reconstruction unit, configured to reconstruct the latent variables output by the encoder into an image using the decoder, and calculate a spatial activation reconstruction error of the reconstructed image weighted by the spatial activation map;

[0218] The model is trained based on a comprehensive loss function that includes the cross entropy loss for production schedule prediction, the triple loss for latent space feature embedding for different production stage feature representations, the negative log-likelihood loss of latent variables in GMM, and the spatial activation reconstruction error:

[0219] The anomaly detection and progress prediction unit is used to process the real-time collected video data using the encoder of the trained model, perform production anomaly detection and predict the production stage based on the obtained latent variables.

[0220] For other details and beneficial effects of this embodiment, please refer to the above related introduction, which will not be repeated here.

[0221] The disclosed embodiments may be applied to a computer system / server that may operate with numerous other general or special computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with a computer system / server include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, and the like.

[0222] Computer systems / servers may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. In general, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

[0223] Fig.14 An electronic device provided for at least one embodiment of the present disclosure includes a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the video-based self-supervised production anomaly detection and progress prediction method described in any embodiment or implementation of the present disclosure when executing the computer instructions.

[0224] At least one embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video-based self-supervised production anomaly detection and progress prediction method described in any embodiment or implementation of the present disclosure.

[0225] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0226] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0227] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0228] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0229] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0230] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0231] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0232] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0233] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0234] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0235] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A video-based self-supervised production anomaly detection and progress prediction method, characterized in that: The method comprises: Acquire video data of the production process; Mapping the image features in the video data to a latent space of a Gaussian mixture model (GMM) using an encoder to obtain latent variables for representing the latent features in the image, wherein the Gaussian components in the GMM correspond one-to-one to the pre-divided production stages; Generate a spatial activation map using the feature map generated before the last convolutional block of the encoder, wherein the spatial activation map is used to extract and represent attention features related to different production stages in the image; Reconstruct the latent variables output by the encoder into an image using the decoder, and calculate the spatial activation reconstruction error of the reconstructed image weighted by the spatial activation map; The model is trained based on a comprehensive loss function that includes the cross entropy loss for production schedule prediction, the triple loss for latent space feature embedding for different production stage feature representations, the negative log-likelihood loss of latent variables in GMM, and the spatial activation reconstruction error: The encoder of the trained model is used to process the real-time collected video data, and production anomalies are detected and the production stage is predicted based on the obtained latent variables.

2. The method according to claim 1, characterized in that The mean μ of each Gaussian component in the GMM y and covariance∑ y By the following piecewise function F μ (k) and F Σ (k) is associated with the continuous production progress k: F μ (k)=μ y if k y <k≤k y+1 ,y∈{0,…,c-1}, F Σ (k)=∑ y if k y <k≤k y+1 ,y∈{0,...,c-1}, Among them, k∈[0,1] represents the production progress in percentage form, c is the number of pre-divided production stages, and k y is the starting point of the production schedule of the yth production stage, μ y is the mean of the latent variable in the y-th production stage, ∑ y is the covariance of the latent variables in the y-th production stage; After each training cycle, the mean μ of each Gaussian component in the GMM is updated based on the maximum likelihood estimate of the latent variable samples output by the current encoder. y and covariance∑ y .

3. The method according to claim 1, characterized in that: Generate a spatial activation map using the feature map generated before the last convolutional block of the encoder, including: Use a multi-layer perceptron (MLP) to process the feature map Q generated before the last convolutional block of the encoder to obtain the feature map Q′; Using the weight matrix f of the fully connected network without bias, calculate the progress activation map for the feature map Q′, where each column of f corresponds to a production stage; The spatial activation map is generated by averaging each production stage of the progress activation map.

4. The method according to claim 3, characterized in that Calculating the progress activation graph includes: The progress activation graph is calculated according to the following formula: Among them, PAM y，i，j represents the progress score at the spatial position (i, j) in the progress activation diagram, y represents the y-th production stage, is the transpose of the y-th column of the weight matrix f, corresponding to the y-th production stage, Q′ ：，i，j represents the feature vector of feature map Q′ at position (i, j), express and Q′ ：，i，j Take the absolute value after dot multiplication.

5. The method according to claim 1, characterized in that Calculate the spatial activation reconstruction error of the reconstructed image weighted by the spatial activation map, including: The spatial activation reconstruction error SARE is calculated according to the following formula: SALT=∑ c ∑ i ∑ j (X′ c，i，j -X c，i，j ) 2 Sat. i，j , Among them, X′ represents the reconstructed image, X represents the original image, c, i, j represent the channel, height, and width index of the image respectively, (X′ c，i，j -X c，i，j ) 2 SAM represents the square of the pixel-level reconstruction error between the reconstructed image and the original image at position (c, i, j). i，j Represents the value of the spatial activation map at position (i, j).

6. The method according to claim 1, characterized in that The comprehensive loss function is calculated according to the following formula: Loss=Loss CE +λ1Loss trip +λ2NLL+λ3SARE, Among them, Loss CE The cross entropy loss for production progress prediction is used to optimize the progress prediction task; Loss trip is the triple loss of latent space feature embedding, which is used to optimize the model's recognition of different production stages; NLL is the negative log-likelihood loss of the latent variable in GMM, which is used to optimize the feature distribution of the latent space; SARE is the spatial activation reconstruction error, which is used to guide the model about the production-related image areas; λ1, λ2, and λ3 are weight coefficients used to balance different loss terms.

7. The method according to claim 1 or 6, characterized in that: The triple loss Loss is calculated according to the following formula trip : Loss trip =max(|z anchor -z positive | 2 -|z anchor -z negative | 2 +m 2 ,0), Among them, z anchor represents the latent variable of the current image sample, z positive For z anchor The latent variable of image samples belonging to the same production stage, z negative For z anchor Latent variables of image samples belonging to different production stages, m is a predefined margin.

8. The method according to claim 1, characterized in that The method also includes introducing random erasures during the training process, including: Randomly select an input image with a predetermined probability; Randomly select a local area on the selected image; Setting the pixel values ​​in the local area to predetermined values ​​to form a partially erased image; Model training using partially erased images.

9. The method according to claim 8, characterized in that The local area is a rectangular area, and the position of the rectangle (r x , r y ) and dimension (r w , r h )satisfy: r x ~U(0,w),r y ~U(0,h), a~U(a l ,a u ),b~U(b l ,b u ), Among them, the dimension of the input image is w×h, U( ) represents uniform distribution, and a l 、a u 、b l 、b u are predefined parameters.

10. The method according to claim 1, characterized in that The production anomaly detection includes: Use the trained encoder to map the image in the real-time collected video data to the latent space of GMM to obtain the latent variable z; The probability density p(z) of the latent variable z in the Gaussian mixture model is calculated based on the following formula: Where c is the number of production stages, α y is the weight of the y-th production stage, The mean is μ y , the covariance matrix is ​​∑ y The probability density function of the multivariate Gaussian distribution; p(z) is used as the anomaly score indicator. When p(z) is lower than the preset anomaly detection threshold, the corresponding image is judged to be an abnormal image.

11. The method according to claim 1, characterized in that: The steps of progress prediction include: Use the trained encoder to map the image in the real-time collected video data to the latent space of GMM to obtain the latent variable z; Predict the current production stage based on latent variable z Production phase of calculation forecast Reliability index when When ∈ is the preset reliability detection threshold, the predicted production stage is determined Unreliable, the prediction result y of the previous moment t-1 As the current prediction result, y t =y t-1 ; when When the current predicted production stage As the current prediction result, 12. A video-based self-supervised production anomaly detection and progress prediction device, characterized in that: The device comprises: A video acquisition unit, used to acquire video data of the production process; A latent feature extraction unit, used for mapping the image features in the video data to the latent space of a Gaussian mixture model (GMM) by using an encoder, to obtain latent variables for representing the latent features in the image, wherein the Gaussian components in the GMM correspond one-to-one to the pre-divided production stages; The SAM generation unit is used to generate a spatial activation map using the feature map generated before the last convolution block of the encoder, and the spatial activation map is used to extract and represent the attention features related to different production stages in the image: An image reconstruction unit, configured to reconstruct the latent variables output by the encoder into an image using the decoder, and calculate a spatial activation reconstruction error of the reconstructed image weighted by the spatial activation map; The model is trained based on a comprehensive loss function that includes the cross entropy loss for production schedule prediction, the triple loss for latent space feature embedding for different production stage feature representations, the negative log-likelihood loss of latent variables in GMM, and the spatial activation reconstruction error: The anomaly detection and progress prediction unit is used to process the real-time collected video data using the encoder of the trained model, perform production anomaly detection and predict the production stage based on the obtained latent variables.

13. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the method according to any one of claims 1 to 11 when executing the computer instructions.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images