Zero-shot standard operating procedure generation method and apparatus, device, and storage medium

Abstract

Disclosed are a zero-shot standard operating procedure generation method and apparatus, a device, and a storage medium. The method comprises: acquiring a plurality of production videos, and using a vision-language model to extract image features of frames in each video; by means of multi-video collaborative processing and dynamic similarity filtering, identifying standard operations in a target video and filtering out irrelevant frames; using a cosine-energy change-point detection algorithm to detect operation boundaries from the filtered frames, and segmenting the target video into a plurality of operation steps; calculating a standard time for each operation step; inputting image features corresponding to each operation step and prompt information into a vision-language model, and generating a textual description for each step; and generating a standard operating procedure document on the basis of the standard times and the textual descriptions. The present disclosure does not require domain-specific annotated data, and enables automatic and efficient generation of accurate standard operating procedures. The present disclosure is applicable to various production environments, particularly frequently changing manufacturing processes.

Description

Zero-sample standard operating procedure generation method and device, equipment and storage medium TECHNICAL FIELD

[0001] The present disclosure relates to the field of advanced manufacturing, and in particular to a method, device, equipment and storage medium for zero-sample standard operating procedure generation. BACKGROUND

[0002] A standard operating procedure (SOP) is a series of repeatable steps designed to perform routine tasks, including the standard time required for each step and the standardized operation description of each step. As a key document, SOP aims to reduce work pressure and errors, and is the basis of production management, especially in the industrial field. The generation of SOP is based on predetermined standards and actual production scenarios or experience to create a document describing the operation process.

[0003] The generation of SOP mainly involves two key steps: determining the process stage and describing the specific operation of each step. The main goal of determining the process stage is to identify the standardized operation steps in production. Traditional methods usually rely on predefined component assembly sequences or subjective classification, both of which require a large amount of manpower and are prone to bias. SUMMARY

[0004] The present disclosure proposes a zero-sample standard operating procedure generation method, which can automatically identify frames related to production in production videos, automatically identify key moments (i.e. operation boundaries) for distinguishing different operation steps, and automatically generate a standard operating procedure document that describes production steps in detail without human annotation. The present disclosure also proposes a corresponding device, electronic equipment and computer readable storage medium.

[0005] According to one embodiment of the present disclosure, a zero-shot standard operation procedure generation method is provided, comprising: for each production video frame in the obtained plurality of production videos, extracting a feature vector of the frame in the production video using a pre-trained first visual language model; selecting one of the plurality of production videos as a target video and the other production videos in the plurality of production videos as reference videos of the target video; based on the feature vectors of the frames in the target video and the feature vectors of the frames in each of the reference videos, filtering out frames irrelevant to production from the frames of the target video to obtain a filtered target video; based on cosine similarity distance and non-parametric permutation test, detecting operation boundaries of a plurality of operation steps from the frames of the filtered target video, and segmenting the target video into the plurality of operation steps according to the operation boundaries; calculating standard time of each operation step in the plurality of operation steps; inputting image features corresponding to the plurality of operation steps and pre-defined prompt information into a pre-trained second visual language model to generate a text description of each operation step in the plurality of operation steps; and generating a production procedure document based on the standard time of each operation step in the plurality of operation steps and the text description.

[0006] In some embodiments, based on the feature vectors of the frames in the target video and the feature vectors of the frames in each of the reference videos, filtering out frames irrelevant to production from the frames of the target video comprises: assigning each frame in the target video to a different cluster; for each frame in each of the reference videos, taking the frame as a reference frame, determining a target frame in the target video matching the reference frame based on the similarity between the feature vector of the reference frame and the feature vectors of each frame in the target video, and assigning the reference frame as an associated frame to the cluster corresponding to the matching target frame; for each frame in the target video, if there is at least one associated frame in the cluster corresponding to the frame, judging that the frame is related to production, if there is no associated frame in the cluster corresponding to the frame, judging that the frame is irrelevant to production, and removing the frame from the target video.

[0007] In some embodiments, based on the similarity between the feature vector of the reference frame and the feature vectors of each frame in the target video, determining a target frame in the target video matching the reference frame comprises: establishing a value function dp[k,t] for dynamic programming: dp[k,t] = min(dp[k,t-1]+d(z i,t-1 ,z 0,k ),dp[k-1][t]+r(i,t,k),dp[k-1][t- 1]+d(z i,t ,z 0,k )),

[0008] wherein k represents a cluster index, t represents a frame index of the reference frame in the reference video, z i,t represents a feature vector of the t-th frame of the reference video i, z 0,k represents a feature vector of a frame of the target video assigned to the k-th cluster, d(, ) represents a cosine similarity distance between two feature vectors, r(i, t, k) is a regularization term, and it is set that dp[0, 0] = 0; for the t-th frame of the reference video i, the value function dp[k, t] is calculated by traversing all values of k, and the frame of the target video corresponding to the cluster index that minimizes the value function dp[k, t] is determined as the frame of the target video matching the t-th frame of the reference video i.

[0009] In some embodiments, for each frame in the target video, if there is at least one associated frame in the cluster corresponding to the frame, it is determined that the frame is production-related, and if there is no associated frame in the cluster corresponding to the frame, it is determined that the frame is production-unrelated, including: if for any reference video i, the following condition is not met then it is determined that the frame of the target video corresponding to the k-th cluster is production-related, wherein l i represents the total number of frames of the reference video i, x i,t,k is a binary variable, x i,t,k = 1 when the t-th frame of the reference video i is assigned to the k-th cluster, and x i,t,k = 0 when the t-th frame of the reference video i is not assigned to the k-th cluster; if there is a reference video i that satisfies then it is determined that the frame of the target video corresponding to the k-th cluster is production-unrelated.

[0010] In some embodiments, based on the cosine similarity distance and the non-parametric permutation test, an operation boundary of a plurality of operation steps is detected from the filtered frames of the target video, including: dividing the filtered frames of the target video into a plurality of non-overlapping time windows; within each time window of the plurality of time windows, calculating a sample energy statistic of each candidate change point based on the following formula

[0011] wherein k c is a candidate change point position, z a and z b respectively represent feature vector sequences of the two time periods before and after the candidate change point k c , T is the total number of frames in the current analysis window, and d(z i , z j ) is a feature vector z i and z ja cosine similarity distance between the sample energy statistics of the two adjacent time windows; identifying, as a candidate boundary point of the time window, a candidate change point in the time window with the largest calculated sample energy statistics; performing a non-parametric permutation test on each candidate boundary point to calculate a statistical significance evaluation value of each candidate boundary point; and determining the operation boundary based on the statistical significance evaluation value of each candidate boundary point.

[0012] In some embodiments, the method further comprises: before detecting the operation boundaries of the plurality of operation steps, obtaining a minimum operation time of the production process based on a method-time measurement (MTM), and determining a length of each time window in the plurality of time windows based on the minimum operation time.

[0013] In some embodiments, calculating the standard time of each operation step in the plurality of operation steps comprises: for each operation step in the plurality of operation steps, identifying a time period corresponding to the operation step in the plurality of production videos; calculating a duration of the operation step in each production video in the plurality of production videos; and calculating an average duration of the operation step based on the duration of the operation step in each production video in the plurality of production videos as the standard time of the operation step.

[0014] In some embodiments, the pre-trained first visual language model takes a CLIP model as a visual backbone network, and / or the pre-trained second visual language model takes a CLIP model as a visual backbone network.

[0015] According to one embodiment of the present disclosure, a zero-shot SOP generation device is provided, comprising: a video acquisition unit configured to, for each production video frame in a plurality of acquired production videos, extract a feature vector of the frame in the production video using a pre-trained first visual language model; a dynamic similarity filtering unit configured to select one of the plurality of production videos as a target video and select other production videos in the plurality of production videos as reference videos of the target video; filter out frames irrelevant to production from frames of the target video based on the feature vector of the frame in the target video and the feature vector of the frame in each of the reference videos, to obtain a filtered target video; a cosine energy change point detection unit configured to detect operation boundaries of a plurality of operation steps from the frames of the filtered target video based on cosine similarity distance and non-parametric permutation test, and segment the target video into the plurality of operation steps according to the operation boundaries; a standard time calculation unit configured to calculate a standard time of each operation step in the plurality of operation steps; a text description winning unit configured to input image features corresponding to the plurality of operation steps and pre-defined prompt information into a pre-trained second visual language model to generate a text description of each operation step in the plurality of operation steps; and an SOP generation unit configured to generate a production program document based on the standard time of each operation step in the plurality of operation steps and the text description.

[0016] According to one embodiment of the present disclosure, an electronic device is provided, the device comprising a memory for storing computer instructions executable on a processor, and a processor for implementing the method of any one of the above when executing the computer instructions.

[0017] According to one embodiment of the present disclosure, a computer-readable storage medium is provided, having stored thereon a computer program executable by a processor to implement the method of any one of the above.

[0018] The zero-shot SOP generation scheme based on the pre-trained visual language model has multiple beneficial effects. First, by utilizing the pre-trained visual language model and innovative algorithms, automatic analysis of production videos and generation of standard operation procedures are realized without the need for labeled data or predefined operation templates in specific fields. This zero-shot learning method significantly improves the efficiency and adaptability of SOP generation, enabling it to be quickly applied to different production environments. Second, by utilizing the Dynamic Similarity Filtering (DSF) algorithm to effectively identify and filter frames unrelated to production, the accuracy of the generated SOP is improved. In addition, by utilizing the Cosine-Energy Change-point Detection (CECD) algorithm to accurately locate the operation boundaries, the division of operation steps is more reasonable. Furthermore, since the standard time for each operation step can be automatically calculated and detailed text descriptions can be generated, the need for human participation is greatly reduced, and the influence of subjective bias is reduced. The automated and accurate SOP generation scheme according to the present disclosure not only improves the efficiency of production management, but also provides a flexible solution for rapidly changing manufacturing environments, which is of great significance for improving product quality, reducing production costs, and optimizing production processes. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present specification and serve to explain the principles of the present specification.

[0020] FIG. 1 illustrates a general concept schematic diagram of a standard operation procedure generation process according to an example embodiment of the present disclosure.

[0021] FIG. 2 illustrates a flowchart of a zero-shot standard operation procedure generation method according to an embodiment of the present disclosure.

[0022] FIG. 3 illustrates production progress changes, non-progress static changes, and non-progress dynamic changes presented by different frames in a certain production video.

[0023] FIG. 4 illustrates a process schematic diagram of dynamic similarity filtering and cosine-energy change-point detection according to an example embodiment of the present disclosure.

[0024] FIG. 5 illustrates a process schematic diagram of generating text descriptions according to an example embodiment of the present disclosure.

[0025] FIG. 6 illustrates a certain visual water valve production process and examples of production-unrelated or abnormal operations to be filtered out.

[0026] FIG. 7 shows an example of partially ground truth SOP and predicted SOP for visualizing the production process of a water valve.

[0027] FIG. 8a shows a spoiler of a certain commercial aircraft; FIG. 8b shows a schematic diagram of the production process of the spoiler of the commercial aircraft.

[0028] FIG. 9 shows an example of partially ground truth SOP and predicted SOP for the production process of the spoiler of the commercial aircraft.

[0029] FIG. 10 shows an example of visualized filtering and segmentation results of a certain water valve production video.

[0030] FIG. 11 is a schematic diagram of the structure of an electronic device according to at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, like reference numerals refer to like elements, unless the context clearly dictates otherwise. The following description of exemplary embodiments is not representative of all possible embodiments consistent with the present disclosure. Rather, it is merely an example of apparatus and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] A standard operating procedure (SOP) is a series of repeatable steps designed to perform routine tasks, including the standard time required for each step and the standardized operation description of each step. As a key document, SOP aims to reduce work pressure, reduce errors, and is the basis of production management, especially in the industrial field. The generation of SOP is based on predetermined standards and actual production scenarios or experience to create a document describing the operation process.

[0033] The generation of SOP mainly involves two key steps: determining the process stage and describing the specific operation of each step. The main goal of determining the process stage is to identify the standardized operation steps in production. Traditional methods usually rely on predefined component assembly sequences or subjective classification, both of which require a large amount of manpower and are prone to bias. Therefore, the demand for automated segmentation of production stages and accurate annotation through high-throughput video analysis becomes particularly urgent.

[0034] While obtaining accurate and up-to-date SOPs is of great value, it still faces many challenges due to the uncertainties in production processes, non-standardized workflows, and irrelevant operations. Traditional SOP generation methods are usually labor-intensive, for example, using standardized work combination tables to manually record operations occurring in the production process. However, these methods are inherently time-consuming and labor-intensive, and cannot be automatically updated when production processes are adjusted, leading to discrepancies between recorded operation data and actual practice. This problem is particularly prominent in industries that frequently make design changes and process optimizations, such as the aerospace manufacturing industry.

[0035] Advances in the field of computer vision have opened up new possibilities for automatically obtaining information from industrial videos. For example, object detection algorithms can be used to identify key elements and actions in videos. In addition, the emergence of large visual language models has made it possible to generate zero-shot image and video captions. However, production processes often contain many operations unrelated to actual manufacturing, such as workers taking breaks or picking up incorrect parts. Existing algorithms have difficulty effectively distinguishing between frames containing production-related operations and frames containing production-unrelated operations. In addition, simple frame-by-frame object detection or general video summaries generated by large language models cannot accurately determine the standard time required for each operation step, nor can they generate sufficiently detailed SOPs.

[0036] Some methods in the related art attempt to automate the SOP generation process, but they often require a large amount of labeled data or pre-trained models specific to a particular field. This dependency limits the flexibility and scope of these methods. Therefore, it is particularly important to develop a "zero-shot" method that can generate accurate SOPs based solely on raw production videos without relying on specific field-labeled data.

[0037] The following first briefly introduces the overall inventive concept of the present disclosure. Fig. 1 illustrates an exemplary overall structure of the zero-shot standard operation procedure generation method proposed by the present disclosure, taking a certain operation as an example.

[0038] As shown in Fig. 1, a plurality of production videos are first processed by an encoder. Production videos refer to videos of the production process. The encoder uses a pre-trained visual language model, such as the CLIP (Contrastive Language-Image Pretraining) model, to extract feature vectors from each frame of each video. These feature vectors capture visual and semantic information in the frames, laying the foundation for subsequent analysis. Next, target video frames can be compared with reference video frames using, for example, the Dynamic Similarity Filtering (DSF) algorithm proposed by the present disclosure to identify production-related frames in the target video frames and filter out production-unrelated frames.

[0039] After DSF, cosine energy change point detection (CECD) is performed, i.e., the filtered target video frame sequence is operated to detect the operation boundary. The cosine energy change point detection algorithm proposed in the present disclosure is used to detect the boundary between different operation steps, so as to divide the video into multiple operation steps.

[0040] After the above processing, a series of video segments showing the production process can be obtained, each video segment corresponding to an operation step, and the standard time of each operation step is calculated.

[0041] In the decoding stage, a pre-trained visual language model (such as the CLIP model) is also used to generate a text description. According to the feature vector of each operation step and the pre-defined prompt information, a comprehensive feature is obtained. According to the comprehensive feature, a text including the step name and detailed operation description is generated. As shown in the figure, for the step "Tightening the valve", a specific operation step description is generated.

[0042] FIG. 2 shows a flowchart of a zero-shot standard operation procedure generation method according to an embodiment of the present disclosure. As shown in FIG. 2, the method includes steps 1-6.

[0043] Step 1, a plurality of production videos are obtained, and a pre-trained first visual language model is used to extract feature vectors of frames in each production video.

[0044] The production videos can be collected by existing monitoring cameras in the workshop, without the need for additional installation of special equipment, thereby ensuring that the normal production process is not disturbed and additional hardware costs are not increased. The plurality of production videos can come from different production cycles of the same production line, or from different workstations performing the same or similar tasks, so as to capture the variability and consistency of the production process, facilitating subsequent identification of standard operations.

[0045] The data collection process can take into account the actual situation of production. For example, data during factory downtime can be excluded to ensure that the collected data truly reflects the production state, thereby improving the quality and relevance of the data.

[0046] A pre-trained first visual language model can be used to extract image features of each production video. In some embodiments, the pre-trained first visual language model uses a CLIP (Contrastive Language-Image Pre-training) model as the visual backbone network. The CLIP model is a powerful visual-language model that is pre-trained on large-scale image-text pair data through contrastive learning, and can extract high-quality image features.

[0047] In some examples, the CLIP model can be used as a feature extractor to process each frame in the video. For each frame image, the CLIP model can output a high-dimensional feature vector to capture the semantic information of the image, so as to not only extract visual features but also understand the semantic content of the image to some extent, which is helpful for subsequent standard operation recognition and description generation.

[0048] Step 2: Select one of the production videos as a target video, use other production videos as reference videos, and identify production-related frames from the target video based on the feature vectors and filter out production-unrelated frames.

[0049] In actual production environments, errors or production-unrelated operations inevitably occur, such as personnel resting or checking mobile phones during production. When generating SOP, these operations should be excluded, and only necessary operation frames related to normal production should be retained. To solve this problem unsupervisedly, the present disclosure proposes a multi-video collaborative processing idea, i.e., selecting one of the production videos as a target video, which is the object of the final standard production program (SOP) generation, and using other videos as reference videos to identify production-related frames in the target video from the frames of the reference videos. Through multi-video comparison, common patterns in different workers or different production cycles can be captured, and abnormal situations in the target video can be filtered out, improving the accuracy and robustness of the identification.

[0050] The present disclosure proposes the following assumption 1: only when a certain action or posture appears in each production video, it is necessary. Therefore, in some embodiments, different clusters can be assigned to the frames in the target video; for each reference video frame, find the target video frame that matches the reference video frame based on the similarity of the feature vectors, and assign the reference video frame to the cluster corresponding to the matching target video frame; after traversing all reference video frames, if there are reference video frames in the cluster corresponding to the target video frame, it is determined that the target video frame is related to production, and if there are no reference video frames in the cluster corresponding to the target video frame, it is determined that the target video frame is unrelated to production, and the target video frame unrelated to production is removed.

[0051] Further, the present disclosure also proposes assumption 2: the production process is standardized, in other words, the order of each production step and the operations performed before and after are fixed, except for abnormal situations.

[0052] Assumptions 1 and 2 are applicable to most non-customized production scenarios.

[0053] It can be assumed that there are n production videos, where video i contains l iframes. Define video 0 as the target video for which SOP is to be generated, but video 0 still contains many frames that are irrelevant to production. As mentioned above, to effectively filter out the frames in video 0 that are irrelevant to production, frames from other production videos are to be utilized to ensure that only the frames in video 0 that are relevant to production are retained.

[0054] Based on assumptions 1 and 2, the present disclosure proposes a matching problem model of a mixed integer programming variant: Minimize Subject to

[0055] For the target video, video 0, a different cluster can be assigned to each frame of the target video, with k representing the cluster index. For video i (i≠0), the feature vector z i,t of the t-th frame of video i is to be associated with the k-th frame of video 0 (equation 3). Here, x i,t,k is a binary variable, with x i,t = 1 when the t-th frame of video i, z i,t,k , is assigned to cluster k, and x i,t,k = 0 otherwise. Based on assumption 2, to maintain the sequential integrity of the production process, the following constraint is imposed: if the t-th frame of video i is matched to cluster k, then the t+1-th frame of video i must appear in cluster k or a cluster after cluster k (equation 2). This constraint helps to prevent the repeated execution of certain steps due to abnormal situations in production.

[0056] The objective function of this model (equation 1) contains two parts. The first part of the objective function, d(z i,t , z 0,k ), is used to measure the loss of matching the t-th frame of video i to the k-th frame of video 0. In some embodiments, a pre-trained CLIP can be used as an image feature extractor of a large visual language model, which is trained using cosine similarity. Therefore, d(z i,t , z 0,k ) can be adopted as the cosine similarity distance (equation 7) to align with the pre-trained encoder.

[0057] The present disclosure further considers that, in some special cases, this model can overfit. For example, if one production video is recorded during the day with sufficient light, and the other is recorded at night with insufficient light, the second video can overfit to the frame with the weakest light in the first video, instead of matching the progress-related features. To avoid overfitting to certain clusters, the present disclosure introduces a regularization mechanism r(i, t, k) that skips frames in video 0 as the second part of equation 1.

[0058] Figure 3 illustrates the changes in production progress (box 301), non-static changes (box 302), and non-dynamic changes (box 303) presented in a production video. The vertical axis represents progress, and the horizontal axis represents the production cycle. As shown in Figure 3, as production progresses, image features change within box 301 (progress); the portion within box 302 also changes continuously in each production cycle (non-static changes), including constant changes in different workers performing operations or background or lighting; the changes within box 303 are dynamic (non-dynamic changes), including random events such as worker breaks or abnormal operations. In some embodiments, it is assumed that the changes are independently and identically distributed along the production timeline in each production cycle.

[0059] In some embodiments, to avoid overfitting to certain clusters, the regularization mechanism r(i,t,k) is set individually for each video. In some implementations, r(i,t,k) can be defined as: Where d(z) i,: ,z 0,: ) α d(z) is estimated through the empirical distribution i,t ,z 0,k The α quantile of ). This means that only when α is found down When matching quantiles, α up Only matching values ​​of quantiles will be skipped (rejected H0). In some implementations, α can be... up Set to 0.75, α down Set it to 0.5.

[0060] While the above optimization model is theoretically reasonable, solving it through constrained programming would present extremely high computational complexity when processing production videos containing thousands of frames. Therefore, this disclosure proposes a dynamic programming algorithm—Dynamic Similarity Filtering (DSF)—that can solve this problem with an optimal solution in polynomial complexity.

[0061] First, the decision variable x of the state variable of segmented video i. i,t,k Since they do not overlap, they can be decomposed into multiple subproblems. Therefore, when building a dynamic programming model, each calculation only requires solving a single model.

[0062] For each video i: stage: [k,t], t∈{0,...,l] i}, k∈{0,...,l0}. The first stage is [0,0], and the stopping criterion is when the stage reaches [l0,l...]. i At time [k, t], the state [k, t] represents the current candidate cluster k and frame t in video i. Actions, costs, and state transitions are as follows:

[0063] Skip: Frame t does not match cluster k, cost is r(i,t,k), xi,t,k = 0. State transitions to [k + 1, t].

[0064] Assign but not change cluster: frame t is matched to cluster k, there can be more frames matching k, x i,t,k = 1. The cost is d(z i,t , z 0,k ). State transitions to [k, t + 1].

[0065] Last assignment: frame t is the last frame matched to cluster k, frame t + 1 cannot be assigned to k, x i,t,k = 1. The cost is d(z i,t , z 0,k ). State transitions to [k + 1, t + 1].

[0066] Thus, in some embodiments, a value function dp[k, t] for dynamic programming is established: dp[k, t] = min(dp[k, t - 1] + d(z i,t-1 , z 0,k ), dp[k - 1][t] + r(i, t, k), dp[k - 1][t - 1] + d(z i,t , z 0,k ),

[0067] wherein k represents a cluster index, t represents a frame index of the reference video, z i,t represents a feature vector of the t-th frame of the reference video i, z 0,k represents a feature vector of a frame of the target video assigned to the k-th cluster, d(,) represents a cosine similarity distance between two feature vectors, r(i, t, k) is a regularization term, and dp[0, 0] = 0 is set.

[0068] For the t-th frame of the reference video i, the value function dp[k, t] is calculated by traversing all values of k, and the cluster index corresponding to the minimum value function dp[k, t] is determined to be the frame of the target video matched to the t-th frame of the reference video i, and the t-th frame of the reference video i is assigned to the cluster corresponding to the matched frame of the target video.

[0069] The present disclosure also proves that the optimal solution of this dynamic programming problem can be efficiently found by the DSF algorithm proposed by the present disclosure, as shown in Proposition 1: solving this dynamic programming problem only requires time complexity. After solving the dynamic programming, the action sequence and the corresponding x i value that achieve the optimal value in the end phase [l0, l i,t,k ] are obtained. The remaining x i,t,kis set to 0. In addition, it can also be proved that proposition 2: solving dynamic programming problems is equivalent to optimizing the model. In order to make the expression clear and brief, the proof of the proposition, theorem, lemma is not presented in detail in this disclosure.

[0070] In some embodiments, if for any reference video i, the following condition is not met then it is determined that the frame of the target video corresponding to the kth cluster is related to production, wherein, l i represents the total number of frames of the reference video i, x i,t,k is a binary variable, when the tth frame of the reference video i is assigned to the kth cluster, x i,t,k = 1, otherwise x i,t,k = 0; if there is a reference video i, which satisfies then it is determined that the frame of the target video corresponding to the kth cluster is not related to production.

[0071] If there is no matching frame in any reference video i which indicates that these frames may be either false or irrelevant operations, the frame of the target video corresponding to the cluster k can be skipped or deleted. z 0,k The classification result of x

[0072] According to the embodiment, by refining the index of the cluster, all the target video frames not related to production are deleted, and only the l frames of the target video related to production are retained.

[0073] The following is the pseudo code of the DSF algorithm according to an embodiment of the present disclosure.

[0074] Step 3, based on the cosine similarity distance and the non-parametric permutation test, the operation boundary of production is detected from the filtered target video frames, and the target video is segmented into multiple operation steps according to the operation boundary.

[0075] The step of detecting the operation or operation boundary in the production video requires identifying which time frames are the turning points of the operation. Traditional turning point detection methods usually introduce a term related to the number of turning points (e.g., a penalty term) when detecting an unknown number of turning points. However, in the production process, some operations show significant changes in image features, while others show minimal changes. This method of globally thresholding the feature values extracted by CLIP can lead to over-segmentation of operations with large feature changes, while ignoring subtle operations with small feature changes. In addition, visual feature extractors like CLIP are usually optimized for cosine similarity, which conflicts with the general kernel function or common normal distribution assumption. To solve these problems, the present disclosure proposes a non-parametric turning point detection algorithm: Cosine Energy Turning Point Detection (CECD), which can adaptively identify the segmentation points of the corresponding steps while taking into account the training objectives of the visual feature extractor and the characteristics of the current operation process.

[0076] According to the present embodiment, the filtered frames of the target video can be first divided into a plurality of non-overlapping time windows, and the operation boundary is detected within each time window to improve computational efficiency and improve local sensitivity of detection. In production video analysis, a frame usually represents less than 0.05 seconds of motion, which is not sufficient for actual operations, so a minimum operable time can be set. To solve this problem, the present disclosure combines a method widely used in the industry - Method-Time-Measurement (MTM), which takes into account the time cost of picking and placing objects in actual implementation, representing the simplest action in the production process. Factors that affect this action time cost include part weight, placement accuracy, and distance range. MTM provides standard time units (Time Motion Unit, TMU) for various types of arm and body movements, with one TMU corresponding to 0.036 seconds. Assuming that in this action, the object is light in weight, needs to be placed accurately, and the distance is between 20 and 50 centimeters. Based on this assumption, the corresponding MTM code is AB3, which is 90 TMU, equivalent to 3.24 seconds. Therefore, in the case of a frame rate of f, the minimum operation time can be set to N = 3.24f frame length in some examples of the present disclosure. In subsequent studies of this case, according to the suggestion of an industrial expert, the minimum operation time is set to 1 minute.

[0077] In some embodiments, the length of the time window for detecting the operation boundary can be determined based on the minimum operation time, so that the present solution is more consistent with production time and actual production environment, and facilitates accurate capture of real operation boundaries. In some examples, the length of the time window can be set to 2N to provide sufficient context information while avoiding the inclusion of too much irrelevant information, and to strike a balance between computational load and computational efficiency.

[0078] Further, the present disclosure proposes Hypothesis 3: the position of step boundaries should align with the change points of image feature distribution.

[0079] According to Hypothesis 3, the structure of energy statistics D(z a ,z b ) as defined in Equation (9) is used to measure the difference between two distributions: D(z a ,z b ) = 2E[d(z a ,z b )] - E[d(z a ,z a )] - E[d(z b ,z b )] (9)

[0080] where d(.,.) is a function to measure the distance between two samples, and E[d(.,.)] is the expectation of the distance. To align with the training objective of large-scale visual language models, d(.,.) can be set as the cosine similarity distance as defined in Equation (7), which is different from the traditional energy statistics using Euclidean distance as d(.,.). To simplify the notation, let d aa = E[d(z a ,z a )], d bb = E[d(z b ,z b )], and d ab = E[d(z a ,z b )].

[0081] Therefore, according to some embodiments of the present disclosure, the sample energy statistics can be fitted using the following Equation (10):

[0082] where k c is the candidate change point position. In this embodiment, each sample (each frame reserved after filtering) within the time window can be considered as a candidate change point.

[0083] In a time window (e.g., a time window with a length of 2N), there will be only one operation boundary, and the problem is simplified to detecting at most one change point in a time series. Therefore, the present disclosure proposes to find the most likely position in each time window and quantify its confidence probability, and find the local maximum of the energy statistics in each time window as a candidate boundary point. To evaluate the confidence level of each candidate boundary point, a non-parametric permutation test can be performed. In one example, under the assumption that there is no other change point, the permutation test is as follows:

[0084] First, for a potential change point, N data points can be taken from both sides of the point and permuted to generate a new sequence of length T. Then the energy statistics are recalculated according to equation (10). This process is repeated several times, and the test statistic value is recorded after each permutation. By comparing with the permutation statistics, the p-value of the candidate boundary point is determined to obtain the quantile.

[0085] If all possible permutations are considered, the exact p-value can be obtained, but the amount of calculation is too large. Therefore, the present disclosure proposes that the p-value can be approximated by performing R random permutations. In some examples, the significance level p0 can be set to 0.1 to approach the boundary point, and the permutation number R = 500 is specified as a constant value.

[0086] To demonstrate the convergence rate of the cosine energy change point detection in detecting step boundaries, the present disclosure also introduces the following assumptions and lemmas.

[0087] Assumption 4: The image features {z k : k e {1, 2,..., k c - 1} and {z k′ : k' e {k c + 1, k c + 2,..., T} are iid sample sequences from two independent multivariate distributions F a and F b , and where c e (0, 1).

[0088] In assumption 4, if there is no change point in the time window, F a = F b .

[0089] Assumption 5: If z a and z b come from two different operations, then assume d ab ≥ d aa + d bb .

[0090] Assumption 5 is a logical assumption for the distribution of image feature values, as it has the structure of the famous triplet loss in the field of contrast learning. And most famous pre-trained visual language encoders are based on contrast learning, including CLIP used in this paper.

[0091] First, it can be proved that the statistics proposed in the present embodiment follow the law of large numbers:

[0092] Lemma 1: Under assumption 4, the sample mean defined by the cosine similarity distance (see equation 7) follows the law of large numbers:

[0093] The disclosure proves that the cosine similarity distance sample statistics contain the property of two different distributions, as shown in Lemma 2.

[0094] Lemma 2: Under the assumption 4 and d(.,.) is cosine similarity distance as shown in equation 7, then:

[0095] By proving the two lemmas, the convergence property of CECD can be proved in Theorem 1 as follows.

[0096] Theorem 1: Under the assumption 4 and assumption 5, if there is a change point k c in the given interval {1,2,...T}, the process of using the cosine energy change point detection (CECD) algorithm finds the maximum value of the energy statistics as an estimated change point, then the estimated change point position converges to the true change point position at a rate of o(1):

[0097] Theorem 1 guarantees the fast convergence of the detected potential step boundary position.

[0098] The disclosure also analyzes the complexity of the proposed cosine energy change point detection algorithm, as shown in Proposition 3, which shows that the complexity of the algorithm is polynomially related to the frame rate (video length):

[0099] Proposition 3: The time complexity of the proposed cosine energy change point detection algorithm is O(l x N 2 ) = O(f 3 ).

[0100] With Theorem 1 and Proposition 3, the accuracy and efficiency of the proposed cosine energy change point detection algorithm are theoretically guaranteed.

[0101] In summary, in some embodiments, the operation boundaries of the operation steps are detected from the filtered target video frames based on cosine similarity distance and non-parametric permutation test, including:

[0102] Divide the filtered target video frames into a plurality of non-overlapping time windows;

[0103] In each time window, calculate the sample energy statistics of each candidate change point based on the following formula

[0104] where k c is the candidate change point position, z a and z b represent the feature vector sequences of the two time periods before and after the candidate change point k c , T is the total number of frames in the current analysis window, and d(z i ,z j) is a feature vector z i and z j between the cosine similarity distance;

[0105] In each time window, the candidate variable point with the largest calculated sample energy statistics is identified as the candidate boundary point of the time window;

[0106] Perform a non-parametric permutation test on each candidate boundary point to calculate the statistical significance evaluation value of each candidate boundary point;

[0107] Based on the statistical significance evaluation value of the candidate boundary point, determine the operation boundary.

[0108] The following shows the pseudo code of the cosine energy variable point detection algorithm according to one exemplary embodiment of the present disclosure.

[0109] Step 4, calculate the standard time of each operation step.

[0110] Calculating the standard time of each operation step is an important part of generating a standard operation procedure (SOP). According to the present embodiment, the standard time of each operation step is calculated based on multiple production video samples to ensure that the calculation result has statistical significance and can reflect the average level of different production cycles or different worker operations.

[0111] For each identified operation step, the calculation method of the standard time is to calculate the average time length between each segmentation point in all production videos. In some embodiments, the standard time of each operation step is calculated, including: identifying the time period corresponding to the operation step in the plurality of production videos; calculating the duration of the operation step in each production video; based on the duration of the operation step in each production video, calculating the average duration of the operation step as the standard time of the operation step.

[0112] In some embodiments, the calculation utilizes the method-time measurement (MTM) system, because one TMU in the MTM system corresponds to 0.036 seconds, which provides an accurate time unit basis for the calculation of the standard time.

[0113] Step 5, input the image features corresponding to the operation steps of the segmented target video and the pre-defined prompt information into the pre-trained second visual language model to generate a text description of each operation step.

[0114] This step utilizes a pre-trained visual language model to generate detailed textual descriptions for each operation step. This process combines the image features of each operation step in the segmented target video and pre-defined prompt information. In some embodiments, a pre-trained second visual language model with CLIP as the visual backbone can be used as a decoder, and the extracted feature vectors can be input into the decoder of the model. At the same time, in order to generate text descriptions that meet specific formats and requirements, prompt engineering techniques are used to guide the visual language model through pre-defined prompt templates, enabling it to generate structured and practically meaningful operation descriptions. For example, in the water valve production case study, two specific prompts were used: "This is a step in the water valve assembly process. Name this assembly step with a short phrase" and "Generate the operation process performed by the left and right hands", which effectively guided the model to generate text descriptions containing step names and detailed operation instructions.

[0115] Through this step, accurate and detailed operation descriptions can be generated based on video features and defined prompt information without specific domain labeled data. By combining visual features and language prompts, the model can understand the context of the operation and generate text descriptions that not only meet the requirements of specific production environments, but are also easy to understand and execute. This not only improves the efficiency of generating SOPs, but also ensures the quality and practicality of the generated content, making the generated standard operation procedures truly guide the actual production process.

[0116] Step 6, based on the standard time and the text description, generating a production procedure document.

[0117] Generating a standard operation procedure (SOP) document is the main goal of this embodiment, which organically combines the standard time and text description obtained in the previous steps. In some embodiments, this process is not just a simple concatenation of information, but a structured and standardized way of organizing content to ensure the practicality and readability of the final document. Specifically, for each identified operation step, the document can include the step name, detailed operation description, and calculated standard execution time. This format not only provides clear operation guidance, but also provides a time benchmark for production management.

[0118] The generated SOP document can serve as a key tool for improving production efficiency, quality, and production management, which not only describes the execution method of each operation step in detail, but also provides quantitative basis for workflow optimization and performance evaluation by including standard time. The SOP automatically generated according to the embodiment can adapt to different production environments, especially in industries such as aviation manufacturing that frequently make design changes and process optimization, it provides an efficient, accurate and easy-to-update method to maintain standard operating procedures, thereby solving the problems of high manual dependence and difficult updating in traditional SOP generation methods, and providing a more flexible and efficient tool for production management.

[0119] The above-described embodiments achieve automatic analysis of production videos and generation of standard operating procedures by utilizing pre-trained visual language models and innovative algorithms, without the need for labeled data or predefined operation templates in specific fields. This zero-shot learning method significantly improves the efficiency and adaptability of SOP generation, enabling it to be quickly applied to different production environments. Further, through the Dynamic Similarity Filtering (DSF) algorithm, this method can effectively identify and filter frames unrelated to production, improving the accuracy of generated SOPs, and through the Cosine-Energy Change-point Detection (CECD) algorithm, it can accurately locate the operation boundaries, making the division of operation steps more reasonable. In addition, the present disclosure can automatically calculate the standard time for each operation step and generate detailed text descriptions, greatly reducing the need for human involvement and reducing the influence of subjective bias. The automated and accurate SOP generation scheme according to the present disclosure not only improves the efficiency of production management, but also provides a flexible solution for rapidly changing manufacturing environments, which is of great significance for improving product quality, reducing production costs, and optimizing production processes.

[0120] FIG. 4 shows a process diagram of dynamic similarity filtering and cosine-energy change-point detection according to an example embodiment of the present disclosure.

[0121] As shown in FIG. 4, first, a plurality of videos (including target videos and reference videos) are processed by an encoder to generate feature vectors. Then, dynamic similarity filtering is used to compare these feature vectors to identify necessary (related to production) and unnecessary (unrelated to production) frames. Next, cosine-energy change-point detection is used to operate on the filtered frame sequence to detect operation boundaries to divide different operation steps (from Step 1 to Step s).

[0122] FIG. 5 shows a process diagram of generating textual descriptions according to an example embodiment of the present disclosure. As shown in FIG. 5, first, the feature vectors and cue information from Step 1 to Step c are combined into comprehensive features, and then the comprehensive features are input into the decoder. The decoder generates two-part output: step name and detailed operation step description. A specific output example is given in the figure, which describes the name and specific operation process of the step of "tightening the valve".

[0123] Next, the standard operating procedure generation scheme according to the present disclosure is compared with various baseline methods, focusing on evaluating the accuracy of action filtering and the quality of generated textual descriptions. Two different datasets will be used for testing: one for a single-station water valve production process and the other for actual commercial aircraft spoiler production.

[0124] The test environment is configured as Python 3.11 and CentOS 7.9. To ensure better performance and efficiency, a single A100 GPU or dual A100 GPU setup is used. All models use a pre-trained CLIP as their backbone (except for the baseline pre-trained large visual language model). The ground truth of the generated SOP is written by a group of industry experts to ensure accuracy and practical relevance.

[0125] The test scenarios include various non-production-related operations such as checking mobile phones or resting during production, as well as some erroneous operations. The tested scheme needs to exclude these non-production-related operations and segment each production operation to generate SOP using a large visual language model.

[0126] The first scenario is the water valve production process at a single-station workstation. The production process is shown in FIG. 6. First, the worker installs the rubber ring into the lower end of the valve body, then inserts the connecting rod, secures it with a screw, and places it into the lower valve body. The following steps include assembling the upper part of the valve, tightening the screw, and clamping. The entire process spans approximately 3000 frames. The dataset is composed of three groups, each containing two to three complete production videos performed by different workers. In each test, one video is designated as Video 0, and the remaining videos come from the same group to assist the DSF algorithm. FIG. 7 shows examples of partial ground truth SOP and predicted SOP.

[0127] The second scenario involves the production of commercial aircraft spoilers, with the finished product shown in FIG. 8a. This scenario is chosen because it involves the handling of mechanical parts and flexible materials. Since the aircraft company does not allow public video data, a schematic diagram of the production process as shown in FIG. 8b is used here. The production process is complex, involving multiple workers collaborating, and involves initial pre-assembly work, primer drying, inspection of auxiliary materials, laying of composite materials, and application of different color films. Subsequent stages include assembling rigid structures, covering isolation films, bagging, and leak testing, and finally curing in an autoclave.

[0128] The test focuses on the production stage to the autoclave process, which spans about 3 hours, with a frame rate of 1 / 3 per second.

[0129] The dataset consists of two groups, each containing three complete production videos, using a mixed production line. Due to the low frame rate, the SOP generated by this case will not describe the specific operations in detail, but will outline the operations performed in each step. Figure 9 shows an example of a partially ground truth SOP and a predicted SOP.

[0130] Next, the performance of the dynamic similarity filtering (DSF) proposed in this disclosure in filtering production-related actions is evaluated.

[0131] This evaluation measures the proportion of correctly retained relevant actions and filtered out irrelevant actions to ensure that the generated captions do not contain irrelevant operations while retaining all necessary operations.

[0132] An operation can be defined as retained if more than 8 frames are retained (as large visual language models typically require at least 8 frames to generate predictions). If an irrelevant operation is retained for a maximum of 2 frames (two boundary frames), it is considered filtered out to minimize the risk of irrelevant operations being generated into the SOP document. This criterion is strictly enforced in the test, as the number of irrelevant frames that need to be filtered out can be as high as several hundred frames.

[0133] The baseline methods include k-means, support vector machines (SVM), and multi-layer perceptron (MLP) using CLIP as a feature encoder. Although the scheme according to the present disclosure is completely unsupervised, SVM, k-means, and MLP require a training set, and therefore, an additional training video is specially annotated for this purpose. All baseline models are carefully tuned to ensure high performance and fair evaluation. The performance of different methods can be evaluated using F1 score and accuracy. Table 1 gives the quantitative evaluation results of action filtering. Table 1 Quantitative results of action filtering

[0134] As can be seen from the results in Table 1, the scheme according to the present disclosure outperforms the supervised baseline methods in all measured indicators without requiring any supervised annotations. This superiority stems from the fact that the baseline methods classify frames or operations individually, while the cosine energy change point detection proposed in this disclosure optimizes across the entire production video, allowing global features of the production to be considered when classifying operations. In addition, the cosine similarity distance measurement proposed in this disclosure is adaptable to the training characteristics of the pre-trained visual encoder.

[0135] Moreover, it can also be observed that k-means performs well in aircraft spoiler production but poorly in water valve production. By examining the classification results, the present disclosure finds that this difference is due to the fact that there are fewer abnormal operations in aircraft production, and the irrelevant operations mainly include workers leaving the workstation or irrelevant personnel passing by the station. These features are relatively easy to cluster because the presence or absence of personnel is a clear feature. In contrast, in water valve production, there are many abnormal and irrelevant operations, and the workers are always within the camera's field of view, making it difficult to capture the features through simple clustering.

[0136] Moreover, Fig. 10 visualizes the filtering and segmentation results of one of the water valve production videos. As can be seen from Fig. 10, most irrelevant operations, such as checking mobile phones, taking a break during production, or workers leaving the workstation to perform other tasks or performing other abnormal operations, are filtered out. Only the error operation in Step 8 is not filtered out.

[0137] To ensure the objectivity and accuracy of the experimental results, no performance evaluation indicators in the field of change point detection are used in the evaluation, the main reason being that because the production video frames are continuous, manually annotating the corresponding step segmentation points will introduce significant subjectivity. Even for the simplest segmentation scenarios, such as a worker stretching out his hand to grab the next step's part, it is difficult to objectively determine whether the segmentation point is when the previous step ends, when the worker stretches out his hand, or when he touches the next part. However, the convergence and optimality of the step segmentation proposed by the present disclosure have been theoretically proven in the above description for Step 3 in Fig. 2. Therefore, numerical experiments can be omitted in the field of change point detection.

[0138] Next, the accuracy of the generated text SOP is evaluated. As described above, after filtering out the frames irrelevant to production and segmenting the video into different operation steps, these feature vectors and prompt information are input into the decoder of the model to generate the text description of the corresponding step. Fig. 7 and Fig. 9 respectively show examples of predicted SOP and ground truth SOP for water valve production and aircraft spoiler production. Three widely used text generation task metrics, BLEU-4, METEOR, and ROUGE-L, are used in the test.

[0139] Since traditional step description generation algorithms are supervised and require specific operation steps to appear in the training set, they cannot meet the needs of SOP caption structuring, so the results are compared with several state-of-the-art large visual language models that can adjust the generated structure according to the prompt. The baselines include Video-LLaVA, Video Chat, and Video-ChatGPT. The quantitative results in Table 2 show that the text description generated according to the present disclosure is significantly better than all baseline models in all measurement indicators, again proving the superiority of the zero-shot standard operation procedure generation method proposed by the present disclosure. Table 2 Quantitative evaluation results of generated text description

[0140] In addition to comparing this disclosure with state-of-the-art large-scale visual language models, ablation studies were conducted to demonstrate the outstanding beneficial effects of the various embodiments of this disclosure. The performance of the method under the normal distribution assumption was further evaluated by comparing the embodiments with versions lacking dynamic similarity filtering, cosine energy change point detection, or both. To this end, the distance function d(z) was used during DSF operation in the tests. i ,z i ) Modify to squared Euclidean distance |z i -z i | 2 Furthermore, under the assumption of normal distribution, change point detection based on the T-test statistic was used to determine the operational boundaries. To ensure fairness and accuracy, all comparison methods underwent comprehensive fine-tuning and optimization.

[0141] Table 3 shows the results. In this table, "P-NA" indicates the model using the normal distribution assumption, "P-CECD" indicates the model without CECD, where the entire video is used as the decoder input; "P-DSF" indicates the model without DSF, where CECD directly segments the unfiltered video; and "P-CECD-DSF" indicates the model without DSF and CECD, using the original video as the decoder input. Table 3: Quantitative Results of Ablation Studies

[0142] As observed in Table 3, removing either DSF or CECD leads to a performance decrease, demonstrating that each component can provide additional beneficial effects. Furthermore, performance also significantly decreases after changing to the widely used normal distribution assumption (P-NA), proving that using cosine similarity distance to measure the distance between samples is more in line with the training objectives of the pre-trained visual encoder.

[0143] According to one embodiment of this disclosure, a video-based self-supervised production anomaly detection and progress prediction device is also proposed, the device comprising:

[0144] The video acquisition unit is used to acquire video data of the production process;

[0145] The latent feature extraction unit is used to map the image features in the video data to the latent space of the Gaussian mixture model (GMM) using the encoder, so as to obtain latent variables for representing the latent features in the image. The Gaussian components in the GMM correspond one-to-one with the pre-divided production stages.

[0146] The SAM generation unit is configured to generate a spatial activation map based on the feature map generated before the last convolution block of the encoder, and the spatial activation map is used to extract and represent the attention features related to different production stages in the image.

[0147] The image reconstruction unit is configured to reconstruct the latent variable output by the encoder into an image by using the decoder, and calculate a spatial activation reconstruction error of the reconstructed image weighted by the spatial activation map.

[0148] The model is trained based on a comprehensive loss function, which includes a cross-entropy loss of production progress prediction, a triple loss of latent space feature embedding for feature representation of different production stages, a negative log-likelihood loss of the latent variable in the GMM, and a spatial activation reconstruction error.

[0149] The anomaly detection and progress prediction unit is configured to process the real-time collected video data by using the encoder of the trained model, and perform production anomaly detection and production stage prediction based on the obtained latent variable.

[0150] Other details and benefits of the embodiments will be described in the foregoing relevant descriptions, and will not be repeated here.

[0151] The embodiments of the present disclosure can be applied to a computer system / server, which can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, and the like.

[0152] The computer system / server can be described in the general context of computer system executable instructions, such as program modules, executed by the computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like, which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment, in which tasks are performed by remote processing devices that are linked through a communication network. In a distributed cloud computing environment, program modules can be located in local or remote computer system storage media, including storage devices.

[0153] FIG. 11 is an electronic device provided by at least one embodiment of the present disclosure, the device including a memory for storing computer instructions executable on a processor, and the processor for implementing the zero-shot SOP generation method according to any embodiment or implementation of the present disclosure when executing the computer instructions.

[0154] At least one embodiment of the present disclosure further provides a computer readable storage medium having stored thereon a computer program, the program being executed by a processor to implement the zero-shot SOP generation method according to any embodiment or implementation of the present disclosure.

[0155] Those skilled in the art will appreciate that one or more embodiments of the disclosure can be provided as a method, a system or a computer program product. Therefore, one or more embodiments of the disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0156] Each of the embodiments in the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the data processing device embodiment is described simply because it is basically similar to the method embodiment, and the relevant part can be referred to the part of the method embodiment.

[0157] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims can be performed in a different order than those described in the embodiments and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown or sequential order in order to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous or necessary.

[0158] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0159] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit), and the apparatus can be implemented as special purpose logic circuitry.

[0160] Computers suitable for the execution of a computer program include, by way of example, general and / or special purpose microprocessors, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The essential elements of a computer are a central processing unit for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0161] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0162] While the specification contains many specifics, these should not be construed as limiting the scope of any invention or application, but as merely providing illustrations of some of the embodiments of the inventions. Certain features that are, for clarity, described above in the context of separate embodiments, can also be provided in combination in a single embodiment. Conversely, various features that are, for brevity, described above in the context of a single embodiment, can also be provided separately or in any suitable subcombination. In addition, while features can be described above as being implemented in digital electronic circuitry, one or more features also can be implemented in analog circuitry, in embodiments of the application. As used herein, the term "circuitry" refers to all of the following: (a) hardware-only circuitry implementations (which will be entirely analog in some embodiments, and / or entirely digital in some embodiments); (b) combinations of circuits and digital circuitry as well as software, such as (as applicable): (i) a combination of processor(s) or (ii) portions of circuits / matrices / processors and associated memory to implement software; as a non-limiting example, a software driver and / or firmware to control one or more processors of a programmable circuit. In some embodiments, multiple software and / or firmware implementations can be possible, to implement various aspects of the application. Also, the term "circuitry" should not be interpreted as applying only to mechanical / electrical circuitry formed from transistors, resistors, capacitors, etc., but can also refer to a circuitry that includes software, as described. Such software can include, be incorporated in, or born from software applications, middleware, firmware, operating systems, etc. In some embodiments, the software can be written in a high-level programming language, such as C, C++, Java, etc., and / or in assembly / machine language. In some embodiments, the software can be stored on a suitable computer readable medium, such as a hard disk, flash memory, etc., and loaded into memory for execution by a machine, such as a computer, etc.

[0163] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such order, nor that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0164] Accordingly, particular embodiments of the subject matter have been described. Other embodiments within the scope of the following claims can be apparent to those of ordinary skill in the art. Under the heading "METHODS OF OPERATION" are to be included programs, routines, algorithms, components, processes, management tasks, etc. that can be implemented to provide one or more features as described herein. Under the heading "ARTICLES OF MANUFACTURE" are to be included articles manufactured and / or composed of a combination of hardware and software elements, such as for example, computer hardware, client devices, consumer electronic devices, multimedia devices, memory media, etc. That work can have multiple embodiments, only one of which is described, and only in the context of specific embodiments.

[0165] The foregoing description of one or more implementations has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the one or more implementations to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. It is intended that the scope of the one or more implementations be limited not by this detailed description, but rather by the claims of this document.

Claims

1. A zero-shot SOP generation method, characterized in that, The method comprises the following steps: extracting a feature vector of each frame in the production video using a pre-trained first visual language model; selecting one of the plurality of production videos as a target video and the other production videos in the plurality of production videos as reference videos of the target video; filtering out frames irrelevant to production from the frames of the target video based on the feature vectors of the frames in the target video and the feature vectors of the frames in each of the reference videos, to obtain a filtered target video; detecting operation boundaries of a plurality of operation steps from the frames of the filtered target video based on cosine similarity distance and non-parametric permutation test, and segmenting the target video into the plurality of operation steps according to the operation boundaries; calculating a standard time of each operation step in the plurality of operation steps; inputting image features corresponding to the plurality of operation steps and pre-defined prompt information into a pre-trained second visual language model to generate a text description of each operation step in the plurality of operation steps; generating a production procedure document based on the standard time of each operation step in the plurality of operation steps and the text description.

2. The method of claim 1, wherein, The method of filtering out frames irrelevant to production from the frames of the target video based on the feature vectors of the frames in the target video and the feature vectors of the frames in each of the reference videos comprises the following steps: assigning each frame in the target video to a different cluster; for each frame in each of the reference videos, taking the frame as a reference frame, determining a target frame in the target video matching the reference frame based on the similarity between the feature vector of the reference frame and the feature vector of each frame in the target video, and assigning the reference frame as an associated frame to the cluster corresponding to the matching target frame; for each frame in the target video, if there is at least one associated frame in the cluster corresponding to the frame, judging that the frame is related to production, if there is no associated frame in the cluster corresponding to the frame, judging that the frame is irrelevant to production, and removing the frame from the target video.

3. The method of claim 2, wherein, The method of determining a target frame in the target video matching the reference frame based on the similarity between the feature vector of the reference frame and the feature vector of each frame in the target video comprises the following steps: establishing a value function dp[k,t] for dynamic programming: dp[k, t] = min(dp[k, t - 1] + d(z i,t-1 ,z 0,k ), dp[k - 1][t] + r(i, t, k), dp[k - 1][t - 1] + r(i, t, k - 1) 1] + d(z i,t ,z 0,k )), wherein k represents a cluster index, t represents the frame index of the reference frame in the reference video, z i,t denotes a feature vector of the t-th frame of the reference video i, z 0,k represents a feature vector of a frame of the target video assigned to the k-th cluster, d(,) represents the cosine similarity distance between two feature vectors, r(i,t,k) is a regularization term, and setting dp[0,0] = 0; for the t-th frame of the reference video i, calculating the value function dp[k,t] for all values of k to determine that the frame of the target video corresponding to the cluster index minimizing the value function dp[k,t] is the frame of the target video matching the t-th frame of the reference video i.

4. The method of claim 3, wherein, The method of judging that each frame in the target video is related to production if there is at least one associated frame in the cluster corresponding to the frame, and judging that the frame is irrelevant to production if there is no associated frame in the cluster corresponding to the frame, comprises the following steps: If for any reference video i, both of the following are not satisfied then determine that the frame of the target video corresponding to the kth cluster is production-related, wherein l i represents the total number of frames of the reference video i, x i,t,k is a binary variable, x i,t,k = 1 when the tth frame of the reference video i is assigned to the kth cluster, and x i,t,k = 0 when the tth frame of the reference video i is not assigned to the kth cluster. If there is a reference video i, satisfying if k > 0, then judging that the frame of the target video corresponding to the k-th cluster is irrelevant to production.

5. The method of claim 1, wherein, detect operation boundaries of the plurality of operation steps from the filtered frames of the target video based on cosine similarity distance and non-parametric permutation test, comprising: dividing the filtered frames of the target video into a plurality of non-overlapping time windows; Within each of the plurality of time windows, a sample energy statistic for each candidate change point is calculated based on the following equation where k c is the candidate point position, z a and z b represent the feature vector sequences of the two time periods before and after the candidate point k c , respectively, T is the total number of frames in the current analysis window, and d(z i , z j ) is the cosine similarity distance between the feature vectors z i and z j . in each time window of the plurality of time windows, identifying a candidate boundary point with the maximum calculated sample energy statistics as the boundary point of the time window; performing non-parametric permutation test on each candidate boundary point to calculate a statistical significance evaluation value of each candidate boundary point; determining the operation boundaries based on the statistical significance evaluation value of each candidate boundary point.

6. The method of claim 5, further comprising: before detecting the operation boundaries of the plurality of operation steps, obtaining a minimum operation time of the production process based on method-time measurement (MTM), and determining a length of each time window of the plurality of time windows based on the minimum operation time. calculating a standard time of each operation step of the plurality of operation steps, comprising:

7. The method of claim 1, wherein, for each operation step of the plurality of operation steps, identifying a time period corresponding to the operation step in the plurality of production videos; calculating a duration of the operation step in each production video of the plurality of production videos; calculating an average duration of the operation step as the standard time of the operation step based on the duration of the operation step in each production video of the plurality of production videos.

8. The method of claim 1, wherein: the pre-trained first visual language model uses a CLIP model as a visual backbone network, and / or the pre-trained second visual language model uses a CLIP model as a visual backbone network. comprising:

9. A zero-shot SOP generation apparatus, characterized by comprising: a video acquisition unit configured to extract a feature vector of each frame of the plurality of production videos using a pre-trained first visual language model; a dynamic similarity filtering unit configured to select one of the plurality of production videos as a target video and select other production videos of the plurality of production videos as reference videos of the target video; filter out frames irrelevant to production from the frames of the target video based on the feature vector of the frames of the target video and the feature vector of the frames of each reference video to obtain a filtered target video; a cosine energy change point detection unit configured to detect operation boundaries of a plurality of operation steps from the filtered frames of the target video based on cosine similarity distance and non-parametric permutation test, and divide the target video into the plurality of operation steps according to the operation boundaries; a standard time calculation unit configured to calculate a standard time of each operation step of the plurality of operation steps; a text description generation unit configured to input image features corresponding to the plurality of operation steps and pre-defined prompt information into a pre-trained second visual language model to generate a text description of each operation step of the plurality of operation steps; an SOP generation unit configured to generate a production program document based on the standard time of each operation step of the plurality of operation steps and the text description. ​ 10. An electronic device, comprising: The device comprises a memory for storing computer instructions executable on a processor, and a processor for implementing the method of any one of claims 1 to 8 when executing the computer instructions.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by a processor, implements the method of any one of claims 1 to 8.

Images