Zero sample standard operation program generation method and device, equipment and storage medium

Abstract

The invention discloses a zero-sample standard operation program generation method and device, equipment and a storage medium. The method comprises the following steps: acquiring a plurality of production videos, and extracting image features of each frame in each video by using a visual language model; through multi-video cooperative processing and dynamic similarity filtering, standard operation in a target video is identified, and irrelevant frames are filtered; detecting an operation boundary from the filtered frame by using a cosine energy change point detection algorithm, and segmenting the target video into a plurality of operation steps; standard time is calculated for each operation step; inputting the image features and the prompt information corresponding to the operation steps into a visual language model, and generating text description of each step; and generating a standard operation program document based on the standard time and the text description. The method does not need labeling data in a specific field, can automatically and efficiently generate an accurate standard operation program, and is suitable for various production environments, especially frequently-changed manufacturing processes.

Description

Technical Field

[0001] The present disclosure relates to the field of advanced manufacturing, and in particular to a method, device, equipment and storage medium for generating a zero-sample standard operating procedure. Background Art

[0002] Standard Operating Procedure (SOP) is a series of repeatable steps designed to perform daily tasks, including the standard time required for each step and the standardized operation description of each step. As a key document, SOP aims to reduce work pressure and errors. It is the basis of production management and is particularly important in the industrial field. The generation of SOP is based on predetermined standards and actual production scenarios or experience to create a document describing the operation process.

[0003] The generation of SOPs mainly involves two key steps: determining the process stages and describing the specific operations of each step. The main goal of determining the process stages is to identify standardized operating steps in production. Traditional methods usually rely on predefined component assembly sequences or subjective classification, both of which require a lot of manpower and are prone to introducing bias. Therefore, the need to automatically segment production stages and achieve accurate annotation through high-throughput video analysis has become particularly urgent.

[0004] Despite the value of having accurate and up-to-date SOPs, there are still many challenges in this regard due to the uncertainty, non-standardized workflows, and unrelated operations in the production process. Traditional SOP generation methods are often labor-intensive, for example, manually recording the operations that occur during the production process using standardized work combination sheets. However, these methods are inherently time-consuming and labor-intensive, and cannot be automatically updated when the production process is adjusted, resulting in discrepancies between recorded operation data and actual practice. This problem is particularly prominent in industries such as aerospace manufacturing where frequent design changes and process optimization are carried out.

[0005] Recent advances in computer vision have opened up new possibilities for automatically acquiring information from industrial videos. For example, object detection algorithms can be used to identify key elements and actions in videos. In addition, the emergence of large visual language models has made zero-shot image and video caption generation possible. However, the production process often contains many operations that are irrelevant to actual manufacturing, such as workers taking breaks or picking up the wrong parts. Existing algorithms have difficulty effectively distinguishing between frames containing production-related operations and irrelevant frames. In addition, simple frame-by-frame object detection or general video summaries generated by large language models cannot accurately determine the standard time required for each operation step, nor can they generate sufficiently detailed SOPs.

[0006] There are also some existing methods that attempt to automate the SOP generation process, but they usually require a large amount of labeled data or pre-trained models in specific fields. This dependence limits the flexibility and applicability of these methods. Therefore, it is particularly important to develop a "zero-shot" method that can generate accurate SOPs based only on raw production videos without domain-specific labeled data. Summary of the invention

[0007] The present disclosure proposes a zero-sample standard operating procedure generation method, which can automatically identify production-related frames in a production video, automatically identify key moments (i.e., operation boundaries) for distinguishing different operation steps, and automatically generate a standard operating procedure document that describes the production steps in detail without manual annotation. The present disclosure also proposes a corresponding device, an electronic device, and a computer-readable storage medium.

[0008] According to an embodiment of the present disclosure, a method for generating a zero-sample standard operating procedure is proposed, comprising:

[0009] Obtain multiple production videos and extract feature vectors of frames in each production video using a pre-trained first visual language model;

[0010] Selecting one of the multiple production videos as a target video, using other production videos in the multiple production videos as reference videos, identifying production-related frames from the target video frames based on feature vectors and filtering out frames that are not related to production;

[0011] Detecting operation boundaries of operation steps from the filtered target video frame based on cosine similarity distance and non-parametric permutation test, and dividing the target video into multiple operation steps according to the operation boundaries;

[0012] Calculate the standard time for each operation step;

[0013] Input the image features and predefined prompt information corresponding to the operation steps of the segmented target video into the pre-trained second visual language model to generate a text description of each operation step;

[0014] Based on the standard time and the text description, a production procedure document is generated.

[0015] In some implementations, identifying production-related frames from target video frames based on the feature vector and filtering out frames not related to production includes:

[0016] Assign a different cluster to each frame in the target video;

[0017] For each reference video frame, a target video frame matching the reference video frame is found based on the similarity of the feature vectors, and the reference video frame is assigned to a cluster corresponding to the matching target video frame;

[0018] After traversing all reference video frames, if there is a reference video frame in the cluster corresponding to the target video frame, it is judged that the target video frame is related to production. If there is no reference video frame in the cluster corresponding to the target video frame, it is judged that the target video frame is not related to production, and the target video frame not related to production is removed.

[0019] In some implementations, searching for a target video frame matching the reference video frame based on the similarity of image features between frames, and assigning the reference video frame to a cluster corresponding to the matching target video frame includes:

[0020] Establish the value function dp[k,t] for dynamic programming:

[0021] dp[k,t]=min(dp[k,t-1]+d(z i,t-1 ,z 0,k ),dp[k-1][t]+r(i,t,k),dp[k-1][t-1]+d(z i,t ,z 0,k ),

[0022] Where k represents the cluster index, t represents the frame index of the reference video, and z i,t represents the feature vector of the tth frame of the reference video i, z 0,k represents the feature vector of the target video frame assigned to the kth cluster, d(,) represents the cosine similarity distance between two feature vectors, r(i,t,k) is the regularization term, and dp[0,0]=0;

[0023] For the t-th frame of reference video i, traverse all values ​​of k to calculate the value function dp[k,t], determine that the target video frame corresponding to the cluster index that minimizes the value function dp[k,t] is the target video frame that matches the t-th frame of reference video i, and assign the t-th frame of reference video i to the cluster corresponding to the matching target video frame.

[0024] In some implementations, if the cluster corresponding to the target video frame contains a reference video frame, then the target video frame is determined to be related to the production; if the cluster corresponding to the target video frame contains no reference video frame, then the target video frame is determined to be unrelated to the production, including:

[0025] If for any reference video i, does not satisfy Then the target video frame corresponding to the kth cluster is judged to be relevant to the production, where l i represents the total number of frames of reference video i, x i,t,kis a binary variable. When the tth frame of the reference video i is assigned to the kth cluster, x i,t,k =1, otherwise x i,t,k =0;

[0026] If there is a reference video i, satisfying It is then determined that the target video frame corresponding to the kth cluster is irrelevant to production.

[0027] In some embodiments, detecting the operation boundary of the operation step from the filtered target video frame based on the cosine similarity distance and the non-parametric permutation test includes:

[0028] Divide the filtered target video frame into multiple non-overlapping time windows;

[0029] In each time window, the sample energy statistics of each candidate change point are calculated based on the following formula:

[0030]

[0031] Among them, k c is the candidate change point position, z a and z b They represent candidate change points k respectively. c The feature vector sequence of the first and last two time periods, T is the total number of frames in the current analysis window, d(z i ,z j ) is the eigenvector z i and z j The cosine similarity distance between ;

[0032] In each time window, the candidate change point with the largest calculated sample energy statistic is identified as the candidate boundary point of the time window;

[0033] Perform a nonparametric permutation test on each candidate boundary point and calculate the statistical significance evaluation value of each candidate boundary point;

[0034] Based on the statistical significance evaluation values ​​of the candidate boundary points, the operational boundary is determined.

[0035] In some embodiments, the method further comprises:

[0036] Before detecting the operation boundary of the operation step, the minimum operation time of the production process is obtained based on method-time measurement (MTM), and the length of the time window is determined based on the minimum operation time.

[0037] In some embodiments, calculating the standard time for each operation step includes:

[0038] Identifying a time period corresponding to the operation step in the multiple production videos;

[0039] Calculate the duration of this operation step in each production video;

[0040] Based on the duration of the operation step in each production video, the average duration of the operation step is calculated as the standard time of the operation step.

[0041] In some embodiments, the pre-trained first visual language model uses the CLIP model as a visual backbone network, and / or the pre-trained second visual language model uses the CLIP model as a visual backbone network.

[0042] According to one embodiment of the present disclosure, a zero-sample standard operating procedure generation device is proposed, comprising:

[0043] a video acquisition unit, configured to acquire a plurality of production videos and extract a feature vector of a frame in each production video using a pre-trained first visual language model;

[0044] A dynamic similarity filtering unit, configured to select one of the multiple production videos as a target video, use other production videos in the multiple production videos as reference videos, identify production-related frames from the target video frames based on feature vectors, and filter frames that are not related to production;

[0045] A cosine energy change point detection unit, used for detecting operation boundaries of operation steps from the filtered target video frame based on cosine similarity distance and non-parametric permutation test, and dividing the target video into multiple operation steps according to the operation boundaries;

[0046] A standard time calculation unit, used to calculate the standard time of each operation step;

[0047] A text description unit is used to input image features and predefined prompt information corresponding to the operation steps of the segmented target video into a pre-trained second visual language model to generate a text description of each operation step;

[0048] The SOP generating unit is used to generate a production procedure document based on the standard time and the text description.

[0049] According to one embodiment of the present disclosure, an electronic device is provided, the device comprising a memory and a processor, the memory being used to store computer instructions executable on the processor, the processor being used to implement any of the above methods when executing the computer instructions.

[0050] According to an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, any of the above methods is implemented.

[0051] The zero-sample standard operating procedure generation scheme based on the pre-trained visual language model proposed in the present disclosure has many beneficial effects. The present disclosure realizes the automatic analysis of production videos and the generation of standard operating procedures by utilizing pre-trained visual language models and innovative algorithms, without the need for annotation data or predefined operation templates in specific fields. This zero-sample learning method significantly improves the efficiency and adaptability of SOP generation, so that it can be quickly applied to different production environments. Further, through the dynamic similarity filtering (DSF) algorithm, the method can effectively identify and filter frames that are not related to production, improve the accuracy of generating SOP, and can achieve accurate positioning of operation boundaries through the cosine energy change point detection (CECD) algorithm, so that the division of operation steps is more reasonable. In addition, the present disclosure can automatically calculate the standard time of each operation step and generate a detailed text description, which greatly reduces the need for manual participation and reduces the influence of subjective bias. The automated and accurate SOP generation scheme disclosed in the present invention not only improves the efficiency of production management, but also provides a flexible solution for the rapidly changing manufacturing environment, which is of great significance for improving product quality, reducing production costs and optimizing production processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0053] Figure 1 Taking a certain operation as an example, a schematic diagram of the overall concept of a standard operating procedure generation process according to an exemplary embodiment of the present disclosure is shown.

[0054] Figure 2 A flowchart of a method for generating a zero-sample standard operating procedure according to an embodiment of the present disclosure is shown.

[0055] Figure 3 The figure shows the production progress changes (blue box), non-progress static changes (red box) and non-progress dynamic changes (green box) presented in different frames of a production video.

[0056] Figure 4 A schematic diagram of a process of dynamic similarity filtering and cosine energy change point detection according to an exemplary embodiment of the present disclosure is shown.

[0057] Figure 5 A schematic diagram of a process of generating a text description according to an exemplary embodiment of the present disclosure is shown.

[0058] Figure 6It shows a visualized water valve production process and examples of operations that are irrelevant to production or abnormal operations to be filtered out.

[0059] Figure 7 An example of partial ground truth SOP and predicted SOP for visualizing water valve production process is shown.

[0060] Figure 8a A commercial aircraft spoiler is shown; Figure 8b A schematic diagram showing the production process of a commercial aircraft spoiler.

[0061] Fig. 9 An example of partial ground truth SOPs and predicted SOPs for a commercial aircraft spoiler production process is shown.

[0062] Fig.10 Shown is a visualization of the filtering and segmentation results for a water valve production video.

[0063] Fig.11 It is a schematic diagram of the structure of an electronic device shown in at least one embodiment of the present disclosure.

[0064] It should be noted that there are English words in some of the drawings in the specification. These English words are screenshot contents and do not affect the understanding of the technical solution, specific features and details of the present disclosure. DETAILED DESCRIPTION

[0065] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0066] The following is a brief introduction to the overall inventive concept of the present disclosure. Figure 1 A certain operation is taken as an example to illustrate the exemplary overall structure of the method for generating a zero-sample standard operating procedure proposed in the present disclosure.

[0067] like Figure 1 As shown, first, multiple production videos are processed through an encoder. The encoder uses a pre-trained visual language model, such as CLIP, to extract feature vectors from each frame of each video. These feature vectors capture the visual and semantic information in the frame, laying the foundation for subsequent analysis. Next, the target video frame can be compared with the reference video frame by, for example, the dynamic similarity filtering (DSF) algorithm proposed in the present disclosure, to identify the frames in the target video frame that are related to production, and filter out the frames that are not related to production.

[0068] After DSF, cosine energy change point detection is performed, that is, operation boundary detection is performed on the filtered target video frame sequence. The cosine energy change point detection (CECD) algorithm proposed in the present disclosure is used to detect the boundaries between different operation steps, thereby segmenting the video into multiple operation steps.

[0069] After the above processing, a series of video segments showing the production process can be obtained, each video segment corresponds to an operation step, and the standard time of each operation step is calculated.

[0070] In the decoding stage, the pre-trained visual language model is also used to generate text descriptions. The feature vector of each operation step is combined with the predefined prompt information to obtain a comprehensive feature. The comprehensive feature generates text including the step name and the detailed operation description (procedure). As shown in the figure, for the step "Tightening the valve", a specific operation step description is generated.

[0071] Figure 2 FIG. 4 is a flowchart of a method for generating a zero-sample standard operating procedure according to an embodiment of the present disclosure. Figure 2 As shown, the method includes steps 1 to 6.

[0072] Step 1, obtain multiple production videos, and use the pre-trained first visual language model to extract feature vectors of frames in each production video.

[0073] Production videos can be collected through existing surveillance cameras in the workshop without the need to install additional specialized equipment, thus ensuring that normal production processes are not disrupted and no additional hardware costs are added. Multiple production videos can come from different production cycles of the same production line, or from different workstations performing the same or similar tasks, to capture the variability and consistency of the production process, facilitating the subsequent identification of standard operations.

[0074] The data collection process can take into account the actual production situation. For example, data during factory downtime will be excluded to ensure that the collected data truly reflects the production status, thereby improving the quality and relevance of the data.

[0075] The pre-trained first visual language model may be used to extract image features of each production video. In some embodiments, the pre-trained first visual language model uses a CLIP (Contrastive Language-Image Pre-training) model as a visual backbone network. The CLIP model is a powerful visual-language model that is pre-trained on large-scale image-text pair data through contrastive learning and can extract high-quality image features.

[0076] In some examples, the CLIP model can be used as a feature extractor to process each frame in the video. For each frame, the CLIP model can output a high-dimensional feature vector to capture the semantic information of the image, so that it can not only extract visual features, but also understand the semantic content of the image to a certain extent, which is helpful for subsequent standard operation recognition and description generation.

[0077] Step 2, selecting one of the multiple production videos as a target video, using other production videos in the multiple production videos as reference videos, identifying production-related frames from the target video frames based on feature vectors and filtering out frames unrelated to production.

[0078] In the actual production environment, it is inevitable that errors or non-productive operations will occur, such as personnel taking a rest or checking their mobile phones in production. When generating SOP, all operations related to these should be excluded, and only the necessary operation frames related to normal production should be retained. In order to solve this problem unsupervisedly, the inventors have proposed the idea of ​​multi-video collaborative processing, that is, one is selected from multiple production videos as a target video, and the target video is the object for eventually generating a production program (SOP), and other videos are used as reference videos, using the frames of the reference videos to identify the frames related to production in the target video. Common patterns in different workers or different production cycles can be captured by multi-video comparison, while filtering out the abnormal situations that may occur in the target video, improving the accuracy and robustness of recognition.

[0079] The inventor proposed the following hypothesis 1 when studying this problem: a certain action or posture is necessary only when it appears in every production video. Therefore, in some embodiments, different clusters can be assigned to each frame in the target video; for each reference video frame, a target video frame matching the reference video frame is found based on the similarity of the feature vector, and the reference video frame is assigned to the cluster corresponding to the matching target video frame; after traversing all reference video frames, if the reference video frame exists in the cluster corresponding to the target video frame, the target video frame is judged to be related to production; if the reference video frame does not exist in the cluster corresponding to the target video frame, the target video frame is judged to be irrelevant to production, and the target video frame irrelevant to production is removed.

[0080] Furthermore, the inventor also proposed hypothesis 2. Hypothesis 1 and 2 are applicable to most non-customized production scenarios.

[0081] Assumption 2: The production process is standardized, in other words, the order of each production step and the operations performed before and after are fixed, except for abnormal situations.

[0082] It can be assumed that there are n production videos, where video i contains l iVideo 0 is defined as the target video for generating SOP, but Video 0 still contains many frames that are not related to the ongoing production. As mentioned above, in order to effectively refine Video 0, frames from other production videos are utilized to ensure that only frames related to production in Video 0 are retained.

[0083] Based on assumptions 1 and 2, the inventors proposed a matching problem model of a mixed integer programming variant:

[0084]

[0085] For the target video, video 0, each frame can be assigned a different cluster, where k represents the cluster index. For video i≠0, the feature vector z of the tth frame of video i i,t must be assigned to cluster k, associating it with the kth frame of video 0 (Equation 3). Here, x i,t,k is a binary variable, when x i,t,k =1, it means the tth frame z of video i i,t is assigned to cluster k, otherwise x i,t,t =0. Based on Assumption 2, in order to maintain the sequential integrity of the production process, the inventors impose a constraint: if the tth frame of video i matches cluster k, then the t+1th frame of video i must appear in cluster k or a later cluster (Equation 2). This constraint helps prevent the repetition of certain steps due to abnormal situations in production.

[0086] The objective function of this model (Equation 1) consists of two parts. The first part of the objective function d(z i,t ,z 0,k ) is used to measure the loss of matching the tth frame of video i to the kth frame of video 0. In a specific example, the inventors used pre-trained CLIP as the image feature extractor of a large visual language model, which was trained using cosine similarity. Therefore, d(z i,t ,z 0,k ) is the cosine similarity distance (Equation 7) to align with the pre-trained encoder.

[0087]

[0088] The inventors further considered that in some special cases, this model may overfit. For example, if one production video is recorded during the day with sufficient light, and the other is recorded at night with insufficient light, the second video may overfit to the frames with the lowest light in the first video instead of matching the features related to progress. To avoid overfitting of certain clusters, the inventors introduced a regularization mechanism r(i,t,k) that skips frames in video 0 as the second part of equation 1.

[0089] Figure 3The figure shows the production progress change (blue frame), non-progress static change (red frame) and non-progress dynamic change (green frame) presented in the production video. The vertical axis represents the progress and the horizontal axis represents the production cycle. Figure 3 As shown, as production progresses, the image features change in the blue box (progress); the part in the red box also changes continuously in each production cycle (non-progress static change), including constant changes in operations performed by different workers or background or lighting; the changes in the green box are dynamic (non-progress dynamic change), including random events such as workers taking breaks or abnormal operations, which are assumed to be independent and identically distributed along the production timeline in each production cycle.

[0090] In order to avoid overfitting some clusters, it is preferred that the regularization mechanism r(i, t, k) is set separately for each video. In some embodiments, r(i, t, k) can be defined as: r(i, t, k) = d(z i,: ,z 0,: ) αup - Where d(z i,: ,z 0,: ) α is estimated from the empirical distribution d(z i,t ,z 0,k ). This means that only when we find α down When the quantile matches, α up The matching value of the quantile will be skipped (reject H0). In some preferred embodiments, α up Set to 0.75, α down Set to 0.5.

[0091] Although the above optimization model is reasonable in theory, it will face extremely high computational complexity when processing production videos containing thousands of frames if solved by constraint programming. Therefore, the inventors proposed a dynamic programming algorithm, the dynamic similarity filtering (DSF) algorithm, which can optimally solve this problem with polynomial complexity.

[0092] First, the decision variable x of the state variable of segment video i i,t,k There is no crossover and it can be decomposed into multiple sub-problems. Therefore, when building a dynamic programming model, only a single model needs to be solved each time.

[0093] For each video i: stage: [k, t], t∈{0,...,l i},k∈{0,...,l0}. The first stage is [0,0], and the stopping criterion is when the stage reaches [l0,l i ] when the state: [k, t] represents the current candidate cluster k and frame t in video i.

[0094] The actions, costs, and state transitions are as follows:

[0095] Skip: Frame t is not matched to cluster k, cost is r(i,t,k), x i,t,k = 0. The state is transferred to [k+1, t].

[0096] Assign but do not change clusters: frame t is matched to cluster k, and there may be more frames matching k, x i,t,k = 1. The cost is d(z i,t ,z 0,k ). The state is transferred to [k,t+1].

[0097] Last assignment: Frame t is the last frame matched to cluster k, and frame t+1 cannot be assigned to k. i,t,k = 1. The cost is d(z i,t ,z 0,k ). The state is transferred to [k+1,t+1].

[0098] Thus, in some implementations, a value function dp[k,t] for dynamic programming is established:

[0099] dp[k,t]=min(dp[k,t-1]+d(z i,t-1 ,z 0,k ),dp[k-1][t]+r(i,t,k),dp[k-1][t-1]+d(z i,t ,z 0,k ),

[0100] Where k represents the cluster index, t represents the frame index of the reference video, and z i,t represents the feature vector of the tth frame of the reference video i, z 0,k represents the feature vector of the target video frame assigned to the kth cluster, d(,) represents the cosine similarity distance between two feature vectors, r(i,t,k) is the regularization term, and dp[0,0]=0;

[0101] For the t-th frame of reference video i, traverse all values ​​of k to calculate the value function dp[k,t], determine that the target video frame corresponding to the cluster index that minimizes the value function dp[k,t] is the target video frame that matches the t-th frame of reference video i, and assign the t-th frame of reference video i to the cluster corresponding to the matching target video frame.

[0102] The inventors have also proved that the optimal solution to this dynamic programming problem can be efficiently found by the DSF algorithm proposed in this disclosure, as shown in Proposition 1: Solving this dynamic programming problem only requires Time complexity. After solving the dynamic programming, we can obtain the time complexity of [l0,l i ] The action sequence and corresponding x that achieve the optimal valuei,t,k The remaining x values ​​that are not in the action path i,t,k Set to 0. In addition, Proposition 2 can also be proved: solving the dynamic programming problem is equivalent to optimizing the model. In order to make the expression clear and concise, the proofs of propositions, theorems, and lemmas are not presented in detail in this disclosure.

[0103] Further, in some implementations, if for any reference video i, Then the target video frame corresponding to the kth cluster is judged to be relevant to the production, where l i represents the total number of frames of reference video i, x i,t,k is a binary variable. When the tth frame of the reference video i is assigned to the kth cluster, x i,t,k =1, otherwise x i,t,k =0; if there is a reference video i, satisfying It is then determined that the target video frame corresponding to the kth cluster is irrelevant to production.

[0104] If there is no matching frame in any of the production videos i Indicates that these frames may be either erroneous or irrelevant operations, and the target video frames corresponding to cluster k can be skipped or deleted. 0,k The classification result can also be expressed by the following formula:

[0105]

[0106] According to this embodiment, by refining the index of the cluster, all target video frames irrelevant to the production are deleted, and only l frame of target video frames relevant to the production is retained.

[0107] The following is a pseudo code of a DSF algorithm according to an embodiment of the present disclosure.

[0108]

[0109]

[0110] Step 3: Detect the generated operation boundary from the filtered target video frame based on the cosine similarity distance and non-parametric permutation test, and divide the target video into multiple operation steps according to the operation boundary.

[0111] Detecting the steps of an operation or operation boundary in a production video requires identifying which time frames serve as change points for the operation. Traditional change point detection methods typically introduce a term related to the number of change points (e.g., a penalty term) when detecting an unknown number of change points. However, during production, some operations show significant changes in image features, while others show minimal changes. This approach of globally thresholding the feature values ​​extracted by CLIP may lead to over-segmentation of operations with large feature changes, while ignoring subtle operations with small feature changes. In addition, visual feature extractors like CLIP are typically optimized for cosine similarity, which conflicts with existing kernel functions or common normal distribution assumptions. To address these issues, the inventors propose a non-parametric change point detection algorithm: cosine energy change point detection (CECD), which can adaptively identify the segmentation points of corresponding steps while taking into account both the training objectives of the visual feature extractor and the characteristics of the current operation process.

[0112] According to the present embodiment, the filtered target video can be first divided into multiple non-overlapping time windows, and the operation boundary can be detected in each time window to improve the computational efficiency and improve the local sensitivity of the detection. In production video analysis, one frame usually represents less than 0.05 seconds of action, which is not enough for actual operation, so a minimum operable time can be set. In order to solve this problem, the inventor combined the method-time measurement (MTM) widely used in the industrial field in actual implementation, and considered the time cost of grabbing and placing objects, representing the simplest action in the production process. Factors affecting the time cost of this action include part weight, placement accuracy and distance range. MTM provides standard time units (TMUs) for various types of arm and body actions, and one TMU corresponds to 0.036 seconds. Assume that in this action, the object is light and needs to be placed accurately, and the distance is between 20 cm and 50 cm. Based on this assumption, the corresponding MTM code is AB3, which is 90 TMUs, equivalent to 3.24 seconds. Therefore, when the frame rate is f, the minimum operation time can be set to N=3.24f frame length in some examples of the present disclosure. In a follow-up study of this case, the minimum operating time was set to 1 minute based on the advice of industry experts.

[0113] In some implementations, the length of the time window for detecting the operation boundary can be determined based on the minimum operation time, so that the solution is more in line with the production time and the actual production environment, and is convenient for accurately capturing the real operation boundary. In some examples, the time window length can be set to 2N to provide sufficient context information while avoiding including too much irrelevant information and to strike a balance between the amount of calculation and the efficiency of calculation.

[0114] After in-depth research, the inventor proposed Hypothesis 3: the location of the step boundary should be consistent with the change point of the image feature distribution.

[0115] According to Assumption 3, the energy statistics D(z a ,z b ), as defined in equation (9), is used to measure the difference between two distributions:

[0116] D(z a ,z b )=2E[d(z a ,z b )]-E[d(z a ,z a )]-E[d(z b ,z b )] (9)

[0117] where d(.,.) is a function that measures the distance between two samples, and E[d(.,.)] is the expected value of the distance. To align with the training objectives of large visual language models, d(.,.) can be set to the cosine similarity distance defined in equation (7), which is different from the traditional use of Euclidean distance as the energy statistic of d(.,.). To simplify the notation, d aa =E[d(z a ,z a )],d bb =E[d(z b ,z b )] and d bb =E[d(z a ,z b )].

[0118] Therefore, according to some embodiments of the present disclosure, the sample energy statistics may be fitted using the sample statistics in the following equation (10):

[0119]

[0120] Among them, k c is the candidate change point position. In this embodiment, each sample in the time window (each frame retained after filtering) can be regarded as a candidate change point.

[0121] In a time window (e.g., a time window of length 2N), there will be only one operational boundary, and the problem is simplified to detecting at most one change point in a time series. Therefore, the inventors propose to find the most likely position in each time window and quantify its confidence probability, and find the local maximum of the energy statistic in each time window as a candidate boundary point. In order to evaluate the confidence level of each candidate boundary point, a nonparametric permutation test can be performed. In one example, under the assumption that there are no other change points, the permutation test is as follows:

[0122] First, for a potential change point, N data points can be taken from both sides of the point and permuted to generate a new sequence of length T. Then the energy statistic is recalculated according to equation (10). This process is repeated multiple times, and the test statistic value is recorded after each permutation. The p-value of the candidate boundary point is determined by comparing with the permutation statistic to obtain the quantile.

[0123] If all possible permutations are considered, an exact p-value can be obtained, but the computation is too large. Therefore, the inventors propose that the p-value can be approximated by performing R random permutations. In some examples, in order to simplify the result presentation and focus on the contribution, the significance level p0 can be set to 0.1 to accept the boundary point, and the number of permutations R = 500 is specified as a constant value.

[0124] In order to prove the convergence rate of cosine energy change point detection at the detection step boundary, the inventors also introduce the following assumptions and lemmas.

[0125] Assumption 4: The image features {z k :k∈{1,2...,k c}} and {z k′ :k′∈{k c +1,k c +2...,T}} are from two independent multivariate distributions F a and F b The iid sample sequence of Where c∈(0,1).

[0126] In Assumption 4, if there is no change point in the time window, then F a =F b .

[0127] Assumption 5: If z a and z b from two different operations, then assuming that d ab ≥d aa +d bb .

[0128] Assumption 5 is a logical assumption for the distribution of image feature values, since it has the structure of the famous triplet loss in the field of contrastive learning. And most of the famous pre-trained visual-language encoders are based on contrastive learning, including CLIP used in this paper.

[0129] First, it can be proved that the statistics proposed in this implementation method follow the law of large numbers:

[0130] Lemma 1: Under Assumption 4, the sample mean defined by the cosine similarity distance (see Equation 7) follows the law of large numbers:

[0131]

[0132] The inventors prove that the cosine similarity distance sample statistics contain properties of two different distributions, as shown in Lemma 2.

[0133] Lemma 2: Under the assumption that 4 and d(.,.) is the cosine similarity distance as shown in Equation 7, we have:

[0134]

[0135] By proving these two lemmas, we can prove the convergence property of CECD in Theorem 1 as follows.

[0136] Theorem 1: Under Assumptions 4 and 5, if there exists a change point k in the given interval {1,2,...T} c , the process of using the cosine energy change point detection (CECD) algorithm to find the maximum energy statistics as the estimated change point, then the estimated change point position Converges to the true change point at a speed of o(1):

[0137]

[0138] Theorem 1 guarantees fast convergence of the detected potential step boundary locations.

[0139] The inventors also analyzed the complexity of the proposed cosine energy change point detection algorithm, as shown in Proposition 3, indicating that the complexity of the algorithm is polynomially related to the frame rate (video length):

[0140] Proposition 3: The time complexity of the proposed cosine energy change point detection algorithm is O(l×N 2 )=O(f 3 ).

[0141] With Theorem 1 and Proposition 3, the accuracy and efficiency of the proposed cosine energy change point detection algorithm are theoretically guaranteed.

[0142] In summary, in some implementations, detecting the operation boundary of the operation step from the filtered target video frame based on the cosine similarity distance and the non-parametric permutation test includes:

[0143] Divide the filtered target video frame into multiple non-overlapping time windows;

[0144] In each time window, the sample energy statistics of each candidate change point are calculated based on the following formula:

[0145]

[0146] Among them, k c is the candidate change point position, za and z b They represent candidate change points k respectively. c The feature vector sequence of the first and last two time periods, T is the total number of frames in the current analysis window, d(z i ,z j ) is the eigenvector z i and z j The cosine similarity distance between ;

[0147] In each time window, the candidate change point with the largest calculated sample energy statistic is identified as the candidate boundary point of the time window;

[0148] Perform a nonparametric permutation test on each candidate boundary point and calculate the statistical significance evaluation value of each candidate boundary point;

[0149] Based on the statistical significance evaluation values ​​of the candidate boundary points, the operational boundary is determined.

[0150] The following shows a pseudo code of a cosine energy change point detection algorithm proposed according to an exemplary embodiment of the present disclosure.

[0151]

[0152]

[0153] Step 4, calculate the standard time for each operation step.

[0154] Calculating the standard time for each operation step is an important step in generating a standard operating procedure (SOP). According to this embodiment, the standard time for each operation step is calculated based on multiple production video samples to ensure that the calculation result has statistical significance and can reflect the average level of different production cycles or different workers' operations.

[0155] For each identified operation step, the standard time is calculated by calculating the average time length between each segmentation point of all production videos. In some embodiments, calculating the standard time of each operation step includes: identifying the time period corresponding to the operation step in the multiple production videos; calculating the duration of the operation step in each production video; and calculating the average duration of the operation step based on the duration of the operation step in each production video as the standard time of the operation step.

[0156] In some embodiments, the calculation utilizes the Method-Time Measurement (MTM) system, because one TMU in the MTM system corresponds to 0.036 seconds, providing an accurate time unit basis for the calculation of standard time.

[0157] Step 5: Input the image features and predefined prompt information corresponding to the operation steps of the segmented target video into the pre-trained second visual language model to generate a text description of each operation step.

[0158] This step uses a pre-trained visual language model to generate a detailed text description of each operation step. This process combines the image features of each operation step in the segmented target video with pre-defined prompt information. In some embodiments, a pre-trained second visual language model with CLIP as the visual backbone can be used as a decoder, and the extracted feature vector is input into the decoder of the model. At the same time, in order to generate text descriptions that meet specific formats and requirements, prompt engineering technology can be used to guide the visual language model through pre-defined prompt templates so that it can generate structured and meaningful operation descriptions. For example, in the water valve production case study, two specific prompts were used: "This is a step in the water valve assembly process. Name this assembly step with a short phrase" and "Generate the operation process performed by the left and right hands". These prompts effectively guided the model to generate a text description containing step names and detailed operation instructions.

[0159] Through this step, accurate and detailed operation descriptions can be generated based only on video features and defined prompt information without specific domain annotation data. By combining visual features and language prompts, the model can understand the context of the operation and generate text descriptions that meet the requirements of specific production environments and are easy to understand and execute. This not only improves the efficiency of generating SOPs, but also ensures the quality and practicality of the generated content, so that the generated standard operating procedures can truly guide the actual production process.

[0160] Step 6: Generate a production procedure document based on the standard time and the text description.

[0161] Generating a standard operating procedure (SOP) document is the main goal of this embodiment, which organically combines the standard time and text description obtained in the previous steps. In some embodiments, this process is not just a simple splicing of information, but an organization of content in a structured and standardized way to ensure the practicality and readability of the final document. Specifically, for each identified operation step, the document may include the name of the step, a detailed description of the operation, and the calculated standard execution time. This format provides both clear operational guidance and a time benchmark for production management.

[0162] The generated SOP document can be used as a key tool to improve production efficiency, quality and production management. It not only describes in detail the execution method of each operation step, but also provides a quantitative basis for workflow optimization and performance evaluation by including standard time. The SOP automatically generated according to this embodiment can adapt to different production environments, especially in industries such as aviation manufacturing where design changes and process optimization are frequently performed. It provides an efficient, accurate and easy-to-update method to maintain standard operating procedures, thereby solving the problems of high manual dependence and difficulty in updating in traditional SOP generation methods, and providing a more flexible and efficient tool for production management.

[0163] The embodiment introduced above realizes the automatic analysis of production video and the generation of standard operating procedures by utilizing pre-trained visual language model and innovative algorithm, without the need for annotation data or predefined operation templates in specific fields. This zero-sample learning method significantly improves the efficiency and adaptability of SOP generation, so that it can be quickly applied to different production environments. Further, by dynamic similarity filtering (Dynamic Similarity Filtering, DSF) algorithm, the method can effectively identify and filter frames unrelated to production, improve the accuracy of generating SOP, and can realize accurate positioning of operation boundaries by cosine energy change point detection (Cosine-Energy Change-point Detection, CECD) algorithm, so that the division of operation steps is more reasonable. In addition, the disclosure can automatically calculate the standard time of each operation step, and generate detailed text description, greatly reducing the demand for manual participation, reducing the influence of subjective bias. According to the automation and accurate SOP generation scheme disclosed in the present invention, not only the efficiency of production management is improved, but also a flexible solution is provided for a rapidly changing manufacturing environment, which is of great significance to improving product quality, reducing production costs and optimizing production processes.

[0164] Figure 4 A schematic diagram of a process of dynamic similarity filtering and cosine energy change point detection according to an exemplary embodiment of the present disclosure is shown.

[0165] like Figure 4 As shown, first, multiple videos (including target videos and reference videos) are processed by the encoder to generate feature vectors. Then, dynamic similarity filtering is used to compare these feature vectors to identify necessary (production-related) and unnecessary (production-irrelevant) frames. Next, cosine energy change point detection is used to operate on the filtered frame sequence and detect the operation boundaries to divide different operation steps (from Step 1 to Step s).

[0166] Figure 5FIG. 2 shows a schematic diagram of a process for generating a text description according to an exemplary embodiment of the present disclosure. Figure 5 As shown in the figure, first, the feature vectors and prompt information from Step 1 to Step c are combined into comprehensive features, and then the comprehensive features are input into the decoder. The decoder generates two parts of output: the step name and the detailed description of the operation steps. The figure gives a specific output example, describing the name and specific operation process of the step "tighten the valve".

[0167] Next, the SOP generation scheme according to the present disclosure is compared with various baseline methods, focusing on evaluating the accuracy of action filtering and the quality of generated text descriptions. Two different datasets will be used for testing: one for a single-station water valve production process and the other for actual commercial aircraft spoiler production.

[0168] The test environment is configured with Python 3.11 and CentOS 7.9. To ensure optimal performance and efficiency, a single A100 GPU or dual A100 GPU setup is used. All models use pre-trained CLIP as their backbone (except for the baseline pre-trained large visual language model). The ground truth of the generated SOP is written by a group of industrial experts to ensure accuracy and practical relevance.

[0169] The test scenarios include various operations that are not related to production, such as looking at the phone or taking a break in production, as well as some erroneous operations. The tested scenario needs to exclude these operations that are not related to production and segment each production operation to generate an SOP using a large visual language model.

[0170] The first scenario is the water valve production process at a single workstation. Figure 6 As shown. First, the worker installs the rubber ring into the lower end of the valve body, then inserts the connecting rod, fixes it with screws, and puts it into the lower valve body. The next steps include assembling the upper part of the valve, tightening the screws, and clamping. The entire process spans about 3000 frames. The dataset consists of three groups, each containing two to three complete production videos performed by different workers. In each test, one video is designated as video 0, and the remaining videos are from the same group to assist the DSF algorithm. Figure 7 Examples of some ground truth SOPs and predicted SOPs are shown.

[0171] The second scenario involves the production of spoilers for commercial aircraft, where the finished product is Figure 8a This scene was chosen because it involves the handling of mechanical parts and flexible materials. Since the aircraft company does not allow the video data to be made public, the following is used here. Figure 8bSchematic diagram of the production process. The production process is complex and involves the collaboration of several workers, involving initial pre-assembly work, drying of primers, checking auxiliary materials, laying of composite materials and application of different colored films. Subsequent stages include assembly of rigid structures, covering with isolation films, bagging and leak testing, and finally curing in an autoclave.

[0172] The tests focused on the spoiler production phase up to the autoclave process, spanning approximately three hours at a frame rate of 1 / 3 per second.

[0173] The dataset consists of two groups, each containing three complete production videos using a mixed production line. Due to the low frame rate, the SOP generated for this case does not describe the specific operations in detail, but rather outlines the operations performed in each step. Fig. 9 Examples of some ground truth SOPs and predicted SOPs are shown.

[0174] Next, the performance of the dynamic similarity filtering (DSF) proposed in this disclosure in filtering production-related actions is evaluated.

[0175] This evaluation measures the proportion of relevant actions correctly retained and irrelevant actions filtered out to ensure that the generated captions do not contain irrelevant actions while retaining all necessary actions.

[0176] An operation can be defined as retained if more than 8 frames are retained (because large visual language models usually need at least 8 frames to generate predictions). If an irrelevant operation retains at most 2 frames (two boundary frames), it is considered filtered to minimize the risk of irrelevant operations being generated into the SOP document. This standard is strictly enforced in the test because the number of irrelevant frames that need to be filtered out can be as high as hundreds of frames.

[0177] Baseline methods include k-means, support vector machine (SVM), and multi-layer perceptron (MLP) using CLIP as feature encoder. Although the scheme according to the present disclosure is completely unsupervised, SVM, k-means, and MLP require a training set, so an additional training video is specially annotated for this purpose. All baseline models are carefully tuned to ensure high performance and fair evaluation. The performance of different methods can be evaluated using F1 score and accuracy. Table 1 gives the quantitative evaluation results of action filtering.

[0178] Table 1 Quantitative results of action filtering

[0179]

[0180] From the results in Table 1, it can be seen that without any supervised annotation, the scheme according to the present disclosure outperforms the supervised baseline method in all measurement indicators. This superiority stems from the fact that the baseline method classifies frames or operations separately, while the cosine energy change point detection proposed in the present disclosure is optimized in the entire production video, and the global characteristics of the production can be considered when classifying operations. In addition, the present disclosure can use cosine similarity to measure distance, which can adapt to the training characteristics of pre-trained visual encoders.

[0181] In addition, it can be observed that k-means performs well in aircraft spoiler production, but performs poorly in water valve production. By examining the classification results, the inventors found that this difference is due to the fact that irrelevant operations in aircraft production mainly include workers leaving the workstation or irrelevant personnel passing through the station, and there are fewer abnormal operations. These features are relatively easy to cluster because the presence or absence of personnel is an obvious feature. In contrast, in water valve production, there are many abnormal and irrelevant operations, and workers are always within the camera's field of view, making it difficult to capture the features through simple clustering.

[0182] also, Fig.10 Visualize the filtering and segmentation results of one of the water valve production videos. Fig.10 As can be seen in Figure 1, most irrelevant operations, such as checking the phone, taking a break during production, or workers leaving the workstation to perform other tasks or performing other abnormal operations, are filtered out. Only the erroneous operation in Step 8 is not filtered out.

[0183] To ensure the objectivity and accuracy of the experimental results, no performance evaluation indicators in the field of change point detection were used in the evaluation. The main reason is that since the production video frames are continuous, manually annotating the corresponding step segmentation points will introduce significant subjectivity. Even in the simplest segmentation scenario, such as a worker reaching out to grab a part for the next operation, it is difficult to objectively determine whether the segmentation point is at the end of the previous step, when the worker reaches out, or when he touches the next part. However, in the above Figure 2 The convergence and optimality of the step segmentation proposed in the present disclosure have been theoretically proved in the description of step 3. Therefore, in the field of change point detection, numerical experiments can be omitted.

[0184] Next, we evaluate the accuracy of the generated text SOP. As mentioned above, after filtering out frames irrelevant to production and segmenting the video into different operation steps, these feature images and prompt information are input into the decoder of the model to generate a text description of the corresponding step. Figure 7 and Fig. 9 The predicted SOP and ground truth SOP examples for water valve production and aircraft spoiler production are shown respectively. Three widely used text generation task metrics, BLEU-4, METEOR and ROUGE-L, are used in the test.

[0185] Since the traditional step description generation algorithm is supervised and requires specific operation steps to appear in the training set, it cannot meet the needs of structuring SOP subtitles. Therefore, the results are compared with several state-of-the-art large-scale visual language models in the test, which can adjust the generated structure according to the prompts. The baselines include Video-LLaVA, Video Chat, and Video-ChatGPT. The quantitative results in Table 2 show that the text description generated according to the present disclosure significantly outperforms all baseline models in all measurement indicators, once again proving the superiority of the zero-shot standard operating procedure generation method proposed by the public.

[0186] Table 2 Quantitative evaluation results of generated text descriptions

[0187]

[0188]

[0189] In addition to comparing the present disclosure with state-of-the-art large-scale visual language models, ablation studies are performed to demonstrate the outstanding benefits of various preferred embodiments of the present disclosure. The preferred embodiments are compared with versions lacking dynamic similarity filtering, cosine energy change point detection, or both. The performance of the method is further evaluated under the assumption of normal distribution. To this end, the distance function d(z i ,z i ) is modified to squared Euclidean distance | z i -z i | 2 , and use change point detection based on T-test statistics under the assumption of normal distribution to determine the operating boundaries. To ensure fairness and accuracy, all comparison methods have been thoroughly fine-tuned and optimized.

[0190] From the results, it can be observed that removing DSF or CECD will lead to a decrease in performance, proving that each of these two parts can bring additional beneficial effects. In addition, after changing to the widely used normal distribution assumption (P-NA), the performance also dropped significantly, proving that using cosine similarity distance to measure the distance between samples is more in line with the training objectives of pre-trained visual encoders.

[0192] According to an embodiment of the present disclosure, a video-based self-supervised production anomaly detection and progress prediction device is also proposed, the device comprising:

[0193] A video acquisition unit, used to acquire video data of the production process;

[0194] A latent feature extraction unit, used for mapping the image features in the video data to the latent space of a Gaussian mixture model (GMM) by using an encoder, to obtain latent variables for representing the latent features in the image, wherein the Gaussian components in the GMM correspond one-to-one to the pre-divided production stages;

[0195] A SAM generation unit, for generating a spatial activation map using a feature map generated before the last convolutional block of the encoder, wherein the spatial activation map is used to extract and represent attention features related to different production stages in the image;

[0196] An image reconstruction unit, configured to reconstruct the latent variables output by the encoder into an image using the decoder, and calculate a spatial activation reconstruction error of the reconstructed image weighted by the spatial activation map;

[0197] The model is trained based on a comprehensive loss function that includes the cross entropy loss for production schedule prediction, the triple loss for latent space feature embedding for different production stage feature representations, the negative log-likelihood loss of latent variables in GMM, and the spatial activation reconstruction error:

[0198] The anomaly detection and progress prediction unit is used to process the real-time collected video data using the encoder of the trained model, perform production anomaly detection and predict the production stage based on the obtained latent variables.

[0199] For other details and beneficial effects of this embodiment, please refer to the above related introduction, which will not be repeated here.

[0200] The disclosed embodiments may be applied to a computer system / server that may operate with numerous other general or special computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with a computer system / server include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, and the like.

[0201] Computer systems / servers may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. In general, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

[0202] Fig.11 An electronic device provided for at least one embodiment of the present disclosure includes a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the zero-sample standard operating procedure generation method described in any embodiment or implementation of the present disclosure when executing the computer instructions.

[0203] At least one embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method for generating a zero-sample standard operating procedure described in any embodiment or implementation of the present disclosure is implemented.

[0204] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0205] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0206] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0207] The embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules in computer program instructions encoded on a tangible non-temporary program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0208] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits), and the apparatus can also be implemented as special purpose logic circuits.

[0209] Computers suitable for executing computer programs include, for example, general and / or special microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to this large-capacity storage device to receive data from it or to transmit data to it, or both. However, the computer does not necessarily have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0210] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0211] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of the specific embodiments of specific inventions. Certain features described in multiple embodiments in this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claim protection, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0212] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0213] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0214] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for generating a zero-sample standard operating procedure, characterized in that: include: Obtain multiple production videos and extract feature vectors of frames in each production video using a pre-trained first visual language model; Selecting one of the multiple production videos as a target video, using other production videos in the multiple production videos as reference videos, identifying production-related frames from the target video frames based on feature vectors and filtering out frames that are not related to production; Detecting operation boundaries of operation steps from the filtered target video frame based on cosine similarity distance and non-parametric permutation test, and dividing the target video into multiple operation steps according to the operation boundaries; Calculate the standard time for each operation step; Input the image features and predefined prompt information corresponding to the operation steps of the segmented target video into the pre-trained second visual language model to generate a text description of each operation step; Based on the standard time and the text description, a production procedure document is generated.

2. The method according to claim 1, characterized in that Identify production-related frames from target video frames based on feature vectors and filter out frames not related to production, including: Assign a different cluster to each frame in the target video; For each reference video frame, a target video frame matching the reference video frame is found based on the similarity of the feature vectors, and the reference video frame is assigned to a cluster corresponding to the matching target video frame; After traversing all reference video frames, if there is a reference video frame in the cluster corresponding to the target video frame, it is judged that the target video frame is related to production. If there is no reference video frame in the cluster corresponding to the target video frame, it is judged that the target video frame is not related to production, and the target video frame not related to production is removed.

3. The method according to claim 1, characterized in that Finding a target video frame matching the reference video frame based on the similarity of inter-frame image features, and assigning the reference video frame to a cluster corresponding to the matching target video frame, including: Establish the value function dp[k,t] for dynamic programming: dp[k,t]=min(dp[k,t-1]+d(z i,t-1 ,z 0,k ),dp[k-1][t]+r(i,t,k),dp[k-1][t-1]+d(z i,t ,z 0,k ), Where k represents the cluster index, t represents the frame index of the reference video, and z i,t represents the feature vector of the tth frame of the reference video i, z 0,k represents the feature vector of the target video frame assigned to the kth cluster, d(,) represents the cosine similarity distance between two feature vectors, r(i,t,k) is the regularization term, and dp[0,0]=0; For the t-th frame of reference video i, traverse all values ​​of k to calculate the value function dp[k,t], determine that the target video frame corresponding to the cluster index that minimizes the value function dp[k,t] is the target video frame that matches the t-th frame of reference video i, and assign the t-th frame of reference video i to the cluster corresponding to the matching target video frame.

4. The method according to claim 3, characterized in that If the cluster corresponding to the target video frame has a reference video frame, then the target video frame is determined to be related to the production; if the cluster corresponding to the target video frame has no reference video frame, then the target video frame is determined to be unrelated to the production, including: If for any reference video i, does not satisfy Then the target video frame corresponding to the kth cluster is judged to be relevant to the production, where l i represents the total number of frames of reference video i, x i,t,k is a binary variable. When the tth frame of the reference video i is assigned to the kth cluster, x i,t,k =1, otherwise x i,t,k =0; If there is a reference video i, satisfying It is then determined that the target video frame corresponding to the kth cluster is irrelevant to production.

5. The method according to claim 1, characterized in that The operation boundaries of the operation steps are detected from the filtered target video frames based on the cosine similarity distance and non-parametric permutation test, including: Divide the filtered target video frame into multiple non-overlapping time windows; In each time window, the sample energy statistics of each candidate change point are calculated based on the following formula: Among them, k c is the candidate change point position, z a and z b They represent candidate change points k respectively. c The feature vector sequence of the first and last two time periods, T is the total number of frames in the current analysis window, d(z i ,z j ) is the eigenvector z i and z j The cosine similarity distance between ; In each time window, the candidate change point with the largest calculated sample energy statistic is identified as the candidate boundary point of the time window; Perform a nonparametric permutation test on each candidate boundary point and calculate the statistical significance evaluation value of each candidate boundary point; Based on the statistical significance evaluation values ​​of the candidate boundary points, the operational boundary is determined.

6. The method according to claim 5, characterized in that The method further comprises: Before detecting the operation boundary of the operation step, the minimum operation time of the production process is obtained based on method-time measurement (MTM), and the length of the time window is determined based on the minimum operation time.

7. The method according to claim 1, characterized in that Calculate the standard time for each operation step, including: Identifying a time period corresponding to the operation step in the multiple production videos; Calculate the duration of this operation step in each production video; Based on the duration of the operation step in each production video, the average duration of the operation step is calculated as the standard time of the operation step.

8. The method according to claim 1, characterized in that The pre-trained first visual language model uses the CLIP model as a visual backbone network, and / or the pre-trained second visual language model uses the CLIP model as a visual backbone network.

9. A zero-sample standard operating procedure generation device, characterized in that: include: a video acquisition unit, configured to acquire a plurality of production videos and extract a feature vector of a frame in each production video using a pre-trained first visual language model; A dynamic similarity filtering unit, configured to select one of the multiple production videos as a target video, use other production videos in the multiple production videos as reference videos, identify production-related frames from the target video frames based on feature vectors, and filter frames that are not related to production; A cosine energy change point detection unit, used for detecting operation boundaries of operation steps from the filtered target video frame based on cosine similarity distance and non-parametric permutation test, and dividing the target video into multiple operation steps according to the operation boundaries; A standard time calculation unit, used to calculate the standard time of each operation step; A text description unit is used to input image features and predefined prompt information corresponding to the operation steps of the segmented target video into a pre-trained second visual language model to generate a text description of each operation step; The SOP generating unit is used to generate a production procedure document based on the standard time and the text description.

10. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to implement the method according to any one of claims 1 to 8 when executing the computer instructions.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images