Method, device and storage medium for detecting operation procedure of production line

Abstract

The application relates to the field of visual detection, and discloses a detection method, equipment and storage medium for a production line operation process. The method comprises the following steps: collecting a work station video stream, and performing frame-by-frame inference on the work station video stream according to a target detection model to obtain a detection result; evaluating the detection result of each video frame according to a trigger condition to obtain a condition signal; statistically obtaining a condition satisfaction rate by taking the proportion of the condition signals representing the condition satisfying the trigger condition in a sliding window; comparing the condition satisfaction rate with an active threshold and an inactive threshold, generating a determination result of the start of a target process step when the condition satisfaction rate is not less than the active threshold; generating a determination result of the completion of the target process step when the condition satisfaction rate is not greater than the inactive threshold; and generating a detection result of the target process step according to the determination result. In the embodiment of the application, the interference is inhibited, the false trigger rate is reduced, and the response speed and the anti-interference capability are decoupled.

Description

Technical Field

[0001] This invention relates to the field of visual inspection, and more particularly to a method, equipment, and storage medium for inspection of production line processes. Background Technology

[0002] In modern manufacturing, standard operating procedures (SOPs) are fundamental to ensuring consistent product quality and safe production line operation. Currently, most production lines rely on manual inspections, post-production video sampling, or sensor solutions to monitor workers' SOP execution. These methods suffer from low monitoring efficiency, inability to correct errors in real time, and inability to perceive specific operational behaviors.

[0003] In recent years, computer vision technology has been introduced into production line SOP monitoring scenarios, using object detection in workstation video streams to determine the execution status of process steps. However, the inventors discovered during implementation that existing visual inspection solutions have at least the following drawbacks: First, single-frame detection is susceptible to interference, leading to false alarms. Production line environments are rife with interference, such as: rapid movement of a worker's arm causing the target to appear and disappear momentarily in a video frame; camera shake or sudden changes in lighting causing the target to be briefly lost; and in collaborative scenarios, other workers' arms or tools repeatedly entering the same detection area. Existing solutions use a frame-by-frame judgment method, lacking measures to suppress jitter in inter-frame detection results, resulting in a persistently high false trigger rate.

[0004] Secondly, the fixed threshold determination method lacks flexibility. A single fixed threshold cannot simultaneously ensure both response speed and anti-interference capability. If the threshold is too low, false alarms will increase; if the threshold is too high, the response will be slow or even result in missed detections. Furthermore, different processes have significantly different requirements for detection sensitivity. Instantaneous actions require rapid response, while continuous actions need to avoid false triggers caused by intermediate jitter. Fixed thresholds are difficult to adapt to these differentiated needs.

[0005] Therefore, in order to address the technical problems in existing technologies, such as the susceptibility of visual inspection of production line operations to false triggering due to inter-frame interference, and the difficulty in balancing response speed and anti-interference capability with fixed threshold judgment methods, a new technology is needed to solve the current problems. Summary of the Invention

[0006] The main objective of this invention is to solve the technical problems that visual inspection of production line processes is easily affected by inter-frame interference, resulting in false triggering, and that fixed threshold judgment methods are difficult to balance response speed and anti-interference capability.

[0007] The first aspect of this invention provides a method for detecting a production line operation process, comprising the steps of: The video stream of the workstation is collected, and the video stream of the workstation is inferred frame by frame according to the preset target detection model to obtain the detection results of the preset category targets in each video frame. According to the preset triggering conditions of the target process step corresponding to the workstation video stream, the detection results of each video frame are evaluated to obtain a frame-by-frame condition signal, wherein the condition signal is used to characterize whether the current video frame meets the triggering conditions of the corresponding target process step. The condition signals are written into a sliding window frame by frame, and the proportion of the condition signals that represent the fulfillment of the triggering condition within the sliding window is counted to obtain the condition satisfaction rate. The condition satisfaction rate is compared with a preset active threshold and a preset inactive threshold. When the condition satisfaction rate is not less than the active threshold, a determination result for the start of the target process step is generated; when the condition satisfaction rate is not greater than the inactive threshold, a determination result for the completion of the target process step is generated, wherein the active threshold is greater than the inactive threshold. Based on the determination result, the detection result of the target process step is generated.

[0008] Optionally, in a first implementation of the first aspect of the present invention, the detection result includes: a preset category target name, a bounding box corresponding to the category name, and the step of evaluating the detection result of each video frame according to the preset triggering condition of the target process step corresponding to the workstation video stream to obtain a frame-by-frame conditional signal includes: The detection results of the current video frame are aggregated according to category name to obtain the bounding box set corresponding to each category name; The preset triggering conditions of the target process steps corresponding to the workstation video stream are represented as a condition expression tree. The leaf nodes of the condition expression tree are atomic conditions for judging the occurrence or location of the preset category target, and the internal nodes of the condition expression tree are nodes for logically combining the judgment results of the child nodes. Based on the bounding box set, the leaf nodes of the condition expression tree are evaluated to obtain the evaluation results of each leaf node. The evaluation results of each leaf node are then recursively combined from bottom to top along the condition expression tree to generate a combined result, and the combined result is determined as the condition signal of the current video frame.

[0009] Optionally, in a second implementation of the first aspect of the present invention, the atomic condition includes: a category occurrence condition, and the step of evaluating each leaf node of the condition expression tree based on the bounding box set to obtain the evaluation result of each leaf node includes:

[0010] in, The condition for the category to appear in the first The evaluation results of video frames. This refers to the frame number of the video frame. The default category name. For the first The type name in the video frame is The corresponding bounding box set, This represents the number of bounding boxes in the bounding box set. This is a preset minimum target quantity threshold. This is an indicator function; the value is 1 if the condition within the parentheses is true, and 0 if it is false.

[0011] Optionally, in a third implementation of the first aspect of the present invention, the atomic condition further includes: a category located in a region condition, and the step of evaluating each leaf node of the condition expression tree based on the bounding box set to obtain the evaluation result of each leaf node includes: Calculate the intersection-union area ratio between the bounding box corresponding to the preset category name and the preset region of interest; When at least one bounding box in the set of bounding boxes corresponding to the preset category name has an intersection-union area ratio not less than a preset area ratio threshold, a determination result is generated indicating that the category is located in the region and the determination result is used as the evaluation result of the corresponding leaf node.

[0012] Optionally, in a fourth implementation of the first aspect of the present invention, the step of calculating the proportion of condition signals representing the fulfillment of the triggering condition within the sliding window to obtain the condition satisfaction rate includes:

[0013] in, This represents the condition satisfaction rate for processing the t-th video frame. This refers to the frame number of the video frame. The length of the sliding window queue. This refers to the frame number corresponding to the condition signal within the sliding window queue. For the first The value of the condition signal of the frame after it is written into the sliding window queue. The value can be 0 or 1. 1 represents the first If the condition signal of the frame satisfies the triggering condition, 0 represents the first The condition signal of the frame does not meet the triggering condition.

[0014] Optionally, in a fifth implementation of the first aspect of the present invention, the step of comparing the condition satisfaction rate with a preset active threshold and a preset inactive threshold includes: Record the current determination status of the target process step, including: not started and in progress. When the current determination state is not started and the condition satisfaction rate is not less than the active threshold, a determination result for the start of the target process step is generated, and the current determination state is updated to the in progress state. When the current determination state is in progress and the condition satisfaction rate is not greater than the inactive threshold, a determination result for the completion of the target process step is generated, and the current determination state is updated to the not started state. When the condition satisfaction rate is between the inactive threshold and the active threshold, the current determination state remains unchanged.

[0015] Optionally, in a sixth implementation of the first aspect of the present invention, the step of generating the detection result of the target process step based on the determination result includes: Based on the process semantics of the target process step, a trigger mode is selected, wherein the trigger mode includes: rising edge trigger mode and falling edge trigger mode; When the rising edge trigger mode is selected, a detection result indicating that the target process step has been completed is generated based on the determination result of the start of the target process step. When the falling edge trigger mode is selected, a detection result indicating that the target process step has been executed is generated based on the determination result of the target process step completion.

[0016] Optionally, in a seventh implementation of the first aspect of the present invention, before acquiring the video stream at the workstation, the following step is further included: Receive configuration file, which includes: trigger conditions for each target process step, window length of the sliding window, active threshold, inactive threshold, and reference resolution; The configuration file is parsed, and when the triggering condition includes a region location condition, the vertex coordinates of the region corresponding to the region location condition are scaled proportionally according to the ratio between the actual inference resolution and the reference resolution of the configuration file to generate scaled vertex coordinates. The scaled vertex coordinates are used to evaluate the detection results of each video frame.

[0017] A second aspect of the present invention provides a detection device for a production line operation process, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the detection device for the production line operation process to perform the aforementioned detection method for the production line operation process.

[0018] A third aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the detection method for the production line operation process described above.

[0019] In this embodiment of the invention, a preset target detection model is used to perform frame-by-frame reasoning on the workstation video stream. The frame-by-frame detection results are evaluated based on preset triggering conditions for the target process step. The condition signals for each frame are written into a sliding window to statistically determine the condition satisfaction rate. This rate is then compared with preset active and inactive thresholds. The target process step is determined to begin when the condition satisfaction rate is not less than the active threshold and to be completed when it is not greater than the inactive threshold, thus generating the detection result for the target process step. The sliding window statistical method transforms single-frame judgment into proportional judgment within a window. By smoothing the instantaneous jitter signal through the window, it suppresses inter-frame interference such as the instantaneous appearance and disappearance of the target and brief intrusion into the detection area, reducing the false trigger rate. Simultaneously, the active and inactive thresholds constitute a hysteresis comparison interval, and the judgment result remains in the previous state between the two thresholds without flipping. The independent setting of the two thresholds decouples response speed and anti-interference capability, allowing for configuration according to the sensitivity requirements of different processes. This solves the technical problems of production line visual inspection being susceptible to false triggers due to inter-frame interference, and the difficulty in balancing response speed and anti-interference capability with fixed threshold judgment methods. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of an embodiment of the detection method for production line operations in this invention; Figure 2 This is a schematic diagram of a specific embodiment of the 102 steps of the detection method in the production line operation process of the present invention; Figure 3 This is a schematic diagram of a specific embodiment of the 104 steps of the detection method for production line operations in this invention; Figure 4 This is a schematic diagram of a specific embodiment of the detection method for production line operation processes in this invention, which includes 105 steps. Figure 5 This is a schematic diagram of an embodiment of the testing equipment for production line operations in this invention. Detailed Implementation

[0021] This invention provides a method, equipment, and storage medium for detecting production line processes.

[0022] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0023] In the description of the embodiments disclosed in this invention, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 An embodiment of the detection method for production line operations in this invention includes the following steps: 101. Collect the workstation video stream, and perform frame-by-frame reasoning on the workstation video stream according to the preset target detection model to obtain the detection results of the preset category targets in each video frame; In this embodiment, the workstation video stream is acquired in real time by a camera deployed at the workstation. The real-time video stream is pulled from the workstation camera according to the RTSP protocol through a preset media pull module. The pulled video stream is decoded to obtain video frames one by one. The decoded video frames are written into a preset frame buffer and a frame ready notification is sent for the inference module to read.

[0025] The inference module reads video frames from a preset frame buffer and loads the target detection model corresponding to the workstation according to the workstation identifier. The target detection model is a pre-trained YOLO single-stage target detection model. In a single forward propagation, the target detection model simultaneously predicts the target bounding box and class probability to meet the frame rate requirements of real-time detection on the production line.

[0026] In one alternative implementation, let the input video frame be... Target detection model The output is:

[0027] in, Let i be the bounding box of the i-th detected target. The coordinates of the top-left corner of the bounding box. The coordinates of the bottom right corner of the bounding box. Let i be the category of the i-th detected target. Let N be the detection confidence score for the i-th target, and N be the total number of targets detected in the current video frame. Optionally, targets with a detection confidence score lower than a preset confidence threshold are filtered out to obtain the detection results for targets of preset categories in each video frame. Preset target categories include, for example, human hands, electric screwdrivers, screws, etc. Specific categories can be obtained by pre-labeling samples and training based on the workstation's process content. After obtaining the detection results for each video frame, the detection results, along with the workstation identifier, are encapsulated into an inference result message and sent to the downstream process processing module for subsequent trigger condition evaluation. For example, the inference result message includes: workstation identifier, bounding boxes of each detected target, and a mapping dictionary from category identifier to category name.

[0028] Specifically, prior to step 101, the following specific implementation methods are also included: 1011. Receive configuration file, the configuration file includes: trigger conditions for each target process step, window length of the sliding window, active threshold, inactive threshold, and reference resolution; 1012. Parse the configuration file. When the triggering condition includes a region location condition, scale the vertex coordinates of the region corresponding to the region location condition proportionally according to the ratio between the actual inference resolution and the reference resolution of the configuration file to generate scaled vertex coordinates. The scaled vertex coordinates are used to evaluate the detection results of each video frame.

[0029] In steps 1011-1012, the configuration file can be in JSON format. In addition to the trigger conditions for each target process step, the window length of the sliding window, the active threshold, the inactive threshold, and the reference resolution, the configuration file can also include: workstation name, model path of the target detection model, region name and vertex coordinates of each region of interest, step identifier and trigger mode of each target process step.

[0030] When parsing the configuration file, a corresponding condition expression tree is constructed based on the trigger conditions of each target process step. Independent sliding window instances are created for each target process step, allowing for independent configuration of the window length, activity threshold, inactivity threshold, and trigger mode to adapt to the detection features of different processes. When the trigger conditions include regional location conditions, the reference resolution of the configuration file is set to [value missing]. The actual inference resolution is Then the horizontal scaling factor and the vertical scaling factor are respectively: .

[0031] Based on the horizontal and vertical scaling factors, the coordinates of each vertex in the region corresponding to the region's location conditions are... Scaling proportionally to This generates scaled vertex coordinates; thus, the vertex coordinates of the region of interest only need to be labeled once according to the reference resolution to adapt to different actual inference resolutions, ensuring consistency in the determination of region position conditions under different inference resolutions. Through the configuration file-driven approach described above, new target process steps and their triggering conditions can be defined at runtime without modifying the source code, reducing the deployment cost of new processes.

[0032] 102. Based on the preset triggering conditions of the target process step corresponding to the workstation video stream, the detection results of each video frame are evaluated to obtain a frame-by-frame condition signal, wherein the condition signal is used to characterize whether the current video frame satisfies the triggering conditions of the corresponding target process step. In this embodiment, the triggering condition is used to characterize the target state that each preset category target in the workstation screen should meet when the target process step starts or is completed. For example, the triggering condition can be "the human hand is in the operation area" or "the screw appears in the screen and the electric screwdriver is in the screw tightening area".

[0033] Evaluating the detection results for each video frame involves substituting the detection result of the current video frame into the trigger condition for logical judgment, obtaining a Boolean value representing whether the current video frame meets the trigger condition, and then using this Boolean value as the condition signal for the current video frame. t is the frame number of the video frame. A value of 1 indicates that the current video frame meets the trigger condition. A value of 0 indicates that the current video frame does not meet the trigger condition. This is the sequence of conditional signals obtained after evaluating each frame. This serves as the input for subsequent writing to the sliding window to calculate the condition satisfaction rate.

[0034] It should be noted that the trigger conditions support the expression of composite conditions. This means that multiple occurrence and location judgments for preset target categories can be arbitrarily nested and combined using logical AND, OR, and NOT operations to express composite spatial category logical conditions, such as "target A is within region R and target B is in the frame, but target C is not in region S". This decouples frame-by-frame geometric condition judgments from subsequent window statistical processing, ensuring that the expression and modification of trigger conditions do not affect the execution logic of the sliding window statistics.

[0035] Specifically, the detection results include: the category name of the preset category target, and the bounding box corresponding to the category name. Please refer to [link / reference]. Figure 2 , Figure 2 This is a schematic diagram of a specific embodiment of step 102 of the detection method in the production line operation process of the present invention. Step 102 includes the following specific implementation methods: 1021. Aggregate the detection results of the current video frame according to the category name to obtain the bounding box set corresponding to each category name; 1022. The preset triggering conditions of the target process steps corresponding to the workstation video stream are represented as a condition expression tree, wherein the leaf nodes of the condition expression tree are atomic conditions for judging the occurrence or location of the preset category target, and the internal nodes of the condition expression tree are nodes for logically combining the judgment results of the child nodes. 1023. Based on the bounding box set, evaluate each leaf node of the condition expression tree to obtain the evaluation result of each leaf node, and recursively combine the evaluation results of each leaf node from bottom to top along the condition expression tree to generate a combined result, and determine the combined result as the condition signal of the current video frame.

[0036] In steps 1021-1023, the detection results of the current video frame are aggregated by category name. This means constructing a dictionary of the detection targets for the current video frame, using the category name as the key and a list of all bounding boxes corresponding to that category name as the value.

[0037] in, Let be the target dictionary for the t-th video frame. The name of the species. The type name in the current video frame is The conditional expression tree is a set of bounding boxes composed of the bounding boxes of each detected target. The node types supported by the conditional expression tree are shown in Table 1 below: Table 1. Node Types of the Conditional Expression Tree

[0038] Among them, the has_class node and class_in_roi node are leaf nodes, i.e. atomic conditions, and nodes, or nodes, and not nodes are internal nodes. The root node of the condition expression tree corresponds to the triggering condition of the target process step. Through nesting combinations at any depth, the complex detection logic in the production line operation can be accurately expressed.

[0039] In step 1023, the evaluation of the conditional expression tree is performed recursively, and the evaluation function is defined. Where T is a node in the conditional expression tree, Let be the target dictionary for the t-th video frame.

[0040] When node T is of type has_class This means evaluating the condition for the occurrence of the category corresponding to node T, which is then used as the evaluation result of node T. In other words, it finds the cn and m categories under the has_class node type. When the type of node T is class_in_roi, The result of evaluating the category corresponding to node T in the region condition is used as the evaluation result of node T, that is, in the region name R of the class_in_roi node type. T Find the cn types that satisfy the intersection-union area ratio threshold τ.

[0041] When node T is of type AND, the results of evaluating each child node of node T are combined using a logical AND operation, i.e. In the list of child nodes, node c. When the type of node T is OR, the evaluation results of each child node of node T are logically ORed together, i.e. When node T is of type NOT, the logical NOT operation is performed on the evaluation results of node T's child nodes, i.e. Where T.type is the node type, T.children is the list of child nodes of node T, T.child is the child node of the not type node, ∧ represents logical AND, and ∨ represents logical OR. Represents the logical NOT operation, the conditional signal for the current video frame. The result of evaluating the root node of the conditional expression tree, i.e. .

[0042] For example, the evaluation process for the trigger condition "and(or(has_class('hand'), has_class('electric screwdriver')), not(has_class('screw')))" involves first evaluating the leaf nodes has_class('hand'), has_class('electric screwdriver'), and has_class('screw') separately. Then, the results of the first two leaf nodes are combined using a logical OR operation, and the result of the third leaf node is combined using a logical NOT operation. Finally, the result of the logical OR operation and the result of the logical NOT operation are combined using a logical AND operation to obtain the condition signal for the current video frame. In this way, by recursively combining the evaluation results of each leaf node from bottom to top, the condition signal for any complex trigger condition in the current video frame can be obtained, and the configuration and modification of the trigger condition do not require changes to the evaluation execution logic.

[0043] Furthermore, the atomic conditions include: category occurrence conditions, which in step 1023 include the following specific implementations:

[0044] in, The condition for the category to appear in the first The evaluation results of video frames. This refers to the frame number of the video frame. The default category name. For the first The type name in the video frame is The corresponding bounding box set, This represents the number of bounding boxes in the bounding box set. This is a preset minimum target quantity threshold. This is an indicator function; the value is 1 if the condition within the parentheses is true, and 0 if it is false.

[0045] It should be noted that the minimum target quantity threshold The default value is 1, meaning that the category condition is met when a target with the corresponding category name appears in the image. In scenarios where it's necessary to eliminate false single-target detection interference, the minimum target quantity threshold can also be set. Set to an integer greater than 1, only when the number of targets with the same name in the image reaches [a certain threshold]. The system determines whether a category occurrence condition is met only when the condition is met, thus improving the reliability of the category occurrence condition determination. For example, for the category occurrence condition has_class('hand'), and When the number of bounding boxes with the category name "hand" in the target dictionary of the t-th video frame is not less than 1. The value is 1; otherwise The value is 0.

[0046] When the triggering condition includes multiple category occurrence conditions, each category occurrence condition corresponds to an independent leaf node in the condition expression tree. Each category occurrence condition is evaluated separately based on the same target detection dictionary, without affecting each other. In this way, through flexible configuration of the minimum target quantity threshold, the category occurrence condition can express both the judgment of target occurrence and the counting judgment of target quantity reaching a specified number. The evaluation result is used as the evaluation result of the corresponding leaf node and participates in the subsequent recursive combination along the condition expression tree.

[0047] Furthermore, the atomic conditions also include: the category is located in the region condition, and step 1023 includes the following specific implementation: 10231. Calculate the intersection-union area ratio between the bounding box corresponding to the preset category name and the preset region of interest; 10232. When there is at least one bounding box in the set of bounding boxes corresponding to the preset category name, and the intersection-union area ratio of the bounding boxes is not less than the preset area ratio threshold, a determination result is generated to determine that the category is located in the region condition is met, and the determination result is used as the evaluation result of the corresponding leaf node.

[0048] In steps 10231-10232, the preset region of interest can be represented as a convex polygon. convex polygon With L vertices The vertices are arranged sequentially, and the coordinates of each vertex are scaled vertex coordinates after being labeled according to the reference resolution and scaled proportionally.

[0049] For the bounding box corresponding to the preset category name Represent the bounding box as a bounding box polygon. Calculate the intersection area of ​​the bounding box polygon and the convex polygon. The intersection region of the bounding box polygon and the convex polygon is calculated using a convex polygon intersection algorithm. The area of ​​the intersection region is then obtained by integrating the area of ​​the vertices of the intersection region, and the area of ​​the bounding box is calculated as a result. for The area of ​​intersection (IoA) between the bounding box and the preset region of interest is:

[0050] in, This represents the area of ​​the intersection of the bounding box polygon and the convex polygon. The area of ​​the bounding box is denoted by , and the intersection-union area ratio represents the proportion of the bounding box area that falls within the preset region of interest to the total area of ​​the bounding box, with a value range of [0, 1].

[0051] The evaluation result of the category location condition in the t-th video frame is:

[0052] in, The preset area ratio threshold is... The category name in the t-th video frame is The corresponding bounding box set, This indicates that it exists. The bounding boxes in the bounding box set, The function is an indicator function; it takes a value of 1 when the condition within the parentheses is true, and a value of 0 otherwise. The above formula indicates that when at least one bounding box in the bounding box set has an intersection-union area ratio (IUAR) with the preset region of interest that is not less than the preset area ratio threshold, the condition for determining that the category is located in the region is met, and the result is 1; otherwise, it is 0. The preset area ratio threshold can be configured according to the accuracy requirements of the process for position determination. For example, in a scenario where the target is required to fall mostly within the region, the preset area ratio threshold can be set to a value between 0.5 and 0.9.

[0053] It should be noted that using the intersection-merge area ratio instead of the bounding box center point to determine the positional relationship can avoid misjudgment when the target part enters the region, making the determination of the region position condition more in line with the actual semantics of the process. The evaluation result of the above categories in the region condition is used as the evaluation result of the corresponding leaf node and participates in the subsequent recursive combination along the condition expression tree.

[0054] 103. Write the condition signals frame by frame into a sliding window, and count the proportion of the condition signals in the sliding window that represent the fulfillment of the triggering condition to obtain the condition satisfaction rate; In this embodiment, the sliding window is a queue of fixed length, implemented using a fixed-length double-ended queue. For each video frame processed, the condition signal of the current frame is written to the tail of the queue. When the queue length reaches the window length, the condition signal at the head of the queue is removed, ensuring that the sliding window always retains the condition signals of the most recent W video frames. The core parameters involved in the statistical determination of the sliding window are shown in Table 2 below: Table 2. Parameters of the sliding window

[0055] Each target process step maintains an independent sliding window instance. The window length, active threshold, inactive threshold, and trigger mode of each sliding window instance can be configured independently to adapt to the detection features of different processes.

[0056] Simultaneously, internal states are maintained for each target process step, including: sliding window queues. and the current determination status of the target process step, where The value obtained after the condition signal of the i-th frame is written into the sliding window queue. The length of the sliding window queue. This refers to the frame number of the video frame.

[0057] The motivation behind the sliding window design is that the conditional signals in a single frame exhibit numerous instantaneous fluctuations, such as pulse-like jumps caused by a hand quickly sweeping across the detection area, temporary target loss due to changes in lighting, or occlusion. Frame-by-frame judgment would directly transmit these instantaneous fluctuations as false triggers. By writing the conditional signals frame-by-frame into the sliding window and statistically analyzing the condition satisfaction rate within the window, the instantaneous judgment of a single frame is elevated to continuous verification within the window, thus suppressing the interference of these instantaneous fluctuations on process judgment from a mechanistic perspective.

[0058] Specifically, step 103, "statistically calculating the proportion of condition signals representing the fulfillment of the triggering condition within the sliding window to obtain the condition satisfaction rate," includes the following specific implementation methods:

[0059] in, This represents the condition satisfaction rate for processing the t-th video frame. This refers to the frame number of the video frame. The length of the sliding window queue. This refers to the frame number corresponding to the condition signal within the sliding window queue. For the first The value of the condition signal of the frame after it is written into the sliding window queue. The value can be 0 or 1. 1 represents the first If the condition signal of the frame satisfies the triggering condition, 0 represents the first The condition signal of the frame does not meet the triggering condition.

[0060] 104. Compare the condition satisfaction rate with a preset active threshold and a preset inactive threshold. When the condition satisfaction rate is not less than the active threshold, generate a determination result for the start of the target process step; when the condition satisfaction rate is not greater than the inactive threshold, generate a determination result for the completion of the target process step, wherein the active threshold is greater than the inactive threshold. In this embodiment, the active threshold is greater than the inactive threshold, and the two thresholds constitute a hysteresis comparison interval. , The inactivity threshold, This is the activity threshold.

[0061] The purpose of the hysteresis comparison interval is to introduce a hysteresis effect. When the condition satisfaction rate falls within the hysteresis comparison interval, the current decision state remains unchanged regardless of whether it is in progress or not. The physical meaning of the hysteresis effect is similar to that of a Schmitt trigger in electronics. If a single fixed threshold is used, when the condition satisfaction rate oscillates around the threshold, the decision state will repeatedly flip, producing a chattering phenomenon. However, by setting active and inactive thresholds, the condition satisfaction rate must cross the entire hysteresis comparison interval to trigger a change in the decision state. Changes in the condition signal of a single or a few frames will not cause the decision state to flip, thus eliminating the chattering phenomenon.

[0062] Meanwhile, the active and inactive thresholds can be configured independently. The active threshold controls the response speed and false alarm resistance at the start of the process, while the inactive threshold controls the stability and false alarm tolerance at the end of the process. By setting these two thresholds independently, response speed and anti-interference capability are decoupled, and can be configured according to the sensitivity requirements of different processes.

[0063] For details, please refer to Figure 3 , Figure 3 This is a schematic diagram of a specific embodiment of step 104 of the detection method for production line operations in this invention. Step 104, "comparing the condition satisfaction rate with a preset active threshold and a preset inactive threshold," includes the following specific implementation methods: 1041. Record the current determination status of the target process step, the current determination status including: not started state, in progress state; 1042. When the current determination state is not started and the condition satisfaction rate is not less than the active threshold, generate a determination result for the start of the target process step and update the current determination state to be in progress. 1043. When the current determination state is in progress and the condition satisfaction rate is not greater than the inactive threshold, generate a determination result that the target process step is completed, and update the current determination state to the not started state. 1044. When the condition satisfaction rate is between the inactive threshold and the active threshold, the current determination state remains unchanged.

[0064] In steps 1041-1044, the initial value of the current judgment state is "not started". Once one work cycle of the target process step is completed, the current judgment state is reset to "not started", awaiting judgment in the next work cycle. The above-mentioned judgment logic based on dual thresholds has a clear ability to suppress typical production line interferences, as analyzed below: 1. Pulse-type false alarm suppression: Assuming a conditional signal that is continuously zero is accompanied by a k-frame pulse-type interference with a value of 1, if... Then the condition satisfaction rate The current judgment state remains in the "not started" state, and impulsive false alarms are suppressed. The length of the sliding window queue. The inactivity threshold, This is the activity threshold.

[0065] 2. Intermittent Missed Detection Tolerance: Assuming that in a conditional signal that is continuously 1, there are intermittent missed detections where the value of d frames is 0, if Then the condition satisfaction rate The current status remains in progress and will not be mistakenly judged as the process completed. The length of the sliding window queue. The inactivity threshold, This is the activity threshold.

[0066] 3. Elimination of critical oscillation: When the condition satisfaction rate oscillates around the active threshold, due to the existence of the hysteresis comparison interval, the condition satisfaction rate must cross the complete hysteresis comparison interval to trigger a change in the judgment state, thus avoiding repeated switching of the judgment state.

[0067] Quantitatively speaking, the window length is At that time, the maximum tolerance The continuous missed detection of frames can prevent false positives, while also filtering out frames with insufficient duration. Pulse-like interference in frames.

[0068] For example, when , , At the same time, it can filter out pulse interference lasting less than 6 frames and tolerate up to 3 consecutive missed detections. In this way, by recording and maintaining the determination state, the start and completion determination of the target process step has clear temporal semantics, and only one start determination result and one completion determination result are output in one operation cycle.

[0069] 105. Based on the judgment result, generate the detection result of the target process step.

[0070] In this embodiment, the completion semantics of different target process steps are different. Some steps are completed when the target appears, such as "the part is placed in the designated position". Some steps are completed when the target disappears, such as "the hand leaves the operation area" and "the tool is removed from the workpiece".

[0071] To accommodate the semantics of the two types of processes mentioned above, a trigger mode parameter is introduced. The trigger mode parameter can be either a rising edge trigger mode or a falling edge trigger mode. The trigger mode of each target process step is specified independently in the configuration file and loaded together when the configuration file is parsed.

[0072] For target process steps that are "completed upon target appearance", the rising edge trigger mode is selected. When the target is detected to appear stably for the first time, i.e., when the determination result for the start of the target process step is generated, the target process step is immediately determined to have been executed.

[0073] For target process steps that require "target disappearance to be considered complete," select the falling edge trigger mode. Only when the target is detected to have stably disappeared from the screen or area, and a determination result indicating completion of the target process step is generated, is the target process step considered to have been executed successfully.

[0074] The detection results may include: the step identifier of the target process step, the frame number or timestamp indicating that the process was completed, and the corresponding workstation identifier. After generating the detection results, the results are sent to the manufacturing execution system or front-end interactive interface through a preset service interface, so that the production line manager can obtain the process execution status in real time. The preset service interface is, for example, a REST API interface and a WebSocket interface.

[0075] It should be noted that for target process steps that select the rising edge trigger mode, the judgment result of the target process step completion will still be generated and used to reset the current judgment state, but will not be output as the detection result of execution in place.

[0076] Similarly, for a target process step that selects the falling edge trigger mode, the initial judgment result of the target process step is used to update the current judgment state to "in progress" as a prerequisite for subsequent judgment completion, but it is not output as the detection result of execution completion. In this way, by selecting the trigger mode, the same set of sliding window dual threshold judgment logic can be adapted to target process steps with different semantics, improving the versatility of the detection method.

[0077] For details, please refer to Figure 4 , Figure 4 This is a schematic diagram of a specific embodiment of step 105 of the detection method for a production line operation process in this invention. Step 105, "generating the detection result of the target process step based on the determination result," includes the following specific implementation methods: 1051. Select a trigger mode according to the process semantics of the target process step, wherein the trigger mode includes: rising edge trigger mode and falling edge trigger mode; 1052. When the rising edge trigger mode is selected, a detection result indicating that the target process step has been completed is generated based on the determination result of the start of the target process step. 1053. When the falling edge trigger mode is selected, a detection result indicating that the target process step has been executed is generated based on the judgment result of the completion of the target process step.

[0078] In steps 1051-1053, the semantics of the process can be determined based on the action type of the target process step. For instantaneous action steps, completion is marked by the stable appearance of the target, and a rising edge trigger mode is selected. For continuous action steps (e.g., tightening a screw), completion is marked by the stable disappearance of the target, and a falling edge trigger mode is selected.

[0079] Optionally, each target process step maintains an independent parameter combination to achieve precise adaptation to different process characteristics. Typical parameter combinations are shown in Table 3 below: Table 3. Typical Parameter Combinations

[0080] Instantaneous action steps are configured with a smaller window length and a lower activity threshold, allowing the condition satisfaction rate to quickly reach the activity threshold for rapid response. Continuous action steps are configured with a larger window length and a higher activity threshold, requiring the triggering condition to remain valid for a longer period before the process step is considered to have started. The process step is confirmed to be complete when the condition satisfaction rate falls below the inactive threshold, avoiding false completion determinations caused by mid-action jitter. The parameter values ​​for intermediate detection steps fall between the two, balancing response speed and anti-interference capability.

[0081] The above parameter combinations are for illustrative purposes only. In actual applications, the parameters of each target process step can be independently adjusted through the configuration file based on the video frame rate, process cycle time, and interference intensity of the production line.

[0082] In step 1052, for example, for the target process step of "the part is placed in the designated position", the trigger condition is "the part is located in the placement area", and the trigger mode is the rising edge trigger mode; when the part first stably appears in the placement area, the determination result of the start of the target process step is generated, and then the detection result of the target process step being executed in place is generated.

[0083] In step 1053, for example, for the target process step of "hand leaving the operating area", the trigger condition is "hand is in the operating area", and the trigger mode is falling edge trigger mode; when the hand stably disappears from the operating area, a judgment result of the completion of the target process step is generated, and then a detection result of the execution of the target process step is generated.

[0084] The following is a specific configuration example: the "Power Detection" step at a certain workstation is triggered by `has_class('meter')`, meaning the meter appears on the screen. The parameters for this step are configured as follows: window length 50, active threshold 0.1, inactive threshold 0.06, and falling edge trigger mode. When the meter appears on the screen and the condition satisfaction rate within the sliding window reaches 0.1, the process enters the in-process state. When the meter moves off the screen and the condition satisfaction rate drops below 0.06, the step is considered complete, and a detection result indicating that the "Power Detection" step has been successfully executed is generated.

[0085] Furthermore, the target process steps can be sequentially linked. After the previous target process step generates a valid test result, the judgment logic for the next target process step is activated, achieving step-by-step inspection of the complete SOP process. Alternatively, the judgments for each target process step can be executed in parallel, with the sliding window instances of each target process step being independent and not interfering with each other.

[0086] In this embodiment of the invention, a preset target detection model is used to perform frame-by-frame reasoning on the workstation video stream. The frame-by-frame detection results are evaluated based on preset triggering conditions for the target process step. The condition signals for each frame are written into a sliding window to statistically determine the condition satisfaction rate. This rate is then compared with preset active and inactive thresholds. The target process step is determined to begin when the condition satisfaction rate is not less than the active threshold and to be completed when it is not greater than the inactive threshold, thus generating the detection result for the target process step. The sliding window statistical method transforms single-frame judgment into proportional judgment within a window. By smoothing the instantaneous jitter signal through the window, it suppresses inter-frame interference such as the instantaneous appearance and disappearance of the target and brief intrusion into the detection area, reducing the false trigger rate. Simultaneously, the active and inactive thresholds constitute a hysteresis comparison interval, and the judgment result remains in the previous state between the two thresholds without flipping. The independent setting of the two thresholds decouples response speed and anti-interference capability, allowing for configuration according to the sensitivity requirements of different processes. This solves the technical problems of production line visual inspection being susceptible to false triggers due to inter-frame interference, and the difficulty in balancing response speed and anti-interference capability with fixed threshold judgment methods.

[0087] Figure 5 This is a schematic diagram of the structure of a detection device for a production line operation process according to an embodiment of the present invention. The detection device 500 for this production line operation process can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 510 and memory 520, and one or more storage media 530 storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the detection device 500 for the production line operation process. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the detection device 500 for the production line operation process.

[0088] The testing equipment 500 based on production line operation processes may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, Free BSD, etc. Those skilled in the art will understand that... Figure 5The illustrated inspection equipment structure for production line operations does not constitute a limitation on inspection equipment based on production line operations. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0089] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the detection method of the production line operation process.

[0090] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0091] Furthermore, although the operations are described in a specific order, this should be understood as requiring that such operations be performed in the specific order shown or in sequential order, or requiring that all illustrated operations be performed to achieve the desired result. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations.

[0092] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for detecting a production line operation process, characterized in that, Including the following steps: The video stream of the workstation is collected, and the video stream of the workstation is inferred frame by frame according to the preset target detection model to obtain the detection results of the preset category targets in each video frame. According to the preset triggering conditions of the target process step corresponding to the workstation video stream, the detection results of each video frame are evaluated to obtain a frame-by-frame condition signal, wherein the condition signal is used to characterize whether the current video frame meets the triggering conditions of the corresponding target process step. The condition signals are written into a sliding window frame by frame, and the proportion of the condition signals that represent the fulfillment of the triggering condition within the sliding window is counted to obtain the condition satisfaction rate. The condition satisfaction rate is compared with a preset active threshold and a preset inactive threshold. When the condition satisfaction rate is not less than the active threshold, a determination result for the start of the target process step is generated; when the condition satisfaction rate is not greater than the inactive threshold, a determination result for the completion of the target process step is generated, wherein the active threshold is greater than the inactive threshold. Based on the determination result, the detection result of the target process step is generated.

2. The detection method for production line operation processes according to claim 1, characterized in that, The detection results include: the category name of the preset target and the bounding box corresponding to the category name. The step of evaluating the detection results of each video frame according to the preset triggering conditions of the target process step corresponding to the workstation video stream to obtain the frame-by-frame condition signal includes: The detection results of the current video frame are aggregated according to category name to obtain the bounding box set corresponding to each category name; The preset triggering conditions of the target process steps corresponding to the workstation video stream are represented as a condition expression tree. The leaf nodes of the condition expression tree are atomic conditions for judging the occurrence or location of the preset category target, and the internal nodes of the condition expression tree are nodes for logically combining the judgment results of the child nodes. Based on the bounding box set, the leaf nodes of the condition expression tree are evaluated to obtain the evaluation results of each leaf node. The evaluation results of each leaf node are then recursively combined from bottom to top along the condition expression tree to generate a combined result, and the combined result is determined as the condition signal of the current video frame.

3. The detection method for production line operation processes according to claim 2, characterized in that, The atomic conditions include: category occurrence conditions. The step of evaluating each leaf node of the condition expression tree based on the bounding box set to obtain the evaluation results of each leaf node includes: in, The condition for the category to appear in the first The evaluation results of video frames. This refers to the frame number of the video frame. The default category name. For the first The type name in the video frame is The corresponding bounding box set, This represents the number of bounding boxes in the bounding box set. This is a preset minimum target quantity threshold. This is an indicator function; the value is 1 if the condition within the parentheses is true, and 0 if it is false.

4. The detection method for production line operation processes according to claim 2, characterized in that, The atomic condition further includes: a category located in a region condition. The step of evaluating each leaf node of the condition expression tree based on the bounding box set to obtain the evaluation result of each leaf node includes: Calculate the intersection-union area ratio between the bounding box corresponding to the preset category name and the preset region of interest; When at least one bounding box in the set of bounding boxes corresponding to the preset category name has an intersection-union area ratio not less than a preset area ratio threshold, a determination result is generated indicating that the category is located in the region and the determination result is used as the evaluation result of the corresponding leaf node.

5. The detection method for production line operation processes according to claim 1, characterized in that, The step of calculating the proportion of condition signals representing the fulfillment of the triggering condition within the sliding window to obtain the condition satisfaction rate includes: in, This represents the condition satisfaction rate for processing the t-th video frame. This refers to the frame number of the video frame. The length of the sliding window queue. This refers to the frame number corresponding to the condition signal within the sliding window queue. For the first The value of the condition signal of the frame after it is written into the sliding window queue. The value can be 0 or 1. 1 represents the first If the condition signal of the frame satisfies the triggering condition, 0 represents the first The condition signal of the frame does not meet the triggering condition.

6. The detection method for production line operation processes according to claim 1, characterized in that, The step of comparing the condition satisfaction rate with a preset active threshold and a preset inactive threshold includes: Record the current determination status of the target process step, including: not started and in progress. When the current determination state is not started and the condition satisfaction rate is not less than the active threshold, a determination result for the start of the target process step is generated, and the current determination state is updated to the in progress state. When the current determination state is in progress and the condition satisfaction rate is not greater than the inactive threshold, a determination result for the completion of the target process step is generated, and the current determination state is updated to the not started state. When the condition satisfaction rate is between the inactive threshold and the active threshold, the current determination state remains unchanged.

7. The detection method for production line operation processes according to claim 1, characterized in that, The step of generating the detection result of the target process step based on the determination result includes: Based on the process semantics of the target process step, a trigger mode is selected, wherein the trigger mode includes: rising edge trigger mode and falling edge trigger mode; When the rising edge trigger mode is selected, a detection result indicating that the target process step has been completed is generated based on the determination result of the start of the target process step. When the falling edge trigger mode is selected, a detection result indicating that the target process step has been executed is generated based on the determination result of the target process step completion.

8. The detection method for production line operation processes according to claim 1, characterized in that, Before acquiring the video stream from the workstation, the following steps are also included: Receive configuration file, which includes: trigger conditions for each target process step, window length of the sliding window, active threshold, inactive threshold, and reference resolution; The configuration file is parsed, and when the triggering condition includes a region location condition, the vertex coordinates of the region corresponding to the region location condition are scaled proportionally according to the ratio between the actual inference resolution and the reference resolution of the configuration file to generate scaled vertex coordinates. The scaled vertex coordinates are used to evaluate the detection results of each video frame.

9. A testing device for a production line operation process, characterized in that, The detection equipment for the production line operation process includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; The at least one processor invokes the instructions in the memory to cause the inspection equipment of the production line operation process to perform the inspection method of the production line operation process as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the detection method for the production line operation process as described in any one of claims 1-8.

Images