A standard operation procedure (SOP) behavior process detection method and system
By continuously acquiring video streams and tracking multi-dimensional targets, combined with SOP multi-constraint models and multi-frame fusion technology, the problem of not being able to dynamically monitor SOP behavior in existing technologies has been solved. This enables real-time, accurate, and multi-dimensional detection and control of the SOP execution process, improving detection stability and data traceability, and supporting lean production optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIANGSU JIUZHONG XINSHI INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2026-04-29
- Publication Date
- 2026-07-24
AI Technical Summary
Existing technologies cannot achieve dynamic and continuous monitoring of operators' behavior in performing standard operating procedures (SOPs) in industrial production. They cannot identify process anomalies such as incorrect operation sequence, missing steps, and illegal operations. They have poor environmental adaptability, cannot quantify compliance assessment, have weak anti-interference capabilities, and lack traceable behavioral data support throughout the entire process.
By continuously acquiring video streams, adaptive preprocessing, multi-dimensional target recognition and tracking, atomic behavior structured coding, SOP multi-constraint model construction, multi-dimensional compliance quantitative judgment, multi-frame fusion enhancement, and anomaly classification real-time intervention, a full-link technology closed loop is constructed to achieve real-time, accurate, and full-dimensional detection and control of the SOP execution process.
It enables real-time, accurate, and comprehensive detection and control of the SOP execution process, identifies operational sequence errors and omissions, reduces hardware costs, improves detection stability and accuracy, provides traceable behavioral data support for the entire process, and supports lean production optimization and personnel skills assessment.
Smart Images
Figure CN122454633A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of detection and analysis technology, and more specifically, to a method and system for detecting standard operating procedure (SOP) behavior processes. Background Technology
[0002] In modern industrial production, Standard Operating Procedures (SOPs) are a core control measure for standardizing personnel operations, ensuring product quality, and reducing production safety risks. With the rapid development of machine vision technology, industrial vision inspection has been widely applied to quality control scenarios in the production process. Existing technologies mostly use industrial cameras to capture static images of the target to be inspected, and use image algorithms to detect and identify the final assembly results and appearance defects of the product, thereby intercepting defective products at the end of the production line. Some technical solutions are beginning to explore using image recognition to detect single human actions to assist in production process control.
[0003] The aforementioned existing technologies have formed a mature application system in the field of product static quality inspection, enabling efficient inspection of product appearance and assembly results under stable artificial lighting conditions, thus improving the efficiency of production quality control to a certain extent. However, in actual industrial applications, there are still unavoidable defects and shortcomings: First, the inspection paradigm has inherent limitations. The inspection objects of existing technologies focus on the final static state of the "product," and can only achieve post-event result verification. They cannot dynamically and continuously monitor the entire process of operators performing SOPs, and cannot identify process anomalies such as incorrect operation sequence, missing steps, and illegal operations, making it difficult to achieve proactive error prevention in the production process. Second, they have poor environmental adaptability. Existing technologies heavily rely on external supplementary lighting devices to construct a stable lighting environment. The effective illumination range of supplementary lighting devices is usually within 1 meter. When the inspection field of view exceeds 1 meter, uneven light distribution is prone to occur. At the same time, the aging of the supplementary lighting device, positional displacement, or external factors can also cause problems. Changes in ambient light can significantly reduce detection stability, resulting in high hardware and maintenance costs. Third, it cannot achieve structured modeling and quantitative compliance determination of Standard Operating Procedures (SOPs). Existing technologies can only identify the presence or absence of a single action, failing to transform text-based SOPs into machine-executable multi-dimensional constraint models. It cannot quantify the compliance of operational sequence, step completeness, and duration, making it difficult to achieve standardized and refined control over SOP execution. Fourth, single-frame detection has weak anti-interference capabilities. Existing technologies, based on single-frame static images, are susceptible to target occlusion, instantaneous noise, and lighting fluctuations, leading to false positives and false negatives, and insufficient stability and reliability of detection results. Fifth, it lacks traceable behavioral data support throughout the entire process. Existing technologies only store the final detection results, failing to structurally record and store the operator's behavior throughout the entire process, thus failing to provide objective and comprehensive data support for lean production optimization and personnel skill assessment.
[0004] To address the numerous shortcomings of existing technologies, this invention proposes a method and system for detecting Standard Operating Procedures (SOPs) behavior processes. By constructing a closed-loop technology chain encompassing continuous video stream acquisition, adaptive preprocessing, multi-dimensional target recognition and tracking, atomic behavior structured encoding, SOP multi-constraint model construction, multi-dimensional compliance quantification, multi-frame fusion enhancement, real-time intervention for anomaly classification, and full-process behavior tracing, the core of visual inspection is shifted from "product static result detection" to "personnel operation behavior process control." This enables real-time, accurate, and comprehensive detection and control of the SOP execution process, effectively solving the problems of lack of process monitoring, poor environmental adaptability, inability to quantify compliance assessment, weak anti-interference capabilities, and insufficient data traceability in existing technologies. Summary of the Invention
[0005] To overcome the aforementioned deficiencies of the prior art, the present invention provides a standard operating procedure (SOP) behavior process detection method and system, which solves the problems mentioned in the background art through the following solutions.
[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for detecting standard operating procedure (SOP) behavior processes, comprising:
[0007] S1. Continuously acquire video streams in the work area, calibrate core acquisition parameters, and complete the calculation of the total number of frames and the calibration of the region of interest coordinates.
[0008] S2, Video stream decoding and frame extraction, complete the entire process of frame preprocessing, and perform brightness normalization formula calculation and resolution unification;
[0009] S3. Analyze the preprocessed image frame by frame to complete multi-dimensional target recognition and screening, calibrate the target bounding box and confidence level, and generate continuous spatiotemporal trajectory;
[0010] S4. Based on the changes in the target's spatiotemporal relationship, complete the definition and triggering determination of atomic behaviors, generate state chains, and calculate the duration formula.
[0011] S5. Convert standard operating procedure text, construct a four-dimensional constraint mathematical model, and complete the definition and assignment of standard state transition matrix;
[0012] S6. Compare the behavioral state chain with the constraint model, perform multi-dimensional compliance analysis, and complete the comprehensive compliance quantification judgment;
[0013] S7. Set an adaptive sliding fusion window, construct a single-frame recognition result sequence, and perform multi-frame fusion formula calculation and result determination.
[0014] S8. Based on the multi-dimensional compliance judgment results, complete the classification of operational anomalies and execute the corresponding graded intervention strategies;
[0015] S9. Perform full-cycle data collection, complete structured storage and unique traceability index establishment, and support multi-dimensional traceability analysis.
[0016] Preferably, the continuous acquisition of the video stream in the work area is achieved by continuously capturing images of the target work area without interruption using at least one video capture camera deployed at the workstation, obtaining video stream data containing the complete operation cycle. The video acquisition is performed in natural light, without the need for external lighting. The core acquisition parameters include the video capture frame rate and the theoretical total duration of the complete operation cycle, both of which are pre-calibrated fixed parameters. The total frame count is calculated using the calibrated core acquisition parameters, and the formula is as follows: In the formula, The total number of frames in the video stream. This is the theoretical total duration of the complete operation cycle. The frame rate for video capture; the region of interest coordinate calibration is used to calibrate the pixel coordinate range of the smallest rectangular area containing all work actions, tools, parts, and products, including the pixel coordinates of the upper left corner and the lower right corner of the region of interest.
[0017] Preferably, the video stream decoding and frame extraction are used to perform real-time decoding and frame extraction on the acquired video stream to obtain a continuous sequence of video image frames; the full-process frame preprocessing includes four consecutive processing steps: region of interest (ROI) cropping, brightness normalization, noise suppression, and resolution unification, which are performed sequentially on each frame image; the ROI cropping is performed on each frame image according to the pre-defined ROI coordinate range, removing irrelevant background areas to obtain the cropped image frame; the brightness normalization is used to standardize the grayscale value of each pixel in the cropped image frame, and the calculation formula is: In the formula, The image frame after brightness normalization in coordinates The pixel grayscale value at that location, The cropped image frame in coordinates The pixel grayscale value at that location, For the first The average grayscale value of all pixels in the image after frame cropping. For the first The standard deviation of all pixel grayscale values in the image after frame cropping; the noise suppression uses a 3x3 Gaussian filter to smooth and denoise the brightness-normalized image frame, resulting in a preprocessed image frame; the resolution unification adjusts all preprocessed image frames to a preset resolution to ensure consistency of input to subsequent detection modules.
[0018] Preferably, the preprocessed image frame-by-frame analysis is used to parse the preprocessed continuous image frame sequence frame by frame, providing an input basis for subsequent target detection and tracking; the multi-dimensional target recognition is based on a pre-trained industrial-grade deep learning target detection model, which performs target detection on each preprocessed image frame and outputs the detection results of all targets in that frame, covering three categories of targets: personnel targets, tool and material targets, and work area targets; the target bounding box is the coordinate parameters of the rectangular region where the target is located, including the top-left pixel coordinates of the target bounding box, the width of the target bounding box, and the target... The bounding box height; the confidence level is a parameter representing the reliability of the detection result output by the target detection, with a value range of 0 to 1; the filtering is used to retain targets with a detection confidence level greater than or equal to 0.85 and remove invalid detection results with insufficient confidence level to form a set of valid targets in a single frame. This threshold is a preferred implementation value and can be adjusted according to the actual operation scenario; the continuous spatiotemporal trajectory uses a Kalman filter algorithm to perform identity matching and trajectory association on valid targets in adjacent frames, assigning a fixed identity ID to each unique target, generating a continuous spatiotemporal trajectory sequence for each target within a complete operation cycle, and realizing uninterrupted tracking of the target.
[0019] Preferably, the change in the spatiotemporal relationship of the target refers to the change in the spatial position, relative relationship, and quantity of each effective target over time in continuous video frames, providing a basis for atomic behavior recognition; the definition of atomic behavior is as follows: a set of atomic behaviors corresponding one-to-one with the standard operating procedure steps is predefined, and each atomic behavior in the set corresponds to a unique human-computer interaction logic, with no repetition or omission; the triggering judgment is applied to each preprocessed image frame, and based on the spatiotemporal position relationship and state change of each target in the effective target set, the atomic behavior corresponding to that frame is determined, clarifying the triggering conditions and judgment logic of the atomic behavior; the method of generating the state chain is as follows: the atomic behavior triggered in each frame is uniquely encoded to generate a behavior state sequence corresponding to the video frame sequence, and consecutive identical atomic behaviors are temporally merged to obtain the atomic behavior state chain within the complete operation cycle, and the state chain format is as follows: In the formula, For atomic behavior state chains, The total number of atomic behaviors in the state chain. arrive This represents the sequence number of the atomic behavior. For the first The starting frame number of each atomic behavior For the first The termination frame number of each atomic behavior The value range is 1 to The duration formula is based on the start frame number, end frame number, and pre-calibrated video capture frame rate of the atomic behavior to calculate the actual duration of each atomic behavior. The calculation formula is as follows: In the formula, For the first The actual duration of an atomic behavior, measured in seconds. For the first The termination frame number of each atomic behavior For the first The starting frame number of each atomic behavior The pre-calibrated video capture frame rate.
[0020] Preferably, the standard operating procedure (SOP) text conversion is used to transform manually formulated text-based SOPs into machine-executable and decision-aware mathematical constraint models, thereby making the SOPs machine-readable; the four-dimensional constraint mathematical model is used to integrate four types of constraints—sequential constraints, mandatory constraints, prohibited constraints, and time constraints—to form a complete multi-constraint behavioral mathematical model of the SOPs; the standard state transition matrix corresponds to the sequential constraints. take Two-dimensional matrix The total number of atomic behavior sets is used to define the temporal transition rules between atomic behaviors. The assignment rules for matrix elements are as follows: In the formula, This is the standard state transition matrix. , This represents the sequence number of the atomic behavior. , The atomic behaviors are defined by their corresponding sequence numbers; the sequence constraints define the standard time-sequence transition links of atomic behaviors in the standard operation process, and the assignment rules for the corresponding standard state transition matrix; the mandatory constraints define the set of atomic behaviors that must be fully executed in the standard operation process, and each atomic behavior in the set must occur at least once in the complete operation cycle; the prohibited constraints define the set of atomic behaviors that are absolutely not allowed to occur in the standard operation process, and atomic behaviors in the set are prohibited from occurring in the complete operation cycle; the time constraints define the allowed duration range for each atomic behavior, including the minimum allowed duration and the maximum allowed duration, both in seconds.
[0021] Preferably, the comparison between the behavior state chain and the constraint model is used to compare the real-time generated atomic behavior state chain with the preset standard operating procedure multi-constraint behavior mathematical model in real time, providing a basis for compliance calculation; the multi-dimensional compliance analysis includes four dimensions: time-series deviation calculation, mandatory completion calculation, prohibited behavior triggering degree calculation, and time compliance calculation. The calculation formulas for each dimension are as follows: Time-series deviation calculation: In the formula, This represents the timing deviation, with a value ranging from 0 to 1. The total number of atomic behaviors in the atomic behavior state chain. For the first state in the state chain The sequence number of the atomic behavior. The standard state transition matrix; the required completion degree calculation: In the formula, The required completion level is 0 to 1. The number of elements in the set. This is the set of atomic behaviors corresponding to mandatory constraints. For the first state in the state chain Individual atomic behaviors; calculation of the trigger degree of the prohibited behavior: In the formula, This is the trigger level for prohibited behaviors, with a value ranging from 0 to 1. The number of elements in the set. For the set of atomic behaviors corresponding to the prohibition constraints, For the first state in the state chain Individual atomic behaviors; the time compliance calculation: In the formula, For time compliance, the value ranges from 0 to 1. The total number of atomic behaviors in the atomic behavior state chain. For the first The minimum allowable duration corresponding to each atomic behavior For the first The maximum permissible duration corresponding to each atomic behavior. For the first The actual duration of each atomic action; the comprehensive compliance quantification judgment is based on the compliance calculation results of four dimensions, and the comprehensive compliance of the operation process is calculated through a nonlinear formula, the calculation formula is as follows: In the formula, To assess overall compliance, the value ranges from 0 to 1. The closer the value is to 1, the higher the overall compliance of the standard operating procedure. Completion level is mandatory. For time compliance, This refers to the time series deviation. This is the trigger level for prohibited behaviors.
[0022] Preferably, the adaptive sliding fusion window is a fixed-length temporal sliding window, with the window unit being frames. The window length is the result of dividing the video capture frame rate by 2 and rounding down. The single-frame recognition result sequence is constructed by building a single-frame recognition result sequence within the sliding window for each atomic behavior. Each element in the sequence represents the recognition result of the corresponding frame for that atomic behavior; a recognition result of 1 indicates that the corresponding frame recognized the atomic behavior, and a recognition result of 0 indicates that the corresponding frame did not recognize the atomic behavior. The multi-frame fusion formula is used to fuse the single-frame recognition result sequence within the sliding window to obtain the final fused recognition result of the atomic behavior. The calculation formula is as follows: In the formula, For the first Atomic behavior within frame window The final fusion recognition result, For floor operations, The length of the sliding merge window. For the first Frame-to-Atomic Behavior The single-frame recognition result; the result determination: when the final fusion recognition result is 1, it is determined that the atomic behavior in the current window has been triggered; when the final fusion recognition result is 0, it is determined that the atomic behavior in the current window has not been triggered.
[0023] Preferably, the multi-dimensional compliance judgment results include the calculation results of comprehensive compliance, time sequence deviation, mandatory completion rate, prohibited behavior triggering rate, and time compliance, providing a basis for anomaly level classification. The operation anomaly level classification, based on the multi-dimensional compliance judgment results, divides anomalies in the operation process into three levels: Level 1 Minor Anomaly, Level 2 Critical Anomaly, and Level 3 Severe Anomaly, clearly defining the judgment criteria for each level. A Level 1 Minor Anomaly: meets the following criteria: comprehensive compliance greater than or equal to 0.8, time sequence deviation less than or equal to 0.2, mandatory completion rate equal to 1, prohibited behavior triggering rate equal to 0, and time compliance greater than or equal to 0.8. It only involves a slight deviation in operation time or a minor deviation in sequence, without affecting the core execution of the standard operating procedure. The logic is as follows: Level 2 critical anomalies: These meet the following criteria: a comprehensive compliance score greater than or equal to 0.6 and less than 0.8; a timing deviation score greater than 0.2 and less than or equal to 0.5; a mandatory completion score greater than or equal to 0.8 and less than 1; or a time compliance score less than 0.8. These anomalies indicate minor omissions of mandatory steps, significant deviations in the operation sequence, or serious exceedances of operation time limits, affecting the standardization of standard operating procedures. Level 3 serious anomalies: These meet the following criteria: a comprehensive compliance score less than 0.6; a timing deviation score greater than 0.5; a mandatory completion score less than 0.8; or a prohibited behavior trigger score greater than 0. These anomalies indicate omissions of core mandatory steps, triggering of prohibited behaviors, or serious disruption of the operation sequence, directly leading to product quality defects. The tiered intervention strategy targets different levels of anomalies and executes accordingly. Corresponding differentiated intervention operations: For Level 1 minor anomalies, the system records the anomaly information, corresponding video clips, and behavioral status data to the behavior log, while simultaneously displaying text prompts on the UI interface of the workstation, without interrupting the work process; for Level 2 critical anomalies, the system immediately triggers an audible and visual alarm, and displays the anomaly type, location, and rectification requirements on the UI interface. The alarm is deactivated after the operator completes the rectification and the system re-determines compliance, allowing continued work; for Level 3 severe anomalies, the system immediately locks the work fixture via IO signals, simultaneously triggering a high-intensity audible and visual alarm, prohibiting the operator from continuing subsequent operations until on-site confirmation and deactivation by management personnel; the method for collecting data throughout the entire operation cycle is as follows: collecting data within the complete operation cycle... The system includes video stream data, atomic behavior state chains, compliance calculation results for each dimension, anomaly types and intervention records, and target spatiotemporal trajectory sequences. The structured storage is used to store the collected full-cycle data in a hierarchical structure according to operation batch, operator, operation time, and compliance level. A unique traceability index is established for each operation batch, with each index corresponding to a specific batch. The multi-dimensional traceability analysis, based on the stored structured data, supports multi-dimensional retrieval and video playback by traceability index code, operator, operation time, anomaly type, and compliance level. It also supports the export of structured behavioral data, providing objective data support for lean production analysis, personnel skills training, and operational bottleneck identification.
[0024] Preferably, a standard operating procedure (SOP) behavior process detection system includes:
[0025] Video acquisition module: continuously acquires video streams in the work area, calibrates core acquisition parameters, and completes the calculation of the total number of frames and the calibration of the region of interest coordinates;
[0026] Image preprocessing module: Video stream decoding and frame extraction, completes full-process frame preprocessing, performs brightness normalization formula calculation and resolution unification;
[0027] Target recognition module: Analyzes preprocessed image frames one by one, completes multi-dimensional target recognition and screening, calibrates target bounding boxes and confidence scores, and generates continuous spatiotemporal trajectories;
[0028] Behavior encoding module: Based on the changes in the spatiotemporal relationship of the target, it completes the definition and trigger determination of atomic behaviors, generates state chains, and calculates the duration formula;
[0029] SOP model building module: standard operating procedure text conversion, construction of four-dimensional constraint mathematical model, and completion of standard state transition matrix definition and assignment;
[0030] Compliance assessment module: compares the behavioral state chain with the constraint model, performs multi-dimensional compliance analysis, and completes a comprehensive compliance quantification assessment;
[0031] Multi-frame fusion optimization module: Sets an adaptive sliding fusion window, constructs a single-frame recognition result sequence, and performs multi-frame fusion formula calculation and result determination;
[0032] Anomaly Classification and Intervention Module: Based on multi-dimensional compliance judgment results, it completes the classification of operational anomalies and executes corresponding classification and intervention strategies;
[0033] Behavior traceability management module: collects data throughout the entire operation cycle, completes structured storage and establishes a unique traceability index, and supports multi-dimensional traceability analysis.
[0034] The technical effects and advantages of this invention are as follows:
[0035] 1. This invention upgrades the paradigm of industrial visual inspection from post-event verification of static product results to in-process control of personnel operation behavior by constructing a closed-loop technology for full-link detection of SOP behavior process based on video stream. It fills the gap in existing technology for monitoring the SOP execution process, and can identify process anomalies such as incorrect operation sequence, missing steps, and illegal operation in real time, so as to achieve proactive error prevention in the production process and eliminate the generation of defective products from the source.
[0036] 2. This invention uses video frame adaptive brightness normalization preprocessing technology, combined with multi-frame fusion enhancement recognition optimization technology, to effectively solve the problem of uneven lighting and light fluctuations affecting the detection results in natural light environments without the need for external supplementary lighting devices. It eliminates the high dependence of existing technologies on artificial supplementary lighting environments and breaks through the limitations of the detection field of view. It can still maintain stable and reliable detection results in large field of view operation scenarios exceeding 1 meter, and significantly reduces hardware costs and maintenance complexity.
[0037] 3. This invention constructs a multi-constraint behavioral mathematical model for SOPs, transforming unstructured text SOPs into a machine-executable and decisionable four-dimensional model of sequential constraints, mandatory constraints, prohibited constraints, and time constraints. Based on this model, it achieves quantitative calculations of timing deviation, mandatory completion, prohibited triggering, and time compliance, and obtains comprehensive compliance through a non-linear formula. This eliminates the need for manually set weighting coefficients, avoiding interference from subjective factors, and enables refined, standardized, and objective quantitative evaluation of SOP execution compliance. This solves the problem that existing technologies cannot quantitatively control the SOP execution process.
[0038] 4. This invention uses atomic behavior structured coding technology to transform continuous video streams into computable atomic behavior state chains. Combined with Kalman filtering multi-target continuous tracking technology, it achieves uninterrupted spatiotemporal trajectory tracking of personnel, tools, materials, and work areas. It can accurately identify atomic behaviors in complex human-computer interaction processes. At the same time, through adaptive sliding window multi-frame fusion technology bound to the acquisition frame rate, it effectively eliminates false detections and missed detections caused by occlusion and instantaneous noise in single-frame detection, and greatly improves the accuracy and stability of behavior recognition.
[0039] 5. This invention, through an anomaly classification and real-time intervention mechanism, accurately classifies operational anomalies based on multi-dimensional compliance judgment results and executes corresponding differentiated intervention strategies. Without affecting the continuity of normal operations, it achieves prompting for minor anomalies, alarm rectification for critical anomalies, and tooling lock-up for serious anomalies, taking into account the needs of production efficiency and quality control. At the same time, through the construction of a full-process structured behavior log, it achieves full-dimensional traceability management of the operation process, providing objective and comprehensive data support for lean production analysis, personnel skills training, and operational bottleneck identification. Attached Figure Description
[0040] Figure 1 This is a schematic diagram of the method steps of the present invention;
[0041] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation
[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] As attached Figure 1 The method for detecting the behavior of a standard operating procedure (SOP) includes the following steps:
[0044] S1. Continuously acquire video streams in the work area, calibrate core acquisition parameters, and complete the calculation of the total number of frames and the calibration of the region of interest coordinates.
[0045] It should be noted that: the continuous acquisition of the video stream in the work area is achieved by continuously capturing images of the target work area without interruption using at least one video capture camera deployed at the workstation, obtaining video stream data containing the complete operation cycle. Video acquisition is conducted in natural light conditions, without the need for external lighting. The core acquisition parameters include the video capture frame rate and the theoretical total duration of the complete operation cycle, both of which are pre-calibrated fixed parameters. The total frame count is calculated using the calibrated core acquisition parameters, and the formula is as follows: In the formula, The total number of frames in the video stream. This is the theoretical total duration of the complete operation cycle. The frame rate for video capture; the region of interest coordinate calibration is used to calibrate the pixel coordinate range of the smallest rectangular area containing all work actions, tools, parts, and products, including the pixel coordinates of the upper left corner and the lower right corner of the region of interest.
[0046] S2, Video stream decoding and frame extraction, complete the entire process of frame preprocessing, and perform brightness normalization formula calculation and resolution unification;
[0047] It should be noted that: the video stream decoding and frame extraction are used to perform real-time decoding and frame extraction on the acquired video stream to obtain a continuous sequence of video image frames; the full-process frame preprocessing includes four consecutive processing steps: region of interest (ROI) cropping, brightness normalization, noise suppression, and resolution unification, which are performed sequentially on each frame; the ROI cropping is performed on each frame according to the pre-defined ROI coordinate range, removing irrelevant background areas to obtain the cropped image frame; the brightness normalization is used to standardize the grayscale value of each pixel in the cropped image frame, and the calculation formula is: In the formula, The image frame after brightness normalization in coordinates The pixel grayscale value at that location, The cropped image frame in coordinates The pixel grayscale value at that location, For the first The average grayscale value of all pixels in the image after frame cropping. For the first The standard deviation of all pixel grayscale values in the image after frame cropping; the noise suppression uses a 3x3 Gaussian filter to smooth and denoise the brightness-normalized image frame, resulting in a preprocessed image frame; the resolution unification adjusts all preprocessed image frames to a preset resolution to ensure consistency of input to subsequent detection modules.
[0048] S3. Analyze the preprocessed image frame by frame to complete multi-dimensional target recognition and screening, calibrate the target bounding box and confidence level, and generate continuous spatiotemporal trajectory;
[0049] It should be noted that: the frame-by-frame analysis of the preprocessed image frames is used to parse the continuous sequence of preprocessed image frames frame by frame, providing an input basis for subsequent target detection and tracking; the multi-dimensional target recognition is based on a pre-trained industrial-grade deep learning target detection model, which performs target detection on each preprocessed image frame and outputs the detection results of all targets in that frame, covering three categories of targets: personnel targets, tool and material targets, and work area targets; the target bounding box is the coordinate parameters of the rectangular region where the target is located, including the top-left pixel coordinates of the target bounding box, the width of the target bounding box, and the coordinates of the target bounding box. The bounding box height is marked; the confidence level is a parameter representing the reliability of the detection result output by the target detection, with a value range of 0 to 1; the filtering is used to retain targets with a detection confidence level greater than or equal to 0.85 and remove invalid detection results with insufficient confidence level to form a set of valid targets in a single frame. This threshold is a preferred implementation value and can be adjusted according to the actual operation scenario; the continuous spatiotemporal trajectory uses the Kalman filter algorithm to perform identity matching and trajectory association on valid targets in adjacent frames, assigning a fixed identity ID to each unique target, and generating a continuous spatiotemporal trajectory sequence for each target within a complete operation cycle to achieve uninterrupted tracking of the target.
[0050] S4. Based on the changes in the target's spatiotemporal relationship, complete the definition and triggering determination of atomic behaviors, generate state chains, and calculate the duration formula.
[0051] It should be noted that: the spatiotemporal relationship change of the target refers to the change in the spatial position, relative relationship, and quantity status of each effective target in continuous video frames over time, providing a basis for atomic behavior recognition; the atomic behavior definition: a set of atomic behaviors corresponding one-to-one with the standard operating procedure steps is predefined, and each atomic behavior in the set corresponds to a unique human-computer interaction logic, with no repetition or omission; the triggering judgment is for each preprocessed image frame, based on the spatiotemporal position relationship and state change of each target in the effective target set, to determine the atomic behavior corresponding to that frame, clarifying the triggering condition and judgment logic of the atomic behavior; the method of generating the state chain is: to uniquely encode the atomic behavior triggered in each frame, generate a behavior state sequence corresponding to the video frame sequence, and temporally merge consecutive identical atomic behaviors to obtain the atomic behavior state chain within the complete operation cycle, the state chain format is: In the formula, For atomic behavior state chains, The total number of atomic behaviors in the state chain. arrive This represents the sequence number of the atomic behavior. For the first The starting frame number of each atomic behavior For the first The termination frame number of each atomic behavior The value range is 1 to The duration formula is based on the start frame number, end frame number, and pre-calibrated video capture frame rate of the atomic behavior to calculate the actual duration of each atomic behavior. The calculation formula is as follows: In the formula, For the first The actual duration of an atomic behavior, measured in seconds. For the first The termination frame number of each atomic behavior For the first The starting frame number of each atomic behavior The pre-calibrated video capture frame rate.
[0052] S5. Convert standard operating procedure text, construct a four-dimensional constraint mathematical model, and complete the definition and assignment of standard state transition matrix;
[0053] It should be noted that: the standard operating procedure (SOP) text conversion is used to transform manually formulated textual SOPs into machine-executable and decision-making mathematical constraint models, thereby making the SOPs machine-readable; the four-dimensional constraint mathematical model is used to integrate four types of constraints—sequential constraints, mandatory constraints, prohibited constraints, and time constraints—to form a complete multi-constraint behavioral mathematical model of the SOP; and the standard state transition matrix corresponds to the sequential constraints. take Two-dimensional matrix The total number of atomic behavior sets is used to define the temporal transition rules between atomic behaviors. The assignment rules for matrix elements are as follows: In the formula, This is the standard state transition matrix. , This represents the sequence number of the atomic behavior. , The atomic behaviors are defined by their corresponding sequence numbers; the sequence constraints define the standard time-sequence transition links of atomic behaviors in the standard operation process, and the assignment rules for the corresponding standard state transition matrix; the mandatory constraints define the set of atomic behaviors that must be fully executed in the standard operation process, and each atomic behavior in the set must appear at least once in the complete operation cycle; the prohibited constraints define the set of atomic behaviors that are absolutely not allowed to appear in the standard operation process, and the atomic behaviors in the set are prohibited from appearing in the complete operation cycle; the time constraints define the allowed duration range for each atomic behavior, including the minimum allowed duration and the maximum allowed duration.
[0054] S6. Compare the behavioral state chain with the constraint model, perform multi-dimensional compliance analysis, and complete the comprehensive compliance quantification judgment;
[0055] It should be noted that the comparison between the behavior state chain and the constraint model is used to compare the real-time generated atomic behavior state chain with the preset standard operating procedure multi-constraint behavior mathematical model in real time, providing a basis for compliance calculation. The multi-dimensional compliance analysis includes four dimensions: time-series deviation calculation, mandatory completion degree calculation, prohibited behavior trigger degree calculation, and time compliance calculation. The calculation formulas for each dimension are as follows: Time-series deviation calculation: In the formula, This represents the timing deviation, with a value ranging from 0 to 1. The total number of atomic behaviors in the atomic behavior state chain. For the first state in the state chain The sequence number of the atomic behavior. The standard state transition matrix; the required completion degree calculation: In the formula, The required completion level is 0 to 1. The number of elements in the set. This is the set of atomic behaviors corresponding to mandatory constraints. For the first state in the state chain Individual atomic behaviors; calculation of the trigger degree of the prohibited behavior: In the formula, This is the trigger level for prohibited behaviors, with a value ranging from 0 to 1. The number of elements in the set. For the set of atomic behaviors corresponding to the prohibition constraints, For the first state in the state chain Individual atomic behaviors; the time compliance calculation: In the formula, For time compliance, the value ranges from 0 to 1. The total number of atomic behaviors in the atomic behavior state chain. For the first The minimum allowable duration corresponding to each atomic behavior For the first The maximum permissible duration corresponding to each atomic behavior. For the first The actual duration of each atomic action; the comprehensive compliance quantification judgment is based on the compliance calculation results of four dimensions, and the comprehensive compliance of the operation process is calculated through a nonlinear formula, the calculation formula is as follows: In the formula, To assess overall compliance, the value ranges from 0 to 1. The closer the value is to 1, the higher the overall compliance of the standard operating procedure. Completion level is mandatory. For time compliance, This refers to the time series deviation. This is the trigger level for prohibited behaviors.
[0056] S7. Set an adaptive sliding fusion window, construct a single-frame recognition result sequence, and perform multi-frame fusion formula calculation and result determination.
[0057] It should be noted that: the adaptive sliding fusion window is a fixed-length temporal sliding window, with the window unit being frames. The window length is the result of dividing the video capture frame rate by 2 and rounding down; the single-frame recognition result sequence construction involves constructing a single-frame recognition result sequence within the sliding window for each atomic behavior. Each element in the sequence represents the recognition result of the corresponding frame for that atomic behavior. A recognition result of 1 indicates that the corresponding frame recognized the atomic behavior, while a recognition result of 0 indicates that the corresponding frame did not recognize the atomic behavior; the multi-frame fusion formula calculation is used to fuse the single-frame recognition result sequence within the sliding window to obtain the final fused recognition result of the atomic behavior. The calculation formula is as follows: In the formula, For the first Atomic behavior within frame window The final fusion recognition result, For floor operations, The length of the sliding merge window. For the first Frame-to-Atomic Behavior The single-frame recognition result; the result determination: when the final fusion recognition result is 1, it is determined that the atomic behavior in the current window has been triggered; when the final fusion recognition result is 0, it is determined that the atomic behavior in the current window has not been triggered.
[0058] S8. Based on the multi-dimensional compliance judgment results, complete the classification of operational anomalies and execute the corresponding graded intervention strategies;
[0059] It should be noted that the multi-dimensional compliance judgment results include the calculation results of comprehensive compliance, time sequence deviation, mandatory completion rate, prohibited behavior triggering rate, and time compliance, providing a basis for the classification of anomaly levels. The operational anomaly level classification, based on the multi-dimensional compliance judgment results, divides anomalies in the operation process into three levels: Level 1 Minor Anomaly, Level 2 Critical Anomaly, and Level 3 Severe Anomaly, clearly defining the judgment criteria for each level. Level 1 Minor Anomaly: Meets the following criteria: comprehensive compliance greater than or equal to 0.8, time sequence deviation less than or equal to 0.2, mandatory completion rate equal to 1, prohibited behavior triggering rate equal to 0, and time compliance greater than or equal to 0.8. It only involves a slight deviation in operation time or a minor deviation in sequence, without affecting the core execution logic of the standard operating procedure. Level 2 Critical Anomaly: Meets the following criteria: comprehensive compliance greater than or equal to 0.6 and less than 0.8, or time sequence deviation greater than 0.2 and less than or equal to 0.5, or mandatory completion rate greater than or equal to 0.8 and less than 1, or time compliance less than 0.8. It involves a slight omission of mandatory steps, a significant deviation in the operation sequence, or an excessively long operation time. The severe exceedance of limits affects the standardization of standard operating procedures. The three levels of severe anomalies are: a comprehensive compliance score of less than 0.6, a timing deviation score of greater than 0.5, a mandatory completion score of less than 0.8, or a prohibited behavior trigger score of greater than 0. These indicate omissions of core mandatory steps, triggering of prohibited behaviors, or serious disruption of the operation sequence, which will directly lead to product quality defects. The graded intervention strategy performs corresponding differentiated intervention operations for different levels of anomalies. For level one minor anomalies, the system records the anomaly information, corresponding video clips, and behavior status data to the behavior log, and provides text prompts on the UI interface of the operation station without interrupting the operation process. For level two critical anomalies, the system immediately triggers an audible and visual alarm, and displays the anomaly type, anomaly location, and rectification requirements on the UI interface. After the operator completes the rectification and the system re-determines compliance, the alarm is deactivated, and the operation can continue. For level three severe anomalies, the system immediately locks the workpiece through IO signals and triggers a high-intensity audible and visual alarm, prohibiting the operator from continuing to perform subsequent operations until management personnel confirm and deactivate the lock on-site.
[0060] S9. Perform full-cycle data collection, complete structured storage and unique traceability index establishment, and support multi-dimensional traceability analysis.
[0061] It should be noted that: the method of collecting full-cycle data for the operation is as follows: collecting video stream data, atomic behavior state chains, compliance calculation results for each dimension, anomaly types and intervention records, and target spatiotemporal trajectory sequences within the complete operation cycle; the structured storage is used to store the collected full-cycle data in a hierarchical structure according to operation batch, operator, operation time, and compliance level; the unique traceability index establishes a unique traceability index code for each operation batch, with a one-to-one correspondence between the traceability index code and the operation batch; the multi-dimensional traceability analysis is based on the stored structured data, supporting multi-dimensional retrieval and video playback by traceability index code, operator, operation time, anomaly type, and compliance level, and also supporting the export of structured behavioral data, providing objective data support for lean production analysis, personnel skills training, and operational bottleneck identification.
[0062] Based on the above scheme and appendix Figure 2 The present invention also provides a standard operating procedure (SOP) behavior process detection system, comprising:
[0063] Video acquisition module: continuously acquires video streams in the work area, calibrates core acquisition parameters, and completes the calculation of the total number of frames and the calibration of the region of interest coordinates;
[0064] Image preprocessing module: Video stream decoding and frame extraction, completes full-process frame preprocessing, performs brightness normalization formula calculation and resolution unification;
[0065] Target recognition module: Analyzes preprocessed image frames one by one, completes multi-dimensional target recognition and screening, calibrates target bounding boxes and confidence scores, and generates continuous spatiotemporal trajectories;
[0066] Behavior encoding module: Based on the changes in the spatiotemporal relationship of the target, it completes the definition and trigger determination of atomic behaviors, generates state chains, and calculates the duration formula;
[0067] SOP model building module: standard operating procedure text conversion, construction of four-dimensional constraint mathematical model, and completion of standard state transition matrix definition and assignment;
[0068] Compliance assessment module: compares the behavioral state chain with the constraint model, performs multi-dimensional compliance analysis, and completes a comprehensive compliance quantification assessment;
[0069] Multi-frame fusion optimization module: Sets an adaptive sliding fusion window, constructs a single-frame recognition result sequence, and performs multi-frame fusion formula calculation and result determination;
[0070] Anomaly Classification and Intervention Module: Based on multi-dimensional compliance judgment results, it completes the classification of operational anomalies and executes corresponding classification and intervention strategies;
[0071] Behavior traceability management module: collects data throughout the entire operation cycle, completes structured storage and establishes a unique traceability index, and supports multi-dimensional traceability analysis.
[0072] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for detecting the behavior process of a standard operating procedure (SOP), characterized in that, include: S1. Continuously acquire video streams in the work area, calibrate core acquisition parameters, and complete the calculation of the total number of frames and the calibration of the region of interest coordinates. S2, Video stream decoding and frame extraction, complete the entire process of frame preprocessing, and perform brightness normalization formula calculation and resolution unification; S3. Analyze the preprocessed image frame by frame to complete multi-dimensional target recognition and screening, calibrate the target bounding box and confidence level, and generate continuous spatiotemporal trajectory; S4. Based on the changes in the target's spatiotemporal relationship, complete the definition and triggering determination of atomic behaviors, generate state chains, and calculate the duration formula. S5. Convert standard operating procedure text, construct a four-dimensional constraint mathematical model, and complete the definition and assignment of standard state transition matrix; S6. Compare the behavioral state chain with the constraint model, perform multi-dimensional compliance analysis, and complete the comprehensive compliance quantification judgment; S7. Set an adaptive sliding fusion window, construct a single-frame recognition result sequence, and perform multi-frame fusion formula calculation and result determination. S8. Based on the multi-dimensional compliance judgment results, complete the classification of operational anomalies and execute the corresponding graded intervention strategies; S9. Perform full-cycle data collection, complete structured storage and unique traceability index establishment, and support multi-dimensional traceability analysis.
2. The method for detecting standard operating procedure (SOP) behavior processes according to claim 1, characterized in that: The core acquisition parameters include the video acquisition frame rate and the theoretical total duration of the complete operation cycle, both of which are pre-calibrated fixed parameters; the total frame count formula calculates the total number of frames in the video stream using the calibrated core acquisition parameters; the region of interest coordinate calibration is used to calibrate the pixel coordinate range of the smallest rectangular area containing all operation actions, tools, parts, and products, including the pixel coordinates of the upper left corner and the lower right corner of the region of interest.
3. The method for detecting standard operating procedure (SOP) behavior processes according to claim 1, characterized in that: The region of interest (ROI) cropping is performed on each frame of the image according to the pre-defined ROI coordinate range, removing irrelevant background areas to obtain cropped image frames; The brightness normalization is used to standardize the grayscale value of each pixel in the cropped image frame; The noise suppression uses a 3x3 Gaussian filter to smooth and denoise the brightness-normalized image frame, resulting in a pre-processed image frame. The resolution unification adjusts all pre-processed image frames to a preset resolution.
4. The method for detecting standard operating procedure (SOP) behavior processes according to claim 1, characterized in that: The multi-dimensional target recognition is based on a pre-trained industrial-grade deep learning target detection model. It performs target detection on each pre-processed image frame and outputs the detection results of all targets in the frame. The detected objects cover three categories: personnel targets, tool and material targets, and work area targets. The target bounding box is the coordinate parameters of the rectangular region where the target is located, including the pixel coordinates of the upper left corner of the target bounding box, the width of the target bounding box, and the height of the target bounding box.
5. The method for detecting standard operating procedure (SOP) behavior processes according to claim 1, characterized in that: The trigger determination is applied to each preprocessed image frame. Based on the spatiotemporal position relationship and state changes of each target in the effective target set, the atomic behavior corresponding to the frame is determined. The method of generating the state chain is as follows: the atomic behavior triggered by each frame is uniquely encoded to generate a behavior state sequence corresponding to the video frame sequence. The consecutive identical atomic behaviors are temporally merged to obtain the atomic behavior state chain within the complete operation cycle. The duration formula is calculated based on the start frame number, end frame number and pre-calibrated video acquisition frame rate of the atomic behavior to calculate the actual duration of each atomic behavior.
6. The method for detecting standard operating procedure (SOP) behavior processes according to claim 1, characterized in that: The four-dimensional constraint mathematical model integrates four types of constraints: sequence constraints, mandatory constraints, prohibited constraints, and time constraints. The sequence constraints define the standard temporal transition links of atomic behaviors in the standard operating procedure, corresponding to the assignment rules of the standard state transition matrix. The mandatory constraints define the set of atomic behaviors that must be fully executed in the standard operating procedure; each atomic behavior in the set must occur at least once within the complete operation cycle. The prohibited constraints define the set of atomic behaviors that are absolutely not allowed to occur in the standard operating procedure; atomic behaviors in the set are prohibited from occurring within the complete operation cycle. The time constraints define the allowed duration range for each atomic behavior, including a minimum allowed duration and a maximum allowed duration.
7. The method for detecting standard operating procedure (SOP) behavior processes according to claim 1, characterized in that: The behavior state chain and constraint model comparison is used to compare the real-time generated atomic behavior state chain with the preset standard operating procedure multi-constraint behavior mathematical model in real time, dimension by dimension; the multi-dimensional compliance analysis includes four dimensions: time sequence deviation calculation, mandatory completion calculation, prohibited behavior triggering calculation, and time compliance calculation. The comprehensive compliance metric is based on the compliance calculation results of four dimensions, and the comprehensive compliance of the operation process is calculated through a non-linear formula.
8. The method for detecting standard operating procedure (SOP) behavior processes according to claim 1, characterized in that: The construction of the single-frame recognition result sequence involves constructing a single-frame recognition result sequence within a sliding window for each atomic behavior. Each element in the sequence represents the recognition result of the corresponding frame for that atomic behavior. A recognition result of 1 indicates that the corresponding frame has recognized the atomic behavior, while a recognition result of 0 indicates that the corresponding frame has not recognized the atomic behavior. The multi-frame fusion formula is used to perform fusion calculations on the single-frame recognition result sequence within the sliding window to obtain the final fusion recognition result of atomic behavior.
9. The method for detecting standard operating procedure (SOP) behavior processes according to claim 1, characterized in that: The multi-dimensional compliance judgment results include the calculation results of comprehensive compliance, time sequence deviation, mandatory completion, prohibited behavior triggering degree, and time compliance, which provide a basis for the classification of abnormality levels. The classification of operational abnormality levels is based on the multi-dimensional compliance judgment results, and abnormalities in the operation process are divided into three levels: Level 1 minor abnormality, Level 2 critical abnormality, and Level 3 severe abnormality.
10. A standard operating procedure (SOP) behavior process detection system, used to implement the standard operating procedure (SOP) behavior process detection method according to any one of claims 1 to 9, characterized in that, include: Video acquisition module: continuously acquires video streams in the work area, calibrates core acquisition parameters, and completes the calculation of the total number of frames and the calibration of the region of interest coordinates; Image preprocessing module: Video stream decoding and frame extraction, completes full-process frame preprocessing, performs brightness normalization formula calculation and resolution unification; Target recognition module: Analyzes preprocessed image frames one by one, completes multi-dimensional target recognition and screening, calibrates target bounding boxes and confidence scores, and generates continuous spatiotemporal trajectories; Behavior encoding module: Based on the changes in the spatiotemporal relationship of the target, it completes the definition and trigger determination of atomic behaviors, generates state chains, and calculates the duration formula; SOP model building module: standard operating procedure text conversion, construction of four-dimensional constraint mathematical model, and completion of standard state transition matrix definition and assignment; Compliance assessment module: compares the behavioral state chain with the constraint model, performs multi-dimensional compliance analysis, and completes a comprehensive compliance quantification assessment; Multi-frame fusion optimization module: Sets an adaptive sliding fusion window, constructs a single-frame recognition result sequence, and performs multi-frame fusion formula calculation and result determination; Anomaly Classification and Intervention Module: Based on multi-dimensional compliance judgment results, it completes the classification of operational anomalies and executes corresponding classification and intervention strategies; Behavior traceability management module: collects data throughout the entire operation cycle, completes structured storage and establishes a unique traceability index, and supports multi-dimensional traceability analysis.