Production process monitoring method, equipment and medium

Abstract

The invention relates to the technical field of production process monitoring, and discloses a production process monitoring method, equipment and a medium. The method comprises the following steps: acquiring a plurality of video frame images in a production monitoring video; performing motion state feature extraction on the production material in the video frame image to obtain a material motion state feature including a position feature and a speed feature of the production material; performing operation state feature extraction on the production operation in the video frame image to obtain an operation comprehensive state feature including an operation type feature, a compliance degree feature and an abnormal type feature of the production operation; and performing feature compliance evaluation on the material motion state features and the operation comprehensive state features, and generating a process compliance evaluation result for the production process based on the feature compliance evaluation result. According to the embodiment of the invention, the monitoring efficiency of production process monitoring and the sequential logic coherence analysis performance can be improved.

Description

Technical Field

[0001] This application relates to the field of production process monitoring technology, and in particular to a production process monitoring method, equipment and medium. Background Technology

[0002] In modern industrial production, especially in discrete manufacturing fields such as automobile assembly and electronic equipment assembly, the compliance, efficiency, and stability of production flow and assembly processes are core factors determining product quality and cost control. Currently, production process monitoring methods primarily rely on manual inspections and paper records, or use isolated sensor networks or single-function video surveillance, which are inefficient and lack the ability to provide coherent analysis of the overall production process's temporal logic. Summary of the Invention

[0003] The purpose of this application is to provide a production process monitoring method, equipment, and medium, which aims to improve the monitoring efficiency and time-series logic coherence analysis performance of production process monitoring.

[0004] This application provides a production process monitoring method, including: Acquire multiple video frame images from production monitoring videos; Motion state features are extracted from the production materials in the video frame image to obtain material motion state features that include the position and velocity features of the production materials; The operation status features of the production operations in the video frame images are extracted to obtain comprehensive operation status features that include operation type features, compliance level features, and abnormality type features of the production operations. The material movement state characteristics and the comprehensive operation state characteristics are evaluated for compliance, and a process compliance evaluation result for the production process is generated based on the evaluation results.

[0005] In some embodiments, the extraction of motion state features from the production materials in the video frame image includes: The positional features of the production materials in the video frame image are extracted to obtain the positional features of the production materials. Based on the location features of the production materials and the temporal features of the video frame images, the velocity features of the production materials in the video frame images are extracted to obtain the velocity features of the production materials. By fitting the position and velocity characteristics of the production material, the motion state characteristics of the material are obtained.

[0006] In some embodiments, the step of extracting velocity features from the production material in the video frame image based on the positional features of the production material and the temporal features of the video frame image includes: Based on the location characteristics of the production material and the temporal characteristics of the video frame images, the motion trajectory of the production material is predicted; Based on the motion trajectory of the production material and the temporal characteristics of the video frame image, the velocity features of the production material in the video frame image are extracted to obtain the velocity features of the production material.

[0007] In some embodiments, the extraction of operation state features from the production operation in the video frame image includes: Spatiotemporal state features of the production operations in the video frame images are extracted to obtain spatiotemporal state features of the production operations that characterize the spatiotemporal features of the production operations. The spatiotemporal state features of the operation are subjected to multi-task classification processing to obtain the operation type features, compliance degree features, and anomaly type features of the production operation. By fitting the operation type feature, the compliance feature, and the anomaly type feature, the comprehensive operation status feature is obtained.

[0008] In some embodiments, the extraction of spatiotemporal state features from the production operations in the video frame image includes: Based on the slow branch of the SlowFast network, continuous convolution operations are performed on the production operations in the video frame image to obtain the spatial domain features of the production operations. Based on the fast branch of the SlowFast network, continuous convolution operations are performed on the production operations in the video frame image to obtain the temporal features of the production operations. By fitting the spatial domain features and the temporal domain features, the spatiotemporal state features of the operation are obtained.

[0009] In some embodiments, the feature compliance evaluation of the material motion state characteristics and the comprehensive operational state characteristics includes: A compliance evaluation of the motion state characteristics of the material is performed to obtain the feature compliance evaluation result of the material motion state characteristics. Local and global compliance evaluations are performed on the overall operational status characteristics to obtain the feature compliance evaluation results of the overall operational status characteristics.

[0010] In some embodiments, the motion state compliance evaluation of the material motion state characteristics includes: Based on the material movement state characteristics, the residence time characteristics of the production material in the preset area, the displacement distance characteristics within the preset time, and the material quantity characteristics within the preset area are determined. Based on the deviations between the residence time feature, the displacement distance feature, and the material quantity feature and the corresponding motion state compliance features, the feature compliance evaluation results of the residence time feature, the displacement distance feature, and the material quantity feature are determined.

[0011] In some embodiments, the step of performing local compliance evaluation and global compliance evaluation on the comprehensive operational status characteristics includes: Based on the deviations between the operation type characteristics, compliance degree characteristics, and anomaly type characteristics of the production operation and the corresponding local compliance characteristics, the characteristic compliance evaluation results of the operation type characteristics, compliance degree characteristics, and anomaly type characteristics are determined; Based on the deviation between the operation type feature sequence and the preset compliant operation feature sequence, the feature compliance evaluation result of the operation type feature is determined; the operation type feature sequence is obtained by arranging the operation type features of the production operation based on the temporal features of the video frame image.

[0012] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described production process monitoring method.

[0013] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described production process monitoring method.

[0014] The beneficial effects of this application are as follows: By acquiring video frame images and extracting features from material movement and operational behavior, the extracted material movement state features and comprehensive operational state features are then integrated and subjected to feature compliance evaluation, ultimately generating a process compliance evaluation result. This improves the monitoring efficiency and temporal logic coherence analysis performance of production process monitoring. Compared to traditional manual inspection, acquiring video frame images and extracting features from material movement and operational behavior enables real-time, objective, and quantitative monitoring of the production process. Precise location and speed feature extraction significantly improves the accuracy and timeliness of monitoring. Compared to isolated sensor networks or single-function video surveillance, integrating the extracted material movement state features and comprehensive operational state features and performing feature compliance evaluation allows for the understanding and evaluation of these complex visual semantic information, overcoming the technical limitations of existing technologies in understanding and parsing rich visual semantic behavioral information. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating the application environment of the production process monitoring method provided in the embodiments of this application.

[0016] Figure 2 This is a flowchart of the production process monitoring method provided in the embodiments of this application.

[0017] Figure 3 This is a flowchart of a method for extracting motion state features of production materials in a video frame image, provided in an embodiment of this application.

[0018] Figure 4 This is a flowchart of a method for extracting operation status features from production operations in video frame images, provided in an embodiment of this application.

[0019] Figure 5 This is a flowchart of a method for evaluating the feature compliance of material movement state characteristics and comprehensive operational state characteristics, provided in an embodiment of this application.

[0020] Figure 6 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and drawings are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0024] The production process monitoring method provided in this application can be executed by computer equipment, which can be a terminal device or a server. Terminal devices include, but are not limited to, mobile phones, computers, smart home appliances, vehicle terminals, and aircraft. The server can be a standalone physical server, a server cluster consisting of multiple physical servers, a distributed system, or a cloud server. Furthermore, all information, data, and signals involved in this application's embodiments are authorized by the relevant parties or have been fully authorized by all parties involved, and the collection, use, and processing of related data comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0025] In existing production flow and assembly process monitoring, the lack of automated semantic parsing capabilities for operational details makes real-time compliance verification of production operations impossible. Existing technologies rely on manual inspections or isolated sensor networks, which struggle to acquire behavioral information rich in visual semantics. This prevents the system from accurately identifying operation types, the correctness of tool selection, and the compliance level of process execution sequence. This issue leads to a lag in production process compliance judgment and insufficient reliability of verification results, directly impacting the stability of product quality control and the accuracy of process parameter execution. For example, in the door installation station of an automotive assembly line, operators must sequentially perform component positioning, bolt tightening, and functional testing. Existing monitoring systems can only record changes in physical parameters and cannot analyze whether the operator used a specified torque wrench, whether the bolt assembly sequence conforms to process specifications, or whether any processes were omitted. When an operator mistakenly uses a regular wrench instead of a torque wrench, the system cannot automatically identify this abnormal operation type or correlate it with material movement characteristics to determine whether the component assembly displacement meets standards. This phenomenon causes a continuous accumulation of process execution deviations, ultimately leading to door sealing defects and increasing the re-inspection pressure in subsequent quality inspection stages.

[0026] If the above problems are not addressed, operational deviations in the production process cannot be captured and corrected in a timely manner, causing process parameters to deviate from preset standard ranges. The disconnect between operational type characteristics and material movement characteristics means that local process anomalies may escalate into global process interruptions, leading to decreased product consistency and increased risk of assembly line downtime. The continued existence of this technical deficiency will weaken the self-diagnostic capabilities of the production system, making the quality control system reliant on post-event manual intervention, and failing to meet the core requirements for process stability in discrete manufacturing.

[0027] Based on this, embodiments of this application provide a production process monitoring method, apparatus, equipment, and medium. By extracting material motion state features and comprehensive operation state features from video frame images and performing feature compliance evaluation, it solves the technical problems of low monitoring efficiency, strong subjectivity, and lack of material-operation correlation analysis, thereby improving the monitoring efficiency and time-series logic coherence analysis performance of production process monitoring.

[0028] Figure 1 This diagram illustrates the application environment of the production process monitoring method provided in this embodiment. (See attached diagram.) Figure 1This method is applied to a production process monitoring system. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal; the mobile terminal can be at least one of a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of several servers. The terminal 110 is used to send multiple video frame images from the production monitoring video to the server 120. The server 120 is used to acquire multiple video frame images from the production monitoring video, extract motion state features from the production materials in the video frame images to obtain material motion state features including position and velocity features, extract operation state features from the production operations in the video frame images to obtain comprehensive operation state features including operation type features, compliance level features, and anomaly type features, perform feature compliance evaluation on the material motion state features and comprehensive operation state features, and generate process compliance evaluation results for the production process based on the feature compliance evaluation results.

[0029] It should be understood that Figure 1 The application scenarios shown are merely examples. In practical applications, the production process monitoring method provided in this application embodiment can also be applied to other scenarios. For example, the above-described production process monitoring method can be applied to terminal 110. Terminal 110 is used to acquire multiple video frame images in the production monitoring video, extract motion state features of the production materials in the video frame images to obtain material motion state features including the position and speed features of the production materials, extract operation state features of the production operations in the video frame images to obtain comprehensive operation state features including operation type features, compliance degree features, and abnormality type features of the production operations, perform feature compliance evaluation on the material motion state features and comprehensive operation state features, and generate process compliance evaluation results for the production process based on the feature compliance evaluation results.

[0030] See Figure 2 In one embodiment, a production process monitoring method is provided. The execution subject of the method can be a terminal or a server, including but not limited to steps S201 to S204.

[0031] Step S201: Acquire multiple video frame images from the production monitoring video.

[0032] Production monitoring video refers to video streams used on industrial production lines to record production operations and materials at various production nodes during the production process; these are typically captured by cameras.

[0033] Production materials refer to raw materials, semi-finished products, or parts that are processed in the production process.

[0034] Production operations refer to the specific actions or behaviors performed by workers or equipment on production materials on the production line.

[0035] A video frame image refers to a single static image taken at consecutive points in time in a production monitoring video.

[0036] Acquiring multiple video frames from production monitoring videos can be achieved by manually extracting keyframes or by manually extracting video frame images from the video stream at fixed time intervals (e.g., 1 frame per second). Alternatively, a simple video decoder can be used to continuously read and store all video frame images from the video file at a preset frame rate (e.g., 25 frames per second). Furthermore, a video capture card can be configured to receive video signals from industrial cameras in real time, convert them into digital image data streams, and then buffer them at a fixed frame rate.

[0037] Step S202: Extract motion state features from the production materials in the video frame image to obtain material motion state features that include the position and velocity features of the production materials.

[0038] Material motion state characteristics refer to the dynamic attribute characteristics of production materials in space and time, including the position characteristics and velocity characteristics of production materials.

[0039] The location characteristics of production materials refer to the spatial coordinate information of production materials in video frame images.

[0040] The velocity characteristic of production materials refers to the amount of displacement of production materials per unit time in a video frame image.

[0041] Motion state features of the production materials in the video frame can be extracted using traditional image processing methods based on background subtraction or color thresholding to identify the production materials and calculate their centroid coordinates as positional features. Velocity features can be approximated by calculating the pixel displacement of the material's centroid between consecutive frames using a simple inter-frame differencing method. Alternatively, a pre-trained general object detection model can be deployed to detect and identify the production materials in each frame, obtaining their bounding box coordinates as positional features. Velocity features can be estimated using a tracking algorithm combined with the material's positional information across consecutive frames.

[0042] Step S203: Extract operation status features from the production operations in the video frame image to obtain comprehensive operation status features that include operation type features, compliance level features, and anomaly type features.

[0043] Material operation status characteristics refer to the attribute characteristics that describe production operations, including operation type characteristics, compliance level characteristics, and abnormality type characteristics.

[0044] Operation type characteristics refer to the predefined category to which a production operation belongs, such as "grab", "place", "tighten", etc.

[0045] Compliance level characteristics refer to the degree to which production operations conform to preset standard procedures.

[0046] Anomaly type characteristics refer to the categories of non-standard or erroneous behaviors that occur in production operations, such as "omission", "misoperation", "timeout", etc.

[0047] Extracting operational status features from production operations in video frame images can be achieved using traditional computer vision methods based on keypoint detection and pose estimation. This involves identifying the operator's skeletal keypoints and determining the operation type based on rule matching according to the sequence of keypoint position changes. Compliance level and anomaly type are then determined using preset thresholds or simple logical judgments. Alternatively, a convolutional neural network (CNN)-based image classification model can be deployed to classify operation regions in video frame images to identify operation types. Compliance level and anomaly type are then determined using a separate classifier or a rule-based expert system.

[0048] Step S204: Perform a feature compliance evaluation on the material movement state characteristics and the comprehensive operation state characteristics, and generate a process compliance evaluation result for the production process based on the feature compliance evaluation results.

[0049] Feature compliance evaluation refers to the assessment of the compliance of extracted material motion state characteristics and comprehensive operational state characteristics.

[0050] Process compliance evaluation results refer to the final judgment on the compliance of the entire production process based on feature compliance evaluation results.

[0051] The system performs feature compliance evaluation on the material movement state characteristics and the overall operation state characteristics, and generates a process compliance evaluation result for the production process based on the feature compliance evaluation results. This can be achieved by pre-setting a series of hard-coded rules and thresholds. For example, if a material stays in a certain area for more than a preset time, it is marked as non-compliant; if the operation type does not conform to the standard process, it is also marked as non-compliant. Then, all non-compliant features are summarized to generate the process compliance evaluation result. Furthermore, a simple decision tree model can be constructed, taking material movement state characteristics (such as position and speed) and overall operation state characteristics (such as operation type, compliance level, and anomaly type) as input, judging them through pre-set decision rules, outputting a compliance score for each feature, and finally aggregating these scores to generate the overall process compliance evaluation result.

[0052] The following example will provide a more detailed explanation of the above technical solution: On an electronics assembly line, it is necessary to monitor the compliance of user A's "chip placement" operation at workstation B. The standard procedure requires that the chip must be placed within a specific area C, and the placement speed must not be too fast or too slow, while the operator must not miss any chips or make any mistakes.

[0053] First, high-definition cameras on the production line continuously record the production process at workstation B and transmit the video stream to the monitoring system. The executing unit acquires and caches these video frames in real time at a rate of 30 frames per second.

[0054] Secondly, the execution unit analyzes each video frame to identify the chip (production material) to be placed. Using a target detection algorithm, the chip's position coordinates within each frame are precisely determined, i.e., its positional characteristics. Based on the chip's positional changes across consecutive frames, the execution unit uses a motion estimation algorithm to calculate the chip's instantaneous velocity, i.e., its velocity characteristics. Thus, the material motion state characteristics, containing both chip position and velocity information, are obtained.

[0055] Next, the executing entity analyzes user A's actions in the video frame image to identify the ongoing "chip placement" operation. Through behavior recognition algorithms, the executing entity determines that the operation's type is "chip placement." Simultaneously, the executing entity assesses whether user A's placement actions conform to preset posture, trajectory, and timing requirements, thus obtaining the operation's compliance characteristics. Furthermore, the executing entity also detects for abnormal behaviors such as "chip falling" or "misalignment" and identifies the abnormality type characteristics. Therefore, a comprehensive operation status feature containing information on operation type, compliance level, and abnormality type is obtained.

[0056] Finally, the executing entity compares the extracted chip material movement status characteristics (position, speed) with preset compliance standards. For example, it checks whether the chip is placed in area C and whether its placement speed is within the allowable range. Simultaneously, the executing entity compares user A's overall operation status characteristics (operation type, compliance level, anomaly type) with standard operating procedures. For example, it confirms whether the operation type is "chip placement," whether the compliance level meets the requirements, and whether any anomalies exist. Based on these comparison results, the executing entity performs compliance evaluations on the material movement status characteristics and overall operation status characteristics separately, generating their respective feature compliance evaluation results. Ultimately, the executing entity integrates the compliance evaluation results of all features to generate a process compliance evaluation result for the entire "chip placement" production process. For example, if the chip placement position deviation is too large or the operator makes a mistake, the process compliance evaluation result may show "non-compliant" with a specific reason.

[0057] This method decomposes the production process into two core dimensions: material movement and personnel operation, and then performs refined feature extraction and compliance evaluation on each. Material movement status features provide an objective quantification of the physical state of the production objects, while comprehensive operational status features reveal the standardization and potential risks of production behavior. The combination of these two features enables comprehensive, multi-dimensional monitoring of the production process. Through real-time acquisition, analysis, and evaluation of these features, the implementing entity can promptly identify non-compliant behaviors or potential problems in the production process, thereby ensuring product quality and production efficiency.

[0058] Based on the above examples, the technical contributions of this solution are reflected in the following aspects: Compared to traditional manual inspection methods, which are inefficient and highly subjective, making it difficult to achieve real-time, objective recording and analysis of operational details at each workstation, this method automates the acquisition of video frame images and extracts features from material movement and operational behaviors, enabling real-time, objective, and quantitative monitoring of the production process. For example, in the chip placement example mentioned above, manual inspection struggles to accurately determine whether the chip placement speed and position are within a small deviation range, and it also fails to capture subtle abnormal movements of the operator in real time. This method, however, can provide refined compliance judgments through precise position and speed feature extraction, as well as the identification of operation type, compliance level, and anomaly type, significantly improving the accuracy and timeliness of monitoring.

[0059] Compared to existing monitoring systems that often employ isolated sensor networks or single-function video surveillance, which can collect some physical parameters but cannot understand and interpret behavioral information containing rich visual semantics, this method achieves in-depth analysis of the behavior of both "materials" and "people" in the production process by extracting the location and speed characteristics of materials, as well as the type, compliance level, and anomaly type characteristics of operations. In the chip placement example, a single video surveillance system cannot automatically identify whether the chip has been placed in place or whether the operator has performed the correct grasping action. This method, however, can understand and evaluate this complex visual semantic information by extracting the characteristics of material movement and the comprehensive characteristics of operation, thus overcoming the limitations of existing technologies in understanding and interpreting rich visual semantic behavioral information.

[0060] This method integrates the characteristics of material movement and overall operational status, performs feature compliance evaluation, and ultimately generates a process compliance evaluation result. This enables the executing entity to conduct a comprehensive and coherent analysis and evaluation of the compliance of the entire production process from two dimensions: material flow and personnel operation. This overcomes the shortcomings of existing technologies in lacking the ability to analyze the overall process's temporal logic coherence. In the example above, the executing entity can not only determine the compliance of individual operations but also comprehensively evaluate the efficiency and quality of the entire chip placement process by combining the material flow situation, thereby providing a more comprehensive insight into the production process.

[0061] See Figure 3 In one embodiment, the method for extracting motion state features of production materials in a video frame image includes, but is not limited to, steps S301 to S303.

[0062] Step S301: Extract the location features of the production materials in the video frame image to obtain the location features of the production materials.

[0063] Step S302: Based on the location features of the production materials and the temporal features of the video frame images, the velocity features of the production materials in the video frame images are extracted to obtain the velocity features of the production materials.

[0064] Step S303: Fit the position and velocity characteristics of the production material to obtain the material motion state characteristics.

[0065] The purpose of extracting location features from production materials in video frame images is to identify and locate these materials and obtain their positional information in the image coordinate system or the actual spatial coordinate system. This process can utilize object detection algorithms, such as the YOLO series or Faster R-CNN, to identify the production materials and obtain their bounding box center coordinates or keypoint coordinates as location features. Alternatively, image segmentation techniques, such as Mask R-CNN or U-Net, can be used to accurately segment the pixel regions of the production materials and calculate their centroid coordinates or contour information as location features.

[0066] Based on the positional characteristics of the production materials and the temporal characteristics of video frames, velocity features of the production materials in the video frames are extracted. The aim is to calculate the velocity of the production materials based on their positional changes and temporal information across consecutive video frames. This step can be implemented using tracking algorithms, such as Kalman filtering or DeepSORT, to track the production materials across frames and then calculate the instantaneous velocity by dividing the change in the material's position across consecutive frames by the inter-frame time interval. Alternatively, optical flow methods, such as Lucas-Kanade optical flow or Farneback optical flow, can be used to directly estimate the motion vectors of pixels in the image, thereby inferring the overall velocity of the production materials.

[0067] Fitting the position and velocity characteristics of production materials aims to integrate independently extracted position and velocity information into a unified and more comprehensive description of the material's motion state. This fitting process can directly concatenate position features (e.g., x, y coordinates) and velocity features (e.g., vx, vy components) into a high-dimensional vector as the material's motion state feature. Alternatively, machine learning models, such as support vector machines or neural networks, can be used to fuse and learn the position and velocity features, generating a more abstract and discriminative representation of the material's motion state.

[0068] The proposed solution first extracts the positional features of the production materials independently from each video frame of the production monitoring video. This step ensures precise spatial positioning of the production materials. Subsequently, using the positional features in these consecutive frames and the inherent temporal features of the video frames, velocity features of the production materials are extracted. By analyzing the positional changes of the materials at different time points, their velocity information can be accurately calculated. Finally, the independently acquired positional and velocity features are fitted together, integrating these two different but interrelated features to obtain a comprehensive and detailed material motion state feature. This step-by-step extraction and integration method avoids the difficulty of directly extracting complex motion state features, enabling more accurate capture and characterization of position and velocity information, and providing a more reliable foundation for subsequent feature compliance evaluation.

[0069] The following is a specific example to illustrate this. As a concrete implementation, after acquiring multiple video frames from the production monitoring video, a deep learning-based object detection model (e.g., YOLOv7 model) can first be used to identify and locate the production materials in each video frame, thereby obtaining the bounding box coordinates of each material (e.g., top-left x, y and bottom-right x, y) as its positional features. Subsequently, for consecutive video frames, a multi-object tracking algorithm (e.g., DeepSORT algorithm) can be used to track these production materials, and the instantaneous velocity components in the horizontal and vertical directions are calculated based on the change in the centroid position of the material in adjacent frames and the frame rate (i.e., the temporal characteristics of the video frame images), thus obtaining the velocity features. Finally, the bounding box center coordinates (positional features) of each material are concatenated with the calculated instantaneous velocity components (velocity features) to form a comprehensive vector containing the material's spatial position and motion trend, serving as the material's motion state feature.

[0070] The above technical solution refines the process of extracting the motion state features of production materials into three stages: position feature extraction, velocity feature extraction, and feature fitting. This staged processing approach allows for more accurate capture and quantification of the spatial location and dynamic motion trends of production materials within video frames. Specifically, independent position feature extraction ensures the accuracy of material spatial information, while velocity feature extraction, based on both position and temporal features, guarantees the reliability of velocity calculation. The final feature fitting step effectively integrates this information, forming a comprehensive and highly discriminative feature set of material motion states, thereby significantly improving the accuracy and reliability of feature compliance evaluation in subsequent production process monitoring.

[0071] In some embodiments, based on the positional features of the production material and the temporal features of the video frame image, velocity features of the production material in the video frame image are extracted, including: predicting the motion trajectory of the production material based on the positional features of the production material and the temporal features of the video frame image; and extracting velocity features of the production material in the video frame image based on the motion trajectory of the production material and the temporal features of the video frame image to obtain the velocity features of the production material.

[0072] Predicting the trajectory of production materials refers to inferring the future or current movement path of materials based on known historical location data and time information. Its purpose is to smooth discrete location data, capture the overall trend and pattern of material movement, and provide a more continuous and accurate basis for subsequent velocity calculations. This trajectory prediction can be achieved in various ways. For example, state estimation algorithms such as Kalman filtering or extended Kalman filtering can be used to establish a material motion model and perform real-time filtering and prediction of location data. Alternatively, deep learning models, such as recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or Transformers, can be used to learn the temporal patterns of material movement, thereby predicting its trajectory.

[0073] Based on the motion trajectory of production materials and the temporal characteristics of video frame images, velocity feature extraction of production materials in video frame images involves using the predicted smooth motion trajectory combined with the temporal information of the video frame images to calculate the instantaneous or average velocity of the materials. This provides more accurate and stable velocity features than calculating directly from the original position data, because trajectory prediction has already processed noise and captured the motion trend. This velocity feature extraction can also be achieved in several ways. For example, velocity can be calculated by numerically differentiating the predicted trajectory. For continuous trajectory functions, the derivative can be directly calculated; for discrete trajectory points, finite difference methods (such as central difference) can be used to approximate the velocity calculation. Alternatively, a velocity output module can be integrated into the trajectory prediction model, allowing the model to directly output the corresponding velocity features while predicting the trajectory.

[0074] In the aforementioned production process monitoring method, to more accurately obtain the velocity characteristics of production materials, this application first introduces the prediction of the production material's motion trajectory before directly extracting velocity based on positional and temporal features. Specifically, the executing entity models and predicts the motion trajectory of the production material using a prediction model based on the acquired positional features of the production material and the temporal features of the video frame images. This prediction process effectively smooths out potential noise in the original positional data and captures the overall trend of material movement, thereby generating a more continuous and accurate motion trajectory. Subsequently, based on this predicted and optimized motion trajectory, combined with the temporal features of the video frame images, the velocity features of the production material are extracted. In this way, the calculation of velocity features no longer relies solely on discrete positional changes between adjacent frames, but is based on a more global and continuous motion trajectory, thus significantly improving the accuracy, stability, and robustness of the extracted velocity features. This effectively avoids velocity calculation deviations caused by noise or irregularities in the original data, providing a more reliable data foundation for subsequent compliance evaluation of material motion status.

[0075] As a specific implementation method, when extracting velocity features based on the positional characteristics of production materials and the temporal characteristics of video frame images, the Kalman filter algorithm can be used first to predict the motion trajectory of the production materials. Specifically, the center coordinates of the production materials identified from the video frame images are used as observation values ​​and input into the Kalman filter. The Kalman filter estimates the current state (including position and velocity) of the material based on a preset motion model (e.g., a uniform linear motion model or a uniformly accelerated motion model) and the observation model, and predicts its position at the next moment, thus forming a smooth motion trajectory. Subsequently, based on this motion trajectory predicted by the Kalman filter, the instantaneous velocity of the production materials is accurately calculated by performing differential calculations on the position changes between consecutive frames on the trajectory and combining the frame rate of the video frame images (i.e., temporal characteristics). For example, if the trajectory predicts that the position of the material at time t is P(t) and the position at time t+Δt is P(t+Δt), then its velocity at time t can be approximated as (P(t+Δt)- P(t)) / Δt.

[0076] By employing the aforementioned technical solution, when extracting the velocity characteristics of production materials, the material's trajectory is first predicted. This effectively filters out measurement noise and instantaneous fluctuations in the original position data, resulting in a smoother and more continuous trajectory. Furthermore, combining the temporal features of video frame images to extract velocity features significantly improves the accuracy and stability of these features. This method avoids the errors introduced by directly calculating velocity from discrete, potentially noisy position data. Consequently, it provides more accurate and reliable input for subsequent fitting of material motion state characteristics and the final compliance evaluation of the production process, thereby enhancing the accuracy and reliability of the entire production process monitoring method.

[0077] See Figure 4 In one embodiment, the method for extracting operation state features from production operations in video frame images includes, but is not limited to, steps S401 to S403.

[0078] Step S401: Extract spatiotemporal state features from the production operation in the video frame image to obtain the spatiotemporal state features of the operation that characterize the spatiotemporal features of the production operation.

[0079] Step S402: Perform multi-task classification processing on the spatiotemporal state features of the operation to obtain the operation type features, compliance degree features, and anomaly type features of the production operation.

[0080] Step S403: Fit the operation type features, compliance features, and anomaly type features to obtain the comprehensive operation status features.

[0081] Extracting spatiotemporal state features from production operations in video frame images aims to capture the dynamic changes and interactions of these operations in time and space from consecutive video frame images. This transforms raw pixel data into high-level, meaningful feature representations, providing a foundation for subsequent operation analysis. For example, consecutive video frame images can be processed directly using a 3D convolutional neural network (3D-CNN). 3D-CNN can learn features simultaneously in both spatial and temporal dimensions, thereby capturing the motion patterns and spatial layout of the operations. As another implementation approach, a two-stream network architecture can be used, where one stream processes spatial information (such as the RGB image of a single frame), and the other stream processes temporal information (such as optical flow). The features from both streams are then fused to obtain comprehensive spatiotemporal features.

[0082] Multi-task classification of operational spatiotemporal state features aims to simultaneously identify multiple attributes of production operations—operation type, compliance level, and anomaly type—using these extracted features. Multi-task classification allows for the sharing of underlying feature representations, improving the model's generalization ability and efficiency while ensuring consistency across different attributes. For example, a deep learning model can be constructed that takes operational spatiotemporal state features as input and has three independent output heads, corresponding to operation type classification, compliance level classification (e.g., compliant / non-compliant / partially compliant), and anomaly type classification (e.g., no anomaly / misoperation / omitted operation). Alternatively, an attention-based multi-task learning framework can be employed. By introducing an attention module, the model can adaptively focus on different parts of the operational spatiotemporal state features according to the needs of different tasks, thus more effectively completing multi-task classification.

[0083] Fitting operation type features, compliance level features, and anomaly type features aims to integrate discrete or continuous operation type, compliance level, and anomaly type features obtained through multi-task classification into a unified and comprehensive operational status feature. This comprehensive feature can more completely describe the current state of production operations, providing richer information for subsequent process compliance evaluation. For example, these three feature vectors can be concatenated to form a longer feature vector as the operational status feature. As another implementation, these three features can also be fused through weighted summation or neural network fusion. For instance, a fully connected layer can be used to map the concatenated features to a more compact comprehensive feature space.

[0084] This application first extracts spatiotemporal state features from production operations in video frames, capturing the dynamic evolution and key information of operations in time and space from the raw video data, thus obtaining spatiotemporal state features characterizing the production operations. This step transforms complex video information into a structured feature representation, laying the foundation for subsequent analysis. Based on this, the acquired spatiotemporal state features undergo multi-task classification processing, enabling the executing entity to simultaneously identify the operation type, assess its compliance level, and detect potential anomalies. This multi-task processing approach not only improves feature utilization efficiency but also ensures the consistency and correlation of operation state information across different dimensions. Finally, by fitting these independent but related operation type features, compliance level features, and anomaly type features, a comprehensive operational state feature is formed. This integration method provides a deeper and more complete understanding of production operations, thereby more accurately supporting subsequent process compliance evaluation and effectively solving the technical problem of efficiently and comprehensively extracting comprehensive features of production operations from video data.

[0085] The following is a concrete example. When extracting spatiotemporal state features from production operations in video frames, a pre-trained video behavior recognition model, such as a 3D-CNN model based on ResNet or Inception architecture, can be used to process continuous video frame sequences. This model learns the spatial texture information and temporal motion patterns of the operations from the video through multi-layer convolution and pooling operations, ultimately outputting a fixed-dimensional feature vector as the spatiotemporal state feature of the operation. This spatiotemporal state feature can then be input into a multi-head classifier. For example, the first classifier head is a fully connected layer followed by a Softmax activation function to predict the operation type (e.g., "tightening screws," "carrying," "inspecting," etc.); the second classifier head is another fully connected layer followed by a Sigmoid activation function to evaluate the compliance level (e.g., "fully compliant," "slightly non-compliant," "severely non-compliant"); and the third classifier head is used to identify the anomaly type (e.g., "tool misplacement," "operation sequence reversed," "missed step"). Finally, the probability distributions or encoding vectors output by these three classification heads can be concatenated. For example, the one-hot encoding vector of operation type classification, the numerical representation of compliance level, and the one-hot encoding vector of exception type can be directly connected to form a unified comprehensive operation status feature that can fully describe the current production operation status.

[0086] Through the above technical solution, this application can efficiently and comprehensively extract the integrated status features of production operations from production monitoring videos. Specifically, by extracting spatiotemporal status features, the dynamic information of the operation can be fully captured; by multi-task classification processing, the operation type, compliance level, and anomaly type can be identified simultaneously, avoiding the limitations of single-task analysis; and by feature fitting, these multi-dimensional information are integrated into unified integrated status features, providing richer, more accurate, and more comprehensive data support for subsequent process compliance evaluation, significantly improving the refinement and intelligence level of production process monitoring.

[0087] In some embodiments, the spatiotemporal state feature extraction of production operations in video frame images includes: performing continuous convolution operations on the production operations in video frame images based on the slow branch of the SlowFast network to obtain the spatial domain features of the production operations; performing continuous convolution operations on the production operations in video frame images based on the fast branch of the SlowFast network to obtain the temporal domain features of the production operations; and fitting the spatial domain features and temporal domain features to obtain the spatiotemporal state features of the operation.

[0088] The SlowFast network is a dual-path video recognition model designed to effectively capture spatial semantic and temporal motion information in videos. It processes video input through two parallel branches: one running at a low frame rate to capture spatial semantics, and the other running at a high frame rate to capture rapidly changing motion. The SlowFast network can be built on a backbone network such as ResNet or ResNeXt, and the slow and fast branches can be differentiated by adjusting the stride, number of channels, and sampling rate of the convolutional and pooling layers. For example, the slow branch can use a larger time step and deeper layers to extract high-level semantic features, while the fast branch can use a smaller time step and shallower layers to capture instantaneous motion. Alternatively, the slow and fast branches can use different input resolutions or different preprocessing strategies, and information can be fused through cross-branch connections.

[0089] The slow branch is primarily responsible for capturing spatial semantic information in video frames, i.e., static or slowly changing visual content such as objects, people, and backgrounds in a scene. It typically processes video at a lower frame rate, allowing for deeper analysis of the visual content of each frame and the extraction of rich spatial features. The slow branch can be designed with a deeper network structure and a larger receptive field, for example, by employing multiple residual blocks and dilated convolutions, to effectively aggregate contextual information in the image, thereby extracting spatial domain features with strong semantic expressive power. The slow branch can also incorporate attention mechanisms, such as a spatial attention module, to focus on key areas in the production process, thus extracting operation-related spatial features more accurately.

[0090] The fast branch is primarily responsible for capturing temporal motion information in video frame sequences, i.e., rapidly changing actions, posture transitions, material movement, and other dynamic visual content in production operations. It typically processes video at a high frame rate, enabling it to capture subtle motion changes. The fast branch can be designed with a shallow network structure and a small receptive field, but with high temporal resolution, for example, using 1x1xT 3D convolutional kernels to efficiently capture inter-frame differences and motion trajectories. The fast branch can also enhance its ability to extract temporal features by introducing techniques such as optical flow estimation, explicitly using motion information as input or auxiliary features.

[0091] Spatial domain features refer to the visual attributes of objects, such as shape, texture, color, and position, extracted from the two-dimensional image content of video frames. These features primarily reflect information about the image in the spatial dimension. Spatial domain features can include low-level visual features such as edges, corners, region shapes, color histograms, and texture descriptors, as well as high-level semantic features extracted by deep convolutional networks, such as object categories and part recognition. In the slow branch of the SlowFast network, spatial domain features extracted through continuous convolutional operations are typically high-level semantic features, capable of describing the visual appearance and relative positional relationships of objects such as tools, materials, and operator body parts involved in production operations.

[0092] Temporal features refer to the features extracted from video frame image sequences that characterize dynamic attributes such as object motion, behavioral changes, and the sequence of events. These features primarily reflect information about the image in the temporal dimension. Temporal features can include low-level motion features such as optical flow vectors, motion trajectories, inter-frame differences, and motion energy, as well as high-level behavioral features extracted by deep convolutional networks, such as action categories and behavioral patterns. In the fast branch of the SlowFast network, temporal features extracted through continuous convolutional operations typically capture rapid and subtle changes in movement during production operations, such as the rate of operator hand movements and the direction and speed of material movement.

[0093] Operational spatiotemporal state features are composite features that integrate spatial and temporal characteristics of production operations, comprehensively representing the dynamic processes of production operations in terms of spatial location, morphological changes, and time series. Operational spatiotemporal state features can be high-dimensional vectors containing information such as the visual appearance, action patterns, duration, and speed of the production operation. This information is crucial for accurately identifying operation types, determining compliance, and detecting anomalies. These features can be designed with a hierarchical structure; for example, low-level features capture local actions and textures, while high-level features capture overall behavioral patterns and semantic information, thus providing rich input for subsequent multi-task classification.

[0094] This application's solution employs a dual-branch parallel processing approach using a SlowFast network. The slow branch processes the video at a lower frame rate, enabling in-depth analysis of the visual content of each frame and capturing static or slowly changing visual semantic information such as objects, tools, and operator postures involved in the production operation, thus obtaining the spatial domain features of the production operation. Simultaneously, the fast branch processes the video at a higher frame rate, capturing rapid and subtle changes in motion and trajectories during the production operation, thereby obtaining the temporal domain features of the production operation. Through this dual-branch parallel processing approach, the slow branch focuses on extracting fine spatial semantics, while the fast branch focuses on capturing rapid temporal motion; the two complement each other, ensuring comprehensive coverage of the spatiotemporal information of the production operation. Finally, the spatial and temporal features are fitted to obtain the spatiotemporal state features of the operation. This fitting process effectively integrates the rich spatial context information extracted by the slow branch with the precise motion information captured by the fast branch, forming a comprehensive spatiotemporal feature representation. In this way, the proposed solution can overcome the limitations of a single branch network in processing complex spatiotemporal information, ensuring that the extracted spatiotemporal state features contain both rich spatial details and accurate temporal dynamic information. This provides high-quality input for subsequent multi-task classification of production operations to obtain operation type features, compliance features, and anomaly type features, thereby significantly improving the accuracy and robustness of production operation state feature extraction.

[0095] The following is a concrete example. When extracting spatiotemporal state features from production operations in video frames, a pre-trained SlowFast network model can be used. This SlowFast network model can be built based on ResNet-50 as the backbone network, where the input video frame sampling rate for the slow branch is 2 frames per second, and the input video frame sampling rate for the fast branch is 16 frames per second. The slow branch can contain 5 residual blocks, each containing multiple convolutional layers to extract spatial semantic features of the production operation, such as identifying the operator's hand movements, tool types, and the shape and position of materials. The fast branch can contain 3 residual blocks, each also containing multiple convolutional layers, but with smaller strides in the temporal dimension, used to capture rapid motion changes in the production operation, such as the instantaneous actions of the operator grasping and placing materials. Lateral connections can be set between different layers of the network. The feature map of the fast branch is downsampled and upsampled through a 1x1x1 3D convolutional layer and then concatenated with the corresponding feature map of the slow branch to achieve information exchange. Finally, at the network's output layer, the final feature vectors of the slow and fast branches are concatenated to form a 2048-dimensional spatiotemporal state feature vector. This feature vector integrates the spatial appearance and temporal dynamics of the production operation, comprehensively representing its complex spatiotemporal state.

[0096] Through the above technical solution, this application effectively addresses the challenge of simultaneously considering spatial details and temporal dynamics when extracting spatiotemporal state features of production operations in production process monitoring. By employing the dual-branch structure of the SlowFast network, the slow branch deeply captures the fine spatial semantic information of production operations, while the fast branch accurately captures rapidly changing actions and motion trajectories. This complementary feature extraction method results in more comprehensive and accurate spatiotemporal state features of production operations, avoiding the loss of details or insufficient motion information that may result from single feature extraction methods. Therefore, in subsequent multi-task classification of production operations, the richer and more accurate spatiotemporal features significantly improve the accuracy and reliability of operation type identification, compliance assessment, and abnormal behavior detection, thereby providing strong support for refined monitoring and management of production processes.

[0097] See Figure 5 In one embodiment, the method for evaluating the compliance of material movement state characteristics and operational comprehensive state characteristics includes, but is not limited to, steps S501 to S502.

[0098] Step S501: Perform a compliance evaluation on the motion state characteristics of the material to obtain the compliance evaluation results of the material motion state characteristics.

[0099] Step S502: Perform local compliance evaluation and global compliance evaluation on the overall operational status characteristics to obtain the characteristic compliance evaluation results of the overall operational status characteristics.

[0100] Material movement compliance evaluation refers to comparing and judging the extracted material movement characteristics against pre-defined material movement specifications to determine whether the material movement meets the requirements. The purpose of this evaluation is to ensure that the flow of production materials on the production line is efficient, orderly, and without anomalies. This evaluation can be implemented by setting threshold rules, such as the upper limit of the dwell time of production materials in a specific area or the speed range on a specific path, and comparing the material movement characteristics with these rules. Alternatively, machine learning classifiers, such as Support Vector Machines (SVMs) or decision trees, can be used to automatically learn and judge the compliance of current material movement by training on historical compliant and non-compliant material movement data.

[0101] Local compliance evaluation refers to an immediate, fine-grained assessment of a single or partial dimension (e.g., operation type, compliance level, anomaly type) within the overall characteristics of production operations. Its purpose is to quickly identify the compliance or anomaly of individual operational behaviors. This evaluation compares operation type characteristics, compliance level characteristics, and anomaly type characteristics against pre-defined local compliance standards. For example, it determines whether an operation belongs to a permitted operation type, whether its compliance level meets the minimum standard, and whether there are known anomaly patterns. Alternatively, a rule-based expert system can be used to trigger corresponding compliance check rules based on the real-time characteristics of the operation and output local evaluation results.

[0102] Global compliance evaluation refers to a holistic and macro-level assessment of the overall state characteristics of production operations over a period of time or a series of operations. The purpose of this evaluation is to assess the coherence, logic, and overall standardization of the entire operational process, identifying instances where individual operations may be compliant but the overall process is non-compliant. This evaluation can be achieved by analyzing the characteristic sequences of operation types and matching them with pre-defined compliant operational process templates, for example, by using Dynamic Time Warping (DTW) algorithms or Hidden Markov Models (HMMs) to assess the similarity of operation sequences. Alternatively, a graph-based operational process can be constructed, mapping the overall state characteristics of operations to graph nodes, and analyzing the graph's structure and paths to determine the global compliance of the operational process.

[0103] This application's solution addresses the issue of insufficient precision and comprehensiveness in production process monitoring by employing differentiated evaluations of different types of characteristics. Specifically, when evaluating the compliance of material movement characteristics and overall operational characteristics, this solution departs from a single evaluation method. Instead, it conducts a dedicated motion state compliance evaluation for material movement characteristics, while simultaneously performing local and global compliance evaluations for overall operational characteristics. This divide-and-conquer strategy allows for independent and in-depth analysis of the physical flow compliance of materials and the behavioral standardization of operators. The motion state compliance evaluation focuses on whether the movement of materials in space and time conforms to preset paths, speeds, and dwell times, ensuring smooth production rhythm and material flow. The local compliance evaluation of overall operational characteristics can instantly capture the correctness, standardization, and presence of anomalies in individual operational behaviors, such as the immediate compliance of operating postures, tool usage, and operational steps. Building upon this, the global compliance evaluation further examines the overall logical sequence and coherence of a series of operational behaviors from a macro perspective, ensuring that the entire operational process complies with process specifications and avoiding situations where individual operations are compliant but the overall process is non-compliant. Through this multi-level and multi-dimensional evaluation mechanism, this solution can more comprehensively and accurately identify potential problems in the production process, providing more instructive compliance evaluation results for production management.

[0104] The following is a concrete example. Suppose an automated assembly line needs to monitor the transport of workpieces and the assembly behavior of operators. First, the executing entity acquires multiple video frames from the production monitoring video. For workpiece transport, its position coordinate sequence and instantaneous speed on the conveyor belt can be extracted as material motion state characteristics. Next, a motion state compliance evaluation is performed. For example, the dwell time of the workpiece in a specific detection area (e.g., a vision inspection station) should be between 5-10 seconds, and its speed through that area should be between 0.1-0.5 m / s. If the actual monitored dwell time of the workpiece is 12 seconds or the speed is 0.05 m / s, it is judged as non-compliant, and a corresponding feature compliance evaluation result is generated. Simultaneously, for the operator's assembly behavior, its operation type (e.g., "tightening screws"), compliance level (e.g., full weld joints, no cold welds), and abnormality type (e.g., no abnormalities) can be extracted as comprehensive operation state characteristics. Based on this, local compliance evaluation and global compliance evaluation are performed. Local compliance assessments can focus on a single "screw tightening" operation, checking whether the operator used the correct electric screwdriver, whether the tightening torque met standards, and whether the screw was fully tightened. For example, if a screw is detected as not fully tightened, the local compliance assessment result might indicate "screw not tightened abnormally." Global compliance assessments, on the other hand, examine the sequence of "screw tightening" operations within the entire assembly process. For example, if the assembly specification requires the "place shim" operation to be completed before the "screw tightening" operation, and the executing entity detects that the "screw tightening" operation occurred before the "place shim" operation, the global compliance assessment result would indicate "operation sequence abnormality." In this way, characteristic compliance assessment results of material movement state characteristics and characteristic compliance assessment results of comprehensive operational state characteristics can be obtained separately.

[0105] Through the aforementioned technical solution, this application enables a refined and multi-dimensional compliance evaluation of material movement and operational behaviors in the production process. This differentiated evaluation method allows for the independent and accurate assessment of the efficiency and standardization of material flow. Simultaneously, the compliance of individual operator actions and the logical coherence of the entire operational process can be comprehensively considered. This effectively avoids the blind spots and inaccuracies that may arise under traditional single-evaluation models, significantly improving the accuracy and targeting of production process monitoring. This allows for more effective identification of production bottlenecks, quality defects, and potential safety hazards, providing production managers with more valuable decision-making support, thereby optimizing production efficiency and product quality.

[0106] In some embodiments, a compliance evaluation of the motion state characteristics of materials is performed, including: determining the residence time characteristics of the production materials in a preset area, the displacement distance characteristics within the preset time, and the quantity characteristics of the materials in the preset area based on the motion state characteristics of the materials; and determining the compliance evaluation results of the residence time characteristics, displacement distance characteristics, and quantity characteristics of the materials based on the deviations between the residence time characteristics, displacement distance characteristics, and quantity characteristics of the materials and the corresponding motion state compliance characteristics.

[0107] Preset areas refer to the spatial ranges in the production environment that are pre-defined and have specific requirements for material behavior. They can be manually defined by operators by drawing polygons or rectangles on the video monitoring interface, or automatically generated by analyzing CAD drawings of the production line layout, or determined by identifying key workstations or buffer zones on the production line through machine learning models.

[0108] The residence time characteristic refers to the length of time that production materials stay in a preset area. It can be calculated by continuously tracking the entry and exit timestamps of production materials in the preset area, or by starting a timer when the production materials enter the preset area and stopping the timer when they leave. The preset duration refers to a specific time window used to measure material displacement. It can be a fixed time interval, such as 5 seconds or 10 seconds, or it can be dynamically adjusted according to the process requirements of different production stages.

[0109] The displacement distance feature refers to the straight-line distance or actual trajectory length of the production material within a preset time period. It can be obtained by calculating the Euclidean distance between the position coordinates of the material at the start and end points of the preset time period, or by accumulating the displacement of the material between each video frame.

[0110] Material quantity characteristics refer to the number of production materials existing in a preset area at a certain moment. This can be achieved by real-time detection and counting of material objects in the preset area in each video frame, or by maintaining a list of active material tracking IDs in the preset area.

[0111] Motion status compliance characteristics are pre-set benchmark values ​​or ranges used to determine whether the movement of materials conforms to the specifications. For example, they can be set by production experts based on experience, or obtained through statistical analysis of a large amount of normal production data.

[0112] The compliance evaluation results of the three characteristics of residence time, displacement distance and material quantity are a quantitative representation of whether each specific characteristic (residence time, displacement distance and material quantity) meets the compliance standards and the degree of deviation. It can be a binary judgment (compliant / non-compliant), a score or a multi-level classification label.

[0113] This application's solution utilizes acquired material movement state characteristics to further refine and quantify key material behaviors in the production process, namely, residence time within a specific preset area, displacement distance within the preset time, and material quantity within the preset area. These specific and measurable characteristics can more accurately reflect the actual movement state of the material. Subsequently, by comparing these refined characteristics with preset movement state compliance characteristics and calculating their deviations, the compliance of each characteristic can be objectively determined. This method concretizes the abstract "movement state compliance evaluation" into a quantitative assessment of multiple key indicators, making the judgment of material movement compliance more refined and accurate. In this way, this application can conduct a comprehensive and in-depth analysis of the material movement state from multiple dimensions, thereby more effectively identifying potential production anomalies or non-compliant behaviors.

[0114] The following is a concrete example. Suppose that on an automated assembly line, a critical inspection station is defined as a preset area. The system continuously monitors the production materials entering this preset area. When a piece of production material enters the preset area, the execution unit starts timing and records the dwell time characteristic when it leaves. Simultaneously, the execution unit tracks the displacement of the material within the preset time (e.g., 5 seconds) and calculates the displacement distance characteristic. Furthermore, the execution unit also counts the number of production materials simultaneously present in the preset area in real time, obtaining the material quantity characteristic. For example, if the compliance characteristics of the motion status of this inspection station stipulate that the dwell time should be between 3 and 5 seconds, the displacement distance within 5 seconds should be greater than 10 centimeters, and the number of materials in the area should not exceed 2. When the executing entity detects that a material's dwell time is 8 seconds (exceeding the 5-second upper limit), or its displacement distance within 5 seconds is only 5 centimeters (less than the 10-centimeter lower limit), or there are 3 materials in the area at the same time (exceeding the 2-material upper limit), the executing entity will determine the dwell time feature, displacement distance feature, or material quantity feature as non-compliant based on these deviations, and generate the corresponding feature compliance evaluation results.

[0115] Through the above technical solution, this application provides a more refined and objective method for evaluating the compliance of material movement status. By introducing characteristics such as residence time, displacement distance, and material quantity, and comparing them with preset compliance standards, specific problems such as prolonged material residence, insufficient movement, or material accumulation in the production process can be effectively identified. This makes the detection of abnormal material movement in the production process more accurate and timely, enabling earlier discovery of production bottlenecks, inefficiencies, or potential faults. It provides production managers with more instructive decision-making support, thereby improving the overall monitoring accuracy and response efficiency of the production process.

[0116] In some embodiments, local and global compliance evaluations are performed on the comprehensive operational status characteristics, including: determining the characteristic compliance evaluation results of the operation type characteristics, compliance degree characteristics, and anomaly type characteristics based on the deviations between the operation type characteristics, compliance degree characteristics, and anomaly type characteristics of the production operation and the corresponding local compliance characteristics; and determining the characteristic compliance evaluation results of the operation type characteristics based on the deviations between the operation type characteristic sequence and the preset compliant operation characteristic sequence.

[0117] An operation type feature sequence refers to an ordered set formed by arranging the type features of consecutive production operations occurring over a period of time according to their chronological order. For example, on an assembly line, an operation type feature sequence might include a series of operations such as "grabbing part A," "placing part A," "tightening screws," and "inspection." Its purpose is to integrate discrete individual operation type features into a whole with temporal context, enabling higher-level process analysis. This sequence can be constructed by sorting the production operation types identified in video frames using timestamps or by arranging them based on frame numbers.

[0118] The operation type feature sequence is obtained by arranging the operation type features of production operations based on the temporal features of video frame images. The operation type feature sequence can be constructed by first identifying each production operation and its corresponding operation type feature from consecutive video frame images. Then, using the temporal information carried by the video frame images themselves (e.g., frame number, shooting timestamp), these identified operation type features are arranged in chronological order along the timeline, thereby constructing a sequence reflecting the actual operation process. For example, if frame 100 identifies "grab," frame 150 identifies "place," and frame 200 identifies "tighten," then the sequence is "grab-place-tighten."

[0119] A pre-defined sequence of compliant operational characteristics refers to an ordered set of compliant operational types defined in advance based on standard production processes or expert experience. It represents the correct order and type of operations that should be followed in a specific production stage. For example, for a product assembly step, the pre-defined sequence of compliant operational characteristics might be "material retrieval - assembly - inspection - packaging". This sequence can be stored in a database as a benchmark for assessing the compliance of actual operational sequences.

[0120] The compliance evaluation results for operation type characteristics can be qualitative (e.g., "compliant", "non-compliant", "abnormal sequence") or quantitative (e.g., compliance score). For example, when the deviation exceeds a preset threshold, the operation type characteristic sequence can be determined to be non-compliant. This result provides a crucial basis for the overall compliance evaluation of the production process.

[0121] This application's solution, building upon local compliance evaluation of production operations, further achieves more comprehensive and accurate monitoring of the production process by introducing a global compliance evaluation of the operation type feature sequence. Specifically, the executing entity first extracts the operation type features of each production operation from consecutive video frame images. Utilizing the temporal characteristics of the video frame images, these discrete operation type features are arranged according to their chronological order of occurrence, thus constructing an "operation type feature sequence" reflecting the actual production process. Subsequently, this actual operation type feature sequence is compared with a predefined "pre-defined compliance operation feature sequence." By calculating the "deviation" between the two, the executing entity can quantify the degree of deviation between the actual operation process and the standard process. For example, if a key step is missing from the actual operation sequence, or the order of the operation steps is reversed, these deviations will be accurately identified and quantified. Based on this deviation, the executing entity can determine the feature compliance evaluation result of that operation type feature and judge whether the entire operation process conforms to the pre-defined specifications. This sequence-level evaluation mechanism compensates for the blind spots that may exist in local compliance evaluation of only a single operation. It can effectively identify process non-compliance issues caused by incorrect operation sequence, missing steps, or redundant operations, and significantly improve the accuracy and reliability of production process monitoring.

[0122] The deviation between the operation type feature sequence and the preset compliant operation feature sequence can be calculated by using the Dynamic Time Warping (DWT) algorithm to determine the minimum alignment cost between the two sequences. The specific process is as follows: First, calculate the i-th operation type feature in the operation type feature sequence. The j-th compliance operation feature in the compliance operation feature sequence The deviation between them is usually expressed using Euclidean distance. measure: , Next, construct the cumulative cost matrix, which is a... matrix ,in Indicates the operation type feature sequence and compliant operation characteristic sequence The minimum cumulative cost between them. This matrix is ​​first initialized: , Then, recursive calculations are performed for... and : , Normal settings and Used to handle boundaries; The final calculated DWT distance for: , This distance measure is the deviation between the operation type feature sequence and a preset compliant operation feature sequence. A smaller DWT distance value indicates a smaller deviation between the operation type feature sequence and the preset compliant operation feature sequence. This method also accurately identifies structural deviations such as omissions, redundancies, or disordered sequences of actions. From this, a sequence similarity score can be obtained. : , in, It is a scaling parameter. When the sequence similarity score is lower than the preset score threshold, it is determined that the operation type feature sequence is in violation.

[0123] As a specific implementation method, on an electronic product assembly line, a standard assembly process is assumed to include five operations: "removing the motherboard," "installing the CPU," "applying thermal paste," "installing the heatsink," and "connecting the power cord." The system first analyzes production monitoring video to identify the various operations performed by workers and extracts the corresponding operation type features. For example, over a period of time, the system might identify an operation sequence such as "removing the motherboard," "installing the CPU," "connecting the power cord," and "installing the heatsink." Simultaneously, a preset compliant operation feature sequence is defined as "removing the motherboard - installing the CPU - applying thermal paste - installing the heatsink - connecting the power cord." The executing entity compares the actually identified operation type feature sequence with the preset compliant operation feature sequence. By calculating the edit distance between sequences, it can be found that the actual sequence lacks the crucial step of "applying thermal paste," and the order of "connecting the power cord" and "installing the heatsink" is reversed. Based on this discrepancy, the executing entity can determine that the operation type feature sequence is non-compliant and generate corresponding feature compliance evaluation results, such as "missing thermal paste application operation" and "abnormal operation sequence."

[0124] Through the above technical solution, this application effectively addresses the problem that relying solely on compliance judgments of individual operations cannot comprehensively assess the compliance of the entire production process. By constructing and evaluating operation type feature sequences, this application can identify complex process anomalies such as incorrect sequence of operation steps, omissions of key steps, or redundant operations. These anomalies are difficult to detect using traditional compliance checks based on individual operations. This makes production process monitoring more refined and intelligent, significantly improving the accuracy and reliability of overall production process compliance judgments. This allows for the timely detection and correction of potential problems in the production process, ensuring product quality and production efficiency.

[0125] Figure 6This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0126] The following reference Figure 6 To describe an electronic device 600 according to such an embodiment of the present disclosure. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0127] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0128] The storage unit stores program code, which can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the above-described production process monitoring method section of this specification according to various exemplary embodiments of this disclosure.

[0129] Storage unit 620 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 6201 and / or cache memory 6202, and may further include a read-only memory (ROM) 6203.

[0130] Storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0131] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0132] Electronic device 600 can also communicate with one or more external devices 600' (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0133] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0134] The production process monitoring method, equipment, and medium provided in this application acquire video frame images and extract features from material movement and operational behavior. Then, the extracted material movement state features and comprehensive operational state features are integrated and subjected to feature compliance evaluation, ultimately generating a process compliance evaluation result. This improves the monitoring efficiency and temporal logic coherence analysis performance of production process monitoring. Compared to traditional manual inspection, acquiring video frame images and extracting features from material movement and operational behavior enables real-time, objective, and quantitative monitoring of the production process. Precise location and speed feature extraction significantly improves the accuracy and timeliness of monitoring. Compared to isolated sensor networks or single-function video surveillance, integrating the extracted material movement state features and comprehensive operational state features and performing feature compliance evaluation allows for the understanding and evaluation of complex visual semantic information, overcoming the technical limitations of existing technologies in understanding and parsing rich visual semantic behavioral information.

[0135] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the methods described above according to the embodiments of this disclosure.

[0136] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0137] Computer-readable storage media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0138] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0139] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A production process monitoring method, characterized in that, include: Acquire multiple video frame images from production monitoring videos; Motion state features are extracted from the production materials in the video frame image to obtain material motion state features that include the position and velocity features of the production materials; The operation status features of the production operations in the video frame images are extracted to obtain comprehensive operation status features that include operation type features, compliance level features, and abnormality type features of the production operations. The material movement state characteristics and the comprehensive operation state characteristics are evaluated for compliance, and a process compliance evaluation result for the production process is generated based on the evaluation results.

2. The production process monitoring method according to claim 1, characterized in that, The step of extracting motion state features from the production materials in the video frame image includes: The positional features of the production materials in the video frame image are extracted to obtain the positional features of the production materials. Based on the location features of the production materials and the temporal features of the video frame images, the velocity features of the production materials in the video frame images are extracted to obtain the velocity features of the production materials. By fitting the position and velocity characteristics of the production material, the motion state characteristics of the material are obtained.

3. The production process monitoring method according to claim 2, characterized in that, The step of extracting velocity features from the production materials in the video frame image based on the positional features of the production materials and the temporal features of the video frame image includes: Based on the location characteristics of the production material and the temporal characteristics of the video frame images, the motion trajectory of the production material is predicted; Based on the motion trajectory of the production material and the temporal characteristics of the video frame image, the velocity features of the production material in the video frame image are extracted to obtain the velocity features of the production material.

4. The production process monitoring method according to claim 1, characterized in that, The step of extracting operation state features from the production operations in the video frame image includes: Spatiotemporal state features of the production operations in the video frame images are extracted to obtain spatiotemporal state features of the production operations that characterize the spatiotemporal features of the production operations. The spatiotemporal state features of the operation are subjected to multi-task classification processing to obtain the operation type features, compliance degree features, and anomaly type features of the production operation. By fitting the operation type feature, the compliance feature, and the anomaly type feature, the comprehensive operation status feature is obtained.

5. The production process monitoring method according to claim 4, characterized in that, The extraction of spatiotemporal state features from the production operations in the video frame images includes: Based on the slow branch of the SlowFast network, continuous convolution operations are performed on the production operations in the video frame image to obtain the spatial domain features of the production operations. Based on the fast branch of the SlowFast network, continuous convolution operations are performed on the production operations in the video frame image to obtain the temporal features of the production operations. By fitting the spatial domain features and the temporal domain features, the spatiotemporal state features of the operation are obtained.

6. The production process monitoring method according to claim 1, characterized in that, The feature compliance evaluation of the material movement state characteristics and the comprehensive operation state characteristics includes: A compliance evaluation of the motion state characteristics of the material is performed to obtain the feature compliance evaluation result of the material motion state characteristics. Local and global compliance evaluations are performed on the overall operational status characteristics to obtain the feature compliance evaluation results of the overall operational status characteristics.

7. The production process monitoring method according to claim 6, characterized in that, The motion state compliance evaluation of the material motion state characteristics includes: Based on the material movement state characteristics, the residence time characteristics of the production material in the preset area, the displacement distance characteristics within the preset time, and the material quantity characteristics within the preset area are determined. Based on the deviations between the residence time feature, the displacement distance feature, and the material quantity feature and the corresponding motion state compliance features, the feature compliance evaluation results of the residence time feature, the displacement distance feature, and the material quantity feature are determined.

8. The production process monitoring method according to claim 6, characterized in that, The process of performing local and global compliance evaluations on the overall operational status characteristics includes: Based on the deviations between the operation type characteristics, compliance degree characteristics, and anomaly type characteristics of the production operation and the corresponding local compliance characteristics, the characteristic compliance evaluation results of the operation type characteristics, compliance degree characteristics, and anomaly type characteristics are determined; Based on the deviation between the operation type feature sequence and the preset compliant operation feature sequence, the feature compliance evaluation result of the operation type feature is determined; the operation type feature sequence is obtained by arranging the operation type features of the production operation based on the temporal features of the video frame image.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the production process monitoring method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the production process monitoring method according to any one of claims 1 to 8.

Images