Complex man-machine cooperation station operation compliance sequential logic verification method and system based on multi-modal video

Abstract

The invention discloses a complex man-machine cooperation station operation compliance sequential logic verification method and system based on a multi-modal video, and the method comprises the steps: obtaining a standard operation instruction (SOP), analyzing the SOP, and converting the SOP into a formal model with sequential logic constraints; synchronously acquiring a video data stream, an audio data stream and an equipment data stream of the station; performing real-time analysis on the video data stream by using a human-object interaction (HOI) deep learning model to generate structured visual event metadata; taking the visual event metadata as input, and driving the formalized model to carry out state evolution so as to judge the compliance of an operation process; and performing fusion cross validation on the audio data stream and the equipment data stream at a preset key verification node by taking the time window of the occurrence of the visual event as a reference. By fusing fine-grained visual interaction recognition, strict sequential logical reasoning and multi-modal cross validation, high-reliability closed-loop verification can be carried out on operational process compliance and physical result authenticity.

Description

Technical Field

[0001] This invention relates to the intersection of manufacturing automation and information technology. Specifically, this invention relates to a method and system for verifying the compliance timing logic of complex human-machine collaborative workstation operations based on multimodal video. Background Technology

[0002] In modern high-end manufacturing, such as the production and assembly lines for aerospace vehicles, precision medical endoscopes, new energy vehicle battery modules, and core communication equipment, despite increasing automation, many critical processes still heavily rely on highly skilled manual operations due to the complex requirements for operational flexibility, precision, and handling of non-rigid components. These processes, such as the precise placement of tiny irregularly shaped gaskets, the tightening of bolts requiring specific torque and tightening sequences, the non-destructive connection of multi-pin flexible circuit boards, or the micron-level alignment and installation of optical lenses, rely on compliance with operational quality standards as the cornerstone of the final product's performance, long-term reliability, and safety. In these operations, a seemingly insignificant operational deviation—such as omitting the assembly of a limit circlip, setting a bolt torque below or above the process specification range, or failing to fully lock a connector—can trigger the failure of the entire system or equipment, causing significant economic losses or even serious safety accidents.

[0003] To achieve rigorous control and quality traceability throughout the entire production process, Manufacturing Execution Systems (MES), as the core information system connecting the Enterprise Resource Planning (ERP) layer and the underlying industrial control layer, are widely used in modern factories. Traditional MES management of manual processes largely remains at the level of result recording, i.e., operators scan workpiece barcodes at workstation terminals and manually click to confirm the completion of a process. This management method is essentially a passive, trust-based recording approach, which cannot provide any effective and automated supervision of the authenticity, accuracy, and standardization of the actual operations occurring in the physical world. Therefore, this method is insufficient to prevent potential quality defects caused by fluctuations in operator skill, fatigue, negligence, or misunderstandings of process documents, and it also cannot provide objective and detailed process evidence for subsequent quality problem tracing.

[0004] To solve this problem, the industry has mainly explored and practiced along two technical directions.

[0005] The first direction is to introduce computer vision-based automated monitoring solutions. These solutions typically deploy industrial cameras above or to the side of the workstation and use computer vision algorithms to analyze the captured video streams to automatically identify operator actions. This technology, to some extent, replaces manual verification and improves the level of automation in monitoring. However, when applied to precision assembly scenarios requiring high accuracy, these solutions have several inherent limitations that are difficult to overcome. First, their recognition granularity is too coarse. It can recognize the action of "grabbing," but it struggles to distinguish whether the operator is grabbing an M3 gasket specified in the SOP or an adjacent M4 gasket from the hopper. It can recognize "tightening," but it cannot determine whether the operator is using the specified torque wrench. This lack of ability to identify interactive objects (parts and tools) makes it unsuitable for sophisticated error-proofing tasks. Second, these solutions have a weak understanding of the temporal logic of work processes. Real SOPs often contain complex process structures, such as the parallel installation of multiple components, selection of different operation paths based on inspection results, or conditional rework cycles. Existing vision-based solutions often employ simple linear step lists or finite state machines for process judgment. For complex nonlinear logic, the accuracy and robustness of these judgments decrease significantly, easily leading to logical confusion when operators skip steps or misjudge sequences. Finally, as a single-modal technology, pure vision solutions are vulnerable to the challenges of complex industrial environments. Operators' arms, bodies, or tools inevitably obstruct critical components or operating areas during operations, resulting in the loss of visual information. Simultaneously, sudden changes in ambient lighting and glare from metal surfaces severely interfere with the stability of image recognition algorithms. More fundamentally, vision, as a non-contact perception method, cannot penetrate object surfaces to verify the achievement of physical properties. The system cannot confirm whether the tightening torque of a bolt precisely meets the process requirement of 2.5 ± 0.1 N·m simply by "looking," nor can it determine from an image whether a snap-fit ​​connector emits the characteristic "click" sound indicating that its internal contacts are fully engaged after insertion.

[0006] The second direction involves applying smart tools and devices based on Internet of Things (IoT) technology. This approach integrates sensors and wireless communication modules into handheld tools (such as torque wrenches and screwdrivers) or directly collects operational data from automated equipment with data interfaces (such as glue-applying robots and press-fitting machines), uploading this precise physical data (such as torque, angle, pressure, and glue application amount) to the MES system in real time. This approach can accurately quantify the results of key processes, representing a significant advancement in quality management. However, its main drawback lies in the separation between the collected data and the actual "physical operational context," creating a new information silo. For example, an operator can operate a digital torque wrench without aligning it with the target bolt, and the wrench will still record and upload a valid torque value. The MES system can only receive this isolated data but cannot determine whether the operation corresponding to this data was actually performed on the specified workpiece and location. In other words, this approach cannot effectively prevent intentional "data cheating" or unintentional misoperation (such as applying the correct torque to the wrong screw hole). When data loses its context in the physical world, its credibility as a quality credential is greatly reduced.

[0007] In summary, existing technological approaches, whether relying solely on visual monitoring of process appearances or solely on IoT tools for physical result collection, have failed to construct a comprehensive solution capable of effectively closing the loop to verify both process compliance and result authenticity. Therefore, there is an urgent need in this field for a new method and system that can deeply understand the intricate spatiotemporal interactions between people, tools, and components; rigorously analyze and enforce the complex temporal logic within Standard Operating Procedures (SOPs); and intelligently integrate information from multiple physical dimensions for cross-validation. This would provide truly comprehensive, in-depth, and reliable digital monitoring and quality assurance for complex human-machine collaborative workstations. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for verifying the compliance of complex human-machine collaborative workstation operations based on multimodal video, aiming to solve the technical problems existing in the prior art, such as coarse granularity of monitoring the manual operation process, weak temporal logic constraints, insufficient reliability of single information sources, and disconnect between process and result.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a method for verifying the compliance of complex human-machine collaborative workstation operations based on multimodal video, comprising the following steps: Obtain the standard operating procedure (SOP) and parse the SOP into a formal model with temporal logic constraints; The video data stream from the workstation is acquired synchronously, along with at least one non-visual modal data stream associated with the operating steps in the standard operating procedure. The video data stream is processed in real time to identify the interaction relationships between the operator, tools and parts, and to generate structured visual event metadata. Based on the current state of the formal model and taking the visual event metadata as input, determine whether the visual event metadata satisfies the triggering conditions for state transition in the formal model, so as to drive the formal model to perform state evolution; Furthermore, in the formal model, if the visual event metadata satisfies the triggering condition for state transition corresponding to the key verification node, the non-visual modality data stream is verified within a preset time window based on the timestamp of the visual event metadata, so as to finally confirm whether the state has transitioned based on the verification result.

[0010] Preferably, identifying the interaction relationships between the operator, tool, and parts includes: The video data stream is analyzed using a pre-trained deep learning model for human-object interaction. The output of the human-object interaction deep learning model is a triplet structure containing the interaction subject, the interaction action, and at least one interaction object.

[0011] Preferably, the human-object interaction deep learning model is a two-stream spatiotemporal attention Transformer network model, the model comprising: Spatial Stream Module for extracting spatial visual features from video data streams; A temporal stream module used to extract temporal motion features from video data streams; And a spatiotemporal attention encoder module for performing relational reasoning and contextual information fusion on the spatial visual features and temporal motion features.

[0012] Preferably, the formal model is a Petri net model, in which the library represents the operation state, the transition represents the operation step, and the flow of tokens represents the evolution of the state; The process of driving the formal model to evolve its state includes: matching the visual event metadata with the output transition of the place currently holding the token in the Petri net model; if the match is successful, triggering the transition so that the token flows from the input place to the output place of the transition.

[0013] Preferably, the non-visual modal data stream includes an audio data stream and / or a device data stream; The verification of the non-visual modality data stream includes: If the non-visual modal data stream is an audio data stream, then detect whether there are target voiceprint features associated with the visual event within the time window; And / or, if the non-visual modal data stream is a device data stream, then within the time window, it is determined whether the value of the device data stream meets a preset physical quantity condition.

[0014] Another aspect of the present invention is to provide a timing logic verification system for compliance of complex human-machine collaborative workstation operations based on multimodal video, which implements the above-mentioned method.

[0015] To achieve the above objectives, the present invention provides the following technical solution: a time-series logic verification system for compliance of complex human-machine collaborative workstation operations based on multimodal video, comprising: The standard operating procedure modeling module is configured to acquire the standard operating procedure and parse the standard operating procedure into a formal model with temporal logic constraints. A multimodal data acquisition module is configured to synchronously acquire video data streams from the workstation and at least one non-visual modal data stream associated with the operating steps in the standard operating procedure. The visual interaction detection module is configured to process the video data stream in real time to identify the interaction relationship between the operator, tools and parts, and generate structured visual event metadata. The temporal logic reasoning module is configured to determine, based on the current state of the formal model and with the visual event metadata as input, whether the visual event metadata satisfies the triggering conditions for state transition in the formal model, so as to drive the formal model to perform state evolution. Additionally, a multimodal fusion verification module is configured to perform verification on a preset key verification node in the formal model. If the visual event metadata satisfies the triggering condition for state transition corresponding to the key verification node, the non-visual modal data stream is verified within a preset time window based on the timestamp of the visual event metadata, so as to finally confirm whether the state has transitioned based on the verification result.

[0016] In summary, this invention acquires Standard Operating Procedures (SOPs), parses and transforms them into a formal model with temporal logic constraints; simultaneously collects video, audio, and equipment data streams from workstations; utilizes a Human-Object Interaction (HOI) deep learning model to perform real-time analysis of the video data streams, generating structured visual event metadata; uses this visual event metadata as input to drive the formal model through state evolution to determine the compliance of the operational process; and at preset key verification nodes, uses the time window of the visual event occurrence as a benchmark to perform fusion cross-validation of the audio and equipment data streams. This invention, by integrating fine-grained visual interaction recognition, rigorous temporal logic reasoning, and multimodal cross-validation, solves the problems of existing technologies such as coarse-grained monitoring of manual operations, weak temporal constraints, susceptibility to interference, and inability to prevent cheating. It enables highly reliable closed-loop verification of the compliance of the operational process and the authenticity of the physical results. Through the above technical solutions, this invention can produce the following beneficial effects: First, this invention improves the accuracy and depth of operation recognition. By employing a human-object interaction (HOI) deep learning model, the system can deepen the recognition granularity from macroscopic actions in the background technology to microscopic interaction events involving specific parts and tools. This capability enables the system to perform refined material and tool error prevention, preventing quality problems caused by misuse or omission at the source.

[0017] Secondly, this invention enables strict timing guarantees for complex workflows. By parsing SOPs into formal models such as Petri nets, the system possesses the ability to rigorously express and reason about complex workflow logics such as parallelism, branching, and synchronization. Using high-level semantic visual interaction events as the sole condition driving the model's state evolution ensures that every step of the operation strictly follows the preset process path, effectively preventing skipping steps, out-of-order operations, and other violations.

[0018] Third, this invention enhances the reliability and anti-interference capability of verification conclusions. The multimodal fusion cross-validation mechanism, by using visual events as time anchors, forces logical self-consistency verification of other modal data within a time window, constructing a complementary and mutually reinforcing three-dimensional evidence chain. This mechanism effectively overcomes the weakness of visual data being susceptible to interference from environmental factors such as occlusion and reflection, and by verifying the operational context, it eliminates potential cheating behaviors that might arise from solely relying on device data, thus significantly improving the reliability of the verification results.

[0019] Fourth, this invention provides in-depth and reliable digital traceability capabilities. The system generates a multimedia quality evidence package for each operation, containing structured event data, keyframe images, and even audio and device data fragments, and reports this package to the MES after binding it with a unique product identifier. This provides an objective and detailed data foundation for subsequent quality analysis, responsibility determination, and continuous process optimization. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the system architecture according to an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the workstation hardware deployment according to an embodiment of the present invention.

[0023] Figure 3 This is a flowchart of a method according to an embodiment of the present invention.

[0024] Figure 4 This is a schematic diagram of the HOI Transformer network model structure in an embodiment of the present invention.

[0025] Figure 5 This is an example diagram illustrating the transformation from SOP to Petri net model in an embodiment of the present invention.

[0026] Figure 6 This is a schematic diagram of the multimodal data fusion verification logic in an embodiment of the present invention.

[0027] Figure 7 This is a schematic diagram comparing the technical effects of embodiments of the present invention.

[0028] Figure 8 This is a graph showing the recognition accuracy of the HOI model in an embodiment of the present invention.

[0029] Figure 9 This is a diagram showing the time synchronization error of multimodal data in an embodiment of the present invention.

[0030] Figure 10 This is a heatmap of visual event confidence in an embodiment of the present invention.

[0031] Figure 11 This is an example diagram of the torque curve waveform in an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0033] It should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0034] Example 1 This embodiment details a method for verifying the compliance of complex human-machine collaborative workstation operations based on multimodal video using sequential logic. Before describing this method, first refer to... Figure 1 and Figure 2 The preferred system architecture and hardware environment for implementing this method are described.

[0035] Please see Figure 1 , Figure 1 This is a schematic diagram of the overall architecture of an operational compliance timing logic verification system 100 provided in an embodiment of the present invention. The architecture of the system 100 is designed as a logically clear and functionally decoupled four-layer model, namely, a perception layer 110, a processing layer 120, a decision layer 130, and an application layer 140.

[0036] The perception layer 110, acting as the system's sensory organ, has the core function of capturing all raw information from the physical workstation environment for compliance analysis. Deployed on the production floor, this layer consists of a series of sensors and data acquisition interfaces. Specifically, the perception layer 110 may include at least one visual sensor 111. Preferably, to acquire rich color and three-dimensional spatial information, this visual sensor 111 is an RGB-D depth camera capable of simultaneously outputting color and depth video streams. Depth information aids in subsequent precise three-dimensional spatial positioning, dimensional measurement, and addressing information incompleteness caused by occlusion. The perception layer 110 may also include at least one auditory sensor 112, such as a cardioid microphone or microphone array, configured to directionally acquire sound signals from critical operational areas to subsequently identify characteristic voiceprints associated with specific process steps (such as component fastening, material breakage, etc.). In addition, the perception layer 110 also includes device data sources 113, which can be a series of intelligent industrial devices or tools capable of outputting their operating status data. Examples include digital torque wrenches with integrated wireless communication modules, automatic glue applicators with data interfaces, or pressure sensors connected via programmable logic controllers (PLCs). A key technical requirement for the perception layer 110 is that all its output data streams, including video, audio, and device data, must be assigned a unified, high-precision timestamp to ensure the accuracy of subsequent time window alignment and fusion analysis.

[0037] The processing layer 120, typically deployed in an edge computing unit adjacent to the workstation, serves as the system's primary neural hub, responsible for real-time preprocessing and feature extraction of massive, heterogeneous raw data from the perception layer 110. This layer transforms unstructured signal streams into structured information streams. Specifically, the processing layer 120 includes a multimodal data preprocessing module 121 and a HOI visual interaction detection engine 122. The multimodal data preprocessing module 121 performs adaptive processing on various input data streams, such as decoding, denoising, color space conversion, and pixel value normalization for video streams; performing digital filtering, background noise suppression, and short-time Fourier transform (STFT) on audio streams to generate spectrograms for speaker analysis; and parsing communication protocols, converting data formats, and filtering out outliers for device data streams. The HOI visual interaction detection engine 122 is the core of the processing layer 120 and an important aspect of this invention. It runs a specially trained deep learning model whose function is no longer traditional, generalized action classification, but rather deep scene understanding of the input video stream, accurately identifying the spatiotemporal interaction relationships between the "operator" (specifically the hand), the "tool," and the "parts." The engine's output is highly structured visual event metadata, such as a JSON object containing timestamps, interaction subjects, interaction actions, interaction objects, and spatial location information.

[0038] The decision layer 130, also deployed in the edge computing unit, is the cognitive core and decision-making brain of the entire system. Based on the structured information output from the processing layer 120 and combined with preset process rules, it performs the highest-level logical judgment and compliance ruling. Specifically, the decision layer 130 includes an SOP modeling and loading module 131, a temporal logic reasoning engine 132, and a multimodal fusion verification module 133. The SOP modeling and loading module 131 is responsible for receiving the standard operating procedure (SOP) file for a specific product from upper-level systems such as MES before the production task begins, and automatically and deterministically parsing and compiling it into a formalized, computer-executable temporal logic model. In a preferred embodiment, this model is a Petri net. The temporal logic reasoning engine 132 uses the loaded Petri net model as its execution blueprint and the visual interaction events output in real time by the HOI engine 122 as its driving signals. It continuously matches the currently occurring visual events with the next operation allowed by the current state of the Petri net model (i.e., the triggering conditions for "transitions") to drive the evolution of the model state. The multimodal fusion verification module 133 serves as a key checkpoint for quality control. It is activated at the key nodes that are predefined in the SOP model and require additional verification. Through a cross-validation mechanism with visual events as time anchors, it performs logical consistency checks on other modal data (such as audio and device data) from the processing layer 120 and feeds back the final "pass / fail" result to the temporal logic reasoning engine 132.

[0039] Application layer 140, serving as the interface between the system and the external world (including field operators and the factory information system), is responsible for transforming the internal judgments of decision layer 130 into meaningful external actions and traceable digital records. Specifically, application layer 140 includes a result judgment and feedback unit 141, a human-machine interface (HMI) 142, and a MES interface 143. The result judgment and feedback unit 141 integrates information from the decision layer to generate final operational result conclusions (such as process qualification, sequence error, torque non-compliance, etc.) and drives corresponding feedback. HMI 142, typically an industrial touchscreen or display next to the workstation, can be linked to audible and visual alarm devices to display work instructions and highlight key operational points to the operator in real time, and provide immediate and specific alarm information and corrective guidelines when anomalies occur. MES interface 143 is responsible for stable and reliable two-way data exchange with the factory's Manufacturing Execution System (MES). It uploads verification results containing detailed diagnostic information and multimedia evidence to the MES to build a digital archive of the product's entire lifecycle, while also receiving new production work orders and SOP instructions from the MES.

[0040] In the above four-layer architecture, the system's information flow ( Figure 1 (Represented by solid arrows) This process embodies a bottom-up data value-added process: from raw multimodal signals at the perception layer, to structured event information at the processing layer, then to logical judgment at the decision-making layer, and finally to business-oriented quality records and feedback at the application layer. The system's control flow ( Figure 1 (Represented by dashed arrows) This embodies a top-down instruction transmission and closed-loop feedback: MES sends SOPs from the application layer to the decision layer for modeling, and the decision layer's judgment results are fed back to HMI through the application layer to guide and constrain operator behavior, thus forming a complete intelligent closed loop of "perception-cognition-decision-execution".

[0041] Please see Figure 2 , Figure 2 This is a schematic diagram of a typical workstation hardware deployment scenario 200 in an embodiment of the present invention. On a typical precision assembly workbench, there is a main operating area 201, a material box 202 for storing parts of different specifications, and a tool rack 203 for organizing tools. To achieve comprehensive monitoring of this workstation, the hardware components in this embodiment are optimized as follows: The RGB-D depth camera 204, serving as the core visual sensor, is mounted directly above the workstation, forming a global view. This deployment minimizes the obstruction of key areas due to operator body or arm movements, while the depth data it provides provides the system with three-dimensional scene information of the entire workstation. A cardioid microphone array 206 is mounted on an adjustable arm, its pickup direction precisely focused on the specific location within the main operating area 201 most likely to produce key sound events (such as latch engagement), thereby improving the signal-to-noise ratio of the target voiceprint. A wireless digital torque wrench 207 held by the operator, representing the device data source, communicates with the system via low-power wireless technology (such as Bluetooth or Wi-Fi). All sensor data acquisition, processing, and decision-making algorithms run on an edge computing unit 205 deployed beneath the workbench. This unit is preferably an industrial-grade embedded AI computer equipped with a high-performance GPU. The human-machine interface (HMI) 208, serving as an industrial touchscreen, is mounted on a cantilever for easy operator observation and operation. The most intuitive status indicator, the audible and visual alarm (Andon light) 209, is placed in a prominent, high position within the workstation. These hardware components are connected to the edge computing unit 205 via wired or wireless means, collectively forming a stable, efficient, and integrated physical platform for data acquisition and interaction.

[0042] After introducing the preferred system and hardware environments described above, the following will refer to... Figures 3 to 7 The embodiments of the method provided by the present invention will be described in detail.

[0043] Please see Figure 3The figure is an overall flowchart 300 of an operation compliance timing logic verification method provided by an embodiment of the present invention.

[0044] Step S301: SOP modeling and loading.

[0045] The initial step of this method is to transform a human-readable Standard Operating Procedure (SOP) into a logical model that can be formally reasoned about by a computer. This step provides a benchmark and basis for all subsequent automated verification work. In a specific implementation, the unstructured or semi-structured SOP document (such as PDF or Word document) first needs to be preprocessed into a structured data format, such as XML or JSON. In this structured file, each independent process or operation step is defined as an object or element, which contains several key attribute fields, such as: a unique "step ID"; a "text description" for HMI display; a "list of predecessor step IDs" defining the relationship between processes; an "interaction event specification" defining the core operation of the step (e.g., subject: hand, action: pick up, object: M3 pad); and a "multimodal verification node specification" defining the additional verification required for the step (e.g., modality: equipment data, characteristic: torque, condition: >= 2.5 N·m).

[0046] In a specific implementation, traditional SOP formats (such as Word and PDF) need to be converted into a structured data format, such as XML (Extensible Markup Language) or JSON (JavaScript Object Notation). In this structured SOP file, each operational step is treated as an independent element and contains the following key fields: StepID: A unique identifier for a step.

[0047] Description: A natural language description of this step.

[0048] Predecessors: A list containing all the StepIDs of the preceding steps, defining the temporal dependencies between the steps.

[0049] Interaction: Defines the core visual interaction events required to perform this step, including elements such as {subject, action, object1, object2}. For example: {hand, 'pick up', 'M3 pad', null}.

[0050] VerificationNodes: A list defining the multimodal verification rules that need to be triggered for this step. Each rule contains {modal type, target feature, threshold / condition, time window}. For example, for the bolt tightening step, this field might contain: [{modal: 'device_data', feature: 'torque', condition: '>=2.5N·m'}, {modal: 'audio', feature: 'ratchet_sound_pattern', condition: 'detected'}].

[0051] The size of the time window is set according to the network latency jitter range of the workstation and the data upload frequency of non-visual sensors, and is usually 2-3 times the average network latency to ensure coverage of data transmission fluctuations.

[0052] Subsequently, upon receiving this structured SOP file, the SOP modeling and loading module 131 in the system will execute an automatic parsing and construction process, transforming it into a Petri net model. As a mature mathematical modeling tool for discrete event systems, Petri nets have a natural advantage in describing complex process logics such as concurrency, asynchrony, and synchronization.

[0053] Please see Figure 5 This diagram is a simplified SOP fragment and its corresponding Petri net model 500. In this model, circular nodes are called "places" and represent a specific state of the system. For example, place P1 represents the "task started" state, and place P2 represents the "planetary gear picked up" state. Rectangular nodes are called "transitions" and represent an action or event to be executed, corresponding to a specific operation step in the SOP. For example, transition T1 corresponds to the operation "picking up planetary gears". Directed arcs define the association between states and events. A transition is eligible to be activated only when all its upstream (input) places hold a "token" (a token indicating state activation). Once a transition is activated, it consumes one token in each of its input places and simultaneously generates a new token in each of its output places, thus completing a state transition.

[0054] exist Figure 5In the example shown, the parallel operations in the SOP (steps T2 and T3 can be executed in any order) are naturally expressed by the Petri net structure as two independent paths branching off from the same place P2. The subsequent synchronous operation (step T4 can only begin after both T2 and T3 are completed) is expressed as transition T4 requiring both places P3 and P4 as inputs; T4 can only be triggered after both places have accumulated tokens. Through this mapping, the complex temporal logic inherent in the SOP document is rigorously and unambiguously encoded into the Petri net's topology and transition rules. This completed Petri net model is then loaded into the temporal logic inference engine 132 as a dynamic blueprint for real-time verification.

[0055] Step S302: Multimodal data acquisition and preprocessing.

[0056] After the SOP model is loaded, the system enters a continuous, real-time multimodal data sensing phase. The various sensors deployed at the workstation's sensing layer 110 begin to work in parallel.

[0057] Specifically, the RGB-D depth camera 204 continuously captures color and depth video streams at a preset frame rate (e.g., 30 frames per second) and resolution (e.g., 1920x1080). The microphone array 206 continuously acquires audio waveform data at a preset sampling rate (e.g., 16kHz). The wireless module of the edge computing unit 205 continuously listens for and receives data packets from smart tools such as the wireless torque wrench 207.

[0058] All of these acquired raw data streams are immediately sent to the multimodal data preprocessing module 121 of the processing layer 120 for processing. This processing is parallel and low-latency, designed to provide clean and standardized input for subsequent high-level cognitive algorithms.

[0059] For video data streams, preprocessing may include: frame decoding, image size normalization, applying the CLAHE (contrast-limited adaptive histogram equalization) algorithm to enhance image sharpness under uneven lighting, and pixel-level alignment and registration of the depth map and color map.

[0060] For audio data streams, preprocessing may include: using bandpass filters to filter out noise that overlaps with the frequency range of industrial ambient noise, applying spectral subtraction or deep learning models (such as RNNoise) for dynamic noise reduction, and converting it into a two-dimensional spectrogram through short-time Fourier transform (STFT), which can more intuitively reflect the energy distribution of sound events in time and frequency.

[0061] For device data streams, preprocessing includes: parsing data frames according to the communication protocol of the specific device, extracting key payloads such as torque value, angle value, and device ID, performing standardized conversion of data units, and filtering out abnormal jump values ​​that may be caused by signal interference.

[0062] Crucially, all preprocessed data, whether video frames, audio feature maps, or device data points, will be precisely associated with timestamps generated by a unified clock source and stored in a high-speed memory buffer for use by subsequent algorithms.

[0063] After the aforementioned preprocessing steps, the heterogeneous raw data streams from different physical sensors are transformed into three standardized, time-aligned, and clean data streams: image frame sequences, audio feature map sequences, and device data point sequences. These are stored in the high-speed memory buffer of the edge computing unit 205, ready to be accessed by the various engines of the processing layer 120 and the decision layer 130, laying a solid data foundation for subsequent intelligent analysis and decision-making.

[0064] Step S303: Real-time interaction recognition based on HOI. This aims to extract high-level semantic, structured information describing human operational intentions from the preprocessed video data stream. This step is performed by the HOI visual interaction detection engine 122 in processing layer 120.

[0065] In a preferred embodiment, the engine embeds a specially designed dual-stream spatiotemporal attention HOITransformer network model, the specific structure of which can be found in [reference needed]. Figure 4 ,like Figure 4As shown, the dual-stream spatiotemporal attention HOI Transformer network model used in this invention takes a short video clip containing T consecutive frames as input. The internal processing flow of the model is divided into four stages: the input layer receives video clip data; the feature extraction stage contains two parallel branches, where the spatial flow module 401 uses a CNN backbone network (such as ResNet or EfficientNet) to extract fine spatial features from keyframes for identifying objects and hands in the scene, and the temporal flow module 402 uses a 3D CNN (such as I3D or SlowFast) to process the entire video clip and learn the motion patterns of pixels in the temporal dimension; the feature fusion stage first locates all instances of people and objects in the image and generates bounding boxes through the object detector, and then constructs local feature vectors of person-object pairing through RoI feature pooling. These feature vectors, along with spatial location encoding information, are sent to the spatiotemporal attention Transformer encoder 403, which learns the complex relationships between instances through a multi-head self-attention mechanism; the output stage outputs the final interaction event triplet by the interaction prediction decoder 404, in the format {interaction subject, interaction action, interaction object}.

[0066] In one specific implementation, the model takes a short video clip containing T consecutive frames (e.g., T=16 or 32) as input. The internal processing flow of the model is as follows: First, the input video clip is fed into two parallel feature extraction branches. The spatial flow module 401, typically a convolutional neural network (CNN, such as ResNet or EfficientNet) pre-trained on a large image dataset (e.g., ImageNet), is responsible for extracting fine spatial features from several keyframes of the video clip to identify objects in the scene (such as various parts, tools), as well as human hands. The temporal flow module 402, typically a 3D convolutional neural network (3D CNN, such as I3D or SlowFast), is responsible for processing the entire video clip, learning the motion patterns of pixels in the temporal dimension, and capturing dynamic information such as rotation, translation, and proximity.

[0067] Next, the feature maps extracted from the spatial flow are fed into an object detector head (e.g., the RPN and classification / regression head of YOLO or Faster R-CNN) to locate all instances of "people" (hands) and "objects" in the image and generate a bounding box for each instance. Subsequently, the system constructs all possible "person-object" pairings. For each pair, the system uses techniques such as RoI Align to precisely "crop" the local feature vectors belonging to this "person-object" pair from the spatial and temporal flow feature maps.

[0068] These extracted feature vectors, representing the visual appearance of each "person-object" pair in space and time, along with their relative spatial location encoding information, are fed into a key component of the model—the spatiotemporal attention Transformer encoder 403. This encoder, through its internal multi-head self-attention mechanism, can dynamically learn and infer the complex relationships between all instances in a video clip. Spatial attention: Within a single frame, the model can determine which objects are in contact with the hand or are in the vicinity, assigning them higher attention weights. For example, the model will highly focus on the "wrench" and "nut" whose bounding boxes overlap or are adjacent to the "hand," while ignoring other parts in the distant bin. Temporal attention: In the time dimension, the model can track the state evolution of a "person-object" pair throughout the entire video clip (T frames). For example, it can observe that the features of the "hand" and "wrench" exhibit a coordinated rotation pattern in the time series, thereby inferring the interaction action of "tightening."

[0069] Finally, the features fused from the depth information of the Transformer encoder are fed into a lightweight interaction prediction decoder 404. This decoder outputs the final recognition result, which is one or more highly structured interaction event triples. Each triple explicitly describes an interaction in the format {interaction subject, interaction action, interaction object}, accompanied by a high-confidence score and the three-dimensional spatial coordinates of the interaction (calculated using data from the fused depth camera). For example, a possible output would be {subject: "right hand", action: "tighten", object: "M5 hex bolt", tool: "A-type torque wrench", confidence: 0.98, position: [x, y, z]}.

[0070] The system will filter out interactive events with a confidence level higher than a preset threshold and send them to the decision layer 130 as the "facts" that drive the next logical reasoning.

[0071] Step S304: Sequential logic reasoning.

[0072] After the HOI visual interaction detection engine 122 begins to stably output structured, highly semantic visual event metadata, this method enters the core logical judgment and process control stage. This stage is executed by the temporal logic reasoning engine 132 in the decision layer 130. The core mission of this engine is to act as a rigorous "process judge." It uses the Petri net model loaded in step S301 as the sole "legal" standard and the visual events generated in real time in step S303 as objective "evidence" to determine whether each step of the operator's operation complies with the SOP and drives the synchronous state evolution of the entire process flow in the digital world.

[0073] In one specific implementation, the internal working mechanism of the temporal logic reasoning engine 132 is based on an "event-driven transition triggering" model that operates cyclically. Its detailed workflow can be broken down into the following sub-steps: First, real-time monitoring of the state and determination of the set of potential actions are performed. At the start of each processing cycle (typically matching the output frequency of the HO-I engine, e.g., 10-15 times per second), the inference engine first checks the global state of the current Petri net model. This state is uniquely determined by the set of all "places" currently holding "tokens". The engine identifies all these token-holding places and looks for all downstream "transitions" derived from these places. These transitions collectively constitute a set of "transitions to be triggered", which logically represents a list of all the next legal actions that the operator is allowed to perform according to the current process progress in the SOP. For example, in reference... Figure 5 In the Petri net model 500 shown, if only place P2 (representing the "planetary gear picked up" state) currently holds the token, then the set of "transitions to be triggered" is precisely {T2: mounting circlip, T3: mounting shim}. This means that the system only expects to receive visual events related to the mounting circlip or mounting shim at this time.

[0074] Secondly, precise matching of visual events with transition triggering conditions is performed. The inference engine retrieves the latest visual event metadata generated by the HO-I engine 122 from an input queue, provided the confidence level is higher than a preset threshold (e.g., 0.9). For each newly arrived visual event, the engine performs a structured, multi-dimensional matching verification against the preset triggering conditions of each transition in the current "to-be-triggered transition" set. This matching process is far more complex than simple string comparison; it is a semantic-level conformity check. Specifically, for a visual event E = {subject_e, action_e, object1_e, object2_e, location_e} and a transition T triggering condition C = {action_c, object1_c, object2_c, region_c}, the matching algorithm verifies: whether action_e is semantically consistent with action_c (e.g., 'tightening' and 'screwing' can be considered consistent); whether the category of object1_e is the category required by object1_c or its subclass (this requires the support of a predefined material / tool ​​knowledge graph); if object2_c exists, whether object2_e also meets its category requirements; and whether the spatial location 'location_e' where the interaction occurs falls within the geographical operating range region_c allowed by the transition. This operating region 'region_c' is usually pre-defined during system initialization by spatially calibrating the workstations.

[0075] Next, the compliant state transition is performed. If, in the previous matching verification, a visual event E successfully and uniquely (i.e., unambiguously) matches a transition T in the set of "transitions to be activated," then the inference engine determines that a fully compliant operation has been successfully executed. At this point, the engine executes the core dynamic rule of the Petri net, namely, "activating" the transition T. This activation operation consists of two indivisible atomic actions: first, consuming one token in each of its upstream input locations; second, generating a new token in each of its downstream output locations. For example, if the token is in P2 and a "install snap ring" event matching T2 is received, then P2's token is consumed, and P3 obtains a new token. This "flow" of tokens in the Petri net topology precisely and synchronously replicates a successful evolution of the process state in the physical world at the digital twin level. Along with this state transition, HMI 142 of application layer 140 will also receive a notification of the state update, and accordingly gray out the completed steps on its interface, while highlighting the next operation guide corresponding to the new "transition to be triggered".

[0076] Finally, various anomalies are captured and handled. The value of the inference engine lies not only in confirming compliant operations, but also in its ability to keenly capture various violations or anomalies. In a preferred embodiment, the engine can identify and handle the following types of anomalies: Firstly, there is an error in the operation content, meaning that the visual event that occurs does not match the triggering conditions of any of the current "transitions to be triggered". For example, when expecting to perform "installing the snap ring" (T2) or "installing the gasket" (T3), the operator mistakenly picks up the bolt. The {hand, pick, M5 bolt} event output by the HOI engine will not match either T2 or T3, and the inference engine will immediately determine it as an "error operation" or "misstep" exception.

[0077] Secondly, there is the issue of incorrect operation sequence. This occurs when a visual event matches a transition in the model, but that transition is not currently in the set of "transitions to be triggered." For example, if the operator attempts to execute "Install and Tighten Bolts" (T4) while the token is still in P2, the event output by the HOI engine may semantically match the trigger condition of T4. However, the inference engine will find that neither P3 nor P4 contains a token for T4's input library, therefore T4 is not eligible to be triggered. This situation will be judged as an "incorrect sequence" or "skip" anomaly.

[0078] Thirdly, there is the operation timeout. The system can set a reasonable upper limit for the dwell time of each vault. If no legitimate visual event occurs within this time to drive the token out of the vault, the system will determine it as an "operation timeout" or "idle" abnormality.

[0079] Once any of the above anomalies are detected, the timing logic inference engine immediately interrupts the normal process evolution, placing the current Petri net model in a "suspended" error state. Simultaneously, it generates an anomaly event containing detailed diagnostic information, including the anomaly type, occurrence time, the content of the erroneous visual event, and the currently expected action, and pushes this event to the application layer 140. The application layer then triggers the red alarm and audible / visual alarm 209 on the HMI 142 interface, and provides clear corrective instructions based on the anomaly type, such as "Incorrect material, please return and pick up the correct snap ring."

[0080] Through this rigorous reasoning mechanism based on formal models and event-driven approaches, this method transforms the execution of SOPs from a behavior that relies on human awareness and memory into a deterministic process that is precisely monitored, guided, and constrained by machines.

[0081] Step S305: Multimodal fusion cross-validation.

[0082] After verifying the compliance of the operational procedures, this method introduces a deeper verification layer to further ensure the authenticity and quality compliance of key operational steps at the physical result level. This step is performed by the multimodal fusion verification module 133 in the decision layer 130. This module does not intervene in all operational steps, but specifically works on those "key verification nodes" that were marked as having a decisive impact on the final product quality in the SOP modeling stage of step S301.

[0083] In a preferred embodiment, when a transition associated with a "critical verification node" (e.g.) Figure 5 When transition T2 "Install the snap ring", whose verification node is to verify the "click" sound; or transition T4 "Install and tighten the bolt", whose verification node is to verify the torque value, is initially triggered by a matching visual event, the temporal logic reasoning engine 132 will not immediately complete the activation of the transition, but will place the transition in a temporary "to be verified" state, and at the same time pass the control and related context information (including the visual event E_vision that triggered the transition and its timestamp t_vision) to the multimodal fusion verification module 133.

[0084] Once activated, the multimodal fusion verification module 133 immediately initiates a mandatory cross-modal data consistency check process centered on t_vision. Please refer to... Figure 6 This diagram illustrates the alignment and fusion verification logic of multimodal data streams along time axis 600. The core of this process is a correlation verification strategy based on "time anchor points + sliding time windows".

[0085] First, a dynamic, strongly correlated multimodal data retrieval time window W 602 is defined. The center of this time window is the timestamp t_vision 601 of the visual event, and its width Δt is a parameter that can be flexibly configured according to specific process characteristics, typically set to between several hundred milliseconds and one or two seconds (e.g., [t_vision - 0.5s, t_vision + 0.5s]). This time window W logically defines a region where "events occur simultaneously," and the system assumes that the timestamps of all other modal events with a strong causal relationship to E_vision should fall within this window.

[0086] Secondly, within the time window W, target features are retrieved and analyzed in parallel on one or more non-visual modal data streams. This process is specifically executed according to the verification rules preset in the SOP for this key verification node.

[0087] If the validation rule involves audio data stream 603, for example, for transition T2 "installing the spring clip," the rule requires detecting a "click" sound representing "full engagement." In this case, the module retrieves the spectrogram of the audio data segment in the buffer whose timestamp falls within W. Then, a pre-trained Sound Event Detection (SED) model (typically a lightweight convolutional neural network) specifically designed for detecting specific industrial voiceprints is invoked to analyze this spectrogram. This model is trained to recognize patterns 604 unique to the "click" sound, characterized by instantaneous bursts of energy within a specific frequency band (e.g., 2kHz-5kHz). If the model detects this voiceprint pattern, it returns "True" along with the detected timestamp t_audio; otherwise, it returns "False."

[0088] If the validation rule involves device data stream 605, for example, for transition T4 "Install and tighten bolts," the rule requires that the peak torque must reach or exceed 2.5 N·m. In this case, the module filters out all data point sequences in the buffer whose timestamps fall within W and whose device ID matches the torque wrench ID identified in the visual event. Then, a simple analysis algorithm iterates through this sequence, finds the peak torque reading 606, and compares it with the threshold of 2.5 N·m specified in the SOP. If the peak torque is greater than or equal to the threshold, it returns "True" along with the timestamp t_device at which the peak was reached; otherwise, it returns "False."

[0089] Next, a final, rigorous compliance decision is made. After all relevant non-visual modal data analysis is completed, the fusion verification module will synthesize all evidence and make a final binary (pass / fail) decision. A key verification node is judged to be "verified" only if all of the following conditions are met simultaneously: First, the main visual event E_vision that triggered this verification exists and has a high degree of confidence.

[0090] Secondly, the SOP specifies that all non-visual modal validation items for this node must have a retrieval and analysis result of "True".

[0091] Third, the timestamps of all successfully detected non-visual modal events (such as t_audio, t_device) must fall strictly within the time window W 602 centered on t_vision. This time compliance requirement is crucial, as it ensures that the "evidence" of all modalities is strongly correlated in time, thereby eliminating interference signals that are coincidental and irrelevant to the current operation.

[0092] The result of the ruling is immediately fed back to the waiting temporal logic inference engine 132. If the result is "verification passed," the inference engine will complete the activation of the transition, allowing the Petri net's state to continue evolving, with the entire process seamlessly connected. Conversely, if the result is "verification failed"—whether due to the failure to detect the necessary voiceprint signal, the torque value not meeting the standard, or the timestamp misalignment—the fusion verification module will return a "reject" signal. Upon receiving this signal, the inference engine will characterize this failure as a "quality non-compliance" anomaly, which is more serious than a process error, and immediately trigger the highest-level alarm and error handling process.

[0093] Through this rigorous, physical-fact-based cross-verification mechanism, the present invention effectively binds the "visual appearance" of the operation with its "physical substance," thereby constructing a highly reliable "quality firewall" that can effectively resist various environmental interferences and human cheating.

[0094] Step S306: Compliance assessment and result reporting.

[0095] As the final step in the entire verification process, the core task of this step is to efficiently, clearly, and reliably transform all the judgments generated within the system into valuable feedback and records for the external world (operators and the MES system). This step is led and coordinated by the result judgment and feedback unit 141 in the application layer 140.

[0096] First, the system generates and maintains a global, clearly categorized compliance status for workstation operations in real time. This status is not simply a "qualified / unqualified" state, but a multi-layered definition with rich context. For example, the status can be categorized as: "Compliance in progress," "Waiting for operator input," "Prompts and guidance," "Minor violation - correctable," and "Severe non-compliance - line stoppage required," etc. For non-compliant statuses, the system also provides detailed, structured exception codes and diagnostic information, such as {Error_Code:'E_SEQ_001', Description:'Incorrect operation sequence',Expected_Step:'T3', Detected_Action:'Tighten'}.

[0097] In a specific implementation, this state can be categorized as follows: Green / Conforming: The operator is currently executing operations according to the path sequence of the Petri net model, and all completed key verification nodes have passed verification. At this time, the system appears to be in a normal state.

[0098] Waiting and Prompting (Yellow / Idle or Prompt): The system is in a compliant state but is waiting for the operator to perform the next action. If the wait time exceeds the preset T_idle, the system may issue a gentle prompt. Alternatively, the system is waiting for the operator to select a step that requires confirmation (e.g., before a branch path in an SOP).

[0099] Anomalies and Alarms (Red / Non-conforming): The system has detected a clear violation. This status is further subdivided into specific anomaly types, such as: Error_Type_1: Sequence_Violation (Incorrect sequence / skip) Error_Type_2: Incorrect_Action (Action Error) Error_Type_3: Incorrect_Part / Tool (Material / Tool Error) Error_Type_4: Quality_Verification_Failed (Quality verification failed, such as insufficient torque or missing sound) Error_Type_5: Operation_Timeout (Operation timed out) Secondly, based on the real-time generated compliance status, the system will activate various human-machine interaction devices at the workstation to provide immediate, multi-channel on-site feedback. Please refer to [link / reference]. Figure 7 The figure is an example of information interaction between the system and HMI and MES in an embodiment of the present invention.

[0100] For HMI 208, under normal operating conditions ( Figure 7 (As shown in the example on the left), the interface displays the overall SOP flowchart 702 to the operator in an intuitive and easy-to-understand manner, highlighting the currently pending step 703 with a bold, enlarged view. Preferably, the interface can also embed a standard operating procedure video or 3D animation 704 for that step, clearly listing key process parameters and quality requirements. This design aims to transform passive monitoring into proactive guidance, helping operators execute each step clearly and accurately.

[0101] Once an anomaly is detected ( Figure 7 (Example on the right) The HMI interface immediately switches to alarm mode, for example, the screen background turns red, and a modal, blocking anomaly alarm window 705 pops up. This window uses non-technical, clear natural language and eye-catching icons to explain to the operator what error occurred (e.g., "Torque verification failed"), what the standard requirements are (e.g., "Target torque: 2.5 ± 0.1 N·m"), what the actual detected value is ("Peak torque detected: 1.8 N·m" 706), and most importantly, it provides clear, actionable corrective guidelines (e.g., "Retighten this bolt using the current tool" 707). This diagnostic, closed-loop feedback mechanism greatly improves the efficiency and accuracy of on-site anomaly handling. Synchronized with the HMI's visual feedback, the audible and visual alarm (Andon light) 209 also changes its color and beeping mode according to the status, providing on-site managers with a clear overview of the workstation status from a distance.

[0102] Secondly, and crucially, the system generates detailed, tamper-proof digital quality records and seamlessly integrates them into the factory's MES system. This integration is bidirectional and real-time. Using standard industrial IoT protocols such as OPC-UA or MQTT, the system pushes detailed verification results for each operational step—including start time, end time, execution time, operator ID, and final compliance conclusion—as independent event messages to the MES database in real time.

[0103] Furthermore, for each assembled workpiece with a unique serial number, the system generates a comprehensive electronic file 708 known as a "digital traveler" or "product passport." This file contains far more information than the textual information recorded in traditional MES systems; it is a collection of multimedia evidence covering the entire process. Specifically, for each step in the SOP, especially those key verification nodes, the file includes a "quality evidence package" 709. This evidence package may contain: image snapshots of one or more key visual frames that trigger the step; original audio waveform clips or their spectrograms within that key time window W during multimodal verification; and complete, unedited equipment data graphs (e.g., torque-angle curves) uploaded by intelligent tools.

[0104] This digital archive allows engineers to access it from the MES system at any time in the future when a product needs a quality investigation. They can then trace back to a real snapshot of the physical world at any point in the entire assembly lifecycle, across all dimensions, enabling them to quickly and accurately pinpoint the root cause of any problem.

[0105] Through this complete closed loop, from real-time judgment to immediate feedback and in-depth traceability, this method not only ensures quality in the production process, but also provides a solid and reliable infrastructure for enterprises to achieve continuous process optimization and data-driven decision-making.

[0106] Example 2 This embodiment provides a time-series logic verification system for compliance of complex human-machine collaborative workstation operations based on multimodal video. For ease of understanding, the overall physical deployment environment of this system can be referred to in Embodiment 1. Figure 2 A detailed description has been provided. This embodiment will primarily refer to... Figure 1 The system functional architecture shown provides a detailed explanation of the functions of each module that constitutes the system and their specific implementation methods, so that those skilled in the art can clearly understand the internal structure and working principle of the system and have the ability to manufacture and use the system.

[0107] The system includes: a Standard Operating Procedure (SOP) modeling module 131, a multimodal data acquisition module, a visual interaction detection module, a temporal logic reasoning module 132, and a multimodal fusion verification module 133. In a preferred embodiment, the visual interaction detection module, the temporal logic reasoning module 132, and the multimodal fusion verification module 133 can be integrated and deployed in an edge computing unit 205 located on the production site to ensure real-time data processing and low latency. The multimodal data acquisition module mainly consists of various sensors deployed at workstations.

[0108] The functions and implementation methods of each of the above modules will be explained in detail below.

[0109] Standard Operating Procedure (SOP) Modeling Module 131: This module serves as the source of rules for all logical judgments made by the system. It is responsible for transforming human-readable process documents into machine-executable formal logical models.

[0110] In one specific implementation, this module can be implemented as a software application with a graphical user interface (GUI) for use by process engineers. This application provides an import function that can read SOP files pre-formatted as structured text such as XML or JSON. Once the file is imported, a built-in SOP parser within the module iterates through the file content, automatically extracting the ID, description, predecessor dependencies, and defined interaction event specifications and multimodal validation specifications for each operation step.

[0111] Subsequently, a Petri net generator dynamically constructs the Petri net topology based on the resolved dependencies. Specifically, each SOP step is mapped to a "transition" node; the dependencies between steps determine the connection methods of "place" nodes and "arcs". For example, if step B depends on step A, the generator creates a place node, making it both the output place of A and the input place of B. If steps C and D both depend on step B, the generator branches two paths from the output place of B, connecting to the transitions corresponding to C and D respectively. If step E depends on both C and D, the generator ensures that the transition corresponding to E has two input places from both paths C and D. The generator is also responsible for permanently binding the "interaction event specifications" and "multimodal validation specifications" defined in the steps as attributes or metadata to their corresponding "transition" nodes.

[0112] Finally, the module outputs a complete, serializable Petri net model object. This object can be saved as a file or sent to the timing logic reasoning module 132 for loading via a network interface. Preferably, the module also has an interface with the MES system 710, enabling it to automatically receive SOP files associated with specific work order numbers from the MES, and dynamically and unmannedly complete the entire modeling and loading process when production tasks switch.

[0113] The multimodal data acquisition module is the foundation of the system's perception. Its function is to synchronously and continuously capture high-quality, multi-dimensional raw data from the physical world. Physically, this module is not a single device, but a distributed system composed of multiple sensor sub-modules and data acquisition interfaces.

[0114] The visual data acquisition submodule is primarily composed of at least one visual sensor 111 deployed at the workstation. In one specific implementation, this sensor can be an industrial-grade RGB camera. To obtain richer scene information, in a preferred implementation, the sensor is an RGB-D depth camera 204, employing technologies such as Active Stereo or Time-of-Flight (ToF). This submodule also includes a device driver and software development kit (SDK) running on the edge computing unit 205. The driver is responsible for initializing the camera, setting acquisition parameters (such as resolution, frame rate, and exposure time), and pulling RGB image frames and depth data frames from the camera hardware in a continuous stream. A key function of this submodule is to achieve hardware time synchronization. Preferably, it can use Precise Time Protocol (PTP) or Network Time Protocol (NTP) to synchronize the system clock of the edge computing unit with the factory's master clock server, and assign a high-precision, globally unified acquisition timestamp to each successfully captured frame of image data.

[0115] The audio data acquisition submodule is centered around an auditory sensor 112 deployed at the workstation, such as one or more MEMS microphones or a microphone array 206. In a preferred embodiment, a microphone array is used because it can form a controllable, highly directional pickup area at the software level using digital beamforming technology, effectively suppressing background noise from non-target directions in the industrial environment. This submodule also includes an audio device driver and data acquisition program running on the edge computing unit 205. This program is responsible for configuring the sampling rate (e.g., 16kHz or 48kHz) and bit depth (e.g., 16-bit or 24-bit) of the audio interface (e.g., I2S or USBAudio Class), and continuously acquiring raw PCM audio waveform data in the form of data blocks (buffers). Similarly, each acquired audio data block is assigned a timestamp synchronized with the video data stream.

[0116] Equipment Data Acquisition Submodule: This submodule is responsible for communicating with and acquiring data from various equipment data sources 113 at the workstation. Its implementation varies depending on the specific equipment's interface type.

[0117] If the device is a smart tool with wireless communication capabilities, such as the wireless torque wrench 207, this submodule is implemented as a wireless receiving service program on the edge computing unit 205. This program is responsible for listening to specific wireless communication protocols, such as the GATT service of Bluetooth Low Energy (BLE) or TCP / UDP sockets on a Wi-Fi network. When a data packet is received, a protocol parser parses the binary packet according to the tool's datasheet and extracts the payload.

[0118] If the device is connected via a wired industrial bus, such as Modbus RTU (RS-485) or CANopen, the submodule requires a corresponding hardware interface card and a driver that implements the corresponding bus protocol stack.

[0119] If the data originates from a PLC, this submodule can be implemented as a software module that implements an OPC-UA client or a specific PLC vendor's communication protocol (such as Siemens S7-Communication).

[0120] Regardless of the implementation method, the final output of this submodule is uniform, structured data with unique device identifiers, timestamps, and standardized physical quantity values.

[0121] All the data streams with synchronization timestamps collected by these sub-modules are sent to the multimodal data preprocessing module 121 of the processing layer 120. This module can be implemented as a high-efficiency data processing pipeline, using multi-threading or asynchronous I / O technology to perform preprocessing operations such as denoising, format conversion, and feature transformation as described in Example 1 on each data stream in parallel, and send the processed standardized data stream into a shared, thread-safe memory data ring buffer for efficient and low-latency consumption by subsequent analysis modules.

[0122] The visual interaction detection module is configured to perform deep parsing of video data streams and output high-level semantic interaction events. In one specific implementation, this module is implemented as a continuously operating inference service running on the GPU of the edge computing unit 205. This service internally loads and executes a pre-trained human-object interaction (HOI) deep learning model.

[0123] In a preferred embodiment, the network structure of the HOI deep learning model is the combination of the one described in Example 1. Figure 4 The dual-stream spatiotemporal attention Transformer network model 400 is described below. The implementation methods of its various sub-modules are further explained below: The backbone networks of the spatial flow module 401 and the temporal flow module 402 can be mature models pre-trained on large public datasets, and fine-tuned on a self-built industrial assembly dataset for the application scenario of this invention through transfer learning, so as to better adapt to the characteristics of lighting, object appearance and other features in industrial environment.

[0124] The target detector part in the instance detection and RoI pooling module can use a lightweight and efficient single-stage detector, such as YOLOv7-tiny or NanoDet, to meet the real-time requirements of edge computing while ensuring detection accuracy.

[0125] The spatiotemporal attention Transformer encoder module 403 is the core of this model. In a specific implementation, it can be composed of multiple (e.g., 6-12 layers) stacked encoder layers. Each layer contains a multi-head self-attention sublayer and a feed-forward network sublayer. The multi-head self-attention mechanism enables the model to learn the association weights between input features (i.e., features of all "person-object" pairs) in parallel from different perspectives (i.e., different "heads"), thereby capturing rich contextual relationships.

[0126] The interactive prediction decoder module 404 can consist of several simple fully connected layers. The number of neurons and activation function of its output layer need to be set according to the classification and regression requirements of the specific task. For example, the part used for interactive action classification has an output layer with the number of neurons equal to the size of the predefined action dictionary and uses the Softmax activation function to output the probability of each category; the part used to output interactive objects may be a pointer to a list of candidate objects or a classifier.

[0127] The visual interaction detection module obtains the preprocessed video frame sequence from the shared memory buffer, sends it into the HOI model for inference, and pushes the structured visual event metadata with qualified confidence obtained by inference to a dedicated output message queue for consumption by the temporal logic inference module 132.

[0128] The temporal logic reasoning module 132 is configured as the "process brain" of the system. In a specific implementation, this module is implemented as a state machine engine, which internally maintains a currently active Petri net model instance. The core of this module is an event processing loop that continuously pulls new visual events from the output message queue of the visual interaction detection module and updates the state of its internal Petri net model according to the "event-driven transition triggering" mechanism described in step S304 of embodiment one.

[0129] In a preferred embodiment, the module internally includes the following sub-components: A model loader is responsible for receiving and instantiating the Petri net model from the SOP modeling module 131 at the start of the task.

[0130] A state manager is responsible for tracking and recording the token distribution of all places in the Petri net in real time.

[0131] An event-transition matcher is responsible for semantically matching the input visual event with the triggering conditions of all "transitions to be triggered" in the current state. To improve matching efficiency, the triggering conditions of transitions can be pre-compiled into an efficient data structure, such as a hash table or a decision tree.

[0132] A transition trigger controller is responsible for executing token consumption and generation operations and updating the state manager's records when a match is successful.

[0133] An anomaly detector is responsible for monitoring anomalies such as operation timeouts, sequence errors, and content errors in parallel.

[0134] A communication interface is responsible for sending notifications to the feedback unit of the application layer 140 and the multimodal fusion verification module 133 when the state changes or an anomaly is detected.

[0135] A multimodal fusion verification module 133 is configured to perform deep, cross-modal cross-validation of critical operational steps. In one specific implementation, this module is implemented as a passively responsive service that is normally dormant and is only activated when it receives a "request verification" instruction from the temporal logic reasoning module 132.

[0136] This module also contains several dedicated sub-components: An audio analyzer internally loads one or more pre-trained Sound Event Detection (SED) models. Each SED model is specifically trained for a particular type of industrial sound signature (such as a "click," "break," or "motor start" sound). When invoked, it extracts the audio feature map within a specified time window from a shared memory buffer and feeds it into the corresponding SED model for inference.

[0137] A device data analyzer contains a series of algorithms for processing time-series data. For example, it can implement peak detection algorithms, threshold crossing detection algorithms, signal slope calculation algorithms, and simple statistical analyses (such as calculating the mean and variance). When invoked, it extracts the data sequence of a specific device within a specified time window from a shared memory buffer and executes the analysis algorithms specified for that node in the Standard Operating Procedure (SOP).

[0138] A fusion adjudicator, which is the core logic of this module, receives a verification request (containing the timestamp t_vision of the visual event) from the temporal logic reasoning module 132 and waits for the processing results from the audio analyzer and / or the device data analyzer. Based on the "time anchor + sliding time window" strategy described in step S305 of Embodiment 1, it performs temporal and logical compliance checks on events of all modalities and makes a final "pass" or "fail" decision, returning the result to the temporal logic reasoning module 132.

[0139] Through the collaborative work of the above modules, this system can form a complete technical closed loop from bottom-level perception to high-level decision-making, from process monitoring to quality verification, thereby effectively achieving the technical objectives of this invention.

[0140] Example 3 This embodiment takes a typical precision manufacturing application scenario—"manual operation verification at the automated assembly station of planetary gear reducers"—as an example to provide a comprehensive and reproducible explanation of the specific application, parameter settings, model construction, and the final technical effects that can be achieved by the present invention.

[0141] On an advanced robotic automated production line, the assembly of a certain type of planetary gear reducer is completed collaboratively by robots and humans. One key human-robot collaborative workstation involves sequentially installing a precision snap ring and a special fiber washer onto the planetary carrier assembly of the reducer, and finally securing it with a single critical bolt. The technical requirements for this workstation are extremely stringent. Material Misuse Prevention: Two types of retaining rings (Type C1 and Type C2) with very similar appearances are stored in the workstation material box area. The SOP clearly stipulates that Type C1 must be used. Misusing Type C2 retaining rings will cause assembly failure in the short term.

[0142] Order Guarantee: The order of "install the retaining ring first, then the washer, and finally the bolt" must be strictly followed. Any deviation from this order will result in the washer not being effectively tightened or the retaining ring not being properly positioned, thus creating serious quality hazards.

[0143] Process confirmation: Correct installation of the retaining ring is accompanied by a crisp, high-frequency "click" sound, indicating that it has snapped into the groove. The washer must be placed flat and without warping.

[0144] Quality parameters: The final tightening torque of the locking bolts must be precisely controlled within the range of 2.5 N·m ± 0.1 N·m. Too low a torque will cause the connection to loosen, while too high a torque may damage the fiber gasket or threads.

[0145] Traditional methods relying on visual inspection and manual data recording are no longer sufficient to meet the zero-defect quality target of this workstation. The system provided by this invention is deployed and applied precisely to address the challenges in this scenario.

[0146] To meet the above technical requirements, the system's hardware deployment plan is as follows: Vision sensor 111: An Azure Kinect DK was selected as the RGB-D depth camera 204. Its color camera resolution was set to 1920x1080@30fps, and the depth camera mode was set to NFOV Unbinned to achieve the highest depth accuracy at a resolution of 640x576. The camera was mounted vertically downwards at a height of 1.2 meters above the worktable surface to ensure its field of view completely covers the entire operating area, including the planetary carrier fixture, all material boxes, and the tool rack.

[0147] Hearing sensor 112: A linear microphone array 206 containing seven MEMS microphones is selected. This array is mounted approximately 0.4 meters above the planetary carrier clamp, and software beamforming technology is used to precisely align the pickup focus with the mounting position of the retaining ring. Audio acquisition is set to 16-bit, 48kHz sampling rate.

[0148] Device data source 113: Select a Desoutter E-LIT series wireless digital torque wrench 207 with integrated Bluetooth 5.0 module, with a range of 0.5-5 N·m and an accuracy of ±1%.

[0149] Edge computing unit 205: An industrial computer equipped with an NVIDIA Jetson AGX Orin 32GB module is selected. The powerful GPU computing power of this computing unit (up to 200 TOPS INT8) is sufficient to run the HOI model and other algorithms described in this invention in real time, and its rich interfaces also meet the requirements for simultaneous access to Azure Kinect, microphone array and Bluetooth module.

[0150] HMI 208 and Andon Light 209: Utilize a standard 15.6-inch industrial touchscreen and a three-color alarm light tower.

[0151] The process engineer used the SOP modeling module 131 software to create a Petri net model for this workstation. The core part of the model is... Figure 5 Similar to the example shown, but containing more detailed specification information: Transition T_Pick_Circlip (Pick-up Circlip): Interaction event specification: {Subject: Hand, Action: Pick up, Object: C1 type snap ring}. The "C1 type snap ring" is defined by its spatial location and visual characteristics within a specific material box.

[0152] Transition T_Install_Circlip (Installing the Circlip): Interactive event specification: {Subject: Hand, Action: Installation, Object: C1 type retaining ring, Target: Planetary carrier groove}.

[0153] Multimodal validation specification: {modality: audio, target feature: 'circlip_click_sound', time window: [-500ms, +500ms]}.

[0154] Transition T_Pick_Gasket (Pick Gasket): Interactive event specification: {Subject: Hand, Action: Pick up, Object: Fiber pad}.

[0155] Transition T_Install_Gasket (Installation Gasket): Interactive event specification: {Subject: Hand, Action: Placement, Object: Fiber pad, Target: Planetary carrier plane}.

[0156] Transition T_Tighten_Bolt (Tighten Bolt): Interaction event specification: {Subject: Hand, Action: Use, Object: M5 locking bolt, Tool: Desoutter_E-LIT}.

[0157] Multimodal verification specification: {Modality: Device data, Target feature: 'peak_torque', Condition: 'in_range(2.4, 2.6)N·m', Time window: [-1000ms, +1000ms]}.

[0158] To ensure the performance of the AI ​​model, a dedicated dataset was built and used, and the specific configuration process is as follows.

[0159] Dataset Construction: After deployment at the workstation, a total of approximately 50 hours of assembly operations were simulated by multiple skilled workers and novices, covering all correct operations as well as various foreseeable errors (such as using the wrong snap ring, reversed order, snap ring not installed properly, insufficient torque / overload, etc.). Azure Kinect cameras recorded all RGB-D video data, microphone arrays recorded audio data, and torque wrenches recorded corresponding torque data. A data annotation team meticulously annotated the collected video data. For each frame, 2D bounding boxes of hands, all parts, and tools were annotated. For video clips containing interactions, the subject, action, object, and start and end points of the interaction were annotated. A total of over 100,000 interaction instances were annotated.

[0160] HOI Model Training: The dual-stream spatiotemporal attention HOI Transformer network model 400 described in Example 2 was used. The spatial-stream CNN backbone network used a YOLOv7 model pre-trained on the COCO dataset, fine-tuned using the dataset from this scene. The temporal-stream 3D CNN backbone network used the I3D model. The Transformer encoder was set to 6 layers with 8 attention heads. Training was performed on a server equipped with 4 NVIDIA A100 GPUs, using the AdamW optimizer with an initial learning rate of 1e-4, and cosine annealing was used for learning rate decay. Training lasted for a total of 100 epochs.

[0161] SED Model Training: From the collected audio data, approximately 2000 clear "click" sound clips indicating that the spring clips are properly installed were manually selected as positive samples, and approximately 10000 other workstation noises (such as tool collision sounds, voices, and background machine noises) were selected as negative samples. Using these samples, a small Sound Event Detection (SED) model based on the MobileNetV2 architecture was trained.

[0162] Model Optimization and Deployment: After training, the HOI and SED models were quantized (INT8 precision) and optimized using the NVIDIA TensorRT toolchain to maximize their inference speed on the Jetson AGX Orin platform. The optimized HOI model took approximately 65 milliseconds for a single inference (processing 16 video frames), and the SED model took approximately 5 milliseconds for inference, fully meeting the performance requirements of real-time verification at the workstation.

[0163] After deployment, the system demonstrated significant advantages over existing technologies. Please refer to [link / reference]. Figure 7 This figure is a schematic diagram comparing the technical effects of an embodiment of the present invention.

[0164] Scenario 1: Torque verification with arm obstructing the view When tightening bolts, the operator's forearm often briefly and partially obscures the bolt head.

[0165] In a comparative experiment, if a purely visual solution using existing technology (such as...) is used... Figure 7 (a) When occlusion occurs, the target detection algorithm will cause its confidence to drop sharply because it cannot see the complete bolt, thus giving an erroneous "NG" alarm and interrupting the normal operation process.

[0166] The system of the present invention (such as Figure 7 (b) In this scenario, even if the visual confidence of the HOI model fluctuates, the temporal logic inference engine 132, being in the "tightening the bolt" verification state, will activate the multimodal fusion verification module 133. Within the time window of the visual event, this module successfully receives a qualified torque data with a peak value of 2.53 N·m from the wireless torque wrench 207. Therefore, the system ultimately correctly determines that this step is "OK," and the operation process can continue smoothly. This scenario fully demonstrates the robustness of the present invention in the face of incomplete visual information.

[0167] Scenario 2: Identification of "Idle Operation" Cheating Behavior To test the system's anti-spoofing capability, the operator was asked to manually tighten the torque wrench without touching the actual bolt, so that it also produced a qualified reading of 2.5 N·m.

[0168] In a comparative experiment, if a simple IoT data acquisition scheme using existing technology (such as...) is used... Figure 7 (c) The MES system will only receive a qualified torque data and therefore incorrectly record the process as "OK", thus missing a fatal quality defect.

[0169] The system of the present invention (such as Figure 7(d) In this scenario, although the multimodal fusion verification module 133 also receives qualified torque data, the judgment basis of the temporal logic reasoning engine 132—that is, the visual event from the HOI model—is {subject: hand, action: use, tool: torque wrench, object: null}, meaning that no effective interaction with the “M5 tightening bolt” has occurred. Due to the lack of necessary visual evidence to trigger the “tightening bolt” transition, the multimodal verification process is not even initiated, and the system directly determines it as “seriously non-compliant” during the temporal logic reasoning stage and immediately issues an alarm. This scenario decisively demonstrates the fundamental advantages of this invention in ensuring data consistency with the physical process and preventing data fraud.

[0170] like Figure 8 As shown, this invention demonstrates the accuracy variation curves of the dual-stream spatiotemporal attention (HOI) Transformer network model during training. The horizontal axis represents the training epochs (0 to 100), and the vertical axis represents the accuracy (0 to 1). The three curves represent the training set accuracy (solid dot), validation set accuracy (dashed square), and test set accuracy (dotted triangle). The curve trends show that the model's accuracy rises rapidly in the early stages of training, reaching over 90% around 20 epochs, and then enters a stable convergence phase. Ultimately, the training set accuracy reaches 99.5%, the validation set accuracy stabilizes at around 97%, and the test set accuracy reaches around 96%, indicating that the model has good generalization performance. The horizontal dashed line in the figure marks the target accuracy threshold of 95%, verifying that the model meets the accuracy requirements for industrial applications.

[0171] like Figure 9As shown, this invention demonstrates the measurement results of time synchronization error between multimodal data streams. The horizontal axis represents the sampling sequence number, and the vertical axis represents the time synchronization error, in milliseconds (ms). The two curves represent the video-audio synchronization error (solid dot) and the video-device synchronization error (dashed square dot), respectively. The gray-filled area represents the system-defined tolerance range of ±10ms. From the curve trends, it can be observed that the video-audio synchronization error fluctuates within approximately ±5ms, and the video-device synchronization error fluctuates within approximately ±10ms, both within the tolerance range. This high-precision time synchronization is the physical basis for the successful implementation of the cross-modal verification strategy proposed in this invention, which uses visual events as time anchors and performs cross-modal verification within a sliding time window. It ensures that the data from each associated modality have a strong causal relationship in time when the system makes fusion judgments, thereby effectively eliminating misjudgments caused by random coincidences. This is a key prerequisite for ensuring the reliability of the fusion verification mechanism of this invention. The Precise Time Protocol (PTP) or Network Time Protocol (NTP) adopted by the system can effectively ensure the uniformity and high-precision alignment of timestamps of data streams of each modality, providing a reliable time reference for subsequent multimodal fusion verification.

[0172] like Figure 10 As shown, this invention demonstrates the distribution of detection confidence of the HOI model for different combinations of interactive actions and objects. The vertical axis of the figure lists six typical interactive actions: picking, placing, tightening, pressing, inserting, and sliding; the horizontal axis lists six typical interactive objects: bolts, washers, snap rings, connectors, wrenches, and clamps. The grayscale value and numerical value of each cell in the heatmap represent the average detection confidence of that action-object combination; a higher value indicates a stronger ability of the model to recognize that type of interaction. The heatmap distribution shows that the model's detection confidence for most action-object combinations is above 0.7, and the confidence for frequently occurring typical interactions (such as "tightening-bolt" and "picking-washer") reaches above 0.95. This verifies the model's accurate recognition ability for various interaction types in industrial assembly scenarios, enabling it to reliably identify and classify various interactive behaviors that may occur during assembly, providing comprehensive and reliable input for subsequent temporal logic reasoning.

[0173] like Figure 11As shown, this invention demonstrates a typical waveform curve of torque changing over time during bolt tightening. The horizontal axis represents time in seconds (s), ranging from 0 to 3 seconds; the vertical axis represents torque value in Newton-meters (N·m), ranging from 0 to 3.2. The gray-filled area in the figure represents the acceptable torque range specified in the SOP (2.5 ± 0.1 N·m). The solid curve represents a successful tightening process: the torque value rises steadily from 0, reaching the target value of 2.5 N·m in approximately 2 seconds and remaining stable. The dashed curve represents a failed tightening process: although the torque curve shape is similar, the peak value only reaches 1.73 N·m, failing to enter the acceptable range. The multimodal fusion verification module of this invention can analyze such torque curve data in real time within the time window triggered by visual events, automatically determining whether the tightening operation meets the process requirements.

[0174] After a three-month trial run, the operational compliance rate of this workstation (automatically calculated by the backend logs) increased from 98.5% before deployment (with a 1.5% risk of errors) to over 99.99%. Downstream process rework or final product scrapping due to human error at this workstation was reduced to zero. This fully demonstrates the technical advantages of this invention in improving the quality assurance level of precision manufacturing.

[0175] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs. When the computer program is loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this disclosure are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program can be transferred from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0176] Those skilled in the art will understand that the various numerical designations such as "first," "second," etc., used in this disclosure are merely for the convenience of description and are not intended to limit the scope of the embodiments of this disclosure, nor do they indicate the order of events.

[0177] At least one of the features described in this disclosure can also be described as one or more, and multiple features can be two, three, four or more, and this disclosure does not impose any limitations. In the embodiments of this disclosure, for a technical feature, the technical features in that technical feature are distinguished by "first", "second", "third", "A", "B", "C" and "D", etc., and there is no sequential order or size order among the technical features described by "first", "second", "third", "A", "B", "C" and "D".

[0178] The correspondences shown in the tables of this disclosure can be configured or predefined. The values ​​of the information in each table are merely examples and can be configured to other values; this disclosure is not limiting. When configuring the correspondences between information and parameters, it is not necessarily required to configure all the correspondences shown in each table. For example, the correspondences shown in some rows of the tables in this disclosure may not be configured. Furthermore, appropriate modifications and adjustments can be made based on the above tables, such as splitting, merging, etc. The names of the parameters shown in the headers of the above tables can also use other names that the communication device can understand, and the values ​​or representations of the parameters can also use other values ​​or representations that the communication device can understand. In the implementation of the above tables, other data structures can also be used, such as arrays, queues, containers, stacks, linear lists, pointers, linked lists, trees, graphs, structures, classes, heaps, hash tables, or hash tables, etc.

[0179] The predefined terms in this disclosure can be understood as defined, pre-defined, stored, pre-stored, pre-negotiated, pre-configured, solidified, or pre-burned. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0180] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0181] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for verifying the compliance of complex human-machine collaborative workstation operations based on multimodal video, characterized in that, include: Obtain the standard operating procedure (SOP) and parse the SOP into a formal model with temporal logic constraints; The video data stream from the workstation is acquired synchronously, along with at least one non-visual modal data stream associated with the operating steps in the standard operating procedure. The video data stream is processed in real time to identify the interaction relationships between the operator, tools and parts, and to generate structured visual event metadata. Based on the current state of the formal model and taking the visual event metadata as input, determine whether the visual event metadata satisfies the triggering conditions for state transition in the formal model, so as to drive the formal model to perform state evolution; Furthermore, in the formal model, if the visual event metadata satisfies the triggering condition for state transition corresponding to the key verification node, the non-visual modality data stream is verified within a preset time window based on the timestamp of the visual event metadata, so as to finally confirm whether the state has transitioned based on the verification result.

2. The method according to claim 1, characterized in that, The identification of the interaction relationships between the operator, tools, and parts includes: The video data stream is analyzed using a pre-trained deep learning model for human-object interaction. The output of the human-object interaction deep learning model is a triplet structure containing the interaction subject, the interaction action, and at least one interaction object.

3. The method according to claim 2, characterized in that, The human-object interaction deep learning model is a two-stream spatiotemporal attention Transformer network model, which includes: Spatial Stream Module for extracting spatial visual features from video data streams; A temporal stream module used to extract temporal motion features from video data streams; And a spatiotemporal attention encoder module for performing relational reasoning and contextual information fusion on the spatial visual features and temporal motion features.

4. The method according to claim 1, characterized in that, The formal model is a Petri net model, in which the library represents the operation state, the transition represents the operation step, and the flow of tokens represents the evolution of the state. The process of driving the formal model to evolve its state includes: matching the visual event metadata with the output transition of the place currently holding the token in the Petri net model; if the match is successful, triggering the transition so that the token flows from the input place to the output place of the transition.

5. The method according to claim 1, characterized in that, The non-visual modal data stream includes audio data streams and / or device data streams; The verification of the non-visual modality data stream includes: If the non-visual modal data stream is an audio data stream, then detect whether there are target voiceprint features associated with the visual event within the time window; And / or, if the non-visual modal data stream is a device data stream, then within the time window, it is determined whether the value of the device data stream meets a preset physical quantity condition.

6. A time-series logic verification system for compliance of complex human-machine collaborative workstation operations based on multimodal video, characterized in that, include: The standard operating procedure modeling module is configured to acquire the standard operating procedure and parse the standard operating procedure into a formal model with temporal logic constraints. A multimodal data acquisition module is configured to synchronously acquire video data streams from the workstation and at least one non-visual modal data stream associated with the operating steps in the standard operating procedure. The visual interaction detection module is configured to process the video data stream in real time to identify the interaction relationship between the operator, tools and parts, and generate structured visual event metadata. The temporal logic reasoning module is configured to determine, based on the current state of the formal model and with the visual event metadata as input, whether the visual event metadata satisfies the triggering conditions for state transition in the formal model, so as to drive the formal model to perform state evolution. Additionally, a multimodal fusion verification module is configured to perform verification on a preset key verification node in the formal model. If the visual event metadata satisfies the triggering condition for state transition corresponding to the key verification node, the non-visual modal data stream is verified within a preset time window based on the timestamp of the visual event metadata, so as to finally confirm whether the state has transitioned based on the verification result.

7. The system according to claim 6, characterized in that, The visual interaction detection module includes a pre-trained deep learning model for human-object interaction. The output of the human-object interaction deep learning model is configured as a triplet structure containing an interactive subject, an interactive action, and at least one interactive object.

8. The system according to claim 7, characterized in that, The human-object interaction deep learning model is a two-stream spatiotemporal attention Transformer network model, which includes: The spatial stream module is configured to extract spatial visual features from video data streams. The temporal stream module is configured to extract temporal motion features from video data streams; Additionally, a spatiotemporal attention encoder module is configured to perform relational reasoning and contextual information fusion on the spatial visual features and temporal motion features.

9. The system according to claim 6, characterized in that, The formal model is a Petri net model, in which the library represents the operation state, the transition represents the operation step, and the flow of tokens represents the evolution of the state. The temporal logic reasoning module is configured to match the visual event metadata with the output transition of the place currently holding the token in the Petri net model. If the match is successful, the transition is triggered, causing the token to flow from the input place of the transition to the output place.

10. The system according to claim 6, characterized in that, The non-visual modal data stream includes audio data streams and / or device data streams; The multimodal fusion verification module is configured to: If the non-visual modal data stream is an audio data stream, detect whether there are target voiceprint features associated with the visual event within the time window; And / or, if the non-visual modal data stream is a device data stream, determine within the time window whether the value of the device data stream meets a preset physical quantity condition.

Images