Computer Vision-Based Operation Column Knob Status Recognition and Analysis System and Method

Abstract

This application relates to the field of industrial automation and intelligent auxiliary technology, and discloses a computer vision-based operating column knob status recognition and analysis system and method. The system includes visual perception, signal acquisition, and data processing modules. The visual perception module uses pose tracking to lock the region of interest of the target knob. The data processing module initiates anti-occlusion tracking when occlusion is detected, reconstructs texture by backtracking historical frames through a circular frame buffer queue, and calculates the instantaneous visual angular velocity and cumulative rotation angle using a dense optical flow field combined with a rigid body rotation constraint model. The signal acquisition module acquires the physical device's operating status signal. The data processing module adaptively selects a verification mode based on the sampling frequency, performs homomorphism verification on the visual data and operating status signal, and triggers feedback. This invention achieves stable tracking under strong occlusion environments and improves recognition accuracy through consistency verification of visual and physical signals.

Description

Technical Field

[0001] This invention relates to the field of industrial automation and intelligent auxiliary technology, specifically to a computer vision-based operating column knob status recognition and analysis system and method. Background Technology

[0002] In industrial production processes such as petrochemicals, power generation, and precision manufacturing, the knobs, switches, and valves on the field control columns are key actuators controlling the process flow. To prevent safety accidents caused by misoperation, traditional management models typically rely on dual-person verification or electronic tagging, but this reduces the efficiency of inspection and operation to some extent. With the development of the Industrial Internet and Augmented Reality (AR) technologies, using wearable smart devices for real-time monitoring and assistance of field operations has become a technological trend. Its core lies in using computer vision algorithms to perceive the operator's hand movements and equipment status in real time.

[0003] However, existing visual recognition technologies face significant challenges in monitoring the operation of knob-type devices due to occlusion. In actual operation, the operator's hands inevitably cover a large portion of the knob, obscuring visual features such as markings and textures. Traditional visual tracking algorithms mostly rely on feature point matching or template matching. Once the target features are covered by a work glove with a uniform texture and non-rigid deformation, the algorithm is prone to feature loss or tracking drift, failing to accurately calculate the actual rotation angle of the knob. This results in stuttering or incorrect overlay of virtual guidance screens, affecting the accuracy of human-computer interaction.

[0004] Furthermore, relying solely on visual perception for decision-making has limitations in industrial safety. Visual algorithms can only capture the operator's intentional actions and cannot confirm whether the physical equipment has actually responded. For example, when the internal mechanical transmission of a knob fails or the electrical connection is interrupted, the vision system may misjudge the operation as successful based on the hand rotation. Existing auxiliary systems often lack a mechanism for deep fusion and verification of visually perceived action data with the actual operating signals of the industrial control system. Moreover, due to the difference between the sampling frequency of industrial signals and the video frame rate, it is difficult to effectively verify the logical consistency between the two under a unified spatiotemporal reference. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a computer vision-based operating knob status recognition and analysis system and method. It solves the problems faced by existing industrial knob operation monitoring technologies, such as loss of visual features and tracking drift due to hand obstruction. Furthermore, it lacks a mechanism for effectively fusing and verifying visual perception data with physical equipment operation signals, making it difficult to guarantee the accuracy and reliability of operation status recognition.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a computer vision-based operation column knob status recognition and analysis system, which mainly consists of a visual perception module, a signal acquisition module, and a data processing module.

[0007] The visual perception module is used to collect environmental image data and inertial measurement data containing the target knob, and uses a pose tracking algorithm to calculate the current spatial pose matrix of the device, thereby realizing digital perception of the physical space. The signal acquisition module is used to connect with the physical device, collect the operating status signal of the physical device corresponding to the target knob, and normalize the signal to unify the data dimensions.

[0008] The data processing module connects the two modules mentioned above. Its core function is to solve the visual occlusion problem during knob operation and the consistency verification problem between visual and physical signals. Specifically, the data processing module first locates the region of interest in the environmental image data based on the spatial pose matrix. When the pixel change rate of the region of interest indicates occlusion, the data processing module activates anti-occlusion tracking logic. This logic does not simply rely on the current visual features, but calculates the visual instantaneous angular velocity of the occluding object based on the current image frame, and determines the visual cumulative rotation angle from the occlusion start time to the current time through integration. At the same time, the data processing module adaptively selects the verification mode according to the sampling frequency of the running status signal, performs homomorphism verification between the motion data obtained from visual calculation and the running status signal of the physical device, and finally generates a status flag and triggers corresponding feedback based on the verification result.

[0009] To achieve precise locking of the target knob, the data processing module obtains the three-dimensional coordinates of the target knob in the world coordinate system. Using the pinhole camera projection model, combined with the real-time updated spatial pose matrix and camera intrinsic parameter matrix, the three-dimensional coordinates are mapped to two-dimensional pixel coordinates. Based on this, the system estimates the projection radius according to the knob's physical radius and depth factor, and constructs an expanded rectangular area as the region of interest to ensure that it can cover the range of hand movements during operation.

[0010] To address the issue of visual loss caused by hand occlusion of the knob during operation, the system employs a texture reconstruction technique based on buffer backtracking. The data processing module maintains a circular frame buffer queue following a first-in-first-out (FIFO) principle. This queue synchronously binds and stores image frames and their corresponding spatial pose matrices. When occlusion is detected, the system stops updating the background reference map and retrieves historical image frames and historical poses from the queue before the occlusion occurred, according to a preset safety backtracking interval. Using the historical pose data and the knob normal vector, a homography transformation matrix is ​​constructed. The system performs inverse perspective transformation on the historical frames to generate a standard circular planar texture that eliminates parallax, providing occluded reference material for subsequent virtual texture overlay.

[0011] In terms of occlusion detection, the data processing module counts the number of pixels whose grayscale difference between the current image and the background reference image in the region of interest exceeds the noise tolerance threshold, and calculates the proportion of this number to the total number of pixels as the occlusion judgment index. When this index exceeds the preset occlusion trigger threshold, it is judged as an occlusion state and the start time is marked.

[0012] During the dynamic tracking phase, to extract the rotation information of the knob from hand movements, the data processing module constructs a dense optical flow field within the region of interest using the current and previous frames. Since the glove surface has wrinkles and lacks rigidity, the system constructs a rigid body rotation constraint model based on the center of the target knob, calculating the residual between the observed optical flow and the model's predicted value. By iteratively solving using a robust statistical method, the optimal value that minimizes the weighted sum of squared residuals is determined and used as the instantaneous visual angular velocity. Subsequently, this angular velocity is integrated in discrete time to obtain the cumulative visual rotation angle. Based on this angle, the system performs a two-dimensional rotation transformation on the aforementioned standard circular plane texture and uses the inverse of the homography transformation matrix to map it back to the perspective plane of the current viewpoint, generating a rendered texture which is then overlaid and displayed in the visual perception module, thus achieving a visual perspective effect.

[0013] To ensure the timing synchronization of visual data and physical signals, the signal acquisition module periodically performs clock synchronization detection with the data processing module, calculates the network round-trip time to estimate the one-way delay, and corrects the physical generation time when uploading signals. For discrete step signals, the system adopts a zero-order hold strategy to reconstruct them into continuous functions.

[0014] In the homomorphism verification stage, the system executes differentiated verification logic based on the comparison results between the signal sampling frequency and the mode switching threshold: when the sampling frequency is high, high-frequency dynamic correlation verification is performed, that is, extracting the visual instantaneous angular velocity sequence over a period of time, and obtaining the signal change rate sequence by differentiating the running state signal, calculating the statistical correlation coefficient between the two sequences, and verifying the consistency of the operation rhythm; when the sampling frequency is low, low-frequency total consistency verification is performed, that is, comparing the total amplitude of the visual cumulative rotation angle with the total change amplitude of the running state signal, and verifying whether the mapping deviation between the two is within the total tolerance threshold; in addition, the system also includes a data platform for collecting samples that have passed the verification, calculating the observation slope, and updating the mapping coefficients in the low-frequency verification using exponentially weighted moving average logic to achieve adaptive calibration of parameters.

[0015] A second aspect of the present invention provides a method for recognizing and analyzing the status of a rotary knob based on computer vision, the method comprising the following steps:

[0016] First, the system collects environmental image data and inertial measurement data containing the target knob through the visual perception module, and determines the spatial pose matrix based on the pose tracking algorithm, thereby locking the region of interest of the target knob.

[0017] Secondly, the region of interest is monitored in real time. Once occlusion is detected, the anti-occlusion tracking logic is immediately activated. This step includes constructing a dense optical flow field within the region of interest, using a rigid body rotation constraint model and robust statistical methods to calculate the visual instantaneous angular velocity of the occluding object from the optical flow field, and calculating the visual cumulative rotation angle through integration.

[0018] Meanwhile, the signal acquisition module acquires the physical device operating status signal corresponding to the target knob, and performs time alignment and numerical normalization processing on the signal.

[0019] Subsequently, the corresponding verification mode is selected according to the sampling frequency of the operating status signal. If it is a high-frequency signal, the waveform trend correlation between the visual instantaneous angular velocity and the signal change rate is compared; if it is a low-frequency signal, the amplitude consistency between the visual cumulative rotation angle and the total change of the signal is compared.

[0020] Finally, a status flag is generated based on the homomorphism verification result, and the corresponding feedback is output through the visual perception module to prompt the operator whether the operation is effective and whether the equipment response is normal.

[0021] This invention provides a computer vision-based system and method for recognizing and analyzing the status of rotary knobs. It offers the following advantages:

[0022] 1. This invention constructs a circular frame buffer queue and a historical frame backtracking mechanism, which can automatically obtain the clear knob texture before the occlusion as a reference when hand occlusion is detected. Combined with a dense optical flow solution method based on rigid body rotation constraints, the motion components that conform to rotation logic are extracted from the optical flow field of hand movement. This design effectively suppresses noise interference introduced by wrinkles and non-rigid deformation on the surface of industrial gloves, and realizes continuous and stable tracking of the knob rotation state in a strong occlusion environment. It solves the technical problem that traditional vision algorithms are prone to losing targets or drifting when operation is occluded.

[0023] 2. This invention establishes a dual-modal homomorphism verification mechanism based on signal frequency adaptation. It can automatically switch between high-frequency dynamic correlation verification and low-frequency total consistency verification modes according to the sampling characteristics of the physical equipment's operating status signal, and is supplemented by a time alignment strategy with network latency compensation. This mechanism ensures that the system is compatible with different types of industrial field equipment, and rigorously verifies the logical consistency between visually perceived operation actions and the actual response of physical equipment under a unified spatiotemporal reference. It effectively eliminates misjudgments caused by accidental touches, sensor failures, or network latency, and improves the reliability of industrial operation confirmation.

[0024] 3. This invention utilizes inverse perspective transformation and projection mapping technology to overlay and display the texture of a virtual knob that rotates according to the real-time calculated angle on the operating area, providing operators with intuitive perspective-based operation guidance and status feedback. At the same time, combined with the parameter adaptive update logic of the data platform, the system uses historical valid samples to dynamically calibrate the mapping coefficient between the visual angle and the physical signal, which can automatically adapt to the nonlinear deviation caused by mechanical wear or changes in idle distance due to long-term use of the equipment, ensuring the recognition accuracy of the system throughout its entire life cycle. Attached Figure Description

[0025] Figure 1 An overall architecture diagram of an AR-assisted control system provided for embodiments of the present invention;

[0026] Figure 2 Main flowchart of the AR-assisted control method provided in the embodiments of the present invention;

[0027] Figure 3 This is a schematic diagram comparing the tracking stability under occlusion conditions according to the present invention;

[0028] Figure 4 This is a schematic diagram of the dual-modal verification waveform consistency analysis of the present invention.

[0029] Among them, 100 is the visual perception module; 200 is the data processing module; 300 is the signal acquisition module; and 400 is the data platform. Detailed Implementation

[0030] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Please see the appendix Figure 1 This invention provides a computer vision-based operation column knob status recognition and analysis system, which mainly consists of a visual perception module 100, a data processing module 200, a signal acquisition module 300, and a data platform 400.

[0032] The visual perception module 100 is an industrial-grade smart glasses device worn on the operator's head, integrating a binocular depth camera module, an inertial measurement unit, and an optical waveguide display module. The visual perception module 100 is used to collect first-view video streams (environmental image data) and head posture data (inertial measurement data) of the operation site in real time. The visual perception module 100 uses a pose tracking algorithm (such as visual inertial SLAM) to calculate its spatial pose matrix relative to the world coordinate system in real time. At the same time, the visual perception module 100 is also used to receive feedback instructions from the data processing module 200 and overlay virtual image information in the operator's field of view.

[0033] The data processing module 200 is the core computing node of the system. It communicates with the visual perception module 100 and the signal acquisition module 300. The data processing module 200 is used to run region of interest locking, anti-occlusion texture processing, dense optical flow calculation and dual-modal verification logic.

[0034] The signal acquisition module 300 is connected to the physical device (process control system) corresponding to the target knob via hardwiring or industrial Ethernet. The signal acquisition module 300 is used to read the operating status signal (including analog signal or digital status signal) of the physical device at a preset frequency, and to perform numerical normalization and time axis alignment on the signal.

[0035] The data platform 400 is used to store equipment ledger information, three-dimensional digital twin models of knobs, and historical operation records, and provides the data processing module 200 with initial equipment baseline data and mapping coefficients.

[0036] Please see the appendix Figure 2 This method relies on the above system architecture for execution, and the specific workflow and algorithm details are as follows:

[0037] First, after entering the operating area, the visual perception module 100 activates the spatial perception function. Based on the spatial pose matrix determined by the visual perception module 100, the data processing module 200 locks the region of interest of the target knob in the environmental image data. Specifically, the data processing module 200 obtains the three-dimensional coordinates of the target knob in the world coordinate system, uses the pinhole camera projection model, and combines the spatial pose matrix at the current moment with the preset camera intrinsic parameter matrix to convert the three-dimensional coordinates into two-dimensional pixel coordinates. Then, based on the physical radius and depth factor of the target knob, the projection radius is estimated, and a rectangular area with the geometric center located at the two-dimensional pixel coordinates and the side length is the projection radius multiplied by a specific multiple (such as 1.5 to 2.5 times) is constructed as the region of interest.

[0038] During this process, the data processing module 200 constructs a circular frame buffer queue in the background that follows the first-in-first-out principle. This queue does not simply store video, but rather binds and stores image frames and the spatial pose matrix corresponding to the acquisition time. This process runs continuously in the background.

[0039] When the operator extends their gloved hand to operate the knob, the data processing module 200 monitors the pixel change rate of the region of interest in real time. The specific logic is as follows: the data processing module 200 counts the number of pixels whose grayscale difference between the current region of interest image and the maintained background reference image exceeds the noise tolerance threshold, and calculates the proportion of the number of such pixels to the total number of pixels in the region of interest to obtain the occlusion judgment index. When the occlusion judgment index exceeds the preset occlusion trigger threshold, the data processing module 200 determines that occlusion has occurred, marks the occlusion start time, and stops updating the background reference image.

[0040] Once occlusion is detected, the data processing module 200 initiates anti-occlusion tracking logic. First, the data processing module 200 retrieves historical image frames and historical spatial pose matrices from the circular frame buffer queue according to the preset safe backtracking interval. Using the historical spatial pose matrix and the pre-stored normal vector of the target knob, the data processing module 200 constructs a homography transformation matrix and performs inverse perspective transformation on the knob area in the historical image frame to generate a standard circular plane texture.

[0041] Subsequently, the dynamic operation monitoring phase begins. The data processing module 200 constructs a dense optical flow field within the region of interest using the current occlusion image frame and the previous frame. To separate the knob motion from the hand movement, the data processing module 200 constructs a rigid body rotation constraint model based on the center of the target knob. It calculates the residual between the observed optical flow in the dense optical flow field and the predicted value of the rigid body rotation constraint model, and uses a robust statistical method to iteratively solve the problem to determine the optimal value that minimizes the weighted sum of squared residuals, which is taken as the visual instantaneous angular velocity of the occluded object. Then, the data processing module 200 performs discrete-time integration on the visual instantaneous angular velocity to calculate the visual cumulative rotation angle from the start of the occlusion to the current moment.

[0042] Meanwhile, the data processing module 200 performs a two-dimensional rotation transformation on the above-mentioned standard circular plane texture according to the visual cumulative rotation angle, and uses the inverse matrix of the homography transformation matrix to map the rotated texture back to the image perspective plane at the current moment to generate a rendered texture. The visual perception module 100 receives the rendered texture and superimposes it on the region of interest in the physical space to achieve a visual perspective effect.

[0043] While performing visual computation, the signal acquisition module 300 continuously acquires operating status signals. To ensure timing synchronization, the signal acquisition module 300 periodically performs clock synchronization detection with the data processing module 200 to calculate the network round-trip time and estimate the one-way delay. When uploading data, the signal acquisition module 300 marks the physical generation time of the operating status signal as the receiving time minus the one-way delay. For discrete step signals (such as switching signals), the signal acquisition module 300 uses a zero-order hold strategy to reconstruct them into a continuous signal function.

[0044] After receiving the signal, the data processing module 200 performs a homomorphism check. First, the data processing module 200 compares the sampling frequency of the operating status signal with the preset mode switching threshold.

[0045] If the sampling frequency is greater than or equal to the mode switching threshold, perform a high-frequency dynamic correlation check: extract the visual instantaneous angular velocity sequence, and differentiate the operating state signal to obtain the signal change rate sequence, then calculate the statistical correlation coefficient between the two sequences. If the statistical correlation coefficient is greater than the correlation threshold, the check is considered passed.

[0046] If the sampling frequency is less than the mode switching threshold, perform a low-frequency total consistency check: calculate the total amplitude of the visual cumulative rotation angle and the total change amplitude of the running status signal, and verify whether the mapping deviation between the two is less than the total tolerance threshold based on the preset mapping coefficient. If it is true, the check is deemed to have passed.

[0047] In addition, the system also includes a data platform 400, which is used to collect operation samples that have passed the verification and calculate the observation slope of a single operation. The data platform 400 adopts the logic of exponential weighted moving average and updates the mapping coefficients used in the low-frequency total consistency verification according to the observation slope to achieve parameter adaptation.

[0048] Finally, the data processing module 200 generates a status flag based on the verification result and sends it to the visual perception module 100. The visual perception module 100 triggers feedback based on the status flag: if the verification passes, it displays a green locking aperture or triggers high-frequency vibration; if the verification fails, it displays a red warning icon or triggers low-frequency pulse vibration.

[0049] Spatial perception and target locking logic of visual perception module 100:

[0050] The visual perception module 100 (which can be embodied as an augmented reality acquisition terminal in this embodiment) serves as the perception front end of the system and is mainly responsible for the digital mapping of the physical environment and the spatial alignment of the human-computer interaction interface. Through a positioning scheme that combines vision and inertia, it achieves precise alignment between virtual information and physical knobs in a dynamic industrial environment.

[0051] S110, Multi-sensor Time Base Alignment and Data Cleaning:

[0052] Because the visual data acquired by the visual perception module 100 and the inertial data have different sampling frequencies, the data processing module 200 first performs hardware-level time synchronization. It uses a unified clock source to mark the exposure center time of the image and the sampling time of the inertial measurement unit (IMU) data, aligning them to the same time axis. Based on this, the data processing module 200 uses pre-calibrated camera intrinsic parameters and distortion coefficients to perform distortion correction on the acquired raw image; simultaneously, it performs zero-bias estimation and gravity vector alignment on the raw inertial measurement data to obtain calibrated motion measurement values.

[0053] S120, a six-DOF pose tracking system based on visual inertial odometry:

[0054] The data processing module 200 runs a visual-inertial SLAM algorithm (pose tracking algorithm), using the feature point method or direct method, combined with motion priors calculated by IMU integration, to track environmental features between consecutive image frames. Through nonlinear optimization methods (such as bundle adjustment), the data processing module 200 calculates the current moment in real time. Visual perception module 100 relative to the world coordinate system Spatial pose matrix The spatial pose matrix includes a rotation matrix and a translation vector, which respectively represent the orientation and position of the visual perception module 100, and serve as the spatial coordinate transformation reference for all subsequent virtual texture overlays.

[0055] S130, Projection mapping and region of interest locking of the target knob:

[0056] The system obtains the target knob's coordinates in the world coordinate system by identifying the device's unique identifier. Pre-stored 3D coordinates The data processing module 200 utilizes a pinhole camera projection model or an equivalent projection mapping algorithm, combined with the spatial pose matrix at the current moment. and preset camera intrinsic parameter matrix , three-dimensional coordinates Convert to two-dimensional pixel coordinates in the current video frame .

[0057] After calculating the pixel coordinates of the knob's center, the data processing module 200 estimates the knob's projection radius in the image based on the target knob's physical radius and depth factor. To cover potential occlusion caused by operator hand intervention, the data processing module 200 constructs an expanded rectangular region of interest (ROI). The geometric center is set to the length of the projected radius. Multiples (i.e., specific multiples) To cover the safety factor, the value is usually between 1.5 and 2.5 to ensure that the hand movement falls completely within the monitoring area. This ROI and its internal data stream will serve as the core input for subsequent occlusion detection.

[0058] Anti-occlusion texture acquisition logic based on buffer backtracking:

[0059] To address the issue of the knob texture being invisible due to hand obstruction during operation, the data processing module 200 employs a time-sliding window-based delayed acquisition strategy.

[0060] S210, Construct a pose-binding-based circular frame buffer queue:

[0061] The data processing module 200 contains a circular frame buffer queue that follows the first-in, first-out (FIFO) principle. This queue synchronously stores visual and spatial data. Each data unit in the queue is bound to store: an image frame acquired at a certain moment and the corresponding spatial pose matrix at that moment. The time span of the buffer queue is usually set to 2 to 5 seconds, which is sufficient to cover the action cycle from the operator's intention to the actual contact with the knob.

[0062] S220, Occlusion detection based on the pixel change rate of the region of interest:

[0063] The data processing module 200 monitors the ROI of the target knob in real time. In order to detect hand intervention with extremely low latency, the system adopts background difference logic. The data processing module 200 counts the number of pixels whose grayscale difference between the current ROI image and the maintained background reference image exceeds the noise tolerance threshold. In addition, the noise tolerance threshold is set to 15 to 30 grayscale levels to filter pixel jitter caused by slight fluctuations in ambient light.

[0064] Then, the proportion of this number to the total number of pixels in the ROI is calculated to obtain the occlusion determination index. .

[0065] when Exceeding the preset occlusion trigger threshold When the occlusion rate is typically 30% to 50%, the data processing module 200 determines that occlusion has occurred and marks the current time as the occlusion start time. And pause the update of the background baseline map.

[0066] S230, Historical reference frame retrospective retrieval and texture orthorectification:

[0067] In determining Then, the data processing module 200 calculates the backtracking time earlier than the occlusion occurred, based on a preset safety backtracking interval (e.g., 300 milliseconds to 800 milliseconds). The backtracking logic aims to skip the partially obscured phase when the hand is near and locate the fully clear historical moment.

[0068] Data processing module 200 retrieves the closest frame from the circular frame buffer queue. The data processing module 200 constructs a homography transformation matrix using the historical image frames and their corresponding historical spatial pose matrices, along with the pre-stored normal vector of the target knob. This matrix performs inverse perspective transformation (orthorectification) on the knob region in the historical image frames, generating a standard circular planar texture to eliminate parallax. This standard texture is the base material for achieving the perspective effect later.

[0069] Dynamic visualization guidance logic based on global optical flow:

[0070] Given that industrial gloves lack texture features and exhibit non-rigid deformation, the system adopts a dense optical flow solution based on global pixel motion constraints.

[0071] S310, Construct a dense optical flow field within the region of interest:

[0072] Data processing module 200 at the current time Using the occluded image frame and the previous frame as input, dense optical flow calculation is performed on the ROI. The system calculates the instantaneous motion vector of each pixel in the ROI and constructs a dense optical flow field covering the entire occluded area of ​​the hand. At this time, the optical flow field is mixed with noise caused by hand rotation, translation and glove wrinkles.

[0073] S320, Rotational angular velocity calculation based on robust estimator:

[0074] In order to extract the true angular velocity representing the knob rotation from the high-noise optical flow field, the data processing module 200 constructs a rigid body rotation constraint model using the knob center determined in S130. This rigid body rotation constraint model is based on the kinematic principle: in pure rotational motion, the linear velocity vector of any pixel should be perpendicular to the radial vector from that point to the rotation center, and the magnitude of the linear velocity is proportional to the radial distance.

[0075] The data processing module 200 calculates the residual between the observed optical flow and the predicted value of the rigid body rotation constraint model in the dense optical flow field, and uses a robust statistical method for iterative solution. The optimal value is determined as the visual instantaneous angular velocity by minimizing the weighted sum of squared residuals. The Huber threshold parameter of this method is typically determined based on the median absolute deviation of the residual distribution, thereby effectively eliminating abnormal motion vectors caused by glove wrinkles within a 95% confidence interval.

[0076] After obtaining the instantaneous visual angular velocity, the data processing module 200 performs discrete-time integration on it to calculate the angular velocity from the start of the occlusion. Up to the current moment visual cumulative rotation angle .

[0077] S330, virtual texture rotation transformation and inverse perspective overlay:

[0078] Data processing module 200 calls the standard circular plane texture generated in S230. Based on the cumulative rotation angle of vision The system for Perform a two-dimensional rotational transformation.

[0079] Subsequently, using the inverse of the homography transformation matrix in S230, the rotated texture is remapped back to the current image perspective plane to generate the final rendered texture. The visual perception module 100 receives the rendered texture and overlays it within the ROI in the physical space, so that the operator can visually observe the virtual knob texture overlaid on the obstruction and rotate synchronously with the hand movements.

[0080] Signal acquisition and timing synchronization mechanism of signal acquisition module 300:

[0081] S410, Acquisition and Numerical Normalization of Multi-Protocol Heterogeneous Signals:

[0082] The signal acquisition module 300 (which in this embodiment can be represented as a gateway connecting to the physical device) periodically reads the operating status signal of the physical device corresponding to the target knob through various industrial communication protocols. For the differences in measurement ranges between different devices, the signal acquisition module 300 obtains the upper and lower limits of the measurement range and uses a linear mapping relationship to convert the original value into a standardized signal value between 0 and 100. This involves numerical normalization. The calculation formula is: .

[0083] S420, signal refresh frequency estimation based on sliding window:

[0084] The signal acquisition module 300 maintains a length of A first-in-first-out timestamp queue (usually 10 to 50) records the most recent timestamps. The timing of the signal change; by calculating the average interval between adjacent timestamps, the sampling frequency of the real-time signal is estimated. If the signal remains unchanged for an extended period, the interval of the heartbeat detection packets will be used instead of the calculation.

[0085] S430, Timestamp Reconstruction and Interpolation Based on Network Delay Detection:

[0086] To compensate for network latency, the signal acquisition module 300 periodically performs clock synchronization detection with the data processing module 200 to calculate the network round-trip time and estimate the one-way latency. When uploading data, the signal acquisition module 300 marks the physical generation time of the running status signal as the receiving time minus [the time of data transmission]. This achieves timeline alignment.

[0087] Furthermore, for discrete step signals, the signal acquisition module 300 employs a zero-order hold strategy to reconstruct them into a continuous signal function. This means that the signal value remains unchanged between two sampling points to avoid false transitions introduced by linear interpolation.

[0088] The dual-modal homomorphism verification logic of the data processing module 200:

[0089] The system constructs a dual-mode verification mechanism based on signal frequency adaptation to be compatible with high-speed analog signals and low-speed digital signals.

[0090] S510, verification mode selection based on signal refresh frequency:

[0091] The data processing module 200 receives the operating status signal and sets the sampling frequency of the signal. With preset mode switching threshold Compare; Based on the upper limit of human hand movement frequency and the sampling theorem, it is usually set to 3Hz to 5Hz.

[0092] like This indicates that the signal has high time resolution, and enters mode A: high-frequency dynamic correlation check;

[0093] like This indicates that the signal is low frequency or step type, and enters mode B: low frequency total consistency check.

[0094] S520, Execution Mode A: High-Frequency Dynamic Correlation Verification:

[0095] In this mode, the system compares the similarity of waveform trends. The data processing module 200 extracts the visual instantaneous angular velocity sequence within the operation window. (from continuous time) (composed of), and the operating status signals Differential analysis is performed to obtain the signal rate of change sequence. This will unify the physical dimensions.

[0096] Data processing module 200 calculates the statistical correlation coefficient between two sequences. (For example, using the Normalized Cross-Correlation (NCC) algorithm), this coefficient characterizes the degree of synchronization between the visually perceived rhythm of movement and the rhythm of changes in signal feedback. If Greater than the set correlation threshold If the value is typically between 0.75 and 0.90, the verification is considered successful.

[0097] S530, Execution Mode B: Low-Frequency Total Consistency Verification:

[0098] In this mode, the system compares the logical equivalence of physical quantities. The data processing module 200 calculates the total amplitude of the visually accumulated rotation angle. The total change amplitude of the operating status signal before and after the operation .

[0099] Using mapping coefficients (Characterizing the signal change corresponding to a unit angle), data processing module 200 verifies mapping deviation. Whether the (i.e., mapping deviation) is less than the total tolerance threshold .

[0100] The value is typically set to 5% to 10% of full scale to accommodate mechanical backlash and estimation errors. If the inequality holds true, the verification is considered successful.

[0101] S540, Verification result output and status feedback:

[0102] The data processing module 200 generates a status flag based on the verification result. The signal is then sent to the visual perception module 100. If the signal is true, a confirmation command is sent; if the signal is false, the system further determines whether the error is due to a wrong direction or a transmission failure based on the sign of the correlation coefficient or the magnitude of the amplitude deviation, and sends a corresponding warning.

[0103] Interactive feedback of visual perception module 100 and support of data platform 400:

[0104] S610, virtual overlay rendering based on transparency blending:

[0105] The visual perception module 100 utilizes alpha blending technology to integrate the virtual knob texture into the physical environment video stream. The system weights and superimposes virtual pixels with real background pixels based on a preset transparency factor (typically 0.6 to 0.8). Simultaneously, it adjusts the received status flags... Draw a green indicator ring (pass) or a red warning box (fail) on top of the image.

[0106] S620, adaptive calibration of mapping coefficients based on historical valid samples:

[0107] The system also includes a data platform 400, which is used to maintain the mapping coefficients using an online parameter identification mechanism. The accuracy of the data platform is ensured. Operation samples that pass the verification are collected and the observation slope for each operation is calculated. An exponentially weighted moving average logic is used to update the mapping coefficients used in the low-frequency total consistency verification based on the observation slope. The new coefficients are the weighted sum of the old coefficients and the currently observed slope. The learning rate factor is typically set to 0.05 to 0.1 to allow the parameters to gradually adapt to mechanical wear.

[0108] S630, haptic feedback based on frequency domain mapping:

[0109] The visual perception module 100 maps the verification result to a tactile signal. When the verification passes, it drives the linear motor to generate a short-term high-frequency vibration (e.g., 150Hz) to simulate a mechanical locking sensation; when the verification fails, it generates a continuous low-frequency pulse vibration (e.g., 40Hz) to indicate an operation error.

[0110] Specific application example: AR-assisted control of high-pressure steam pipeline regulating valve

[0111] Scene background:

[0112] The central control room of a chemical plant needs to perform both remote monitoring and on-site operation verification of the high-pressure steam pipeline regulating valve with the field number V-305. The on-site operator wears an AR-Glass-Pro augmented reality terminal (corresponding to the visual perception module 100 in this solution) and thickened non-slip work gloves while performing the operation.

[0113] Detailed Explanation of Implementation Process:

[0114] Step 1: Initialization and Locking (corresponding logic S110-S130)

[0115] When the operator enters the area where the V-305 valve is located, the camera of the visual perception module 100 automatically recognizes the QR code next to the valve and retrieves the absolute spatial coordinates of the valve in the factory's digital twin model from the server.

[0116] The system uses the pose tracking algorithm described in S120 to lock the valve position and generates a rectangular region of interest (ROI) with a side length of 20cm in the video frame based on its physical diameter of 10cm. At this time, the current valve opening reading of 45% is accurately superimposed and displayed in the AR image, realizing the initial alignment between virtual information and physical device.

[0117] Step 2: Obtaining anti-occlusion textures (corresponding to logic S210-S230)

[0118] The operator extends his hand, which is wearing a gray work gloved hand, to prepare to rotate the valve. When the hand enters the ROI and covers more than 40% of the valve area, the computing unit detects a surge in pixel change rate and determines that occlusion has occurred.

[0119] The system immediately triggers the S230 buffer backtracking mechanism, pauses background updates, and retrieves a clear valve image from the circular buffer 0.5 seconds ago (i.e., before the hand touched the valve). The system performs orthorectification on this historical frame to generate a standard circular valve texture that eliminates parallax, thus preparing the material for subsequent perspective display.

[0120] Step 3: Dynamic bootstrapping and optical flow tracing verification (corresponding logic S310-S330)

[0121] The operator begins to rotate the valve clockwise. Due to the lack of texture features such as corners on the surface of the work gloves and the presence of wrinkles and deformations, it is easy to lose the target if traditional algorithms are used. This system uses a dense optical flow algorithm combined with the robust estimator of S320 to calculate hand movements in real time.

[0122] Combination Figure 3 The experimental data shown in the comparison of tracking stability under occlusion conditions analyzes the effectiveness of this operation:

[0123] exist Figure 3 On the horizontal axis representing the operation timeline, the gray background area marks the time period when the operator's hand completely covers the knob.

[0124] During this period, if the traditional feature point method is used (as shown by the black dotted line in the figure), the angle tracking error will diverge sharply due to the occlusion of the feature points, causing the system to be unable to maintain tracking.

[0125] The proposed solution (shown by the black solid line in the figure) uses optical flow field calculation and historical texture backtracking. Although the error fluctuates slightly, it remains at a low level, proving the effectiveness of the algorithm under strong occlusion.

[0126] Based on this stable tracking data, the AR glasses project a clear, semi-transparent valve texture onto the operator's hand, allowing them to see, as if through a mirror, the virtual knob rotating in sync with their hand movements.

[0127] Step 4: Dual-modal signal verification and analysis (corresponding logic S410-S520)

[0128] At the same time, the background data gateway collects the valve opening signal from the PLC at a high speed of 50ms (20Hz). The system determines that the sampling frequency (20Hz) is greater than the preset threshold (3Hz) and automatically enters mode A: high-frequency dynamic correlation verification.

[0129] Combination Figure 4 The data comparison process shown in (dual-modal waveform consistency analysis) is as follows:

[0130] The system maps the two sets of collected data to the 0-1 normalized amplitude range shown on the vertical axis.

[0131] The black solid line in the diagram represents the speed and rhythm of the movement calculated by the visual perception module 100 (visual instantaneous angular velocity).

[0132] The black dotted lines in the diagram represent the actual signal change rate (sensor feedback rate of change) received by the industrial control system after time alignment compensation.

[0133] On the sampling time axis (horizontal axis), the system monitors in real time that the peaks and troughs of the two curves are highly aligned. This high degree of overlap in waveform rhythm (correlation coefficient reaches 0.92) indicates that the manual operation seen by AR and the signal feedback received by PLC are synchronous and related in physical logic, thus determining that the operation is valid and eliminating the risk of sensor failure or accidental touch.

[0134] Step 5: Feedback and parameter adaptation (corresponding logic S540-S630)

[0135] After successful verification, the AR glasses render a green dynamic halo around the valve, while the linear motor at the handle emits short pulse vibrations at a frequency of 150Hz to simulate the tactile sensation of mechanical gear meshing, confirming to the operator that the operation was successful.

[0136] After the operation was completed, the system background analysis found that the ratio of the physical rotation angle to the total amplitude of the signal feedback was slightly off (mainly due to the mechanical wear clearance of the valve). Therefore, the S620 algorithm was used to automatically fine-tune the mapping coefficient. This ensures higher accuracy in the next operation.

[0137] Experimental verification and effect comparison:

[0138] To verify the effectiveness of this solution in complex industrial environments, we conducted comparative tests on a simulation experimental platform.

[0139] Experimental setup:

[0140] Control group (traditional approach): A visual tracking algorithm based on KLT (Kanade-Lucas-Tomasi) sparse feature points is used, without historical frame backtracking mechanism and tactile feedback.

[0141] Experimental group (this scheme): adopts a tracking algorithm based on historical buffer backtracking + dense optical flow, and enables dual-modal verification.

[0142] Test object: an industrial knob with a diameter of 10cm and severe surface wear.

[0143] Interference conditions: Operators wear untextured gray rubber gloves, and ambient light intensity fluctuates between 100 Lux and 500 Lux.

[0144] Analysis of experimental results:

[0145] Comparison of tracking stability under occlusion: The experiment recorded the process of the operator rotating the knob with his hand completely covering it (for 3 seconds).

[0146] Traditional solution: At the moment the hand touches the knob (about 0.5 seconds), the tracking error increases significantly because the feature points on the knob surface are blocked and the glove lacks corner points. Tracking is lost at 0.8 seconds, causing the virtual overlay image to freeze or drift.

[0147] This solution: After detecting occlusion, the optical flow field is quickly used to calculate the hand movement. Although there is deformation noise on the glove surface, the angle tracking error is always controlled within 5 degrees through the correction of the robust estimator, and no tracking loss occurs.

[0148] Accuracy comparison of operation intent verification: We injected simulated network latency (200ms) and signal noise into the system.

[0149] Traditional approach: Simply compares endpoint thresholds (i.e., only looks at the angle and value after rotation). Due to timestamp misalignment caused by network latency, the false alarm rate reaches 15%.

[0150] This solution employs waveform correlation verification combined with timestamp interpolation reconstruction. Even with delays, the rhythm of the visual velocity waveform still matches the signal change waveform, reducing the false alarm rate to below 1%.

Claims

1. A computer vision-based operating column knob status recognition and analysis system, characterized in that, include: The visual perception module (100) is used to collect environmental image data and inertial measurement data containing the target knob, determine the spatial pose matrix using a pose tracking algorithm, and receive rendering data to perform the superposition and display of virtual information in physical space. The signal acquisition module (300) is used to acquire the operating status signal of the physical device corresponding to the target knob and perform normalization processing; A data processing module (200), connected to the visual perception module (100) and the signal acquisition module (300), is used for: Based on the spatial pose matrix, the region of interest of the target knob is located in the environmental image data; Monitor the pixel change rate of the region of interest, start anti-occlusion tracking when occlusion is detected, calculate the visual instantaneous angular velocity based on the image frame at the current moment and integrate to determine the visual cumulative rotation angle; Select the verification mode according to the sampling frequency of the running status signal, perform isomorphism verification between the visual instantaneous angular velocity or the visual cumulative rotation angle and the running status signal, generate a status flag according to the verification result and send it to the visual perception module (100) to trigger the corresponding virtual visual feedback or operation prompt feedback.

2. The computer vision-based rotary knob status recognition and analysis system according to claim 1, characterized in that, The data processing module (200) locks the region of interest of the target knob in the following way: The three-dimensional coordinates of the target knob in the world coordinate system are obtained. Using the pinhole camera projection model, combined with the spatial pose matrix at the current moment and the preset camera intrinsic parameter matrix, the three-dimensional coordinates are converted into two-dimensional pixel coordinates. The projection radius is estimated based on the physical radius and depth factor of the target knob, and a rectangular region with its geometric center located at the two-dimensional pixel coordinates and a side length equal to the projection radius multiplied by a preset multiple is constructed as the region of interest.

3. The computer vision-based rotary knob status recognition and analysis system according to claim 1, characterized in that, The data processing module (200) has a circular frame buffer queue that follows the first-in-first-out principle; The circular frame buffer queue is used to bind and store image frames and the spatial pose matrix corresponding to the acquisition time. After determining that occlusion has occurred, the data processing module (200) stops updating the background reference map of the pre-maintained region of interest and retrieves historical image frames and historical spatial pose matrices from the circular frame buffer queue according to the preset safety backtracking interval. Using the historical spatial pose matrix and the pre-stored normal vector of the target knob, a homography transformation matrix is ​​constructed, and an inverse perspective transformation is performed on the knob area in the historical image frame to generate a standard circular planar texture.

4. The computer vision-based rotary knob status recognition and analysis system according to claim 1, characterized in that, The logic for occlusion detection performed by the data processing module (200) is as follows: The number of pixels whose grayscale difference exceeds the noise tolerance threshold between the image of the currently defined region of interest and the maintained background reference image is counted. The proportion of the number of pixels to the total number of pixels in the region of interest is calculated to obtain the occlusion determination index; The occlusion determination index is compared with the preset occlusion trigger threshold. If the occlusion determination index exceeds the occlusion trigger threshold, occlusion intervention is determined and the occlusion start time is marked.

5. The computer vision-based rotary knob status recognition and analysis system according to claim 1, characterized in that, The data processing module (200) calculates the visual instantaneous angular velocity of the occluded object in the following way: A dense optical flow field within the region of interest is constructed using the current image frame and the previous image frame. A rigid body rotation constraint model is constructed based on the center of the target knob, and the residual between the observed optical flow in the dense optical flow field and the predicted value of the rigid body rotation constraint model is calculated. The robust statistical method is used to iteratively solve the problem and determine the optimal value that minimizes the weighted sum of squared residuals as the visual instantaneous angular velocity of the occluded object; The visual instantaneous angular velocity is integrated in discrete time to calculate the visual cumulative rotation angle from the moment the occlusion intervention is detected to the current moment.

6. The computer vision-based rotary knob status recognition and analysis system according to claim 3, characterized in that, The data processing module (200) is also used for: The standard circular plane texture is subjected to a two-dimensional rotation transformation based on the visual cumulative rotation angle. Using the inverse of the homography transformation matrix, the rotated texture is mapped back to the current image perspective plane to generate the rendered texture; The visual perception module (100) receives the rendered texture and overlays the rendered texture on the region of interest in the physical space.

7. The computer vision-based rotary knob status recognition and analysis system according to claim 1, characterized in that, The logic for time synchronization performed by the signal acquisition module (300) is as follows: Periodically perform clock synchronization detection with the data processing module (200), calculate the network round-trip time by the time difference between the sending and receiving of the detection packet, and determine half of the network round-trip time as the one-way delay; When uploading the running status signal, the physical generation time of the running status signal is marked as the receiving time minus the one-way delay; For discrete step signals, a zero-order hold strategy is used to reconstruct them into continuous signal functions.

8. The computer vision-based rotary knob status recognition and analysis system according to claim 1, characterized in that, The logic for the data processing module (200) to perform homomorphism verification is as follows: Compare the sampling frequency of the operating status signal with a preset mode switching threshold; If the sampling frequency is greater than or equal to the mode switching threshold, perform a high-frequency dynamic correlation check: extract the visual instantaneous angular velocity over a period of time, differentiate the running state signal to obtain the signal change rate sequence, calculate the statistical correlation coefficient between the two sequences, and if the statistical correlation coefficient is greater than the preset correlation threshold, the check is deemed to have passed. If the sampling frequency is less than the modal switching threshold, perform a low-frequency total consistency check: calculate the total amplitude of the visual cumulative rotation angle and the total change amplitude of the running status signal, and verify whether the mapping deviation between the two is less than the preset total tolerance threshold based on the preset mapping coefficient. If it is true, the check is deemed to have passed.

9. The computer vision-based rotary knob status recognition and analysis system according to claim 8, characterized in that, It also includes a data platform (400) that is communicatively connected to the data processing module (200); The data platform (400) is used for: The data processing module (200) collects operation samples that pass the verification and calculates the observation slope of a single operation; The mapping coefficients used in the low-frequency total consistency verification are updated according to the observed slope using an exponentially weighted moving average logic, and the updated mapping coefficients are fed back to the data processing module (200).

10. A computer vision-based method for recognizing and analyzing the state of a rotary knob, applied to the computer vision-based system for recognizing and analyzing the state of a rotary knob as described in any one of claims 1-9, characterized in that... Includes the following steps: The visual perception module (100) acquires environmental image data and inertial measurement data containing the target knob, and locks the region of interest of the target knob based on the pose tracking algorithm; The region of interest is monitored, and when occlusion is detected, anti-occlusion tracking logic is activated to construct a dense optical flow field and calculate the visual instantaneous angular velocity of the occluding object, as well as obtain the visual cumulative rotation angle through integration. The signal acquisition module (300) acquires the physical device operating status signal corresponding to the target knob, and performs time alignment and numerical normalization on the physical device operating status signal; The verification mode is selected according to the sampling frequency of the running status signal, and the visual cumulative rotation angle or the visual instantaneous angular velocity of the occluded object is compared with the running status signal for homomorphism verification. A status flag is generated based on the verification result, and the corresponding feedback is output.

Images