A robot vision pattern matching system in dynamic scenes

Abstract

This invention discloses a robot visual pattern matching system for dynamic scenes, specifically addressing the problems of unstable visual feature extraction, low matching accuracy, and lack of real-time motion coordination in dynamic environments. The system includes an environment perception module, a feature processing module, a motion compensation module, and a decision control module. It collects scene and robot motion data through multi-sensor fusion, constructs stable visual features using multi-scale feature extraction and anti-interference feature descriptors, compensates for image distortion caused by robot motion based on a dynamic distortion model, and finally generates closed-loop motion control commands based on the feature matching results. This application improves the accuracy, robustness, and real-time coordination capabilities of robot visual matching in dynamic scenes.

Description

Technical Field

[0001] This invention relates to the field of robot vision technology, and in particular to a robot vision pattern matching system for dynamic scenes. Background Technology

[0002] Robot vision technology, as a key branch of artificial intelligence and automation, has been widely applied in industrial inspection, autonomous navigation, and intelligent operations. Among these technologies, visual pattern matching is one of the core technologies for enabling robots to perceive their environment and manipulate targets. Its goal is to enable robots to accurately identify and locate specific targets through image processing and feature comparison.

[0003] In dynamic robotic applications, such as ash removal and descaling tasks inside incinerators, existing visual pattern matching systems face significant challenges. Current technologies typically rely on pre-defined static environment models for pattern recognition, making it difficult to adapt to image quality degradation caused by temperature changes, dust interference, and lighting fluctuations within the furnace. This leads to unstable feature extraction and decreased matching accuracy. Furthermore, the robot arm's extension process generates its own motion and vibration, further introducing image blurring and target displacement. Traditional matching algorithms lack robust compensation mechanisms for these dynamic distortions.

[0004] Furthermore, existing systems struggle to achieve real-time coordination between visual recognition and robotic arm extension operations when handling high-altitude tasks due to the lack of joint modeling of the three-dimensional spatial structure and the robot's motion state. In complex structures such as those inside furnaces twenty meters high, these issues can easily lead to positioning deviations or operational failures of the cleaning tools, severely impacting operational efficiency and safety. Therefore, there is an urgent need for a robotic system capable of adapting to dynamic scene changes and improving the stability and accuracy of visual pattern matching. Summary of the Invention

[0005] The purpose of this invention is to provide a robot vision pattern matching system for dynamic scenes, which solves the problems mentioned in the background art.

[0006] This invention is implemented as follows: a robot vision pattern matching system for dynamic scenes, comprising an environment perception module, a feature processing module, a motion compensation module, and a decision control module. The environment perception module collects visual data of the work scene and robot motion state data. The feature processing module, connected to the environment perception module, performs multi-scale feature extraction on the visual data and constructs anti-interference feature descriptors. The motion compensation module, connected to both the environment perception module and the feature processing module, establishes a dynamic distortion model based on the robot's motion state data to correct the feature space mapping relationship. The decision control module integrates the feature processing module and the motion compensation module, generating robot motion control commands based on the corrected feature matching results and achieving closed-loop coordination between visual recognition and robotic arm operation.

[0007] Furthermore, the environment perception module includes a binocular vision sensor, an inertial measurement unit (IMU), and a laser ranging unit. The binocular vision sensor synchronously acquires left and right view data of the scene at a rate of 30 frames per second. The IMU detects the robot's three-axis angular velocity and three-axis acceleration data in real time. The laser ranging unit acquires the distance information of the robot's end effector relative to the target point at a frequency of 1000 Hz. The environment perception module also includes a data preprocessing submodule, which performs grayscale conversion and histogram equalization on the raw visual data, and uses the fourth-order Runge-Kutta method to calculate the attitude of the IMU data.

[0008] Furthermore, the feature processing module adopts a hierarchical feature extraction architecture. The first-layer feature extraction network performs three downsampling operations on the input image. After each downsampling, a 3x3 convolutional kernel is used for feature mapping, and a modified linear unit is used as the activation function. The second-layer feature extraction network constructs a spatial pyramid structure based on the output of the previous layer, calculating the histogram of oriented gradients (HARGs) and local binary pattern features at four different scales. The feature processing module also includes a feature fusion unit, which concatenates the multi-scale feature vectors and inputs them into the fully connected layer, generating a robust feature descriptor with a dimension of 1024 through weight allocation.

[0009] Furthermore, the motion compensation module includes a pose estimation unit and a distortion correction unit. The pose estimation unit receives attitude data calculated by the inertial measurement unit and laser ranging data, fuses multi-source information through an extended Kalman filter, and outputs the six-DOF pose estimation value of the robot's end effector at a 50-millisecond cycle. The distortion correction unit constructs a dynamic distortion model based on the pose estimation value. This model represents the image changes caused by robot motion as an affine transformation matrix and performs spatial transformation on the feature map using a bilinear interpolation algorithm, ensuring that the feature descriptors maintain scale invariance and rotation invariance.

[0010] Furthermore, the decision control module includes a matching degree calculation unit and a motion planning unit. The matching degree calculation unit uses an improved Hamming distance metric to calculate the similarity between the anti-interference feature descriptor extracted from the current frame and the baseline features in the template library. When the similarity exceeds a preset threshold of 0.85, a successful match is determined. The motion planning unit calculates the target angles of each joint of the robotic arm based on the matching results, generates a smooth trajectory using a fifth-order polynomial interpolation algorithm, and implements closed-loop position control through a proportional-integral-derivative controller.

[0011] Furthermore, the system also includes an adaptive update module. This module continuously monitors the success rate and accuracy of feature matching. When the average similarity of 10 consecutive matches falls below 0.8, it automatically triggers a template feature update mechanism. This mechanism adds the currently successfully matched feature descriptors to the template library using a sliding window approach, while removing the five oldest template features to maintain a constant template library capacity of 50 feature descriptors.

[0012] Furthermore, the system adopts a multi-threaded parallel processing architecture. The visual data processing thread and the motion control thread run on independent processor cores, and the two threads exchange data through shared memory and use a mutex lock mechanism to ensure data synchronization. The system sets the priority of the visual processing thread higher than that of the motion control thread to ensure the real-time performance of visual perception in dynamic scenes.

[0013] Furthermore, the feature processing module also integrates an illumination robustness enhancement unit. This unit dynamically adjusts the gamma correction parameters by calculating the gray-level variance of local image regions, and automatically activates the auxiliary illumination system when the overall image brightness is detected to be below 50 lux, ensuring the stability of feature extraction under different illumination conditions.

[0014] Compared with existing technologies, the advantages and positive effects of this invention are as follows: This invention achieves comprehensive perception of dynamic scenes through multi-sensor data fusion in the environmental perception module; the multi-scale hierarchical architecture and anti-interference feature descriptor design of the feature processing module effectively improve feature stability under dust interference and light fluctuation conditions; the dynamic distortion model established by the motion compensation module can accurately compensate for image changes caused by the robot's own motion, significantly improving the accuracy of feature matching; the closed-loop collaborative mechanism of the decision control module ensures real-time linkage between visual recognition results and robotic arm operations, making it particularly suitable for complex high-altitude working environments; the adaptive update module enables the system to continuously optimize and maintain matching performance over a long period; and the multi-threaded parallel architecture ensures the system's real-time response capability in dynamic scenes. Through the synergistic effect of its various modules, the overall system fundamentally solves the challenges faced by existing technologies in dynamic scenes, possessing significant engineering application value.

[0015] Attached Figure Description Figure 1 This is a schematic diagram of the overall technical architecture of the robot vision pattern matching system in dynamic scenes proposed in this invention;

[0016] Figure 2 This is a system block diagram of the environment perception module of the robot vision pattern matching system in dynamic scenes proposed in this invention.

[0017] Figure labels: 1. Environment perception module; 2. Feature processing module; 3. Motion compensation module; 4. Decision control module; 5. Adaptive update module; 6. Multi-threaded parallel processing architecture.

[0018] Detailed Implementation Example 1

[0019] Please refer to the attached document. Figure 1 This embodiment details the specific technical implementation of a robot vision pattern matching system in dynamic scenarios. The system consists of an environment perception module, a feature processing module, a motion compensation module, a decision control module, an adaptive update module, and a multi-threaded parallel processing architecture. Each module interacts with shared memory via a data bus to ensure high-precision vision pattern matching and real-time motion control in dynamic work scenarios.

[0020] The environmental perception module is responsible for collecting visual information about the work environment and the robot's motion parameters. This module includes a binocular vision sensor, an inertial measurement unit (IMU), and a laser ranging unit. The binocular vision sensor uses two 1920x1080 resolution global shutter cameras to simultaneously acquire left and right view data at a rate of 30 frames per second, and transmits the raw images to the data preprocessing submodule via a gigabit Ethernet interface. The IMU integrates a three-axis MEMS gyroscope and a three-axis MEMS accelerometer, with an angular velocity range of ±2000 degrees per second, an acceleration range of ±16g, and a data output frequency of 400 Hz. The laser ranging unit, based on the time-of-flight principle, measures the distance between the robot's end effector and the target object surface at a frequency of 1000 Hz, with a ranging accuracy of ±1 mm. The data preprocessing submodule performs grayscale conversion and histogram equalization on the raw images acquired by the binocular vision sensor. Grayscale conversion uses a weighted average method, merging the red, green, and blue channels of the color image into a single-channel grayscale image with weights of 0.299, 0.587, and 0.114, respectively. Histogram equalization reconstructs the pixel distribution of the image using a cumulative distribution function, enhancing image contrast. For inertial measurement unit (IMU) data, the data preprocessing submodule uses the fourth-order Runge-Kutta method for attitude calculation, obtaining the robot's real-time pitch, roll, and yaw angles from integrated angular velocity and acceleration data, with a calculation period of 2.5 milliseconds.

[0021] The feature processing module receives the preprocessed visual data from the environment perception module and performs multi-scale feature extraction and descriptor generation. This module adopts a hierarchical feature extraction architecture, including a first-layer feature extraction network, a second-layer feature extraction network, and a feature fusion unit. The first-layer feature extraction network performs three downsampling operations on the input grayscale image. Each downsampling uses a max-pooling layer with a stride of 2, successively reducing the image size to half, a quarter, and an eighth of the original image. After each downsampling, a convolutional layer is applied with a 3x3 kernel size and 64, 128, and 256 channels respectively. The activation function is a modified linear unit, whose mathematical expression is: This function outputs zero for negative inputs and maintains linear growth for positive inputs, effectively mitigating the vanishing gradient problem. The second-layer feature extraction network constructs a spatial pyramid structure on the first-layer output feature map, calculating histogram of oriented gradients (HARQs) and local binary pattern features at four different scales. The HARQ feature divides the image at each scale into 8x8 cells, calculates the gradient orientation histogram within each cell, and concatenates the histograms of four adjacent cells into a block feature. The local binary pattern feature uses uniform pattern encoding, comparing each pixel with its eight neighboring pixels to generate a 58-dimensional feature vector. The feature fusion unit concatenates the multi-scale feature vectors along the channel dimension to generate a joint feature vector with a total dimension of 4096. This joint feature vector is then input into the fully connected layer, where the dimensionality is reduced to 1024 through the weight matrix, ultimately outputting a robust feature descriptor. The weight parameters of the fully connected layer are optimized during training using backpropagation, and the loss function is cross-entropy loss.

[0022] The motion compensation module performs dynamic distortion correction on the feature map output by the feature processing module based on the robot motion state data provided by the environmental perception module. This module includes a pose estimation unit and a distortion correction unit. The pose estimation unit receives attitude angle data calculated by the inertial measurement unit (IMU) and distance data from the laser ranging unit, fusing multi-source information through an extended Kalman filter (EPF). The EPF state vector contains the robot end effector's 3D position, 3D velocity, quaternion pose, and angular velocity deviation, totaling 13 dimensions; the observation vector contains laser ranging data and IMU acceleration data. The filter prediction step size is set to 50 milliseconds, and the nonlinear motion model is linearized using the Jacobian matrix to update the state covariance matrix. The distortion correction unit constructs a dynamic distortion model based on the six-DOF pose data output by the pose estimation unit. This model represents the image deformation caused by robot motion as an affine transformation matrix.

[0023]

[0024] Where x and y are the original feature map pixel coordinates, x′ and y′ are the transformed coordinates, a_{11} to a_{22} are rotation and scaling parameters, and t_x and t_y are translation parameters. The distortion correction unit performs spatial transformation on the feature map using a bilinear interpolation algorithm, calculating the pixel value at the new coordinates based on the weighted average of four adjacent pixels, thus ensuring that the feature descriptor maintains scale invariance and rotation invariance under different viewpoints.

[0025] The decision control module generates robot motion commands based on the feature matching results corrected by the motion compensation module. This module includes a matching degree calculation unit and a motion planning unit. The matching degree calculation unit uses an improved Hamming distance metric method, performing a bitwise XOR operation between the 1024-dimensional anti-interference feature descriptor extracted from the current frame and 50 sets of baseline feature descriptors stored in the template library. The number of different bits is counted and divided by the total number of bits to obtain the normalized distance. A successful match is determined when the similarity value corresponding to the minimum normalized distance exceeds a preset threshold of 0.85. The motion planning unit calculates the target angles of each joint of the robotic arm based on the successfully matched feature positions and generates a continuous, smooth trajectory using a fifth-order polynomial interpolation algorithm. The fifth-order polynomial expression is:

[0026] Where q(t) is the joint angle, t is the time, and a_0 to a_5 are coefficient matrices, solved by boundary conditions (angle, velocity, and acceleration between the starting point and the target point). The motion planning unit implements position closed-loop control through a proportional-integral-derivative controller, with the proportional coefficient set to 2.5, the integral coefficient set to 0.1, the derivative coefficient set to 0.05, and the control period set to 10 milliseconds.

[0027] The adaptive update module continuously monitors the system's matching performance. When the average similarity of 10 consecutive matches falls below 0.8, it automatically triggers a template feature update mechanism. This mechanism adds the currently successfully matched feature descriptors to the template library using a sliding window approach, while removing the five oldest template features to ensure the template library capacity remains constant at 50 sets of feature descriptors. The sliding window update strategy follows a first-in, first-out (FIFO) principle: new feature descriptors are stored at index 0 in the template library, existing feature descriptors are shifted sequentially to the next index, and the feature at index 49 is permanently deleted.

[0028] The system employs a multi-threaded parallel processing architecture, with the visual data processing thread and motion control thread running on two independent processor cores. The visual data processing thread is responsible for image acquisition, feature extraction, and matching calculations, with its priority set to real-time to ensure a processing latency of less than 33 milliseconds per frame. The motion control thread is responsible for trajectory planning and servo control, with its priority set to high. Data exchange between the two threads occurs through a shared memory area, which is divided into a feature descriptor buffer and a motion command buffer. Thread synchronization utilizes a mutex mechanism. When the visual data processing thread writes a new feature descriptor, it automatically locks the feature descriptor buffer, preventing the motion control thread from reading it; similarly, when the motion control thread generates a new motion command, it locks the motion command buffer. The mutex timeout is set to 5 milliseconds; if the timeout occurs, a thread priority escalation strategy is triggered.

[0029] The feature processing module also integrates an illumination robustness enhancement unit, which assesses illumination uniformity by calculating the gray-level variance of local image regions. When the overall image brightness is detected to be below 50 lux, an infrared auxiliary illumination system is automatically activated to compensate for insufficient ambient light. The illumination robustness enhancement unit dynamically adjusts the gamma correction parameter, whose value is mapped to a floating-point number between 1.0 and 2.5 based on the average gray-level value of the image. The calculation formula is as follows: Where γ is the gamma correction parameter and ī is the average gray value of the image. When ī is less than 50, γ takes a value greater than 2.0, which significantly improves details in dark areas; when ī is greater than 150, γ takes a value close to 1.0, which preserves the original appearance of the image.

[0030] The system's modules communicate via a high-speed serial bus. The data packet format includes a 32-bit header identifier, a 16-bit data length, a 1024-bit payload, and a 32-bit cyclic redundancy check (CRC) code. The communication protocol uses a master-slave acknowledgment mode. After the master module sends a request command, the slave module must return an acknowledgment signal within 5 milliseconds. If the timeout occurs, a retransmission mechanism is triggered, with a maximum of 3 retransmissions. During system initialization, each module performs a self-test, checking sensor connection status, memory allocation, and thread startup status. After passing the self-test, it enters standby mode.

[0031] In the specific implementation process, the binocular vision sensor of the environment perception module needs to undergo stereo calibration. The camera intrinsic parameter matrix and distortion coefficients are obtained through Zhang Zhengyou's calibration method, and the rotation matrix and translation vector between the left and right cameras are calculated. The weights of the convolutional neural network in the feature processing module need to be trained offline using a public dataset. The training cycle is 100 rounds, the batch size is 32, the optimizer is Adam, and the initial learning rate is 0.001. The extended Kalman filter parameters of the motion compensation module need to be calibrated using measured data. The process noise covariance matrix and the observation noise covariance matrix are set as diagonal matrices diag[0.01, 0.01, 0.01, 0.001, 0.001, 0.001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001] and diag[0.1, 0.1, 0.1, 0.01, 0.01, 0.01], respectively. The matching threshold of the decision control module can be dynamically adjusted according to the application scenario. It can be increased to 0.9 in environments with stable lighting and simple backgrounds, and decreased to 0.75 in environments with strong interference.

[0032] This embodiment achieves high precision and real-time performance in robot vision pattern matching under dynamic scenes through the precise coordination of the aforementioned modules. The multi-source data fusion of the environment perception module provides robust environmental perception capabilities; the hierarchical architecture and anti-interference design of the feature processing module ensure the stability of feature extraction; the dynamic distortion correction of the motion compensation module effectively suppresses errors introduced by the robot's own motion; the closed-loop collaborative mechanism of the decision control module ensures precise linkage between visual recognition and robotic arm operation; the adaptive update module enables the system to evolve over a long period; and the multi-threaded parallel architecture guarantees the overall system performance from the ground up. All modules have undergone rigorous parameter optimization and interface verification, and can run stably on industrial-grade computing platforms.

[0033] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A robot vision pattern matching system for dynamic scenes, characterized in that, include: The environment perception module (1) is used to collect visual data of the work scene and motion state data of the robot body; the feature processing module (2) is connected to the environment perception module and is used to extract multi-scale features from the visual data and construct anti-interference feature descriptors; the motion compensation module (3) is connected to both the environment perception module and the feature processing module and is used to establish a dynamic distortion model based on the robot motion state data to correct the feature space mapping relationship. The decision control module (4) integrates the feature processing module and the motion compensation module, and is used to generate robot motion control commands based on the corrected feature matching results and realize closed-loop collaboration between visual recognition and robotic arm operation; the environment perception module (1) includes: a binocular vision sensor, an inertial measurement unit and a laser ranging unit; the binocular vision sensor synchronously collects left and right view data of the scene at a rate of 30 frames per second; the inertial measurement unit detects the robot's three-axis angular velocity and three-axis acceleration data in real time; the laser ranging unit obtains the distance information of the robot's end effector relative to the target point at a frequency of 1000 Hz; the environment perception module (1) also has a data preprocessing submodule, which performs grayscale and histogram equalization processing on the original visual data, and uses the fourth-order Runge-Kutta method to calculate the attitude of the inertial measurement unit data; the feature processing module adopts a hierarchical feature extraction architecture; The first-layer feature extraction network performs three downsampling operations on the input image. After each downsampling, a 3x3 convolutional kernel is used for feature mapping, and a modified linear unit is used as the activation function. The second-layer feature extraction network constructs a spatial pyramid structure based on the output of the previous layer, calculating the directional gradient histogram features and local binary pattern features at four different scales. The feature processing module also includes a feature fusion unit, which concatenates the multi-scale feature vectors and inputs them into a fully connected layer, generating a 1024-dimensional anti-interference feature descriptor through weight allocation. The motion compensation module includes a pose estimation unit and a distortion correction unit. The pose estimation unit receives the attitude data calculated by the inertial measurement unit and the laser ranging data, fuses the multi-source information through an extended Kalman filter, and outputs the six-DOF pose estimation value of the robot's end effector at a 50-millisecond period. The distortion correction unit constructs a dynamic distortion model based on the pose estimation value. This model represents the image changes caused by robot motion as an affine transformation matrix and performs spatial transformation on the feature map through a bilinear interpolation algorithm, so that the feature descriptor maintains scale invariance and rotation invariance.

2. The robot vision pattern matching system in a dynamic scene according to claim 1, characterized in that, The decision control module (4) is equipped with a matching degree calculation unit and a motion planning unit. The matching degree calculation unit uses an improved Hamming distance metric method to calculate the similarity between the anti-interference feature descriptor extracted from the current frame and the benchmark features in the template library. When the similarity exceeds the preset threshold of 0.85, it is determined that the matching is successful. The motion planning unit calculates the target angle of each joint of the robotic arm according to the matching result, generates a smooth trajectory using a fifth-order polynomial interpolation algorithm, and realizes position closed-loop control through a proportional-integral-derivative controller.

3. The robot vision pattern matching system in a dynamic scene according to claim 1, characterized in that, The robot vision pattern matching system in this dynamic scene is also equipped with an adaptive update module (5); the adaptive update module (5) continuously monitors the success rate and accuracy of feature matching. When the average similarity of 10 consecutive matches is less than 0.8, the template feature update mechanism is automatically triggered. This mechanism incorporates the currently successfully matched feature descriptors into the template library in a sliding window manner, while removing the 5 oldest template features to keep the template library capacity constant at 50 sets of feature descriptors.

4. The robot vision pattern matching system in a dynamic scene according to claim 1, characterized in that, The system adopts a multi-threaded parallel processing architecture (6); the visual data processing thread and the motion control thread run on independent processor cores, and the two threads exchange data through shared memory and use a mutex lock mechanism to ensure data synchronization; the system sets the priority of the visual processing thread to be higher than that of the motion control thread to ensure the real-time performance of visual perception in dynamic scenes.

5. The robot vision pattern matching system in a dynamic scene according to claim 1, characterized in that, The feature processing module also integrates an illumination robustness enhancement unit; this unit dynamically adjusts the gamma correction parameters by calculating the gray-scale variance of local areas of the image, and automatically activates the auxiliary illumination system when the overall brightness of the image is detected to be below 50 lux, so as to ensure the stability of feature extraction under different illumination conditions.

6. The robot vision pattern matching system in a dynamic scene according to claim 2, characterized in that, The extended Kalman filter state vector includes the robot end effector's three-dimensional position, three-dimensional velocity, quaternion attitude, and angular velocity deviation, totaling 13 dimensions; the observation vector includes laser ranging data and inertial measurement unit acceleration data; the filter prediction step size is set to 50 milliseconds, and the state covariance matrix is ​​updated by linearizing the nonlinear motion model using the Jacobian matrix.

7. The robot vision pattern matching system in a dynamic scene according to claim 2, characterized in that, The fifth-order polynomial interpolation algorithm solves the coefficient matrix through boundary conditions; the boundary conditions include the angle, velocity, and acceleration between the starting point and the target point; the proportional coefficient of the proportional-integral-derivative controller is set to 2.5, the integral coefficient is set to 0.1, the derivative coefficient is set to 0.05, and the control period is 10 milliseconds.

Images