Robot vision mode matching system in dynamic scene

Abstract

The invention discloses a robot vision mode matching system in a dynamic scene, particularly discloses a robot vision mode matching system in a dynamic scene, and aims to solve the problems of unstable robot vision feature extraction, low matching accuracy and lack of real-time motion collaboration in a dynamic environment. The system comprises an environment sensing module, a feature processing module, a motion compensation module and a decision control module, scene and robot motion data are collected through multi-sensor fusion, and stable visual features are constructed by adopting multi-scale feature extraction and anti-interference feature descriptors; and compensating image distortion caused by robot motion based on a dynamic distortion model, and finally generating a closed-loop motion control instruction according to a feature matching result. According to the invention, the precision, robustness and real-time cooperation capability of robot vision matching in a dynamic scene are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot vision, in particular to a robot vision pattern matching system in dynamic scene. BACKGROUND

[0002] As a key branch of artificial intelligence and automation, robot vision technology has been widely applied in industrial detection, autonomous navigation and intelligent operation scenarios. Among them, vision pattern matching is one of the core technologies to realize robot environment perception and target operation, which aims to accurately identify and locate specific targets through image processing and feature comparison.

[0003] In robot operation applications in dynamic scenes, such as dust removal and decoking tasks inside incinerators, existing vision pattern matching systems face significant challenges. Existing technologies usually perform pattern recognition based on pre-set static environment models, which are difficult to adapt to image quality degradation caused by temperature changes, dust interference and light fluctuations in the furnace, resulting in unstable feature extraction and decreased matching accuracy. At the same time, the robot operation arm will produce its own motion and vibration during stretching, further introducing image blur and target displacement, and traditional matching algorithms lack robust compensation mechanisms for such dynamic distortions.

[0004] In addition, existing systems lack joint modeling of three-dimensional spatial structure and robot motion state when handling high-space operation tasks, making it difficult to realize real-time collaboration of vision recognition and robot arm stretching operation. In complex structures such as twenty-meter-high furnaces, the above problems can easily lead to positioning deviation or operation failure of the dust removal tool, seriously affecting operation efficiency and safety. Therefore, there is an urgent need for a robot system that can adapt to dynamic scene changes and improve the stability and accuracy of vision pattern matching. SUMMARY

[0005] The purpose of the present application is to provide a robot vision pattern matching system in dynamic scene, which solves the problems mentioned in the background art.

[0006] The present application is implemented as follows: a robot vision pattern matching system in dynamic scene, the system includes an environment perception module, a feature processing module, a motion compensation module and a decision control module. The environment perception module is used to collect the visual data of the operation scene and the motion state data of the robot body. The feature processing module is connected to the environment perception module and is used to extract multi-scale features from the visual data and construct an anti-interference feature descriptor. The motion compensation module is connected to the environment perception module and the feature processing module, and is used to establish a dynamic distortion model based on the robot motion state data to correct the feature space mapping relationship. The decision control module is integrated with the feature processing module and the motion compensation module, and is used to generate robot motion control instructions based on the corrected feature matching results and realize closed-loop collaboration of vision recognition and robot arm operation.

[0007] Further, the environment perception module comprises a binocular vision sensor, an inertial measurement unit, and a laser ranging unit. The binocular vision sensor synchronously acquires left and right view data of a scene at a rate of 30 frames per second. The inertial measurement unit detects three-axis angular velocity and three-axis acceleration data of the robot in real time. The laser ranging unit obtains distance information of an end effector of the robot relative to a target point at a frequency of 1000 Hz. The environment perception module is further provided with a data preprocessing submodule that performs grayscale and histogram equalization processing on original vision data, and adopts a fourth-order Runge-Kutta method for attitude solving on inertial measurement unit data.

[0008] Further, the feature processing module adopts a hierarchical feature extraction architecture. The first layer feature extraction network performs three downsampling operations on the input image, and after each downsampling, a convolution kernel with a size of 3 by 3 is used for feature mapping, and a rectified linear unit is used as an activation function. The second layer feature extraction network constructs a spatial pyramid structure based on the output of the previous layer, and calculates a histogram of oriented gradients feature and a local binary pattern feature at four different scales respectively. The feature processing module is further provided with a feature fusion unit that splices the multi-scale feature vectors and inputs them into a fully connected layer to generate an anti-interference feature descriptor with a dimension of 1024 through weight distribution.

[0009] Further, the motion compensation module comprises a pose estimation unit and a distortion correction unit. The pose estimation unit receives attitude data and laser ranging data solved by the inertial measurement unit, fuses multi-source information through an extended Kalman filter, and outputs a six-degree-of-freedom pose estimation value of the end effector of the robot at a period of 50 milliseconds. The distortion correction unit constructs a dynamic distortion model based on the pose estimation value, which represents the image changes caused by robot motion as an affine transformation matrix, and performs spatial transformation on the feature map through a bilinear interpolation algorithm, so that the feature descriptor maintains scale invariance and rotational invariance.

[0010] Further, the decision control module is provided with a matching degree calculation unit and a motion planning unit. The matching degree calculation unit adopts an improved Hamming distance measurement method to calculate the similarity between the anti-interference feature descriptor extracted from the current frame and the reference feature in the template library, and determines that the matching is successful when the similarity exceeds a preset threshold of 0.85. The motion planning unit calculates the target angles of each joint of the robot arm according to the matching result, generates a smooth trajectory using a quintic polynomial interpolation algorithm, and realizes position closed-loop control through a proportional-integral-derivative controller.

[0011] Further, the system is also provided with an adaptive updating module. The adaptive updating module continuously monitors the success rate and accuracy index of feature matching, and automatically triggers the template feature updating mechanism when the average similarity of 10 consecutive matches is lower than 0.8. The mechanism includes the current successfully matched feature descriptors in the template library in a sliding window manner, while removing the 5 oldest template features, keeping the template library capacity constant at 50 feature descriptors.

[0012] Further, the system adopts a multi-thread parallel processing architecture. The visual data processing thread and the motion control thread run on independent processor cores, data exchange between the two threads is carried out through shared memory, and a mutual exclusion lock mechanism is used to ensure data synchronization. The system sets the priority of the visual processing thread higher than that of the motion control thread, ensuring the real-time performance of visual perception in dynamic scenes.

[0013] Further, the feature processing module also integrates a light robustness enhancement unit. The unit dynamically adjusts the gamma correction parameter by calculating the gray variance of the local region of the image, and automatically enables the auxiliary lighting system when detecting that the overall brightness of the image is lower than 50 lux, ensuring the stability of feature extraction under different lighting conditions.

[0014] Compared with the prior art, the advantages and positive effects of the present application are that: through multi-sensor data fusion of the environment perception module, the present application realizes comprehensive perception of dynamic scenes; the multi-scale hierarchical architecture of the feature processing module and the anti-interference feature descriptor design effectively improve the feature stability under the conditions of dust interference and light fluctuation; the dynamic distortion model established by the motion compensation module can accurately compensate for the image changes caused by the robot's own motion, significantly improving the accuracy of feature matching; the closed-loop collaborative mechanism of the decision control module ensures the real-time linkage of visual recognition results and robot operation, especially suitable for high-altitude complex operation environment; the adaptive updating module enables the system to have continuous optimization capability, which can maintain the matching performance for a long time; the multi-thread parallel architecture ensures the real-time response capability of the system in dynamic scenes. The overall system solves the challenges faced by the prior art in dynamic scenes through the collaborative action of various modules, and has important engineering application value. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is the overall technical scheme architecture schematic diagram of the robot visual pattern matching system in a dynamic scene according to the present application; Figure 2 is the system block diagram of the environment perception module of the robot visual pattern matching system in a dynamic scene according to the present application.

[0016] Marked: 1, environment perception module; 2, feature processing module; 3, motion compensation module; 4, decision control module; 5, adaptive updating module; 6, multi-thread parallel processing architecture. DETAILED DESCRIPTION

[0017] Embodiment 1 Reference is made to the accompanying drawings Figure 1 This embodiment describes a specific technical implementation of a robot vision pattern matching system in a dynamic scene. The system is composed of an environment perception module, a feature processing module, a motion compensation module, a decision control module, an adaptive update module and a multi-thread parallel processing architecture. Each module interacts through a data bus and shared memory to ensure high-precision vision pattern matching and real-time motion control in a dynamic work scene.

[0018] The environment perception module is responsible for collecting visual information of the work scene and the robot's motion parameters. This module includes a binocular vision sensor, an inertial measurement unit and a laser ranging unit. The binocular vision sensor uses two global shutter cameras with a resolution of 1920 by 1080 to synchronously collect left and right view data at a rate of 30 frames per second, and transmits the raw images to the data preprocessing submodule through a gigabit Ethernet interface. The inertial measurement unit integrates a three-axis MEMS gyroscope and a three-axis MEMS accelerometer, with an angular velocity range of ±2000 degrees per second, an acceleration range of ±16g, and a data output frequency of 400 Hz. The laser ranging unit measures the distance between the robot's end effector and the target object's surface based on the time-of-flight principle, with a frequency of 1000 Hz and a ranging accuracy of ±1 mm. The data preprocessing submodule performs grayscale conversion and histogram equalization on the raw images collected by the binocular vision sensor. The grayscale conversion uses the weighted average method to combine the red, green and blue channels of the color image into a single channel grayscale image with weights of 0.299, 0.587 and 0.114. Histogram equalization enhances image contrast by reconstructing pixel distribution using the cumulative distribution function. For inertial measurement unit data, the data preprocessing submodule uses the fourth-order Runge-Kutta method for attitude solving, which obtains the robot's real-time pitch angle, roll angle and yaw angle by integrating angular velocity and acceleration data, with a solving period of 2.5 ms.

[0019] The feature processing module receives the preprocessed visual data from the environment perception module and performs multi-scale feature extraction and descriptor generation. This module uses a hierarchical feature extraction architecture, including a first layer feature extraction network, a second layer feature extraction network and a feature fusion unit. The first layer feature extraction network performs 3 times down-sampling operation on the input grayscale image, each time using a max-pooling layer with a step size of 2 to reduce the image size to one-half, one-fourth and one-eighth of the original image, respectively. After each down-sampling, a convolution layer is connected, with a convolution kernel size of 3 by 3, channel numbers of 64, 128 and 256 respectively, and a rectified linear unit (ReLU) activation function with a mathematical expression of: The function outputs zero for negative input and keeps linear growth for positive input, effectively alleviating the gradient vanishing problem. The second layer feature extraction network constructs a spatial pyramid structure on the output feature map of the first layer, and calculates the direction gradient histogram feature and the local binary pattern feature at 4 different scales respectively. The direction gradient histogram feature divides the image at each scale into 8x8 cell units, calculates the gradient direction histogram in each unit, and concatenates the histograms of the adjacent 4 units into block features; the local binary pattern feature uses uniform pattern encoding, compares each pixel with its 8 neighboring pixels, and generates a 58-dimensional feature vector. The feature fusion unit concatenates the multi-scale feature vectors in the channel dimension to generate a joint feature vector with a total dimension of 4096, which is then input into the fully connected layer, reduced to 1024 dimensions through the weight matrix, and finally outputs the anti-interference feature descriptor. The weight parameters of the fully connected layer are optimized in the training stage through the back propagation algorithm, and the loss function uses cross entropy loss.

[0020] The motion compensation module corrects the feature map output by the feature processing module based on the robot motion state data provided by the environment perception module. This module includes a pose estimation unit and a distortion correction unit. The pose estimation unit receives the attitude angle data calculated by the inertial measurement unit and the distance data from the laser ranging unit, and fuses multi-source information through an extended Kalman filter. The state vector of the extended Kalman filter includes the three-dimensional position, three-dimensional velocity, quaternion attitude and angular velocity bias of the robot end effector, with a total of 13 dimensions; the observation vector includes laser ranging data and inertial measurement unit acceleration data. The filter prediction step is set to 50 milliseconds, and the state covariance matrix is updated by linearizing the nonlinear motion model through the Jacobian matrix. The distortion correction unit constructs a dynamic distortion model based on the six-degree-of-freedom pose data output by the pose estimation unit, which represents the image distortion caused by robot motion as an affine transformation matrix:

[0021] where x and y are the original feature image coordinates, x' and y' are the transformed coordinates, a_{11} to a_{22} are the rotation and scaling parameters, t_x and t_y are the translation parameters. The distortion correction unit performs spatial transformation on the feature map through the bilinear interpolation algorithm, calculates the pixel value at the new coordinate according to the weighted average of the adjacent four pixels, so that the feature descriptor maintains scale invariance and rotational invariance under different viewing angles.

[0022] The decision control module generates robot motion instructions according to the feature matching results corrected by the motion compensation module. The module is provided with a matching degree calculation unit and a motion planning unit. The matching degree calculation unit adopts an improved Hamming distance measurement method, performs bit-by-bit exclusive OR operation on the 1024-dimensional anti-interference feature descriptor extracted from the current frame and the 50 groups of reference feature descriptors stored in the template library, counts the number of different bit positions and divides the total number of bits to obtain a normalized distance. When the similarity value corresponding to the minimum normalized distance exceeds the preset threshold value 0.85, it is determined that the matching is successful. The motion planning unit calculates the target angles of each joint of the robot arm according to the successfully matched feature positions, and generates a continuous and smooth trajectory by using a quintic polynomial interpolation algorithm. The quintic polynomial expression is: where q(t) is the joint angle, t is the time, a_0 to a_5 are coefficient matrices, which are solved by boundary conditions (angle, velocity, acceleration of the starting point and target point). The motion planning unit realizes position closed-loop control through a proportional-integral-derivative controller, the proportional coefficient is set to 2.5, the integral coefficient is set to 0.1, the differential coefficient is set to 0.05, and the control period is 10 milliseconds.

[0023] The adaptive update module continuously monitors the system matching performance, and automatically triggers the template feature update mechanism when the average similarity of 10 consecutive matches is less than 0.8. The mechanism includes the current successfully matched feature descriptors in the template library in a sliding window manner, while removing the 5 oldest template features, ensuring that the template library capacity is constant at 50 feature descriptors. The sliding window update strategy adopts the first-in-first-out principle, the new feature descriptor is stored in the index position 0 of the template library, the original feature descriptor is shifted backward, and the feature in the index position 49 is permanently deleted.

[0024] The system adopts a multi-thread parallel processing architecture, and the visual data processing thread and the motion control thread run on two independent processor cores respectively. The visual data processing thread is responsible for image acquisition, feature extraction and matching calculation, and its priority is set to real-time level to ensure that the processing delay of each frame is less than 33 milliseconds. The motion control thread is responsible for trajectory planning and servo control, and its priority is set to high level. The two threads exchange data through a shared memory area, which is divided into a feature descriptor buffer and a motion instruction buffer. Thread synchronization adopts a mutex mechanism, when the visual data processing thread writes new feature descriptors, the feature descriptor buffer is automatically locked to prevent the motion control thread from reading; when the motion control thread generates new motion instructions, the motion instruction buffer is also locked. The timeout time of the mutex is set to 5 milliseconds, if it is exceeded, the thread priority promotion strategy is triggered.

[0025] The feature processing module also integrates a light robustness enhancement unit that evaluates the light uniformity by calculating the gray variance of local regions of the image. When the overall brightness of the image is detected to be lower than 50 lux, the infrared auxiliary lighting system is automatically enabled to compensate for insufficient ambient light. The light robustness enhancement unit dynamically adjusts the gamma correction parameter, whose value is mapped to a floating-point number between 1.0 and 2.5 according to the average gray value of the image, with the calculation formula being: where γ is the gamma correction parameter, and \bar{I} is the average gray value of the image. When \bar{I} is lower than 50, γ takes a value greater than 2.0, significantly improving dark details; when \bar{I} is higher than 150, γ takes a value close to 1.0, maintaining the original appearance of the image.

[0026] The system modules communicate through a high-speed serial bus, and the data packet format includes a 32-bit header identifier, a 16-bit data length, a 1024-bit payload, and a 32-bit cyclic redundancy check code. The communication protocol uses a master-slave response mode, and after the master module sends a request command, the slave module needs to return a response signal within 5 milliseconds, otherwise the retransmission mechanism is triggered, with a maximum of 3 retransmissions. During system initialization, each module performs a self-checking process to detect the sensor connection state, memory allocation, and thread start state, and after passing the self-checking, it enters standby mode.

[0027] In the specific implementation process, the binocular vision sensor of the environment perception module needs to be stereo calibrated to obtain the camera intrinsic matrix and distortion coefficients through Zhang Zhengyou's calibration method, and to calculate the rotation matrix and translation vector between the left and right cameras. The convolutional neural network weights of the feature processing module need to be trained using public datasets in the offline stage, with a training period of 100 rounds, a batch size of 32, an Adam optimizer, and an initial learning rate of 0.001. The extended Kalman filter parameters of the motion compensation module need to be calibrated through measured data, with the process noise covariance matrix and the observation noise covariance matrix set to diag[0.01, 0.01, 0.01, 0.001, 0.001, 0.001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001] and diag[0.1, 0.1, 0.1, 0.01, 0.01, 0.01], respectively. The matching threshold of the decision control module can be dynamically adjusted according to the application scenario, which can be increased to 0.9 in a stable light and simple background environment, and reduced to 0.75 in a strong interference environment.

[0028] Through the fine cooperation of the above modules, this embodiment realizes high precision and high real-time of robot vision pattern matching in dynamic scene. The multi-source data fusion of the environment perception module provides the system with robust environment perception capability; the layered architecture and anti-interference design of the feature processing module guarantee the stability of feature extraction; the dynamic distortion correction of the motion compensation module effectively suppresses the error introduced by the robot's own motion; the closed-loop collaborative mechanism of the decision control module ensures the precise linkage of visual recognition and robot operation; the adaptive update module enables the system to have long-term evolution capability; the multi-thread parallel architecture guarantees the overall performance of the system from the bottom. All modules have undergone strict parameter tuning and interface verification and can run stably on an industrial-grade computing platform.

[0029] The above merely describes preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A robot vision pattern matching system under dynamic scene, characterized in that, The application relates to a robot motion control system based on visual recognition, which comprises an environment perception module (1) for collecting visual data of a working scene and robot body motion state data, a feature processing module (2) connected with the environment perception module and used for multi-scale feature extraction of the visual data and construction of an anti-interference feature descriptor, a motion compensation module (3) connected with the environment perception module and the feature processing module and used for establishing a dynamic distortion model based on the robot motion state data to correct a feature space mapping relationship, and a decision control module (4) integrated with the feature processing module and the motion compensation module and used for generating a robot motion control instruction according to the corrected feature matching result and realizing closed-loop cooperation of visual recognition and mechanical arm operation. The environment perception module (1) comprises a binocular vision sensor, an inertial measurement unit and a laser ranging unit; the binocular vision sensor synchronously collects left and right view data of the scene at a rate of 30 frames per second; the inertial measurement unit detects three-axis angular velocity and three-axis acceleration data of the robot in real time; and the laser ranging unit obtains distance information of a robot end effector relative to a target point at a frequency of 1000 Hz; the environment perception module (1) is further provided with a data preprocessing sub-module which performs gray processing and histogram equalization processing on original visual data and adopts a fourth-order Runge-Kutta method to solve the attitude of the inertial measurement unit data; the feature processing module adopts a hierarchical feature extraction architecture; a first layer feature extraction network performs three times of downsampling operations on an input image, adopts a convolution kernel with a size of 3*3 to perform feature mapping after each downsampling, and adopts a rectified linear unit as an activation function; a second layer feature extraction network constructs a spatial pyramid structure based on the output of the previous layer, and respectively calculates a direction gradient histogram feature and a local binary pattern feature at four different scales; and the feature processing module is further provided with a feature fusion unit which splices multi-scale feature vectors and inputs the same into a full connection layer to generate an anti-interference feature descriptor with a dimension of 1024 through weight distribution. The motion compensation module comprises a pose estimation unit and a distortion correction unit; the pose estimation unit receives attitude data solved by the inertial measurement unit and laser ranging data, fuses multi-source information through an extended Kalman filter, and outputs a six-degree-of-freedom pose estimation value of the robot end effector at a period of 50 milliseconds; and the distortion correction unit constructs a dynamic distortion model based on the pose estimation value, the model represents image changes caused by robot motion as an affine transformation matrix, and performs spatial transformation on a feature map through a bilinear interpolation algorithm, so that the feature descriptor remains scale invariance and rotation invariance.

2. The dynamic scene based robot vision pattern matching system of claim 1, wherein, The decision control module (4) is provided with a matching degree calculation unit and a motion planning unit; the matching degree calculation unit adopts an improved Hamming distance measurement method to calculate the similarity of the anti-interference feature descriptor extracted from a current frame and reference features in a template library, and determines that matching is successful when the similarity exceeds a preset threshold value 0.85; and the motion planning unit calculates target angles of each joint of the mechanical arm according to the matching result, generates a smooth trajectory through a quintic polynomial interpolation algorithm, and realizes position closed-loop control through a proportional-integral-derivative controller.

3. The dynamic scene based robot vision pattern matching system of claim 1, wherein, ​ 4. The dynamic scene based robot vision pattern matching system of claim 1, wherein, The robot vision pattern matching system in the dynamic scene is also provided with an adaptive updating module (5); the adaptive updating module (5) continuously monitors the success rate and accuracy index of feature matching, and when the average similarity of 10 consecutive matching is lower than 0.8, the template feature updating mechanism is automatically triggered; the mechanism includes the current successfully matched feature descriptor into the template library in a sliding window manner, and removes the 5 oldest template features, so that the capacity of the template library is kept constant at 50 feature descriptors.

5. The dynamic scene based robot vision pattern matching system of claim 1, wherein, The system adopts a multi-thread parallel processing architecture (6); the visual data processing thread and the motion control thread run on independent processor cores, data exchange is carried out between the two threads through shared memory, and a mutual exclusion lock mechanism is used to ensure data synchronization; the system sets the priority of the visual processing thread higher than that of the motion control thread, so as to ensure the real-time performance of visual perception in the dynamic scene.

6. The dynamic scene based robot vision pattern matching system of claim 1, wherein, The feature processing module also integrates a light robustness enhancement unit; the unit dynamically adjusts the gamma correction parameter by calculating the gray variance of the local area of the image, and automatically enables the auxiliary lighting system when detecting that the overall brightness of the image is lower than 50 lux, so as to ensure the stability of feature extraction under different lighting conditions.

7. The dynamic scene based robot vision pattern matching system of claim 2, wherein, The state vector of the extended Kalman filter contains the three-dimensional position, three-dimensional velocity, quaternion attitude and angular velocity deviation of the robot end effector, with a total of 13 dimensions; the observation vector contains laser ranging data and inertial measurement unit acceleration data; the filter prediction step is set to 50 milliseconds, the nonlinear motion model is linearized through the Jacobian matrix, and the state covariance matrix is updated.

8. The dynamic scene based robot vision pattern matching system of claim 3, wherein, The quintic polynomial interpolation algorithm solves the coefficient matrix through boundary conditions; the boundary conditions include the angle, velocity and acceleration of the starting point and the target point; the proportional coefficient of the proportional-integral-derivative controller is set to 2.5, the integral coefficient is set to 0.1, the differential coefficient is set to 0.05, and the control period is 10 milliseconds.

Images