Visual-tactile dual-channel control hardware method and system for robot impedance adaptation

Abstract

The application provides a kind of hardware method and system for robot impedance self-adapting visual touch dual-channel control, comprising: obtaining visual data and tactile data, storing the visual data and tactile data and respectively pre-processing;Through convolutional neural network CNN, visual features are extracted and calculated by FPGA DSP unit acceleration, through graph neural network GNN, tactile features are extracted for graph convolution operation;Through the multi-head attention mechanism of Transformer architecture, the visual features and tactile features are weighted and fused to generate global features;According to the global features, control parameters are generated based on Actor-Critic architecture;Based on the control parameters, motor drive signals for robot motion are generated, and motion tasks are realized through motor execution control.The application converts the software operation process of multimodal visual and tactile information into FPGA parallel computing hardware circuit logic, and uses hardware circuit to implement data calculation of multimodal information, with high processing efficiency and less noise between signals.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a visual-tactile dual-channel control hardware method and system for robot impedance adaptation. Especially, it relates to a visual-tactile fusion dual-channel control hardware method and system for robot impedance adaptation. BACKGROUND

[0002] With the wide application of intelligent robots in industry, medical and service fields, their perception and adaptation to complex environments are put forward with higher requirements. As two key channels for robots to perceive the outside world, vision and touch have the following advantages and limitations: visual perception can provide global information of the environment, but it is easily affected by light changes and occlusions. Tactile perception directly reflects the contact force and surface characteristics, which is suitable for local interaction scenarios, but it is difficult to provide global understanding of the environment.

[0003] At present, robot systems complete motion control through visual or tactile single-modal perception. However, in dynamic and complex environments, relying on a single modality often cannot meet the real-time and robustness requirements. Multimodal fusion of vision and touch has been proven to significantly improve the comprehensiveness and accuracy of perception. However, existing multimodal fusion systems mainly rely on GPUs or high-performance processors, which face the following problems in practical applications: insufficient real-time performance: the computational delay of processing high-dimensional visual and tactile data is large, which is difficult to meet the real-time control requirements. High power consumption: the existing multimodal fusion system runs inefficiently on embedded platforms and cannot run for a long time. Hardware architecture limitation: the parallel hardware characteristics are not fully utilized for algorithm optimization, and the resource utilization rate is low.

[0004] To solve the above problems, FPGA-based hardware implementation scheme has gradually become an ideal choice. FPGA has high parallel computing capability and low power consumption, which provides new possibilities for the fusion processing and real-time control of visual and tactile data. However, the specific method of efficiently implementing multimodal fusion and motion control on FPGA hardware still needs further research.

[0005] Patent document CN119550335A discloses a robot motion planning method based on multimodal information fusion, including: information acquisition: acquiring multimodal data of the robot through multiple sensors; preprocessing: normalizing, denoising and synchronizing the multimodal data; fusion: encoding and fusing the preprocessed data using a feature-level fusion method, and generating an environment model based on an information fusion algorithm; motion planning: combining environment modeling and path planning algorithms to generate a motion path based on the fused multimodal information; feedback adjustment: based on real-time perception of environmental changes, using a feedback control mechanism to dynamically adjust the motion trajectory.

[0006] However, the multi-modal data fusion method of the patent document CN119550335A has large calculation delay in processing high-dimensional visual and tactile data, is difficult to meet the real-time control requirement, and does not fully utilize the parallel hardware characteristics for algorithm optimization, only obtains the processing result of multi-modal information through a software algorithm, and resource utilization is low. The software operation process of the visual and tactile information of the multi-modal is converted into a hardware circuit logic of FPGA parallel calculation, the data calculation of the multi-modal information is implemented by using the hardware circuit, the processing efficiency is high, and the noise between signals is small. SUMMARY

[0007] In view of the defects in the prior art, the purpose of the present application is to provide a visual and tactile dual-channel control hardware method and system for robot impedance adaptation.

[0008] According to the visual and tactile dual-channel control hardware method for robot impedance adaptation provided by the present application, the following steps are included:

[0009] Step S1: acquire visual data and tactile data, store the visual data and tactile data and pre-process them respectively;

[0010] Step S2: extract visual features through a convolutional neural network (CNN) and accelerate calculation through a FPGA DSP unit, and extract tactile features through a graph neural network (GNN) for graph convolution operation;

[0011] Step S3: perform weighted fusion on the visual features and the tactile features through a multi-head attention mechanism of a Transformer architecture, and generate global features;

[0012] Step S4: generate control parameters based on an Actor-Critic architecture according to the global features;

[0013] Step S5: generate motor driving signals of robot motion based on the control parameters, perform control through a motor, and realize a motion task.

[0014] Preferably, the pre-processing of the visual data includes:

[0015] Step S101: convert the RGB format data in the visual data into a gray scale format;

[0016] Step S102: filter the converted data using a fast median filtering algorithm with a 3*3 template;

[0017] Step S103: use a histogram equalization module to instantiate a block random access memory (blockram) for the filtered data, and perform equalization operation on the histogram, and the equalization formula is as follows:

[0018]

[0019] representing a gray level a new gray value after mapping, representing a gray level a cumulative distribution function value; representing a maximum value of a gray level; representing a floor operation.

[0020] Preferably, the preprocessing of the tactile data includes band-pass filtering and amplitude normalization.

[0021] The band-pass filtering selects a FIR filter, calculates the output of the filter, filters out low-frequency drift and high-frequency noise in the tactile data, and only retains the effective signal frequency band of 20Hz to 450Hz, and the output signal y[n] of the FIR filter is calculated according to the formula:

[0022]

[0023] wherein x[n] represents an input signal, h[k] represents an impulse response of the filter, and M represents a filter length, representing the current input signal and the previous M-1 sampling points;

[0024] The amplitude normalization includes standardizing the amplitude range of the tactile data to [0, 1], and ensuring the signal consistency between the sensors, and the normalization formula is:

[0025]

[0026] wherein, and respectively represent the maximum value and the minimum value of the current tactile data.

[0027] Preferably, the convolutional neural network CNN is composed of three convolutional layers, three activation layers and three pooling layers alternately.

[0028] The convolutional layer uses a 3x3 convolution kernel to support a dynamic number of channels, and the convolution adopts a Winograd optimization algorithm.

[0029] The activation operation of the activation layer is completed by a hardware comparator.

[0030] The pooling layer uses a FPGA comparator array to realize a maximum pooling operation to reduce the dimension of the feature map while retaining significant features.

[0031] Quantization technology is used to convert the weights and activation values from floating-point numbers to low-bit numbers INT8.

[0032] Preferably, the graph neural network GNN is composed of three layers of graph convolution layers, the tactile data is modeled as a graph structure, the nodes represent sensor positions, and the edge weights represent adjacent sensors. The graph structure is dynamically generated according to the correlation between the position distribution of the tactile sensor and the contact force, and the edge weight of the graph is calculated as follows:

[0033]

[0034] wherein, represents the weight between and , represents the physical coordinates of nodes and , represents a weight control parameter; the node features are updated by domain propagation and aggregation:

[0035]

[0036] wherein, represents the features of the nodes of the layer, represents the neighborhood node set of the node , represents the weight matrix of the layer, (*) represents an activation function;

[0037] The global feature of the graph is calculated by aggregating the features of all nodes:

[0038]

[0039] wherein, K represents the number of layers of the GNN, N represents the number of nodes, represents a global aggregation operation;

[0040] The weight calculation is realized by hardware parallelization, and the edge weight is stored in the on-chip cache BRAM. The propagation formula of each layer is calculated by sparse matrix multiplication, and the update of the node feature is calculated in parallel using a pipeline architecture. The activation function is realized by a hardware comparison unit. The node feature is realized by an addition tree for global aggregation, and the aggregated feature vector is stored in the register of the FPGA. The sparse optimization is used to calculate only the non-zero edge weight, and the index table is used to skip the zero weight edge.

[0041] Preferably, the step S3 comprises:

[0042] Step S3.1: combining the visual features with the tactile features Dimension alignment is performed to unify the dimensions of the two, and the formula is as follows:

[0043] ,

[0044] wherein, denotes the aligned reconstructed visual feature, denotes the aligned reconstructed haptic feature, and are trainable weight matrices;

[0045] Step S3.2: The input features Q, K, and V are stored in the on-chip cache BRAM, and the multi-attention mechanism formula is as follows:

[0046]

[0047] wherein Q, K, and V are query, key, and value matrices respectively, denotes the dimension of the key vector;

[0048] Step S3.3: The visual feature and the haptic feature are mapped to query, key, and value through the DSP unit, and are processed in parallel by row blocking. Multiple attention heads generate output on the basis of parallel calculation:

[0049]

[0050] Each head: , , , are the projection weight matrices of query, key, and value respectively; denotes the output transformation matrix;

[0051] Step S3.4: The result is divided by the square root of the dimension of the key vector for scaling. The square root calculation is implemented through a lookup table LUT, and the calculation is accelerated using a bit shift operation. The attention score mechanism is as follows:

[0052]

[0053] Step S3.5: The lookup table LUT is used to prestore exponential values, and the approximate results of the exponential operation of each element in Softmax are quickly found;

[0054] Step S3.6: The matrix multiplication is calculated in parallel through the DSP unit, and the results are accumulated by head and stored in the BRAM. The weighted summation formula is as follows:

[0055]

[0056] Step S3.7: Splice each head output into a matrix by column in BRAM, and use the DSP unit to perform linear transformation of the output:

[0057] .

[0058] Preferably, in the Actor-Critic architecture, the Actor module generates control parameters according to global features, and controls the behavior of the robot; the Critic module evaluates the effect of the action, and updates the strategy optimization control performance through the mean square error.

[0059] Preferably, the control parameters include control parameters output by the Actor module , and real-time data from position sensors, force / torque sensors;

[0060] The control parameters include pose parameters and impedance parameters .

[0061] Preferably, the pose parameters are used to generate control parameters of servo control motors through PID control, and the formula is as follows:

[0062]

[0063] In the formula, represents the difference between the target value of the pose parameter and the feedback value of the pose parameter; the accumulator stores the integral value, which is updated in real time, and the difference is calculated by using a sliding window, and the output of the PID controller is used for PWM signal generation;

[0064] According to the impedance parameters , impedance control force is generated:

[0065]

[0066] In the formula, , represents the target position and the actual position, represents the real-time speed feedback by the sensor.

[0067] According to the present application, a visual and tactile dual-channel control hardware system for robot impedance adaptation is provided, comprising:

[0068] The FPGA data acquisition and preprocessing module acquires visual data and tactile data, stores the visual data and tactile data and pre-processes them respectively;

[0069] Feature extraction module: extract visual features through convolutional neural network (CNN) and accelerate calculation through FPGA DSP unit, extract haptic features through graph neural network (GNN) for graph convolution operation;

[0070] Feature fusion module: weight and fuse visual features and haptic features through multi-head attention mechanism of Transformer architecture to generate global features;

[0071] Reinforcement learning module: generate control parameters based on Actor-Critic architecture according to the global features;

[0072] Motor drive module: generate motor drive signals for robot motion based on the control parameters, execute control through motor to achieve motion task.

[0073] Compared with the prior art, the present application has the following beneficial effects:

[0074] 1. The FPGA-based visual and haptic fusion hardware system of the present application is used for real-time impedance adaptation and pose control of robots in complex dynamic environments, and through hardware acceleration processing of dual-channel multi-modal data, the adaptability and control accuracy of robots to the external environment are improved.

[0075] 2. The FPGA parallel computing unit and hardware acceleration module of the present application realize fast processing and feature extraction of visual and haptic data, significantly improving the computing efficiency.

[0076] 3. The present application can realize high-precision motion control and adjust the motion strategy according to real-time sensing data to meet the task requirements in complex environments. It has strong real-time performance, low delay and high precision, and is suitable for complex robot control tasks, especially in dynamic interaction and multi-modal perception tasks, showing a wide application prospect. BRIEF DESCRIPTION OF DRAWINGS

[0077] Other features, objects and advantages of the present application will become more apparent through reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0078] Figure 1 The figure is a schematic diagram of the working method of the present application;

[0079] Figure 2 The figure is a schematic diagram of the feature extraction module architecture of the hardware system;

[0080] Figure 3 The figure is a schematic diagram of the feature fusion module architecture of the hardware system;

[0081] Figure 4 The figure is a schematic diagram of the reinforcement learning module architecture of the hardware system. DETAILED DESCRIPTION

[0082] The application will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be pointed out that those skilled in the art can make several changes and improvements without departing from the concept of the application. These are all within the scope of protection of the application.

[0083] The application collects visual and tactile data through a CMOS camera and a surface electromyography sensor, and uses FPGA data processing to pre-process the collected data in real time. Visual features are extracted through a convolutional neural network (CNN), and tactile features are extracted through a graph neural network (GNN). The features are fused through a multi-head attention mechanism of a Transformer algorithm to generate high-dimensional feature representations with global spatio-temporal dependencies. Then, control parameters are generated based on an Actor-Critic architecture to optimize the robot motion control strategy. Finally, motor drive signals for robot motion are generated based on the control parameters, and the motion control is implemented through the motor to achieve the motion task.

[0084] In the application, the FPGA hardware uses an Xilinx FPGA platform, and the FPGA DSP unit is used to accelerate all matrix operations, convolution calculations and graph convolution calculations. The RAM storage module is used to store intermediate calculation results, network weights and training parameters to reduce memory access delay and improve data processing efficiency. The parallel computing unit processes multiple computing tasks simultaneously through a pipeline architecture and parallel computing units to achieve efficient parallel computing. The clock management module accurately controls the timing of each module to ensure data synchronization between modules.

[0085] Embodiment 1

[0086] According to the visual-tactile dual-channel control hardware method for robot impedance adaptation provided by the application, as shown in Figure 1 , comprising:

[0087] Step S1: acquiring visual data and tactile data, storing the visual data and tactile data and pre-processing them respectively. The visual data is collected by a CMOS camera, and the visual data includes environmental image data. The tactile data is acquired by a surface electromyography sensor and a force / displacement sensor, and the tactile data includes tactile signals.

[0088] The pre-processing of the visual data includes color space conversion, noise filtering and histogram equalization, and the steps include:

[0089] Step S101: converting the RGB format data in the visual data to grayscale format;

[0090] Step S102: The converted data is filtered using a fast median filtering algorithm with a 3*3 template.

[0091] Step S103: A histogram equalization module is instantiated using a random memory blockram for the filtered data, and a histogram equalization operation is performed, with the equalization formula as follows:

[0092]

[0093] denotes the gray level , the new gray value after mapping, denotes the gray level , the cumulative distribution function value; denotes the maximum value of the gray level; denotes the floor operation.

[0094] The preprocessing of the tactile data includes band-pass filtering and amplitude normalization. The band-pass filtering selects an FIR filter, uses a shift register to store the latest M input signals, dynamically updates the data, uses a multi-path parallel multiplier to simultaneously calculate h[k]*x[n-k], and then uses a pipeline structure to accumulate all the products to generate the output y[n] of the filter. The low-frequency drift and high-frequency noise in the tactile signal are filtered out, and only the effective signal frequency band of 20Hz to 450Hz is retained.

[0095] Let the input signal be x[n], the filter coefficient be h[k] (the filter length is M), and the output signal y[n] of the FIR filter be calculated according to the formula:

[0096]

[0097] wherein denotes the current input signal and the previous M-1 sampling points. h[k] denotes the impulse response of the filter.

[0098] The amplitude normalization includes standardizing the amplitude range of the tactile signal to [0, 1], ensuring the signal consistency between the sensors, and facilitating subsequent processing and feature extraction. The maximum value and the minimum value of the current signal are calculated by a hardware comparator, and the normalization calculation is completed by using a subtraction unit and a division pipeline. At the same time, fixed-point operation is used instead of floating-point operation, and the maximum and minimum values are stored in the on-chip register (BRAM) to reduce the storage access delay.

[0099] The normalization implementation formula is:

[0100]

[0101] Step S2: extract visual features, i.e. spatial features of visual data, through a convolutional neural network CNN, and accelerate calculation through an FPGA DSP unit to improve calculation efficiency, and extract haptic features, i.e. spatio-temporal features of haptic data, through a graph neural network GNN to perform graph convolution operation. As shown in FIG. 8, spatial features in an image are extracted through a convolutional neural network CNN, and visual features such as edges, textures, shapes, etc. in the image are recognized at the same time, Winograd optimization algorithm is used to accelerate convolution calculation, and hardware resource occupation is reduced; spatial and temporal dependence of haptic signals is modeled through a graph neural network GNN, haptic force field features are extracted, and graph convolution calculation is accelerated through sparse matrix operation. Figure 2

[0102] The convolutional neural network CNN is composed of three convolutional layers, three activation layers and three pooling layers in an alternating manner, the convolutional layer uses a 3x3 convolution kernel, supports a dynamic number of channels, and the convolution uses a Winograd optimization algorithm to reduce the amount of multiplication operation. The activation operation of the activation layer is completed through a hardware comparator; the pooling layer uses an FPGA comparator array to realize a maximum pooling operation (2x2 window) to reduce the dimensionality of the feature map while retaining significant features. Quantization technology is used to convert weights and activation values from floating-point numbers to low-bit numbers (INT8) to reduce calculation complexity.

[0103] The graph neural network GNN is composed of three graph convolutional layers, the haptic data is modeled as a graph structure, the nodes represent sensor positions, and the edge weights represent adjacent sensors. According to the correlation between the position distribution of the haptic sensor and the contact force, the graph structure is dynamically generated, and the edge weight calculation formula of the graph is as follows:

[0104]

[0105] wherein represents the weight between and ; , represents the physical coordinates of nodes and ; represents a weight control parameter.

[0106] The node features are updated through domain propagation and aggregation:

[0107]

[0108] wherein represents the features of nodes in the layer; represents a set of neighboring nodes of node ; represents the layer. ​weight matrix of the layer; (*) denotes an activation function.

[0109] The global feature of the graph is calculated by aggregating the features of all nodes:

[0110]

[0111] K represents the number of layers of the GNN; N represents the number of nodes; represents a global aggregation operation.

[0112] The weight calculation is realized by hardware parallelization, and the edge weight is stored in the on-chip cache (BRAM), and the propagation formula of each layer is calculated by sparse matrix multiplication. The update of the node feature is calculated in parallel using a pipeline architecture. The activation function is realized by a hardware comparison unit. The node feature is realized by an addition tree for global aggregation. The aggregated feature vector is stored in the register of the FPGA. Sparse optimization is used to calculate only the non-zero edge weight. The index table is used to skip the zero weight edge to reduce the calculation amount.

[0113] Step S3: The visual feature and the tactile feature are weighted and fused through the multi-head attention mechanism of the Transformer architecture to generate a global feature. As shown in Figure 3 , the Transformer fusion algorithm is adopted, and the Transformer algorithm with a multi-head attention mechanism is used to dynamically weight and fuse the visual feature and the tactile feature. By calculating the weighted sum of multiple attention heads in parallel, the spatio-temporal dependency between different modalities is captured. The weights of the features of each modality are adjusted according to different task requirements, thereby generating a high-dimensional feature representation with global spatio-temporal dependency. The step S3 includes:

[0114] Step S3.1: The visual feature is dimensionally aligned with the tactile feature , and the dimensions of the two are unified, and the formula is as follows:

[0115] ,

[0116] wherein, represents the reconstructed visual feature after dimension alignment, represents the reconstructed tactile feature after dimension alignment, and are trainable weight matrices.

[0117] Step S3.2: The input features Q, K, and V are stored in the on-chip cache BRAM, and the formula of the multi-head attention mechanism is as follows:

[0118]

[0119] where Q, K, V are Query, Key and Value matrices respectively, denotes the dimension of the Key vector.

[0120] Step S3.3: Visual features and tactile features are mapped to Query, Key and Value through the DSP unit, and processed in parallel by row blocking. Multiple attention heads generate outputs based on parallel computing:

[0121]

[0122] Each head: , , , are the projection weight matrices of Query, Key and Value respectively; denotes the output transformation matrix.

[0123] Step S3.4: The result is scaled by dividing by the square root of the dimension of the Key vector. The square root calculation is implemented through a lookup table LUT, and the calculation is accelerated using a bit shift operation.

[0124] Attention score mechanism:

[0125]

[0126] Step S3.5: Use the lookup table LUT to pre-store exponential values, and quickly find the approximate results of the exponential operation of each element in Softmax. The denominator summation and normalization operation is completed in the adder tree.

[0127] Step S3.6: Parallel computing of matrix multiplication is performed through the DSP unit, and the results are accumulated by head and stored in BRAM. The weighted summation formula is as follows:

[0128]

[0129] Step S3.7: The outputs of each head are spliced into a matrix by column in BRAM, and linear transformation of the output is performed using the DSP unit:

[0130]

[0131] Step S4: Generate actions based on the global features and evaluate the effects of the actions to optimize the control strategy. As shown in Figure 4 the control based on reinforcement learning, a reinforcement learning hardware module (Actor-Critic architecture) is introduced. The Actor module generates control parameters according to the global features to control the behavior of the robot, and the Critic module evaluates the action effect and updates the strategy through the mean square error to optimize the control performance.

[0132] The global feature vector of the Transformer fusion module is accepted by the on-chip cache BRAM, and then normalized to ensure that the input distribution is adapted to the hardware calculation. The Actor-Critic architecture is a method in reinforcement learning, and the algorithm is described as follows:

[0133] The Actor module generates control parameters a according to the current state s (represented by the global variable F):

[0134]

[0135] The Critic module calculates the value function according to the current state s :

[0136]

[0137] Evaluates the goodness of the current state and outputs the value score.

[0138] Gradient optimization Advantage calculation:

[0139]

[0140] Where r represents the current reward generated by sensor feedback; denotes the discount factor; denotes the next state.

[0141] Update the Actor through policy gradient.

[0142] The Critic is updated through the mean square error:

[0143]

[0144] The linear transformation of the Actor network is implemented through the DSP unit:

[0145]

[0146] The generated probability distribution is sampled to obtain the specific action control parameters.

[0147] The Critic network calculates the state value through the DSP unit:

[0148]

[0149] Use the adder and multiplier pipeline to calculate the loss function and update the Critic network weights.

[0150] Step S5: Based on the control parameters, generate motor driving signals for robot motion, and execute control through motors to achieve the motion task. Generate motor control signals through the PWM signal generation module to drive the robot to execute the motion task, and adjust the motor motion through sensor feedback to ensure accurate execution.

[0151] The control parameters include the control parameters output by the Actor module , including pose parameters (target position and attitude angle) and impedance parameters (stiffness and damping), as well as real-time data from position sensors and force / torque sensors for closed-loop control.

[0152] The difference between the target value and the feedback value is calculated using an adder:

[0153]

[0154] The control parameters for servo control motors are generated using PID control:

[0155]

[0156] The accumulator stores the integral value, which is updated in real time, and calculates the difference using a sliding window, while the output of the PID controller is used to generate PWM signals. PWM signals generate pulses of fixed frequency through counters, while the high and low level outputs are adjusted according to the duty cycle.

[0157] Impedance control force is generated according to stiffness and damping parameters:

[0158]

[0159] , represents the target and actual position; represents the real-time speed feedback by the sensor. The stiffness term and the damping term are calculated through a multiplier, and the stiffness and damping terms are added through an adder to generate , K and D are respectively called position gain coefficient and speed gain coefficient.

[0160] Receive the calculated and store it in the FPGA on-chip cache (BRAM). Use the multiplier and adder pipeline to complete the calculation from force to torque and speed, and convert the force to duty cycle through fixed-point operation to ensure real-time calculation. PWM signals generate pulses of fixed frequency through counters, while the high and low level outputs are adjusted according to the duty cycle.

[0161] The present application aims to solve the problems of poor real-time performance, high power consumption and insufficient robustness in the prior art. By integrating the pre-processing, feature extraction, fusion and control signal generation modules of visual and tactile data in FPGA hardware, efficient adaptation of robots to dynamic environments is achieved. Through hardware acceleration and multi-modal sensor fusion, efficient and accurate robot control is achieved.

[0162] Embodiment 2

[0163] The present application also provides a visual-tactile dual-channel control hardware system for robot impedance adaptation, which can be realized by executing the flow steps of the visual-tactile dual-channel control hardware method for robot impedance adaptation, that is, the visual-tactile dual-channel control hardware method for robot impedance adaptation can be understood by those skilled in the art as the preferred embodiment of the visual-tactile dual-channel control hardware system for robot impedance adaptation.

[0164] According to the present application, a visual-tactile dual-channel control hardware system for robot impedance adaptation is provided, which comprises: an FPGA data acquisition and preprocessing module, a feature extraction module, a feature fusion module, a reinforcement learning module and a motor driving module.

[0165] The FPGA data acquisition and preprocessing module acquires visual data and tactile data, stores the visual data and tactile data and respectively pre-processes them. The FPGA data acquisition and preprocessing module comprises a CMOS camera data acquisition module and a surface electromyography sensor data acquisition module. The CMOS camera data acquisition module comprises a camera driving module, a data read-write control module, a color space conversion module, a median filter module and a histogram equalization module. The CMOS camera acquires visual data, generates a video stream and transmits the video stream to a visual data channel of a data processing module of the FPGA through a digital interface. Meanwhile, the sEMG sensor monitors electromyography signals in real time, captures tactile feedback, converts analog signals into digital signals through an analog-to-digital converter (ADC) and transmits the digital signals to a tactile data channel of the data processing module of the FPGA. The data read-write module controls reading and writing of the acquired visual and tactile data in a DDR3 storage module, and the acquired visual and tactile data are stored in the DDR3 storage module. The camera driving module controls the operation of the CMOS camera through an interface and acquires image data in real time. The color space conversion module converts the acquired RGB format data into a grayscale image for subsequent feature extraction. The filter module uses a median filter algorithm to remove noise in the image and ensure the image definition. The histogram equalization module enhances the contrast of the image, so that the features are more prominent and subsequent processing is facilitated. The preprocessing of the visual data comprises: module M101: converting the RGB format data in the visual data into grayscale format. Module M102: using a 3x3 template to filter the converted data using a fast median filter algorithm. Module M103: using a histogram equalization module to instantiate a random memory blockram to perform histogram equalization operation on the filtered data, and the histogram equalization formula is as follows:

[0166]

[0167] denotes the gray level the new gray value after mapping, denotes the gray level the cumulative distribution function value. denotes the maximum value of the gray level. denotes the floor operation.

[0168] The surface electromyography sensor data acquisition module includes a data read-write module, a band-pass filtering module, and an amplitude normalization module. The tactile signal is obtained through the surface electromyography sensor and the force / displacement sensor, and the analog signal is converted into a digital signal through an analog-to-digital converter (ADC). The band-pass filtering module filters out the low-frequency drift and high-frequency noise in the tactile signal, and only retains the effective frequency band. The amplitude normalization module standardizes the amplitude range of the tactile signal to [0, 1], ensuring that the signals of each sensor are consistent and facilitating subsequent processing. The preprocessing of the tactile data includes band-pass filtering and amplitude normalization. The band-pass filtering selects an FIR filter, calculates the output of the filter, filters out the low-frequency drift and high-frequency noise in the tactile data, and only retains the effective signal frequency band of 20 Hz to 450 Hz. The output signal y[n] of the FIR filter is calculated as follows:

[0169]

[0170] where x[n] represents the input signal, h[k] is the impulse response of the filter, M represents the filter length, represents the current input signal and the previous M-1 sampling points. The amplitude normalization includes standardizing the amplitude range of the tactile data to [0, 1], ensuring the signal consistency between each sensor, and the normalization formula is as follows:

[0171]

[0172] where, and represent the maximum value and the minimum value of the current tactile data, respectively.

[0173] The feature extraction module extracts visual features through a convolutional neural network (CNN) and accelerates the calculation through an FPGA DSP unit, and extracts tactile features through a graph neural network (GNN) for graph convolution operation. The feature extraction module includes a visual feature extraction module and a tactile feature extraction module. The visual feature extraction module: extracts spatial features from image data using a convolutional neural network (CNN). To improve computational efficiency, the convolutional layer uses the Winograd optimization algorithm, which accelerates the calculation through the FPGA DSP unit and reduces the hardware resource occupation. Visual features are extracted through multiple convolutional layers, pooling layers, and activation layers. Through hardware optimization, the convolutional calculation is processed in parallel on the FPGA, improving the speed. The convolutional neural network (CNN) is composed of three convolutional layers, three activation layers, and three pooling layers alternately. The convolutional layer uses a 3x3 convolutional kernel to support dynamic channel numbers, and the convolution uses the Winograd optimization algorithm. The activation operation of the activation layer is completed through a hardware comparator. The pooling layer uses an FPGA comparator array to realize the maximum pooling operation to reduce the dimension of the feature map while retaining significant features. Quantization technology is used to convert the weights and activation values from floating-point numbers to low-bit numbers INT8.

[0174] The haptic feature extraction module uses a graph neural network (GNN) to model the spatio-temporal dependency of the haptic signal, from which the haptic force field features are extracted. Each haptic sensor is taken as a graph node, and the graph convolution calculation is accelerated by sparse matrix operation. The graph convolution network (GCN): the node represents the sensor position, and the edge weight represents the physical connection between the sensors. The graph neural network GNN is composed of three layers of graph convolution layers, and the haptic data is modeled as a graph structure, the node represents the sensor position, and the edge weight represents the adjacent sensor. According to the correlation between the position distribution of the haptic sensor and the contact force, the graph structure is dynamically generated, and the edge weight calculation formula of the graph is as follows:

[0175]

[0176] wherein, represents and the weight between , represents the physical coordinates of the nodes and , represents the weight control parameter. The node features are updated by domain propagation and aggregation:

[0177]

[0178] wherein, represents the features of the nodes of the layer, represents the neighborhood node set of the node , represents the weight matrix of the layer, (*) represents the activation function. The global feature of the graph is calculated by aggregating the features of all nodes:

[0179]

[0180] wherein, K represents the number of layers of the GNN, N represents the number of nodes, represents the global aggregation operation. The weight calculation is realized by hardware parallelization, and the edge weight is stored in the on-chip cache BRAM. The propagation formula of each layer is calculated by sparse matrix multiplication, and the update of the node features is calculated in parallel using a pipeline architecture. The activation function is realized by a hardware comparison unit. The node features are realized by an addition tree for global aggregation, and the feature vector after aggregation is stored in the register of the FPGA. The sparse optimization is used to calculate only the non-zero edge weight, and the index table is used to skip the zero weight edge.

[0181] The feature fusion module fuses the visual feature and the tactile feature through the multi-head attention mechanism of the Transformer architecture to generate a global feature. The multi-head attention mechanism captures the spatial and temporal dependencies between different modalities through parallel calculation of the weighted sum of multiple attention heads. The feature fusion module includes:

[0182] Module M3.1: Align the dimensions of the visual feature and the tactile feature, and unify the dimensions of the two features. The formula is as follows:

[0183]

[0184] wherein represents the aligned reconstructed visual feature, represents the aligned reconstructed tactile feature, and and are trainable weight matrices.

[0185] Module M3.2: The input features Q, K, and V are stored in the on-chip cache BRAM. The multi-attention mechanism formula is as follows:

[0186]

[0187] wherein Q, K, and V are query, key, and value matrices, respectively, and represents the dimension of the key vector.

[0188] Module M3.3: The visual feature and the tactile feature are mapped to the query, key, and value through the DSP unit, and are processed in parallel by row blocking. Multiple attention heads generate outputs based on parallel calculation:

[0189]

[0190] Each head: , , , is the projection weight matrix of the query, key, and value, respectively. represents the output transformation matrix.

[0191] Module M3.4: The result is divided by the square root of the dimension of the key vector for scaling. The square root calculation is implemented through a lookup table LUT, and the calculation is accelerated using a bit shift operation. The attention score mechanism is as follows:

[0192]

[0193] ​​​​​​​​Module M3.5: Use lookup table LUT to pre-store exponential values, quickly find the approximate result of each element exponential operation in Softmax.

[0194] Module M3.6: Parallel calculation of matrix multiplication by DSP unit, results are accumulated by head, stored in BRAM, weighted sum formula as follows:

[0195]

[0196] Module M3.7: The head outputs are spliced into a matrix by column in BRAM, and linear transformation of the output is performed using the DSP unit:

[0197] .

[0198] The reinforcement learning module generates control parameters based on the global features using an Actor-Critic architecture, optimizing the robot motion control strategy. The Actor module generates robot control actions based on the fused features. A fully connected neural network (FCN) is used for action generation, and FPGA hardware acceleration is used to improve computational efficiency. The Critic module evaluates the effectiveness of the current action and provides feedback through mean square error, optimizing the strategy of the Actor module. Training and optimization are performed through reinforcement learning algorithms (policy gradient method). In the Actor-Critic architecture, the Actor module generates control parameters based on global features to control the behavior of the robot. The Critic module evaluates the action effect and updates the strategy through mean square error to optimize control performance.

[0199] The motor drive module generates motor drive signals for robot motion based on the control parameters, and executes control through the motor to achieve the motion task. According to the action parameters generated by the Actor module, the motor control signal is generated through the PWM signal generation module to control the robot to achieve the motion task. The precise control signal of the motor is generated according to the control parameters to ensure the accuracy of the robot action. The control parameters include the control parameters output by the Actor module , and real-time data from position sensors, force / torque sensors. The control parameters include pose parameters and impedance parameters . The pose parameters are used to generate control parameters for servo control motors using PID control, with the formula as follows:

[0200]

[0201] where, represents the difference between the target value of the pose parameter and the feedback value of the pose parameter. The accumulator stores the integral value, which is updated in real time, and the difference is calculated with a sliding window, while the output of the PID controller for PWM signal generation. According to the impedance parameter Generate impedance control force:

[0202]

[0203] wherein, , represents the target position, the actual position, represents the real-time speed fed back by the sensor.

[0204] The present application realizes efficient and precise robot impedance adaptive control through FPGA hardware acceleration, combined with multi-modal sensor fusion, reinforcement learning control and robot motion control. The system design focuses on real-time performance and computing efficiency, and adopts parallel computing, hardware optimization and efficient data flow management, providing strong technical support for complex robot control tasks. Through the implementation of the present application, precise control and interaction of the robot can be realized in various dynamic environments, meeting the needs of high real-time performance, low delay and high precision.

[0205] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device, module and unit thereof in a pure computer readable program code manner, the same functions can also be realized by logically programming the method steps in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers. Therefore, the system provided by the present application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for realizing various functions can also be considered as structures within the hardware component; the devices, modules and units for realizing various functions can also be considered as both software modules realizing methods and structures within the hardware component.

[0206] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essential content of the present application. In the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for visual-tactile dual-channel control hardware for robot impedance adaptation, characterized in that, The method comprises the following steps: Step S1: acquiring visual data and tactile data, storing the visual data and tactile data and pre-processing the visual data and tactile data respectively; Step S2: extracting visual features through a convolutional neural network (CNN) and accelerating the calculation through an FPGA DSP unit, and extracting tactile features through a graph neural network (GNN) for graph convolution operation; Step S3: weighting and fusing the visual features and tactile features through a multi-head attention mechanism of a Transformer architecture to generate global features; Step S4: generating control parameters based on an Actor-Critic architecture according to the global features; Step S5: generating motor driving signals of robot movement based on the control parameters, and executing control through a motor to realize a movement task; The pre-processing of the visual data comprises the following steps: Step S101: converting the RGB format data in the visual data into grayscale format; Step S102: filtering the converted data using a 3*3 template and a fast median filtering algorithm; Step S103: using a histogram equalization module to instantiate a block random access memory (blockram) for equalizing the histogram, and the equalization formula is as follows: representing a gray level mapped new gray value, representing a gray level cumulative distribution function value; representing a maximum value of a gray level; representing a floor operation; The pre-processing of the tactile data comprises band-pass filtering and amplitude normalization; The band-pass filtering selects an FIR filter, calculates the output of the filter, filters out low-frequency drift and high-frequency noise in the tactile data, and only retains the effective signal frequency band of 20Hz to 450Hz, and the output signal y[n] of the FIR filter is calculated according to the formula: where x[n] represents the input signal, h[k] the impulse response of the filter, and M represents the filter length, representing the current input signal and the previous M-1 sample points; The amplitude normalization comprises standardizing the amplitude range of the tactile data to [0, 1] to ensure the signal consistency between the sensors, and the normalization formula is: wherein, and max and min represent the maximum and minimum values of the current haptic data, respectively.

2. The hardware-in-the-loop method for visual-haptic dual-channel control for robot impedance adaptation according to claim 1, wherein, The convolutional neural network (CNN) comprises three convolutional layers, three activation layers and three pooling layers alternately; The convolutional layer uses a 3*3 convolutional kernel to support a dynamic number of channels, and the convolution adopts a Winograd optimization algorithm; The activation operation of the activation layer is completed through a hardware comparator; The pooling layer uses an FPGA comparator array to realize a maximum pooling operation to reduce the dimension of the feature map while retaining significant features; Quantization technology is used to convert the weights and activation values from floating-point numbers to low-bit numbers INT8.

3. The hardware-in-the-loop method for visual-haptic dual-channel control for robot impedance adaptation according to claim 1, wherein, The graph neural network (GNN) comprises three layers of graph convolutional layers, the tactile data is modeled as a graph structure, the nodes represent sensor positions, and the edge weights represent adjacent sensors, the graph structure is dynamically generated according to the correlation between the position distribution of the tactile sensor and the contact force, and the edge weight calculation formula of the graph is as follows: wherein, denotes and between the weights, , denotes the physical coordinates of the nodes and , denotes the weight control parameter; the node features are updated by domain propagation and aggregation: wherein, represents the layer node characteristic, represents the neighborhood node set of the node represents the weight matrix of the (*) represents the activation function; The global features of the graph are calculated by aggregating the features of all nodes: where K represents the number of layers of the GNN, and N represents the number of nodes, denotes a global aggregation operation; The weight calculation is realized by hardware parallelization, and the edge weight The propagation formula of each layer is calculated by sparse matrix multiplication, the update of the node feature is calculated in parallel by using a pipeline architecture, the activation function is realized by a hardware comparison unit, the global aggregation of the node feature is realized by an addition tree, and the aggregated feature vector is stored in the register of the FPGA.

4. The hardware-in-the-loop method for visual-haptic dual-channel control for robot impedance adaptation according to claim 1, wherein, The step S3 comprises: Step S3.1: Aligning the dimensions of the visual features with the haptic features by dimension alignment, unifying the dimensions of both, as follows: , wherein, denotes the aligned reconstructed visual feature, denotes the aligned reconstructed haptic feature, and is a trainable weight matrix; Step S3.2: inputting the features Q, K and V into an on-chip buffer BRAM, and the multi-attention mechanism formula is as follows: where Q, K, V are Query, Key and Value matrices respectively, denotes the dimension of the Key vector. Step S3.3: mapping the visual features and tactile features into queries, keys and values through a DSP unit, processing in parallel according to rows, and generating outputs based on parallel calculation of multiple attention heads: Each head: , , , are projection weight matrices for queries, keys, and values, respectively; denotes the output transformation matrix; Step S3.4: dividing the results by the square root of the key vector dimension for scaling, implementing square root calculation through a lookup table (LUT), using a bit shift operation to accelerate the calculation, and the attention score mechanism: Step S3.5: using a lookup table LUT to pre-store exponential values, quickly finding the approximate result of the exponential operation of each element in Softmax; Step S3.6: performing parallel calculation on matrix multiplication by the DSP unit, accumulating the results by head, and storing them in the BRAM, and the weighted summation formula is as follows: Step S3.7: splicing the outputs of each head into a matrix by column in the BRAM, and performing linear transformation of the output by the DSP unit: 。 5. The hardware-in-the-loop method for visual-haptic dual-channel control for robot impedance adaptation according to claim 1, wherein, In the Actor-Critic architecture, the Actor module generates control parameters according to global features to control the behavior of the robot, and the Critic module evaluates the action effect and updates the strategy to optimize the control performance through mean square error.

6. The hardware-in-the-loop method for visual-haptic dual-channel control for robot impedance adaptation according to claim 1, wherein, The control parameters include control parameters output by an Actor module and real-time data from position sensors, force / torque sensors The control parameters including pose parameters and impedance parameters .

7. The hardware-in-the-loop method for visual-haptic dual-channel control of robot impedance adaptation according to claim 6, wherein, By means of the pose parameters The control parameters of the servo control motor are generated using PID control, and the formula is as follows: In the formula, represents the difference between the pose parameter target value and the pose parameter feedback value; The accumulator stores the integral value, which is updated in real time, and the difference is calculated with a sliding window, while the output of the PID controller for the generation of the PWM signal; According to the impedance parameter Generating impedance control force: In the formula, , represents the target position, the actual position, represents the real-time speed fed back by the sensor, and K and D represent the position gain coefficient and the speed gain coefficient, respectively.

8. A visual-tactile dual-channel control hardware system for robot impedance adaptation, characterized in that, It comprises: An FPGA data acquisition and preprocessing module for acquiring visual data and tactile data, storing the visual data and tactile data and preprocessing them respectively; A feature extraction module for extracting visual features through a convolutional neural network CNN and accelerating the calculation through an FPGA DSP unit, and extracting tactile features through a graph neural network GNN for graph convolution operation; A feature fusion module for weighting and fusing visual features and tactile features through a multi-head attention mechanism of a Transformer architecture to generate global features; A reinforcement learning module for generating control parameters based on an Actor-Critic architecture according to the global features; A motor driving module for generating motor driving signals for robot motion based on the control parameters, and executing control through a motor to achieve a motion task; The preprocessing of visual data comprises: Module M101: converting the RGB format data in the visual data to grayscale format; Module M102: filtering the converted data using a 3x3 template using a fast median filtering algorithm; Module M103: using a histogram equalization module to instantiate a block RAM for the filtered data, and performing histogram equalization operation, and the equalization formula is as follows: representing a gray level mapped new gray value, representing a gray level cumulative distribution function value; representing a maximum value of a gray level; representing a floor operation; The preprocessing of tactile data comprises band-pass filtering and amplitude normalization; The band-pass filtering selects a FIR filter to calculate the output of the filter, filters out low-frequency drift and high-frequency noise in the tactile data, and only retains the effective signal frequency band of 20Hz to 450Hz, and the output signal y[n] of the FIR filter is calculated as follows: where x[n] represents the input signal, h[k] the impulse response of the filter, and M represents the filter length, denotes the current input signal and the previous M-1 sample points; The amplitude normalization comprises normalizing the amplitude range of the tactile data to [0, 1] to ensure the signal consistency between the sensors, and the normalization formula is as follows: wherein, and max and min represent the maximum and minimum values of the current haptic data, respectively.

Images