Visual touch double-channel control hardware method and system for robot impedance self-adaption

Abstract

The invention provides a visual touch double-channel control hardware method and system for robot impedance self-adaption, and the method comprises the steps: obtaining visual data and touch data, storing the visual data and the touch data, and carrying out the preprocessing of the visual data and the touch data; visual features are extracted through a convolutional neural network CNN, calculation is accelerated through an FPGA DSP unit, and tactile features are extracted through a graph neural network GNN for graph convolution operation; performing weighted fusion on the visual features and the tactile features through a multi-head attention mechanism of a Transform architecture to generate global features; according to the global features, control parameters are generated based on an Actor-Critic architecture; and generating a motor driving signal of robot motion based on the control parameters, and performing control through a motor to realize a motion task. According to the invention, a software operation process of multi-modal visual and tactile information is converted into hardware circuit logic of FPGA parallel calculation, data calculation of the multi-modal information is implemented by using a hardware circuit, the processing efficiency is high, and noise between signals is low.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and more particularly to a hardware method and system for vision-touch dual-channel control of robot impedance adaptation. In particular, it relates to a hardware method and system for vision-touch fusion dual-channel control of robot impedance adaptation. Background Art

[0002] The widespread application of intelligent robots in industry, healthcare, and service sectors has placed higher demands on their ability to perceive and adapt to complex environments. Vision and touch, as two key channels for robots to perceive the outside world, each have their own advantages and limitations. Visual perception can provide global information about the environment but is susceptible to lighting variations and occlusions. Tactile perception directly reflects contact forces and surface properties, making it suitable for localized interaction scenarios but struggling to provide a global understanding of the environment.

[0003] Currently, robotic systems mostly complete motion control through single-modal perception of vision or touch. However, in dynamic and complex environments, relying solely on a single modality often cannot meet the requirements of real-time and robustness. The multimodal fusion of vision and touch has been proven to significantly improve the comprehensiveness and accuracy of perception. However, existing multimodal fusion systems mainly rely on GPUs or high-performance processors, which face the following problems in practical applications: Insufficient real-time performance: The computational delay in processing high-dimensional visual and tactile data is large, making it difficult to meet real-time control requirements. High power consumption: Existing multimodal fusion systems have low operating efficiency on embedded platforms and cannot run for a long time. Hardware architecture limitations: The parallel hardware characteristics are not fully utilized for algorithm optimization, resulting in low resource utilization.

[0004] To address these issues, FPGA-based hardware implementations are becoming an ideal choice. FPGAs, with their high parallel computing capabilities and low power consumption, offer new possibilities for the fusion processing and real-time control of visual and tactile data. However, specific methods for efficiently implementing multimodal fusion and motion control on FPGA hardware still require further research.

[0005] Patent document CN119550335A discloses a robot motion planning method based on multimodal information fusion, including: information acquisition: acquiring multimodal data of the robot through multiple sensors; preprocessing: normalizing, denoising and synchronizing the multimodal data; fusion: encoding and fusing the preprocessed data using a feature-level fusion method, and generating an environmental model based on an information fusion algorithm; motion planning: combining environmental modeling with a path planning algorithm to generate a motion path based on the fused multimodal information; feedback adjustment: dynamically adjusting the motion trajectory based on real-time perceived environmental changes using a feedback control mechanism.

[0006] However, the multimodal data fusion method in patent document CN119550335A suffers from significant computational delays when processing high-dimensional visual and tactile data, making it difficult to meet real-time control requirements. Furthermore, the patent document fails to fully utilize the characteristics of parallel hardware for algorithm optimization, relying solely on software algorithms to obtain multimodal information processing results, resulting in low resource utilization. The present invention transforms the software computation process for multimodal visual and tactile information into hardware circuit logic for FPGA parallel computing. This hardware circuit then performs multimodal data computation, resulting in high processing efficiency and minimal noise between signals. Summary of the Invention

[0007] In view of the defects in the prior art, the purpose of the present invention is to provide a visual-touch dual-channel control hardware method and system for robot impedance adaptation.

[0008] According to the present invention, a visual-touch dual-channel control hardware method for robot impedance self-adaptation is provided, comprising: Step S1: acquiring visual data and tactile data, storing the visual data and tactile data, and preprocessing them respectively; Step S2: Extract visual features through the convolutional neural network (CNN) and accelerate the calculation through the FPGA DSP unit, and extract tactile features through the graph neural network (GNN) to perform graph convolution operations; Step S3: Through the multi-head attention mechanism of the Transformer architecture, the visual features and tactile features are weightedly fused to generate global features; Step S4: generating control parameters based on the Actor-Critic architecture according to the global features; Step S5: generating a motor drive signal for the robot to move based on the control parameters, and performing control through the motor to achieve the motion task.

[0009] Preferably, the preprocessing of visual data includes: Step S101: converting the RGB format data in the visual data into grayscale format; Step S102: Using a 3×3 template and a fast median filter algorithm to filter the converted data; Step S103: A histogram equalization module is used to instantiate a random access memory (BlockRAM) for the filtered data, and an equalization operation is performed on the histogram. The equalization formula is as follows:

[0010] Indicates grayscale The new grayscale value after mapping, Indicates grayscale The cumulative distribution function value of ; Indicates the maximum value of the gray level; Indicates a floor operation.

[0011] Preferably, the preprocessing of the tactile data includes band-pass filtering and amplitude normalization; The band-pass filter selects an FIR filter, calculates the filter output, filters out low-frequency drift and high-frequency noise in the tactile data, and retains only the effective signal frequency band from 20 Hz to 450 Hz. The output signal y[n] of the FIR filter is calculated as follows:

[0012] Where x[n] represents the input signal, h[k] is the impulse response of the filter, and M represents the filter length. Represents the current input signal and the previous M-1 sampling points; The amplitude normalization includes normalizing the amplitude range of the tactile data to [0, 1] to ensure signal consistency between various sensors. The normalization formula is:

[0013] in, and Respectively represent the maximum and minimum values ​​of the current tactile data.

[0014] Preferably, the convolutional neural network (CNN) is composed of three convolutional layers, three activation layers, and three pooling layers alternatingly; The convolution layer uses a 3×3 convolution kernel to support dynamic channel numbers, and the convolution adopts the Winograd optimization algorithm; The activation operation of the activation layer is completed through the hardware comparator; The pooling layer uses an FPGA comparator array to implement a maximum pooling operation to reduce the dimensionality of the feature map while retaining significant features; Use quantization techniques to convert weights and activation values ​​from floating point numbers to low-bit INT8.

[0015] Preferably, the graph neural network GNN is composed of three layers of graph convolutional layers. The tactile data is modeled as a graph structure. The nodes represent the sensor positions and the edge weights represent the adjacent sensors. The graph structure is dynamically generated based on the correlation between the position distribution of the tactile sensors and the contact force. The edge weight calculation formula of the graph is as follows:

[0016] in, express and The weight between , Representation node and The physical coordinates of Represents the weight control parameter; node features are updated through domain propagation and aggregation:

[0017] in, Indicates the Layer Node Features, Representation node The set of neighboring nodes, Indicates the The weight matrix of the layer, (*) indicates activation function; The global features of the graph are calculated by aggregating the features of all nodes:

[0018] Among them, K represents the number of layers of GNN, N represents the number of nodes, Represents a global aggregation operation; Weight calculation is achieved through hardware parallelization, edge weight Stored in on-chip cache BRAM, the propagation formula of each layer is calculated through sparse matrix multiplication, and the node feature update is calculated in parallel using a pipeline architecture; the activation function is implemented through a hardware comparison unit; the node features are globally aggregated through an addition tree, and the aggregated feature vectors are stored in the FPGA register. Sparse optimization is used to calculate only non-zero edge weights, and zero-weight edges are skipped using an index table.

[0019] Preferably, step S3 includes: Step S3.1: Visual features and tactile characteristics Perform dimension alignment to unify the dimensions of the two. The formula is as follows: ,

[0020] in, represents the reconstructed visual features after alignment, represents the reconstructed tactile features after alignment, and is a trainable weight matrix; Step S3.2: The input features Q, K, V are stored in the on-chip cache BRAM. The multi-attention mechanism formula is as follows:

[0021] Where Q, K, and V are query, key, and value matrices respectively. represents the dimension of the key vector; Step S3.3: Visual features and tactile features are mapped into queries, keys, and values ​​through the DSP unit, and processed in parallel by row blocks. Multiple attention heads generate outputs based on parallel computing:

[0022] Each head: , 、 、 are the projection weight matrices for query, key, and value, respectively; represents the output transformation matrix; Step S3.4: Divide the result by the dimension of the square root key vector to scale it, implement the square root calculation through the lookup table LUT, use the shift operation to accelerate the calculation, and the attention score mechanism:

[0023] Step S3.5: Use the lookup table (LUT) to store the exponential values ​​and quickly find the approximate result of the exponential operation of each element in the Softmax. Step S3.6: The matrix multiplication is parallelized by the DSP unit, and the results are accumulated by head and stored in the BRAM. The weighted summation formula is as follows:

[0024] Step S3.7: Concatenate the outputs of each head into a matrix by column in the BRAM, and use the DSP unit to perform a linear transformation on the output: .

[0025] Preferably, the Actor module in the Actor-Critic architecture generates control parameters based on global features to control the behavior of the robot; the Critic module evaluates the effect of the action and optimizes the control performance through the mean square error update strategy.

[0026] Preferably, the control parameters include the control parameters output by the Actor module , as well as real-time data from position sensors and force / torque sensors; The control parameters Including pose parameters and impedance parameters .

[0027] Preferably, through the pose parameters Use PID control to generate the control parameters of the servo control motor. The formula is as follows:

[0028] Where, Indicates the difference between the pose parameter target value and the pose parameter feedback value; the accumulator stores the integral value, which is updated in real time. The sliding window is used to calculate the difference. At the same time, the output of the PID controller Used for PWM signal generation; According to the impedance parameters Generate impedance-controlled forces:

[0029] Where, 、 Indicates the target position, actual position, Indicates the real-time speed of sensor feedback.

[0030] According to the present invention, a visual-touch dual-channel control hardware system for robot impedance self-adaptation is provided, comprising: FPGA data acquisition and preprocessing module: acquires visual data and tactile data, stores the visual data and tactile data, and preprocesses them respectively; Feature extraction module: extracts visual features through convolutional neural networks (CNN) and accelerates calculations through FPGA DSP units, and extracts tactile features through graph neural networks (GNN) and performs graph convolution operations; Feature fusion module: Through the multi-head attention mechanism of the Transformer architecture, visual features and tactile features are weightedly fused to generate global features; Reinforcement learning module: generates control parameters based on the Actor-Critic architecture according to the global features; Motor drive module: generates motor drive signals for robot movement based on the control parameters, and performs control through the motor to achieve the movement task.

[0031] Compared with the prior art, the present invention has the following beneficial effects: 1. The FPGA-based vision and tactile fusion hardware system of the present invention is used for real-time impedance adaptation and posture control of robots in complex dynamic environments. Through hardware-accelerated processing of dual-channel multimodal data, the robot's adaptability to the external environment and control accuracy are improved.

[0032] 2. The present invention realizes rapid processing and feature extraction of visual and tactile data through FPGA parallel computing units and hardware acceleration modules, significantly improving computing efficiency.

[0033] 3. This invention enables high-precision motion control and can adjust motion strategies based on real-time sensor data to meet the demands of tasks in complex environments. With strong real-time performance, low latency, and high precision, it is suitable for complex robotic control tasks, particularly in dynamic interaction and multimodal perception tasks, showing broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings: Figure 1 It is a schematic flow chart of the working method of the present invention; Figure 2 This is a schematic diagram of the hardware system feature extraction module architecture; Figure 3 This is a schematic diagram of the hardware system feature fusion module architecture; Figure 4 Schematic diagram of the hardware system reinforcement learning module architecture. DETAILED DESCRIPTION

[0035] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0036] This invention uses a CMOS camera and surface electromyography sensors to collect visual and tactile data, and uses FPGA data processing to perform real-time preprocessing on the collected data. Visual features are extracted using a convolutional neural network (CNN), and tactile features are extracted using a graph neural network (GNN). These features are then fused using the Transformer algorithm's multi-head attention mechanism to generate a high-dimensional feature representation with global spatiotemporal dependencies. Control parameters are then generated based on an actor-critic architecture to optimize the robot's motion control strategy. Finally, motor drive signals for the robot's motion are generated based on these control parameters, and the motors are used to execute control to achieve the motion task.

[0037] The FPGA hardware in this invention utilizes the Xilinx FPGA platform, employing FPGA DSP units to accelerate all matrix operations, convolution calculations, and graph convolution calculations. The RAM storage module is used to store intermediate calculation results, network weights, and training parameters, reducing memory access latency and improving data processing efficiency. The parallel computing unit utilizes a pipeline architecture and parallel computing units to simultaneously process multiple computing tasks, achieving efficient parallel computing. The clock management module precisely controls the timing of each module, ensuring data synchronization between modules.

[0038] Example 1 According to the present invention, a visual-touch dual-channel control hardware method for robot impedance self-adaptation is provided. Figure 1 Shown, including: Step S1: Acquire visual data and tactile data, store the visual data and tactile data, and pre-process them separately. Visual data, including environmental image data, is collected via a CMOS camera; tactile data, including tactile signals, is acquired via a surface electromyography sensor and a force / displacement sensor.

[0039] The preprocessing of visual data includes color space conversion, noise filtering and histogram equalization. The steps include: Step S101: converting the RGB format data in the visual data into grayscale format; Step S102: Using a 3×3 template and a fast median filter algorithm to filter the converted data.

[0040] Step S103: A histogram equalization module is used to instantiate a random access memory (BlockRAM) for the filtered data, and an equalization operation is performed on the histogram. The equalization formula is as follows:

[0041] Indicates grayscale The new grayscale value after mapping, Indicates grayscale The cumulative distribution function value of ; Indicates the maximum value of the gray level; Indicates a floor operation.

[0042] Tactile data preprocessing includes bandpass filtering and amplitude normalization. Bandpass filtering uses an FIR filter, uses a shift register to store the most recent M input signals, dynamically updates the data, and uses multiple parallel multipliers to simultaneously calculate h[k]*x[nk]. A pipeline structure then accumulates all these products to generate the filter output y[n]. This removes low-frequency drift and high-frequency noise from the tactile signal, retaining only the valid signal frequency band from 20Hz to 450Hz.

[0043] Assume the input signal is x[n], the filter coefficient is h[k] (filter length is M), the output signal y[n] of the FIR filter is calculated as follows:

[0044] in represents the current input signal and the previous M-1 sampling points. h[k] represents the impulse response of the filter.

[0045] Amplitude normalization includes normalizing the amplitude range of the tactile signal to [0, 1] to ensure signal consistency between sensors and facilitate subsequent processing and feature extraction. The maximum value of the current signal is calculated through the hardware comparator. and minimum value , using the subtraction unit and division pipeline to complete the normalization calculation. At the same time, fixed-point operations are used instead of floating-point operations, and the on-chip register (BRAM) is used to store the maximum value to reduce the storage access delay.

[0046] Normalization implementation formula:

[0047] Step S2: extract visual features, i.e., spatial features of visual data, through convolutional neural network (CNN), and accelerate calculation through FPGA DSP unit to improve calculation efficiency. Extract tactile features, i.e., spatiotemporal features of tactile data, through graph neural network (GNN), and perform graph convolution operation. Figure 2 As shown in the figure, the convolutional neural network (CNN) is used to extract spatial features in the image, and at the same time, visual features such as edges, textures, and shapes in the image are identified. The Winograd optimization algorithm is used to accelerate convolution calculations and reduce hardware resource usage. The spatial and temporal dependencies of tactile signals are modeled through the graph neural network (GNN), tactile force field features are extracted, and sparse matrix operations are used to accelerate graph convolution calculations.

[0048] The convolutional neural network (CNN) consists of three alternating convolutional layers, three activation layers, and three pooling layers. The convolutional layers use 3×3 convolution kernels to support a dynamic number of channels, and the convolutions employ the Winograd optimization algorithm to reduce the number of multiplications. Activation operations in the activation layers are performed using hardware comparators; the pooling layers use an FPGA comparator array to implement a maximum pooling operation (2×2 window) to reduce the dimensionality of the feature map while preserving significant features. Quantization techniques are used to convert weights and activation values ​​from floating-point numbers to low-bit integers (INT8) to reduce computational complexity.

[0049] The graph neural network (GNN) consists of three layers of graph convolutional layers. The tactile data is modeled as a graph structure, where nodes represent sensor locations and edge weights represent adjacent sensors. The graph structure is dynamically generated based on the correlation between the location distribution of tactile sensors and contact force. The edge weight calculation formula of the graph is as follows:

[0050] in express and The weight between , Representation node and The physical coordinates of Represents the weight control parameter.

[0051] Node features are updated through domain propagation and aggregation:

[0052] in Indicates the Layer Node characteristics; Representation node The set of neighboring nodes; Indicates the The weight matrix of the layer; (*) indicates the activation function.

[0053] The global features of the graph are calculated by aggregating the features of all nodes:

[0054] K represents the number of layers of GNN; N represents the number of nodes; Represents a global aggregation operation.

[0055] Weight calculation is achieved through hardware parallelization, edge weight The data is stored in the on-chip cache (BRAM). The propagation formula of each layer is calculated through sparse matrix multiplication, and the node feature updates are calculated in parallel using a pipeline architecture. The activation function is implemented through a hardware comparison unit. The node features are globally aggregated through an addition tree, and the aggregated feature vectors are stored in the FPGA register. Sparse optimization is used to calculate only non-zero edge weights, and an index table is used to skip zero-weight edges to reduce the amount of computation.

[0056] Step S3: Through the multi-head attention mechanism of the Transformer architecture, the visual features and tactile features are weighted and fused to generate global features. Figure 3 As shown, the Transformer fusion algorithm is adopted, and the Transformer algorithm with a multi-head attention mechanism is used to dynamically weight the visual features and tactile features. By computing the weighted sum of multiple attention heads in parallel, the spatiotemporal dependencies between different modalities are captured, and the weights of each modal feature are adjusted according to the different requirements of the task, thereby generating a high-dimensional feature representation with global spatiotemporal dependencies. The step S3 includes: Step S3.1: Visual features and tactile characteristics Perform dimension alignment to unify the dimensions of the two. The formula is as follows: ,

[0057] in, represents the reconstructed visual features after dimension alignment, represents the reconstructed tactile features after dimension alignment, and is the trainable weight matrix.

[0058] Step S3.2: The input features Q, K, V are stored in the on-chip cache BRAM. The multi-head attention mechanism formula is as follows:

[0059] Where Q, K, and V are query, key, and value matrices respectively. Indicates the dimension of the key vector.

[0060] Step S3.3: Visual features and tactile features are mapped into queries, keys, and values ​​through the DSP unit, and processed in parallel by row blocks. Multiple attention heads generate outputs based on parallel computing:

[0061] Each head: , 、 、 are the projection weight matrices for query, key, and value, respectively; Represents the output transformation matrix.

[0062] Step S3.4: Divide the result by the dimension of the square root key vector for scaling, implement square root calculation through the lookup table LUT, and use bit shift operation to accelerate the calculation.

[0063] Attention scoring mechanism:

[0064] Step S3.5: Use the lookup table (LUT) to store the exponential values ​​and quickly find the approximate result of the exponential operation for each element in the softmax. The denominator summation and normalization operations are completed in the adder tree.

[0065] Step S3.6: The matrix multiplication is parallelized by the DSP unit, and the results are accumulated by head and stored in the BRAM. The weighted summation formula is as follows:

[0066] Step S3.7: Concatenate the outputs of each head into a matrix by column in the BRAM, and use the DSP unit to perform a linear transformation on the output:

[0067] Step S4: Generate actions based on the global features and evaluate the effects of the actions to optimize the control strategy. Figure 4 As shown in the figure, reinforcement learning-based control introduces a reinforcement learning hardware module (Actor-Critic architecture). The Actor module generates control parameters based on global features to control the robot's behavior; the Critic module evaluates the effect of the action and updates the strategy through the mean square error to optimize the control performance.

[0068] The on-chip cache BRAM receives the global feature vector of the Transformer fusion module and then normalizes it to ensure that the input distribution is suitable for hardware calculation. The Actor-Critic architecture is a method in reinforcement learning. The algorithm is described as follows: The Actor module generates the control parameter a based on the current state s (represented by the global variable F):

[0069] The Critic module calculates the value function based on the current state s :

[0070] Evaluate the current status and output a value score.

[0071] Advantage calculation of gradient optimization:

[0072] Where r represents the current reward, which is generated by sensor feedback; represents the discount factor; Indicates the next state.

[0073] Update the Actor via policy gradient.

[0074] Critic is updated by mean square error:

[0075] The linear transformation of the Actor network is implemented by the DSP unit:

[0076] The generated probability distribution is sampled to obtain the control parameters of specific actions.

[0077] The Critic network calculates the state value through the DSP unit:

[0078] Use the adder and multiplier pipeline to calculate the loss function and update the critic network weights.

[0079] Step S5: Based on the control parameters, a motor drive signal is generated to control the robot's movement. The motor is then controlled to achieve the motion task. The PWM signal generation module generates a motor control signal to drive the robot to perform the motion task. Sensor feedback adjusts the motor motion to ensure accurate execution.

[0080] The control parameters include the control parameters output by the Actor module , including pose parameters (target position and attitude angle) and impedance parameters (stiffness and damping), as well as real-time data from position sensors and force / torque sensors for closed-loop control.

[0081] Use the adder to calculate the difference between the target value and the feedback value:

[0082] Generate control parameters for the servo motor using PID control:

[0083] The accumulator stores the integral value, which is updated in real time. The sliding window is used to calculate the difference. At the same time, the output of the PID controller Used for PWM signal generation. The PWM signal generates fixed-frequency pulses through a counter, while adjusting the high and low level outputs according to the duty cycle.

[0084] Generate impedance-controlled forces based on stiffness and damping parameters:

[0085] 、 Indicates target and actual position; Indicates the real-time speed of sensor feedback. The stiffness term and damping term are calculated by the multiplier, and the stiffness and damping terms are accumulated by the adder to generate , K and D are called position gain coefficient and speed gain coefficient respectively.

[0086] Receive calculated , and stored in the FPGA on-chip cache (BRAM), using a multiplier and adder pipeline to complete the calculation from force to torque and speed, and converting force into duty cycle through fixed-point operations to ensure real-time calculation. The PWM signal generates a fixed-frequency pulse through a counter, and adjusts the high and low level outputs according to the duty cycle.

[0087] This invention aims to address the existing challenges of poor real-time performance, high power consumption, and insufficient robustness. By integrating visual and tactile data preprocessing, feature extraction, fusion, and control signal generation modules within FPGA hardware, it enables efficient robot adaptation to dynamic environments. Through hardware acceleration and multimodal sensor fusion, efficient and precise robot control is achieved.

[0088] Example 2 The present invention also provides a visual-touch dual-channel control hardware system for robot impedance self-adaptation. The visual-touch dual-channel control hardware system for robot impedance self-adaptation can be implemented by executing the process steps of the visual-touch dual-channel control hardware method for robot impedance self-adaptation, that is, those skilled in the art can understand the visual-touch dual-channel control hardware method for robot impedance self-adaptation as a preferred implementation of the visual-touch dual-channel control hardware system for robot impedance self-adaptation.

[0089] According to the present invention, a visual-touch dual-channel control hardware system for robot impedance adaptation includes: an FPGA data acquisition and preprocessing module, a feature extraction module, a feature fusion module, a reinforcement learning module, and a motor drive module.

[0090] The FPGA data acquisition and preprocessing module acquires visual and tactile data, stores them, and preprocesses them separately. The FPGA data acquisition and preprocessing module includes a CMOS camera data acquisition module and a surface electromyography sensor data acquisition module. The CMOS camera data acquisition module includes a camera driver module, a data read / write control module, a color space conversion module, a median filter module, and a histogram equalization module. The CMOS camera acquires visual data, generates a video stream, and transmits it to the visual data channel of the FPGA data processing module via a digital interface. Simultaneously, the sEMG sensor monitors electromyographic signals in real time, captures tactile feedback, converts the analog signals into digital signals via an analog-to-digital converter (ADC), and transmits them to the tactile data channel of the FPGA data processing module. The data read / write module controls the reading and writing of the collected visual and tactile data to and from the DDR3 memory module, which also stores the collected visual and tactile data. The camera driver module controls the operation of the CMOS camera via an interface and acquires image data in real time. The color space conversion module converts the collected RGB format data into a grayscale image for subsequent feature extraction. Filtering module: Uses the median filter algorithm to remove noise from the image and ensure image clarity. Histogram equalization module: Enhances the contrast of the image, making the features more prominent and facilitating subsequent processing. The preprocessing of visual data includes: Module M101: Converts the RGB format data in the visual data into grayscale format. Module M102: Uses a 3×3 template and a fast median filter algorithm to filter the converted data. Module M103: Uses the histogram equalization module to instantiate a random access memory blockram on the filtered data and perform equalization on the histogram. The equalization formula is as follows:

[0091] Indicates grayscale The new grayscale value after mapping, Indicates grayscale The cumulative distribution function value of . Indicates the maximum value of the grayscale level. Indicates a floor operation.

[0092] The surface electromyography sensor data acquisition module includes a data reading and writing module, a bandpass filtering module, and an amplitude normalization module. Tactile signals are obtained through surface electromyography sensors and force / displacement sensors, and the analog signals are converted into digital signals through an analog-to-digital converter (ADC). The bandpass filtering module filters out low-frequency drift and high-frequency noise in the tactile signal, retaining only the valid frequency band. The amplitude normalization module standardizes the amplitude range of the tactile signal to [0,1] to ensure that the signals of each sensor are consistent and facilitate subsequent processing. The preprocessing of tactile data includes bandpass filtering and amplitude normalization. The bandpass filtering selects an FIR filter, calculates the output of the filter, filters out low-frequency drift and high-frequency noise in the tactile data, and retains only the valid signal frequency band from 20Hz to 450Hz. The output signal y[n] of the FIR filter is calculated as follows:

[0093] Where x[n] represents the input signal, h[k] is the impulse response of the filter, and M represents the filter length. Represents the current input signal and the previous M-1 sampling points. The amplitude normalization includes normalizing the amplitude range of the tactile data to [0, 1] to ensure the signal consistency between each sensor. The normalization formula is:

[0094] in, and Respectively represent the maximum and minimum values ​​of the current tactile data.

[0095] The feature extraction module extracts visual features using a convolutional neural network (CNN) and accelerates computation using the FPGA DSP unit. It extracts tactile features using a graph neural network (GNN) and performs graph convolution operations. The feature extraction module consists of a visual feature extraction module and a tactile feature extraction module. The visual feature extraction module uses a convolutional neural network (CNN) to extract spatial features from image data. To improve computational efficiency, the convolution layer uses the Winograd optimization algorithm, which is accelerated by the FPGA DSP unit to reduce hardware resource usage. Visual features are extracted through multiple convolutional, pooling, and activation layers. Hardware optimization enables parallel processing of convolution operations on the FPGA, improving speed. The convolutional neural network (CNN) consists of three alternating convolutional, activation, and pooling layers. The convolution layers use 3×3 convolution kernels to support a dynamic number of channels, and the convolutions use the Winograd optimization algorithm. Activation operations in the activation layers are performed using hardware comparators. The pooling layer uses the FPGA comparator array to implement max pooling operations to reduce the dimensionality of the feature map while preserving significant features. Quantization techniques are used to convert weights and activation values ​​from floating-point numbers to low-bit INT8.

[0096] The tactile feature extraction module uses a graph neural network (GNN) to model the spatiotemporal dependencies of tactile signals and extract tactile force field features from them. Each tactile sensor is used as a graph node, and sparse matrix operations are used to accelerate graph convolution calculations. Graph Convolutional Network (GCN): Nodes represent sensor locations, and edge weights represent the physical connections between sensors. The GNN consists of three layers of graph convolution layers. Tactile data is modeled as a graph structure, with nodes representing sensor locations and edge weights representing adjacent sensors. The graph structure is dynamically generated based on the correlation between the position distribution of tactile sensors and contact force. The graph edge weight calculation formula is as follows:

[0097] in, express and The weight between , Representation node and The physical coordinates of Represents the weight control parameter. Node features are updated through domain propagation and aggregation:

[0098] in, Indicates the Layer Node Features, Representation node The set of neighboring nodes, Indicates the The weight matrix of the layer, (*) indicates the activation function. The global features of the graph are calculated by aggregating the features of all nodes:

[0099] Among them, K represents the number of layers of GNN, N represents the number of nodes, Represents a global aggregation operation. Weight calculation is implemented through hardware parallelization, and edge weight The data is stored in on-chip BRAM cache. The propagation formula for each layer is calculated using sparse matrix multiplication, and node feature updates are parallelized using a pipelined architecture. Activation functions are implemented using hardware comparison units. Node features are globally aggregated using an adder tree, and the aggregated feature vectors are stored in FPGA registers. Sparse optimization is employed to calculate only non-zero edge weights, and zero-weight edges are skipped using an index table.

[0100] The feature fusion module uses the Transformer architecture's multi-head attention mechanism to perform a weighted fusion of visual and tactile features to generate global features. The multi-head attention mechanism captures the spatiotemporal dependencies between different modalities by computing the weighted sum of multiple attention heads in parallel. The feature fusion module includes: Module M3.1: Visual Features and tactile characteristics Perform dimension alignment to unify the dimensions of the two. The formula is as follows: ,

[0101] in, represents the reconstructed visual features after alignment, represents the reconstructed tactile features after alignment, and is the trainable weight matrix.

[0102] Module M3.2: Input features Q, K, V are stored in the on-chip cache BRAM. The multi-attention mechanism formula is as follows:

[0103] Where Q, K, and V are query, key, and value matrices respectively. Indicates the dimension of the key vector.

[0104] Module M3.3: Maps visual and tactile features into queries, keys, and values ​​through DSP units, processes them in parallel by row, and generates outputs based on parallel computation by multiple attention heads:

[0105] Each head: , 、 、 are the projection weight matrices for query, key, and value respectively. Represents the output transformation matrix.

[0106] Module M3.4: Divide the result by the dimension of the square root key vector to scale, implement square root calculation through the lookup table LUT, use bit shift operation to accelerate calculation, attention scoring mechanism:

[0107] Module M3.5: Use the lookup table (LUT) to pre-store exponential values ​​and quickly find the approximate result of the exponential operation of each element in Softmax.

[0108] Module M3.6: Matrix multiplication is performed in parallel through the DSP unit. The results are accumulated by head and stored in BRAM. The weighted summation formula is as follows:

[0109] Module M3.7: Concatenate the outputs of each head into a matrix by column in the BRAM and use the DSP unit to perform linear transformation on the output: .

[0110] The reinforcement learning module generates control parameters based on the global features using an actor-critic architecture to optimize the robot's motion control strategy. The actor module generates robot control actions based on the fused features. Action generation is performed using a fully connected neural network (FCN), with FPGA hardware acceleration improving computational efficiency. The critic module evaluates the effectiveness of the current action and provides feedback using mean squared error (MSE) to optimize the actor module's strategy. Training and optimization are performed using a reinforcement learning algorithm (policy gradient method). In the actor-critic architecture, the actor module generates control parameters based on global features to control the robot's behavior. The critic module evaluates the effectiveness of the action and uses the mean squared error to update the strategy to optimize control performance.

[0111] The motor drive module generates the motor drive signal for the robot's motion based on the control parameters, and controls the motor to achieve the motion task. Based on the action parameters generated by the Actor module, the PWM signal generation module generates the motor control signal to control the robot to achieve the motion task. Based on the control parameters, the precise control signal for the motor is generated to ensure the accuracy of the robot's motion. The control parameters include the control parameters output by the Actor module. , and real-time data from position sensors and force / torque sensors. The control parameters Including pose parameters and impedance parameters . Through the pose parameters Use PID control to generate the control parameters of the servo control motor. The formula is as follows:

[0112] Where, Indicates the difference between the target value of the posture parameter and the feedback value of the posture parameter. The accumulator stores the integral value, which is updated in real time. The sliding window is used to calculate the difference. At the same time, the output of the PID controller Used for PWM signal generation. According to the impedance parameters Generate impedance-controlled forces:

[0113] Where, 、 Indicates the target position, actual position, Indicates the real-time speed of sensor feedback.

[0114] This invention achieves efficient and precise adaptive impedance control of robots through FPGA hardware acceleration, combined with multimodal sensor fusion, reinforcement learning control, and robotic motion control. The system design focuses on real-time performance and computational efficiency, employing parallel computing, hardware optimization, and efficient data flow management to provide strong technical support for complex robotic control tasks. This implementation enables precise control and interaction of robots in various dynamic environments, meeting the requirements of high real-time performance, low latency, and high precision.

[0115] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0116] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A visual-touch dual-channel control hardware method for robot impedance adaptation, characterized in that: include: Step S1: acquiring visual data and tactile data, storing the visual data and tactile data, and preprocessing them respectively; Step S2: Extract visual features through the convolutional neural network (CNN) and accelerate the calculation through the FPGA DSP unit, and extract tactile features through the graph neural network (GNN) to perform graph convolution operations; Step S3: Through the multi-head attention mechanism of the Transformer architecture, the visual features and tactile features are weightedly fused to generate global features; Step S4: generating control parameters based on the Actor-Critic architecture according to the global features; Step S5: generating a motor drive signal for the robot to move based on the control parameters, and performing control through the motor to achieve the motion task.

2. The visual-touch dual-channel control hardware method for robot impedance adaptation according to claim 1, characterized in that: The preprocessing of visual data includes: Step S101: converting the RGB format data in the visual data into grayscale format; Step S102: Using a 3×3 template and a fast median filter algorithm to filter the converted data; Step S103: A histogram equalization module is used to instantiate a random access memory (BlockRAM) for the filtered data, and an equalization operation is performed on the histogram. The equalization formula is as follows: Indicates grayscale The new grayscale value after mapping, Indicates grayscale The cumulative distribution function value of ; Indicates the maximum value of the gray level; Indicates a floor operation.

3. The visual-touch dual-channel control hardware method for robot impedance adaptation according to claim 1, characterized in that: The preprocessing of tactile data includes bandpass filtering and amplitude normalization; The band-pass filter selects an FIR filter, calculates the filter output, filters out low-frequency drift and high-frequency noise in the tactile data, and retains only the effective signal frequency band from 20 Hz to 450 Hz. The output signal y[n] of the FIR filter is calculated as follows: Where x[n] represents the input signal, h[k] is the impulse response of the filter, and M represents the filter length. Represents the current input signal and the previous M-1 sampling points; The amplitude normalization includes normalizing the amplitude range of the tactile data to [0, 1] to ensure signal consistency between various sensors. The normalization formula is: in, and Respectively represent the maximum and minimum values ​​of the current tactile data.

4. The visual-touch dual-channel control hardware method for robot impedance adaptation according to claim 1, characterized in that: The convolutional neural network CNN is composed of three convolutional layers, three activation layers and three pooling layers alternating; The convolution layer uses a 3×3 convolution kernel to support dynamic channel numbers, and the convolution adopts the Winograd optimization algorithm; The activation operation of the activation layer is completed through the hardware comparator; The pooling layer uses an FPGA comparator array to implement a maximum pooling operation to reduce the dimensionality of the feature map while retaining significant features; Use quantization techniques to convert weights and activation values ​​from floating point numbers to low-bit INT8.

5. The visual-touch dual-channel control hardware method for robot impedance self-adaptation according to claim 1, characterized in that: The graph neural network (GNN) consists of three layers of graph convolutional layers. The tactile data is modeled as a graph structure, where nodes represent sensor locations and edge weights represent adjacent sensors. The graph structure is dynamically generated based on the correlation between the location distribution of tactile sensors and contact force. The edge weight calculation formula of the graph is as follows: in, express and The weight between , Representation node and The physical coordinates of Represents the weight control parameter; node features are updated through domain propagation and aggregation: in, Indicates the Layer Node Features, Representation node The set of neighboring nodes, Indicates the The weight matrix of the layer, (*) indicates activation function; The global features of the graph are calculated by aggregating the features of all nodes: Among them, K represents the number of layers of GNN, N represents the number of nodes, Represents a global aggregation operation; Weight calculation is achieved through hardware parallelization, edge weight Stored in on-chip cache BRAM, the propagation formula of each layer is calculated through sparse matrix multiplication, and the node feature update is calculated in parallel using a pipeline architecture; the activation function is implemented through a hardware comparison unit; the node features are globally aggregated through an addition tree, and the aggregated feature vectors are stored in the FPGA register. Sparse optimization is used to calculate only non-zero edge weights, and zero-weight edges are skipped using an index table.

6. The visual-touch dual-channel control hardware method for robot impedance adaptation according to claim 1, characterized in that: The step S3 comprises: Step S3.1: Visual features and tactile characteristics Perform dimension alignment to unify the dimensions of the two. The formula is as follows: , in, represents the reconstructed visual features after alignment, represents the reconstructed tactile features after alignment, and is a trainable weight matrix; Step S3.2: The input features Q, K, V are stored in the on-chip cache BRAM. The multi-attention mechanism formula is as follows: Where Q, K, and V are query, key, and value matrices respectively. represents the dimension of the key vector; Step S3.3: Visual features and tactile features are mapped into queries, keys, and values ​​through the DSP unit, and processed in parallel by row blocks. Multiple attention heads generate outputs based on parallel computing: Each head: , 、 、 are the projection weight matrices for query, key, and value, respectively; represents the output transformation matrix; Step S3.4: Divide the result by the dimension of the square root key vector to scale it, implement the square root calculation through the lookup table LUT, use the shift operation to accelerate the calculation, and the attention score mechanism: Step S3.5: Use the lookup table (LUT) to store the exponential values ​​and quickly find the approximate result of the exponential operation of each element in the Softmax. Step S3.6: The matrix multiplication is parallelized by the DSP unit, and the results are accumulated by head and stored in the BRAM. The weighted summation formula is as follows: Step S3.7: Concatenate the outputs of each head into a matrix by column in the BRAM, and use the DSP unit to perform a linear transformation on the output: 。 7. The visual-touch dual-channel control hardware method for robot impedance adaptation according to claim 1, characterized in that: In the Actor-Critic architecture, the Actor module generates control parameters based on global features to control the robot's behavior; the Critic module evaluates the effect of the action and optimizes the control performance through a mean square error update strategy.

8. The visual-touch dual-channel control hardware method for robot impedance adaptation according to claim 1, characterized in that: The control parameters include the control parameters output by the Actor module , as well as real-time data from position sensors and force / torque sensors; The control parameters Including pose parameters and impedance parameters .

9. The visual-touch dual-channel control hardware method for robot impedance self-adaptation according to claim 8, characterized in that: Through pose parameters Use PID control to generate the control parameters of the servo control motor. The formula is as follows: Where, Indicates the difference between the pose parameter target value and the pose parameter feedback value; The accumulator stores the integral value, which is updated in real time. The sliding window is used to calculate the difference. At the same time, the output of the PID controller Used for PWM signal generation; According to the impedance parameters Generate impedance-controlled forces: Where, 、 Indicates the target position, actual position, It represents the real-time speed of sensor feedback, K and D represent the position gain coefficient and speed gain coefficient respectively.

10. A visual-touch dual-channel control hardware system for robot impedance adaptation, characterized in that: include: FPGA data acquisition and preprocessing module: acquires visual data and tactile data, stores the visual data and tactile data, and preprocesses them respectively; Feature extraction module: extracts visual features through convolutional neural networks (CNN) and accelerates calculations through FPGA DSP units, and extracts tactile features through graph neural networks (GNN) and performs graph convolution operations; Feature fusion module: Through the multi-head attention mechanism of the Transformer architecture, visual features and tactile features are weightedly fused to generate global features; Reinforcement learning module: generates control parameters based on the Actor-Critic architecture according to the global features; Motor drive module: generates motor drive signals for robot movement based on the control parameters, and performs control through the motor to achieve the movement task.

Images