Multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation

Abstract

The present invention discloses a multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation, comprising a post-disaster detection robot and a remote control platform. The system is combined with a gesture recognition algorithm based on vision-electromyography signal fusion, a robot contactless remote control algorithm based on vision teleoperation, a camera adaptive adjustment algorithm based on multi-sensor fusion, and a visualization interface construction technology to build a post-disaster detection robot system. The remote control platform recognizes the robot's teleoperation gestures by algorithmically fusing the myoelectric signals collected by the MYO bracelet with the gesture signals collected by the camera. The remote control platform performs algorithmic analysis on the position of the sensing glove captured by the camera, and sends the coordinate information of the obtained marker in the image to the robot. The control board sends movement instructions based on the coordinate information to control the motor drive board, which then drives the motor to realize the movement of the robot and conduct injury search.

Description

Technical Field

[0001] The present invention relates to robot remote control technology, and in particular to a multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation. Background Art

[0002] Post-disaster detection, as a crucial component of post-disaster rescue efforts, directly determines rescue efficiency. However, the post-disaster environment is often complex and fraught with danger, posing significant risks to rescue workers and significantly reducing rescue efficiency. Therefore, ensuring the safety of rescue workers and efficiently rescuing affected victims are crucial considerations.

[0003] With the development of artificial intelligence (AI), intelligent robots are now being applied in various fields. However, due to the confined spaces, unstable structures, and toxic gases typically found in post-disaster environments, automated navigation technology is difficult to apply. Therefore, teleoperation has become the preferred choice for post-disaster search and rescue. However, a fully mature technical framework for visual teleoperation and fusion control algorithms has yet to be established.

[0004] According to the survey, the current post-disaster remote control robots in the market have the following main problems:

[0005] 1. Severe terrain constraints. Post-disaster environments are often complex, shifting, and intricate. Remotely operated robots face rugged and steep terrain, making navigation difficult. Furthermore, during detection, the robot's camera has poor adaptability to the terrain, resulting in poor monitoring results.

[0006] 2. Complex operation. Traditional button and joystick remote control devices are complex to operate and have numerous buttons. Myoelectric remote control devices can only achieve direction control, but have difficulty in achieving speed control, and have a limited number of function commands.

[0007] 3. Visual teleoperation is easily affected by factors such as lighting. Visual teleoperation mainly uses gesture information to instruct the robot, making it difficult to control speed. In addition, the gesture recognition process is greatly affected by factors such as lighting.

[0008] This invention takes the safety of post-disaster rescue personnel and the efficient rescue of disaster victims as its starting point. In order to solve the problems of complex post-disaster terrain, narrow space, expensive remote control equipment, and complex operation, a multi-terrain post-disaster detection robot system based on vision-muscle fusion remote control is proposed. The fusion technology based on vision and electromyography signals is studied to realize the joint control of the robot by vision-myoelectricity remote control; and a visual interface is further developed to realize remote monitoring and decision-making. Summary of the Invention

[0009] In order to solve the deficiencies in the prior art, the object of the present invention is to provide a multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation.

[0010] To achieve the purpose of the present invention, the technical solution adopted by the present invention is:

[0011] A multi-terrain post-disaster exploration robot system based on vision-muscle fusion teleoperation includes a post-disaster exploration robot and a remote control platform. The post-disaster exploration robot includes a housing, a microcontroller, a mobile chassis, and a two-axis robotic arm. The housing is fixed to the mobile chassis, and the two-axis robotic arm is arranged at the front end of the housing. The remote working platform includes remote sensing gloves, a MYO wristband, a camera, a Bluetooth module, and a PC.

[0012] The PC in the remote control platform uses an algorithm to analyze the position of the sensing glove captured by the camera to capture the calibration object on the sensing glove. The coordinate information of the obtained marker in the image is then sent to the control board on the robot's mobile chassis via the Bluetooth module. Based on this coordinate information and its own GPS positioning information, the control board sends movement instructions to control the motor driver board, which then drives the motor to realize the movement of the robot and conduct injury search.

[0013] A camera is installed on the two-axis robotic arm to capture the disaster scene. The microcontroller integrates the YOLOVX algorithm to identify injured targets in the disaster scene captured by the camera. When an injured person is found, the current coordinates are sent to the PC, which then sends the coordinates to the rescuers.

[0014] The PC recognizes the robot's remote control gestures by performing algorithmic fusion on the electromyographic signals collected by the MYO bracelet and the gesture signals collected by the camera. The PC then sends the recognized gesture signals to the control panel on the robot's mobile chassis via the Bluetooth module to complete the corresponding operation.

[0015] Furthermore, an infrared obstacle avoidance sensor is installed on the mobile chassis of the post-disaster detection robot, which sends the obstacle data detected by infrared to the control board to control the motor to control the robot to avoid obstacles; obstacle avoidance has a higher priority than remote control instructions.

[0016] Furthermore, the mobile chassis is also provided with a posture sensor to upload the body posture data information to the control board for controlling the stepper motor to adaptively adjust the camera on the two-axis robotic arm of the robot.

[0017] Furthermore, the PC uses an algorithm to fuse the electromyographic signals collected by the MYO bracelet with the gesture signals collected by the camera to create a gesture recognition algorithm based on vision-electromyographic signal fusion;

[0018] The gesture recognition algorithm based on vision-electromyography signal fusion includes the following steps:

[0019] (1) Information collection and feature extraction: Image acquisition is performed through a camera, and the acquired image is grayscaled to generate a single-channel gesture map; at the same time, an 8-channel electromyographic sequence is collected at the same time, and five time-domain features are extracted from the acquired electromyographic sequence as the initial data for distinguishing different gestures;

[0020] (2) performing data dimensionality reduction on the obtained electromyographic signal feature data; using linear transformation to transform the original high-dimensional feature data into a set of linearly independent feature data in each dimension, so as to extract important feature components in the data; using this as the initial data for data imaging, and performing data imaging processing;

[0021] (3) The obtained 1D sequence is imaged; the scaled 1D sequence data is converted from the rectangular coordinate system to the polar coordinate system to obtain the GAFs matrix gasf, gadf; then the recursive value matrix for imaging is calculated, and then the recursive value matrix is scaled by the nearest neighbor interpolation method. Finally, the recursive value matrix and the pixel values of the image are used as the R, B and G channel values in the RGB image to generate a fusion image;

[0022] (4) Perform model training; input the collected fusion image into the MobileNetV3 model for training and classification to generate a gesture classification model based on electromyography and visual information.

[0023] Furthermore, five time-domain statistical features are used as classification criteria for different gestures: mean absolute value MAV, slope change number SSC, zero crossing number ZC, waveform length WL, and root mean square value RMS.

[0024] Furthermore, the control board performs adaptive adjustment control on the camera on the robot's two-axis manipulator arm, using a camera adaptive adjustment algorithm based on multi-sensor fusion;

[0025] The algorithm calculates the posture offset through the initial pitch and roll angles of the two-axis robotic arm and the real-time pitch and roll angles of the posture sensor; then converts the pulse offset of the stepper motor into a position offset; and uses the PID algorithm to calculate the angle at which the stepper motor needs to move, which is input into the stepper motor to achieve real-time adjustment of the two-finger robotic arm, thereby keeping the camera in a relatively stable state.

[0026] Furthermore, the mobile chassis includes a chassis frame, crawler tracks, driving wheels, guide wheels, driven wheels, a motor, a motor drive board, a posture sensor, a control board, a power display screen, a GPS positioning module, a GPS tray, an infrared obstacle avoidance sensor, an obstacle avoidance module bracket, a voltage stabilizing module, a chassis support plate, a shock absorber rod, a microcontroller battery, and a rocker switch;

[0027] The chassis frame is a box-shaped structure with six slots on each side, providing elastic rotational connections for the four damper rods on one side, and for the driving and driven wheels to rotate. The guide wheels are rotatably connected to the other side of the damper rods. The tracks are connected to the driving, guide, and driven wheels on the outside. The left and right driven wheels are connected by guide rods, and the driving wheels are connected to the motors in a rolling manner. The motors are fixed to the rear of the chassis frame. Inside the chassis frame are the motor drive board for driving the control motor, the attitude sensor for outputting the vehicle body's attitude, and the control board for control.

[0028] The power display screen is fixed above the convex groove on the top side of the chassis frame. An elevated copper column is provided on the bottom side of the chassis support plate, and the other side of the copper column is fixed to the top side of the chassis frame. A boat-shaped switch is provided on the top side of the chassis support plate to control the start and stop of the two-axis robotic arm, motor, and control board. The microcontroller battery and voltage regulator module are respectively fixed on the top of the chassis support plate. A pillar is provided at the bottom of the obstacle avoidance module bracket to fix it to the chassis support plate. A GPS tray is provided on the top side to fix it to the infrared obstacle avoidance sensor. The GPS tray is located above the infrared obstacle avoidance sensor, and the top is fixed to the GPS positioning module.

[0029] Furthermore, the two-axis robotic arm includes a first joint, a first joint flange, a first joint stepper motor, a first joint bearing, a ball bearing, a second joint, a second joint stepper motor, a robotic arm base, a first joint driver, a motor battery, a wireless Bluetooth, a camera, a first joint rotation axis, a second joint driver, and a second joint flange;

[0030] The base of the robotic arm is fixed above the chassis support plate. The rear side is provided with a groove for storing the power supply battery. The side is provided with the first joint driver, wireless Bluetooth, and the second joint driver. The first joint driver and the second joint driver are used to drive the first joint stepper motor and the second joint stepper motor. The motor battery is used to power the motor, the first joint stepper motor, the second joint stepper motor, the motor driver board, the control board, and the sensor.

[0031] The second joint is a shell, which is fixed on the top of the robotic arm base; the second joint stepper motor is fixed inside the second joint; the second joint motor rotating shaft is fixed to the second joint flange, and the second joint flange is fixedly connected to the first joint rotating shaft; the second joint and the first joint rotating shaft are provided with ball bearings for rotating stable joints; the first joint rotating shaft is provided with two bosses, and the bosses are provided with two holes; the first joint bearing is fixed to the boss holes on one side of the rotating shaft, and the bosses on the other side are fixedly connected to the first joint flange through the holes; the first joint bearing is rollingly connected to the first joint; the first joint is a shell, and a bracket is provided inside to fix it to the first joint stepper motor; the rotating shaft of the first joint stepper motor is fixedly connected to the first joint flange; the front side of the first joint is provided with a hole to fix it to the camera.

[0032] Furthermore, the system is constructed using a visual interface, including a remote operation module, a GPS module, a function command module, an obstacle warning module and a car imaging module; the remote operation module visualizes the operation position information, the GPS module visualizes the rescue position, the function command module visualizes whether the gesture command is recognized correctly, the obstacle warning module is used to visualize infrared obstacle avoidance information, and the car imaging module is used for remote operation during the driving process of the car.

[0033] The beneficial effects of the present invention are that, compared with the prior art, the present invention adopts the myoelectric-visual teleoperation method to control the independently designed tracked post-disaster detection robot, and builds a multi-terrain post-disaster detection robot based on myoelectric-visual teleoperation. The main advantages are:

[0034] 1. The robot based on visual-muscle fusion teleoperation realizes contactless remote control. In order to ensure the stability, diversity and convenience of visual teleoperation and solve the problems of visual signals being greatly affected by interference and the difficulty in fusing visual signals with multi-dimensional time series signals, the present invention realizes myoelectric-visual teleoperation of the robot by building a remote teleoperation visual interface; and innovatively uses electromyographic signal sequence imaging technology to fuse time series electromyographic signals with visual signals to generate a fusion graph, which is then input into the MobileNetV3 model for training and classification, greatly improving the accuracy of human-computer gesture interaction.

[0035] 2. In order to cope with the complex terrain environment after a disaster, a tracked post-disaster detection robot was designed, which includes functions such as multi-terrain driving, obstacle avoidance, GPS positioning, remote control and monitoring. Through the integration of various functions, remote operation in multi-terrain environments after a disaster is realized, ensuring the safety of rescue personnel and accelerating rescue efficiency.

[0036] 3. To ensure the stability of the robot car's camera shooting in multiple terrains, a two-axis robotic arm for adaptive camera adjustment was designed. Based on multi-sensor fusion technology such as attitude sensors and encoders, stable camera shooting was achieved through the PID algorithm.

[0037] 4. Build a post-disaster detection and rescue system and implement a visual interface. To ensure operational portability and system visibility, the post-disaster detection robot and myoelectric-visual teleoperation were integrated to independently build a post-disaster detection and rescue system and implement a visual interface. This system showcases the teleoperation module, GPS module, function command module, obstacle warning module, and vehicle imaging module. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is an overall schematic diagram of the multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation according to the present invention;

[0039] Figure 2This is the front view of the post-disaster exploration robot;

[0040] Figure 3 This is the rear view of the post-disaster exploration robot;

[0041] Figure 4 It is a schematic diagram of the interior of the mobile chassis;

[0042] Figure 5 It is a schematic diagram of the mobile chassis;

[0043] Figure 6 It is a schematic diagram of a two-axis robotic arm;

[0044] Figure 7 This is a schematic diagram of the assembly of the mobile chassis and the two-axis robotic arm;

[0045] Figure 8 This is a schematic diagram of the interior of a two-axis robotic arm;

[0046] Figure 9 This is a schematic diagram of a gesture recognition algorithm based on visual-electromyographic signal fusion;

[0047] Figure 10 There are 5 types of hand gesture diagrams;

[0048] Figure 11 This is a schematic diagram of contactless remote control of the robot;

[0049] Figure 12 It is the recognition map of the hand information of the perception glove;

[0050] Figure 13 It is a schematic diagram of the working principle of the multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation described in the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are not intended to limit the scope of protection of this application.

[0052] like Figure 1 As shown, the multi-terrain post-disaster detection robot system based on vision-muscle fusion remote operation described in the present invention is based on the hardware of the post-disaster detection robot and the remote control platform, and is combined with a gesture recognition algorithm based on vision-electromyography signal fusion, a robot contactless remote control algorithm based on vision remote operation, a camera adaptive adjustment algorithm based on multi-sensor fusion and a visual interface construction technology to build a post-disaster detection robot system.

[0053] like Figure 2 and 3As shown, the post-disaster exploration robot consists of a housing 1, a microcontroller 2, a mobile chassis 3, and a two-axis robotic arm 4. The housing 1 is fixed to the mobile chassis 3, the two-axis robotic arm 4 is located at the front end of the housing 1, and a camera 412 is located on the two-axis robotic arm. This enables wireless remote control, positioning, obstacle avoidance, and adaptive camera adjustment of the robot. The remote work platform, consisting of remote sensing gloves, a MYO wristband, a camera, and a PC, is used to remotely control the exploration robot.

[0054] like Figure 4 and 5 As shown, the mobile chassis 3 includes a chassis frame 31, tracks 32, driving wheels 33, guide wheels 34, driven wheels 35, motors 36, motor drive boards 37, attitude sensors 38, control boards 39, power display screens 310, GPS positioning modules 311, GPS trays 312, infrared obstacle avoidance sensors 313, obstacle avoidance module brackets 314, voltage stabilizing modules 315, chassis support plates 316, shock absorber rods 317, microcontroller batteries 318, and rocker switches 319.

[0055] The chassis frame 31 is a box-shaped structure with six empty slots on each side, allowing for elastic rotational connection of one side of the four shock-absorbing rods 317 and for rotational connection of the driving wheel 33 and the driven wheel 35. The guide wheel 34 is rotationally connected to the other side of the shock-absorbing rod 317. The track 32 is rollingly connected to the outer sides of the driving wheel 33, guide wheel 34, and driven wheel 35. The left and right driven wheels 35 are connected by guide rods to ensure the stability of the vehicle body. The driving wheel 33 is rollingly connected to the motor 36. The motor 36 is a reduction DC motor fixed to the rear side of the chassis frame 31. The inside of the chassis frame 31 is equipped with a motor drive board 37 for driving and controlling the motor, a posture sensor 38 for outputting the vehicle body posture, and a control board 39 for control and communication. The control board 39 is the main control board and uses the STM32F407 chip.

[0056] The posture sensor 38 is connected to the control board 39 and uploads the body posture data information to the main control board for controlling the stepper motor and the adaptive adjustment control of the robot's two-axis mechanical arm and the camera thereon.

[0057] The power display screen 310 is fixed above the convex groove on the top side of the chassis frame 31. A raised copper column is provided on the bottom side of the chassis support plate 316, and the other side of the copper column is fixed to the top side of the chassis frame 31. A rocker switch 319 is provided on the top side of the chassis support plate 316 to control the start and stop of the two-axis robotic arm 4, the motor 36, and the control board 39. The microcontroller battery 318 and the voltage regulator module 315 are respectively fixed on the top of the chassis support plate 316, which are used for power supply and voltage stabilization control of the microcontroller 2. The obstacle avoidance module bracket 314 is provided with a pillar at the bottom to be fixed to the chassis support plate 316, and a GPS tray 312 is provided on the top side to be fixedly connected to the infrared obstacle avoidance sensor 313. There are 5 obstacle avoidance sensors, which are distributed on the front side for obstacle detection. The GPS tray 312 is located above the infrared obstacle avoidance sensor 313 and is fixedly connected to the GPS positioning module 311 on the top.

[0058] The infrared obstacle avoidance sensor 313 and the GPS positioning module 311 are both connected to the control board 39, and send the infrared detected obstacle data and positioning data information to the main control board for controlling the motor 36 for positioning and obstacle avoidance control of the robot.

[0059] The infrared obstacle avoidance sensor 313 is used to identify surrounding obstacles, giving the robot a certain degree of autonomous obstacle avoidance capability. At the same time, control instructions from the remote end assist in controlling the robot's movement direction. Obstacle avoidance has a higher priority than remote control instructions. This can avoid damage to the robot during operation due to problems such as untimely information, thereby realizing the integration of autonomous detection and remote operation of the robot during detection work.

[0060] like Figure 6 、 Figure 7 and Figure 8 As shown, the two-axis robotic arm 4 includes a first joint 41, a first joint flange 42, a first joint stepper motor 43, a first joint bearing 44, a ball bearing 45, a second joint 46, a second joint stepper motor 47, a robotic arm base 48, a first joint driver 49, a motor battery 410, a wireless Bluetooth 411, a camera 412, a first joint rotation axis 413, a second joint driver 414, and a second joint flange 415.

[0061] The robot arm base 48 is fixed above the chassis support plate 316. A recess for the power supply battery 410 is located on the rear side. Side panels include a first joint driver 49, a wireless Bluetooth connector 411, and a second joint driver 414. The first and second joint drivers 49, 414 drive the first and second joint stepper motors 43, 47, respectively. The motor battery 410 powers the motor 36, the first and second joint stepper motors 43, 47, the motor driver board 37, the control board 39, and the sensors. The wireless Bluetooth connector 411 enables communication between the robot and the remote control platform.

[0062] The second joint 46 is a housing fixed to the top of the robotic arm base 48. The second joint stepper motor 47 is fixed inside the second joint 46. The rotating shaft of the second joint motor 47 is fixed to the second joint flange 414, and the second joint flange 414 is fixedly connected to the first joint rotating shaft 413. The second joint 46 and the first joint rotating shaft 413 are provided with ball bearings 45 for rotational stability of the rotating joint. The first joint rotating shaft 413 is provided with two bosses, and the bosses are provided with two holes. The first joint bearing 44 is fixed to the boss holes on one side of the rotating shaft 413, and the boss on the other side is fixedly connected to the first joint flange 42 through the holes. The first joint bearing 44 is in rolling connection with the first joint 41. The first joint 41 is a housing, and a bracket is provided inside to fix it to the first joint stepper motor 43. The rotating shaft of the first joint stepper motor 43 is fixedly connected to the first joint flange 42. A hole is provided on the front side of the first joint 41 to fix it to the camera 412.

[0063] To ensure stable camera recording during the robot's exploration, a two-axis robotic arm was developed for adaptive camera angle adjustment in various terrains. To ensure smooth camera angle adjustment, the two-axis robotic arm is controlled by a stepper motor. Compared to conventional motors and servos, stepper motors offer low precision errors and no cumulative error. A posture sensor is installed within the robot to continuously monitor its posture during operation. Once the robot's posture is determined, a PID algorithm is used to calculate the required stepper motor displacement angle, thereby maintaining a relatively stable camera position.

[0064] A camera is mounted on the two-axis robotic arm 4 to capture the disaster scene. The microcontroller 2 is secured to the chassis frame 31. The housing 1 is secured to the top of the chassis support plate 316, with a screen 21 located on the rear. The post-disaster detection robot uses the YOLOVX algorithm, integrated into the microcontroller 2, to identify casualties in real time. Upon discovering a casualty or the latest disaster situation, the algorithm sends its coordinates to a PC. The PC control platform transmits these coordinates to rescuers, who then obtain the location and conduct precise search and rescue operations. The microcontroller 2 communicates with the PC via wireless Bluetooth 411.

[0065] The remote control platform consists of a sensing glove, an MYO wristband, a camera, a bracket, a Bluetooth module, and a PC. The camera, fixed to the bracket, collects hand signals. The MYO wristband collects myoelectric signals, generating eight channels of myoelectric timing information. A square-shaped calibration object is placed in the center of the sensing glove to capture hand information. The PC integrates the sensing glove, MYO wristband, Bluetooth module, and camera to enable remote control of the robotic vehicle.

[0066] The PC recognizes the robot's remote control gestures by performing algorithmic fusion on the electromyographic signals collected by the MYO bracelet and the gesture signals collected by the camera. The PC then sends the recognized gesture signals to the control board 39 on the robot's mobile chassis 3 via the Bluetooth module to complete the corresponding operation.

[0067] Control board 39 is based on the pinout features of the STM32F407VGT6. The PCB was drawn using Altium Designer software. It features 67 I / O ports, including 15 ADC interfaces, 4 USART serial ports, 19 TIM ports, and 25 general I / O ports. Six on / off switches are also included on the PCB to separate the power supplies for the microcontroller, sensors, and driver modules, further ensuring hardware operational security. USART1 is used for communication with the infrared obstacle avoidance sensor, USART2 is the PC Bluetooth serial port, USART3 is used for communication with the attitude sensor, and USART6 is used for GPS communication. TIM2's CH3 and TIM1's CH1 are used as motor control interfaces. TIM3's CH2 and TIM4's CH1 are used as stepper motor control interfaces. Ten general I / O ports are used.

[0068] The software solutions of the present invention mainly include a gesture recognition algorithm based on vision-electromyography signal fusion, a robot contactless remote control algorithm based on vision-electromyography teleoperation, a camera adaptive adjustment algorithm based on multi-sensor fusion, and a visualization interface construction technology.

[0069] like Figure 9 As shown in the figure, the gesture recognition algorithm based on vision-electromyography signal fusion.

[0070] The present invention collects 2000 sets of gesture data, each set consists of a gesture picture and an 8-channel electromyographic sequence with a sequence length of 150. The data set is gesture information of (0, 5, 6, 7, 8), a total of 5 categories, such as Figure 10 As shown, there are 400 sets of data for each category.

[0071] The gesture recognition algorithm based on vision-electromyography signal fusion described in the present invention includes the following steps:

[0072] (1) Information collection and feature extraction: Image acquisition is performed through a camera, and the acquired image is grayscaled to generate a single-channel gesture map; at the same time, an 8-channel electromyographic sequence is collected at the same time, and five time-domain features are extracted from the acquired electromyographic sequence as the initial data for distinguishing different gestures;

[0073] The present invention uses a camera to capture images and grayscales them to generate a single-channel gesture image I1. The MYO wristband collects 100 samples before the photo is taken, with each sample interval being 10ms. The length of the collected time series is 100, the total time is 1s, and the collected time series is 8 channels.

[0074] After collecting the EMG sequence, the next step is to extract features from the information obtained. Extracting effective EMG signals is crucial for the subsequent classification of EMG signals of different gestures. If gesture action pattern recognition is to be performed, the key step of feature extraction must be solved. The purpose of feature extraction is to use specific gesture information to distinguish different gesture actions.

[0075] Currently, there are many effective methods for extracting surface electromyographic (EMG) signal features for signal classification. These methods can be broadly categorized into four main categories: time-domain analysis, frequency-domain analysis, time-frequency domain analysis, and nonlinear dynamics. Research by scholars both domestically and internationally suggests that time-domain analysis is superior to other methods in characterizing gesture features. Furthermore, time-domain feature analysis offers simple computation, rapid acquisition, and excellent real-time performance.

[0076] This invention uses five time-domain statistical features as the classification criteria for different gestures: mean absolute value (MAV), slope change (SSC), zero crossings (ZC), waveform length (WL), and root mean square (RMS). The meanings and calculation formulas of these five time-domain statistical features are listed below.

[0077] Mean absolute value (MAV): First, the surface electromyographic signal data collected by each channel are superimposed in sequence and then the average value is calculated. The obtained MAV is the mean absolute value of each channel, which is used to reflect the amplitude of each electromyographic signal. The calculation formula is as shown in the formula:

[0078]

[0079] In the above formula, k=8, x(k) is the electromyographic signal data collected each time, and N is the number of sampling points.

[0080] Slope change (SSC): Select a sampling point and two consecutive points on its left and right and set them as x k-1 ,x k ,x k+1 , set a threshold. When the value obtained is greater than the threshold, it proves that there is a slope change. The slope change reflects the number of peaks and troughs. The formula is as follows:

[0081]

[0082] Zero Crossing Count (ZC): As a crucial time-domain statistical feature of surface EMG signals, it is used to count the frequency of the surface EMG signal waveform crossing the zero axis and characterize the fluctuation of the surface EMG signal. The formula is as follows:

[0083]

[0084] In the above formula, x k ,x k+1 are two consecutive points, and ε is the preset threshold.

[0085] Waveform length (WL): represents the cumulative length of the waveform within N data lengths and can be used to estimate the waveform amplitude, frequency, and duration. The formula is as follows:

[0086]

[0087] Root mean square (RMS): Also known as effective value, when used as a time-domain statistical feature, the root mean square value is obtained by first squaring all values, then summing them, and then finding the average. The formula is as follows:

[0088]

[0089] The five time domain features mentioned above are extracted from the collected EMG sequence as the initial data for distinguishing different gestures. The time domain features of the EMG signal under one gesture are shown in Table 1 below.

[0090] Table 1

[0091]

[0092] (2) performing data dimensionality reduction on the obtained electromyographic signal feature data; using linear transformation to transform the original high-dimensional feature data into a set of linearly independent feature data in each dimension, so as to extract important feature components in the data; using this as the initial data for data imaging, and performing data imaging processing;

[0093] The electromyographic signal used is an eight-channel electromyographic signal sequence. After feature extraction, the obtained sample sequence is an 8×5 matrix. For subsequent data imaging processing, the obtained electromyographic signal must first be subjected to data dimensionality reduction processing.

[0094] As a commonly used data analysis method, PCA mainly uses linear transformation to transform the original high-dimensional feature data into a set of linearly independent feature data in each dimension to extract important feature components in the data.

[0095] The specific steps of PCA dimensionality reduction are as follows:

[0096] The original data is synthesized into an m×n matrix X. The characteristics of each electromyographic channel are u1,u2,u3,u4,u5,u6,u7,u8. The matrix X is constructed as follows:

[0097]

[0098] By formula:

[0099]

[0100] After calculating the average value of each row of data, decentralize each eigenvalue by its own average value to obtain the matrix X1, and then use the formula:

[0101]

[0102] Calculate the covariance matrix X2 and find the maximum eigenvalue λ max and the corresponding eigenvectors as row vectors to construct the matrix P by:

[0103] Y=PX

[0104] Get the one-dimensional PCA sequence Y k (k=1,2,3…,n). The 8×5 matrix sequence is reduced to a one-dimensional sequence through PCA, which is used as the initial data for data imaging processing.

[0105] (3) The obtained 1D sequence is imaged; the scaled 1D sequence data is converted from the rectangular coordinate system to the polar coordinate system to obtain the GAFs matrices gasf and gadf; then the recursive value matrix used for imaging is calculated, and then the recursive value matrix is scaled by the nearest neighbor interpolation method. Finally, the recursive value matrix and the pixel values of the image are used as the R, B, and G channel values in the RGB image to generate a fusion image.

[0106] The resulting one-dimensional data is used to calculate the Gram Angular Field (GAF) matrix for imaging. GAFs essentially converts scaled 1D sequence data from a rectangular coordinate system to a polar coordinate system. The temporal correlation between different time points is then identified by considering the sum and difference of angles between different points. Depending on whether angle sums or differences are used, there are two implementation methods: GASF (for angle sums) and GADF (for angle differences).

[0107] First scale the data to [-1,1];

[0108]

[0109] Among them, y i is an element in the one-dimensional sequence Y, and after scaling, a new sequence is obtained is a one-dimensional sequence Then convert the scaled sequence data to the polar coordinate system, that is, consider the value as the cosine of the angle and the timestamp as the radius. The formula is as follows:

[0110]

[0111] Finally, the formula is used to calculate:

[0112]

[0113]

[0114] After getting the GAFs matrix gasf and gadf, use the formula:

[0115] gasf R i，j =255×gasf i，j

[0116] gadf R i，j =255×gadf i，j

[0117] Calculate the matrix used for imaging, where R i，j is an N×N matrix, where N is gasf i The number of vectors, then gasf R i，j , gadf R i，j The matrix is scaled to a 120×160 matrix using the nearest neighbor interpolation method. gadf R i，j , gadf R i，j The pixel values of image I1 are used as the R, B and G channel values in the RGB image to generate a fusion image. Figure 10 Shown is a fusion diagram of the gestures.

[0118] (4) Perform model training; input the collected fusion image into the MobileNetV3 model for training and classification to generate a gesture classification model based on electromyography and visual information.

[0119] It is further compared with Convnext, Efficientnet, Regnetx, Repvgg, Res2net-50 and MobileNetV3-based single electromyography data (MobileNetV3-EMG) and single image data (MobileNetV3-IMG).

[0120] The present invention uses various evaluation indicators calculated by the confusion matrix (CM) to compare various models, where TP (True Positive) is the number of samples correctly predicted for the current class, TN (True Negative) is the number of samples correctly predicted for other classes, FN (False Negative) is the number of samples incorrectly predicted for the current class, and FP (False Positive) is the number of samples predicted for the current class by other classes. The evaluation indicators of the classification model include accuracy (Acc), precision (Precision), recall (Recall), specificity (Specificity), F1 index (F1), and ROS line area (AUC).

[0121]

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128] Table 2 shows a comparison table of various models.

[0129] Table 2

[0130]

[0131] The model of the present invention is superior to other models, and it is worth mentioning that the fusion graph constructed based on visual and electromyographic signals has achieved good results in various deep learning models, among which the MobileNetV3 model has the best effect. The MobileNetV3 model based on single electromyography and single image has a lower accuracy than other models. Therefore, the electromyographic and visual signal fusion method based on the data imaging method realizes the migration of electromyographic and visual fusion signals in the field of image depth models, with excellent results. And the classification model based on electromyographic-visual fusion is better than the classification model based on single electromyography and single image. The present invention applies the model based on electromyographic-visual signals to the electromyographic-visual remote control system, which solves the problem that the visual signal is greatly affected by interference and the visual signal is difficult to fuse with the multi-dimensional time series signal.

[0132] like Figure 11 As shown in the figure, the contactless remote control algorithm of the robot based on vision-myoelectric teleoperation.

[0133] To ensure the stability and convenience of visual teleoperation, a vision-electromyography-based teleoperation method was designed. By sensing the glove's position and gestures in the video, the algorithm controls the robot's speed, direction, and functions. The algorithm applies Gaussian blur, HSV image conversion, erosion, color gamut selection, and calibration object selection to the gesture images captured by the camera, ensuring the camera accurately captures the calibration objects used by the glove.

[0134] The coordinate information of the obtained marker in the image is then sent to the robot's control board 39 single-chip computer through the Bluetooth module. The single-chip computer realizes the movement of the robot based on the coordinate information and in combination with its own GPS positioning module 311. Furthermore, the algorithm is combined with a gesture model based on electromyography-visual signals to realize the sending of characteristic commands for the car, such as braking, taking pictures, turning left 180 degrees on the spot, etc. The PC obtains coordinate information by shooting the sensing gloves at different positions through the camera, and the single-chip computer uses the coordinate information to control the different movement intentions of the robot during exploration. In different areas, the robot obtains different control signals and completes different action instructions. Within the entire area, the closer the calibration object of the sensing glove is to the outer circle, the faster the robot is controlled to move. As Figure 12 The figure shows the hand information recognition diagram of the perception glove.

[0135] The control board performs adaptive adjustment control on the camera on the robot's two-axis robotic arm, using a camera adaptive adjustment algorithm based on multi-sensor fusion.

[0136] To ensure the stability of the robot's camera shooting in various terrains, a two-axis robotic arm is designed for adaptive camera adjustment. Based on the pitch and roll angle information of the attitude sensor and the stepper motor pulse position, a PID algorithm is used to achieve stable camera shooting. The specific algorithm is as follows:

[0137] The two-axis robotic arm is first reset to zero and the current posture is recorded. The attitude sensor value range is [-180, 180]. The initial positions of the pitch and roll angles are set to pitch_start and roll_start. The real-time pitch and roll angles of the sensor are pitch and roll. Due to the range limitations of the attitude sensor, the attitude offset needs to be calculated using the following formula:

[0138] When pitch_start>0:

[0139]

[0140] When pitch_start≤0:

[0141]

[0142] When roll_start>0:

[0143]

[0144] When roll_start≤0:

[0145]

[0146] Then assume that the pulse offset of the stepper motor is converted into position offset:

[0147]

[0148]

[0149] Among them, i is the reduction ratio, n is the subdivision number, the pulse positions of the two stepper motors are current_pos_p and current_pos_r, and the angular positions of the two stepper motors are current_p and current_r.

[0150] Then calculate the errors error_p and error_r:

[0151] error_p=current_p-error_pitch

[0152] error_r=current_r-error_roll

[0153] Afterwards, the output is calculated by the PID algorithm:

[0154] pid_p=P×error_p+I×(error_p-last_error_p)+D×initial_p

[0155] pid_r=P×error_r+I×(error_r-last_error_r)+D×initial_r

[0156] Where last_error_p and last_error_r are the last error, initial_p and initial_r are the accumulated errors, P, I, and D are set to their own values, and the output values are then clipped. Finally, the stepper motor is input, and the two-finger robotic arm is used to adjust the camera in real time on various terrains.

[0157] To ensure operational portability and system visibility, the project implements a visual interface that displays the teleoperation module, GPS module, function command module, obstacle warning module, and vehicle imaging module. The teleoperation module visualizes the operation position information, further facilitating robot control; the GPS module is used to visualize the rescue location; the function command module is used to visualize whether gesture commands are correctly recognized; the obstacle warning module is used to visualize infrared obstacle avoidance information, flashing when encountering objects; and the vehicle imaging module is used for teleoperation while the vehicle is in motion.

[0158] In practical applications, when a disaster strikes, ground collapses, and toxic gases leak, a post-disaster detection robot is deployed. Controlled via visual and myoelectric teleoperation, it begins searching for injured people. Upon discovering a casualty or the latest disaster situation, it transmits its current coordinates. A PC-based control platform transmits these coordinates to rescuers, who then receive the location and conduct precise search and rescue operations. The robot then continues its exploration until the post-disaster rescue mission is complete.

[0159] like Figure 13 As shown, the multi-terrain post-disaster detection robot system based on vision-muscle fusion remote control described in the present invention, the PC in the remote control platform accurately captures the calibration object of the perception glove through algorithm analysis of the position and gesture of the perception glove in the video captured by the camera; then the coordinate information of the obtained marker in the image is sent to the robot's control board 39 through the Bluetooth module. Based on the coordinate information and combined with its own GPS positioning information, the control board sends a movement instruction to control the motor drive board 37, and then drives the motor 36 to realize the movement of the robot to search for injuries.

[0160] The PC recognizes the robot's remote control gestures by performing algorithmic fusion on the electromyographic signals collected by the MYO bracelet and the gesture signals collected by the camera. The PC then sends the recognized gesture signals to the control board 39 on the robot's mobile chassis 3 via the Bluetooth module to complete the corresponding operation.

[0161] The robot is also equipped with an infrared obstacle avoidance sensor 313, which sends infrared obstacle detection data and positioning data to the main control board for controlling the motor 36 to control the robot to avoid obstacles. Obstacle avoidance has a higher priority than remote control commands.

[0162] A posture sensor 38 is connected to a control board 39, transmitting vehicle posture data to the main control board. This data is used to control the stepper motors that control the robot's two-axis robotic arm and its attached camera. The two-axis robotic arm uses the chassis posture information from the posture sensor to adaptively adjust the robot in various terrains, ensuring stable camera capture.

[0163] The present invention adopts a vision-muscle fusion remote operation method to control an independently designed tracked post-disaster detection robot. The present invention is superior to other devices in terms of all terrains used and available functions, and is contactless, easy to operate, and can be used for multiple purposes. The accuracy of the myoelectric-visual fusion algorithm is better than other algorithms, and it solves the problem that image, sequence data fusion and visual recognition are affected by external factors such as light. In terms of application scenarios, it can be used not only for post-disaster rescue in narrow and complex multi-terrain environments, but also for the exploration of abandoned mines, closed spaces that are not suitable for human entry, search and rescue missions in narrow spaces after disasters, and in industry, it can also realize some special tasks such as logistics allocation of warehouses.

[0164] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation plans of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, and is not a limitation on the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation, characterized in that: The invention comprises a post-disaster detection robot and a remote control platform; the post-disaster detection robot comprises a housing (1), a microcontroller (2), a mobile chassis (3) and a two-axis mechanical arm (4); the housing (1) is fixed on the mobile chassis (3), and the two-axis mechanical arm (4) is arranged at the front end of the housing (1); the remote working platform comprises a remote sensing glove, a MYO wristband, a camera, a Bluetooth module and a PC; The PC in the remote control platform performs algorithmic analysis on the position of the sensing glove captured by the camera to capture the calibration object on the sensing glove; then the coordinate information of the obtained marker in the image is sent to the control board (39) on the robot mobile chassis (3) through the Bluetooth module. Based on the coordinate information and its own GPS positioning information, the control board sends a movement instruction to control the motor drive board (37), which then drives the motor (36) to realize the movement of the robot and conduct injury search; A camera is provided on the two-axis robotic arm (4) for photographing the disaster scene; a YOLOVX algorithm is integrated on the microcontroller (2) to identify the injured target in the disaster scene photographed by the camera, and when the injured are found, the current coordinate position is sent to the PC, and the PC sends the coordinate position to the rescuer; The PC recognizes the robot's remote control gestures by performing algorithmic fusion on the electromyographic signals collected by the MYO wristband and the gesture signals collected by the camera. The PC then sends the recognized gesture signals to the control board (39) on the robot's mobile chassis (3) via the Bluetooth module to complete the corresponding operation. The PC uses an algorithm to fuse the electromyographic signals collected by the MYO bracelet with the gesture signals collected by the camera to create a gesture recognition algorithm based on vision-electromyographic signal fusion; The gesture recognition algorithm based on vision-electromyography signal fusion includes the following steps: (1) Information collection and feature extraction: Image acquisition is performed through a camera, and the acquired image is grayscaled to generate a single-channel gesture map; at the same time, an 8-channel electromyographic sequence is collected at the same time, and five time-domain features are extracted from the acquired electromyographic sequence as the initial data for distinguishing different gestures; (2) performing data dimensionality reduction on the obtained electromyographic signal feature data; using linear transformation to transform the original high-dimensional feature data into a set of linearly independent feature data in each dimension, so as to extract important feature components in the data; using this as the initial data for data imaging, and performing data imaging processing; (3) The obtained 1D sequence is imaged; the scaled 1D sequence data is converted from the rectangular coordinate system to the polar coordinate system to obtain the GAFs matrix gasf, gadf; then the recursive value matrix for imaging is calculated, and then the recursive value matrix is scaled by the nearest neighbor interpolation method. Finally, the recursive value matrix and the pixel values of the image are used as the R, B and G channel values in the RGB image to generate a fusion image; First scale the data to [-1,1]; Among them, y i is an element in the one-dimensional sequence Y, and after scaling, a new sequence is obtained is a one-dimensional sequence An element in ; then convert the scaled sequence data to the polar coordinate system, that is, regard the value as the cosine of the angle and the timestamp as the radius, the formula is as follows: Finally, the formula is used to calculate: After getting the GAFs matrix gasf and gadf, use the formula: gasfR i，j =255×gcsf i，j gadfR i，j =255×gadf i，j The matrix used for imaging is calculated, where R i，j is an N×N matrix, where N is gasf i The number of vectors, then gasfR i,j ,gadfR i,j The matrix is scaled to a 120×160 matrix by the nearest neighbor interpolation method, and finally the gasfR i,j ,gadfR i,j The pixel values of the image I1 are used as the R, B, and G channel values in the RGB image to generate a fusion image; (4) Perform model training; input the collected fusion image into the MobileNetV3 model for training and classification to generate a gesture classification model based on electromyography and visual information.

2. The multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation according to claim 1 is characterized in that: An infrared obstacle avoidance sensor (313) is also provided on the mobile chassis (3) of the post-disaster detection robot to send infrared detected obstacle data to a control panel (39) for controlling a motor (36) to control the robot to avoid obstacles; the priority of obstacle avoidance is higher than that of remote control instructions.

3. The multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation according to claim 1 is characterized in that: The mobile chassis (3) is also provided with a posture sensor (38) for uploading the body posture data information to the control board (39) for controlling the stepper motor to perform adaptive adjustment control on the camera on the two-axis mechanical arm of the robot.

4. The multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation according to claim 1 is characterized in that: Five time-domain statistical features are used as classification criteria for different gestures: mean absolute value MAV, slope change number SSC, zero crossing number ZC, waveform length WL, and root mean square value RMS.

5. The multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation according to claim 3 is characterized in that: The control board performs adaptive adjustment control on the camera on the robot's two-axis robotic arm, using a camera adaptive adjustment algorithm based on multi-sensor fusion; The algorithm calculates the posture offset through the initial pitch and roll angles of the two-axis robotic arm and the real-time pitch and roll angles of the posture sensor; then converts the pulse offset of the stepper motor into a position offset; and uses the PID algorithm to calculate the angle at which the stepper motor needs to move, which is input into the stepper motor to achieve real-time adjustment of the two-finger robotic arm, thereby keeping the camera in a relatively stable state.

6. The multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation according to claim 1 is characterized in that: The mobile chassis (3) includes a chassis frame (31), a crawler track (32), a driving wheel (33), a guide wheel (34), a driven wheel (35), a motor (36), a motor drive board (37), a posture sensor (38), a control board (39), a power display screen (310), a GPS positioning module (311), a GPS tray (312), an infrared obstacle avoidance sensor (313), an obstacle avoidance module bracket (314), a voltage stabilizing module (315), a chassis support plate (316), a shock absorber rod (317), a microcontroller battery (318) and a rocker switch (319); The chassis frame (31) is a box-shaped structure, with six empty slots on each side for elastic rotation connection of four shock-absorbing rods (317) on one side, and for rotation connection of the driving wheel (33) and the driven wheel (35); the guide wheel (34) is rotationally connected to the other side of the shock-absorbing rod (317); the crawler track (32) is rollingly connected to the outer side of the driving wheel (33), the guide wheel (34) and the driven wheel (35); the left and right driven wheels (35) are connected by a guide rod, the driving wheel (33) is rollingly connected to the motor (36), and the motor (36) is fixed to the rear side of the chassis frame (31); the inner side of the chassis frame (31) is provided with a motor drive board (37) for driving and controlling the motor, a posture sensor (38) for outputting the posture of the vehicle body, and a control board (39) for control; A power display screen (310) is fixed above the convex groove on the top side of the chassis frame (31); a raised copper column is provided on the bottom side of the chassis support plate (316), and the other side of the copper column is fixed to the top side of the chassis frame (31); a rocker switch (319) for controlling the start and stop of the two-axis mechanical arm (4), the motor (36), and the control panel (39) is provided on the top side of the chassis support plate (316); a microcontroller battery (318) and a voltage stabilizing module (315) are respectively fixed above the chassis support plate (316); a pillar is provided at the bottom of the obstacle avoidance module bracket (314) and fixed to the chassis support plate (316); a GPS tray (312) is provided on the top side and fixedly connected to the infrared obstacle avoidance sensor (313); the GPS tray (312) is located above the infrared obstacle avoidance sensor (313), and the top is fixedly connected to the GPS positioning module (311).

7. The multi-terrain post-disaster exploration robot system based on vision-muscle fusion teleoperation according to claim 1 is characterized in that: The two-axis robotic arm (4) includes a first joint (41), a first joint flange (42), a first joint stepping motor (43), a first joint bearing (44), a ball bearing (45), a second joint (46), a second joint stepping motor (47), a robotic arm base (48), a first joint driver (49), a motor battery (410), a wireless Bluetooth (411), a camera (412), a first joint rotation axis (413), a second joint driver (414), and a second joint flange (415); The two-axis robotic arm (4) includes a first joint (41), a first joint flange (42), a first joint stepping motor (43), a first joint bearing (44), a ball bearing (45), a second joint (46), a second joint stepping motor (47), a robotic arm base (48), a first joint driver (49), a motor battery (410), a wireless Bluetooth (411), a camera (412), a first joint rotation axis (413), a second joint driver (414), and a second joint flange (415); The second joint (46) is a housing fixed to the top of the robotic arm base (48); the second joint stepper motor (47) is fixed inside the second joint (46); the second joint stepper motor (47) rotation axis is fixed to the second joint flange (415), and the second joint flange (415) is fixedly connected to the first joint rotation axis (413); the second joint (46) and the first joint rotation axis (413) are provided with a ball bearing (45) for rotational stability of the rotation joint; the first joint rotation axis (413) is provided with two bosses, the bosses Two holes are provided; a first joint bearing (44) is fixed to a hole of a boss on one side of a rotating shaft (413), and the boss on the other side is fixedly connected to a first joint flange (42) through the hole; the first joint bearing (44) is rollingly connected to the first joint (41); the first joint (41) is a shell, and a bracket is provided inside to be fixed to the first joint stepper motor (43); the rotating shaft of the first joint stepper motor (43) is fixedly connected to the first joint flange (42); a hole is provided on the front side of the first joint (41) to be fixedly connected to the camera (412).

8. The multi-terrain post-disaster detection robot system based on vision-muscle fusion teleoperation according to claim 1 is characterized in that: The system is constructed using a visual interface, including a remote operation module, a GPS module, a function command module, an obstacle warning module and a vehicle imaging module; the remote operation module visualizes the operation position information, the GPS module visualizes the rescue position, the function command module visualizes whether the gesture command is recognized correctly, the obstacle warning module is used to visualize infrared obstacle avoidance information, and the vehicle imaging module is used for remote operation during the vehicle's driving process.

Images