An automatic inspection robot based on multimodal technology
Through multimodal technology combined with tic-tac-shaped frame chassis, cross-roller bearing gimbal and deep learning, the problem of insufficient perception and obstacle avoidance capabilities of automatic patrol robots in complex environments is solved, and more efficient and safe navigation and patrol.
Patent Information
- Application Number
- CN202311853004.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2043-12-29
AI Technical Summary
Existing automatic inspection robots cannot accurately perceive complex, dynamic or harsh environments, and lack flexibility and obstacle avoidance capabilities, resulting in uncertainty in navigation decision making and collision risks.
Using automatic inspection robot based on multimodal technology, combining tic-tac frame chassis, cross-roller bearing gimbal, GM motor drive, STM32 series development board, high-resolution camera and lidar, through motion resolution and PID control , deep learning and computer vision technology to realize real-time environmental perception and obstacle detection, dynamically avoid obstacles and optimize navigation paths.
It improves the flexibility, passability and environmental perception of the automatic patrol robot, enhances the accuracy and stability of navigation, and ensures efficient and safe patrols in complex environments.
Smart Images

Figure CN117681989B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of automatic inspection, and in particular to an automatic inspection robot based on multimodal technology. Background Art
[0002] With the development of automatic inspection technology, the technologies contained in automatic inspection robots are becoming more and more abundant and advanced, and autonomous navigation is one of the key technologies of such robots, which provides them with the ability to patrol autonomously. At present, the commonly used navigation methods are laser navigation and visual navigation, both of which are targeted at specific application environments. In addition, the application of automatic inspection robots in the power industry is becoming increasingly important. For example, as a key node in the power system, how to ensure the normal operation of the substation is directly related to the stability and safety of the entire power system. Traditional substation monitoring and inspection mainly rely on manual methods, but this method has problems such as high labor intensity and low work efficiency. Therefore, it has become a trend to use automatic inspection robots for real-time monitoring and maintenance. Furthermore, explosion-proof inspection robots represent the forefront of intelligent protection technology. These inspection robots have shown great potential in improving public safety through advanced perception systems, autonomous navigation capabilities, rugged design and efficient communication, and have also played a key role in responding to natural disasters and performing dangerous tasks.
[0003] However, the existing automatic inspection robots have the following disadvantages:
[0004] (1) Existing automatic inspection robots are unable to accurately perceive their surrounding environment, especially under complex, dynamic or harsh conditions, which leads to uncertainty in navigation decisions and reflects insufficient environmental perception.
[0005] (2) Existing automatic inspection robots are not very flexible and have low passability. When encountering obstacles, it is difficult for the automatic inspection robots to effectively avoid the obstacles, resulting in collisions or inability to avoid complex environments, which reflects the lack of obstacle avoidance capabilities. Summary of the invention
[0006] In order to overcome the shortcomings of the prior art, the present invention provides an automatic inspection robot based on multimodal technology, which is used to solve the technical problems of the existing automatic inspection robots that are unable to accurately perceive the surrounding environment and have difficulty in effectively avoiding obstacles. The purpose of improving the obstacle avoidance capability and the accuracy and stability of navigation is achieved by improving the flexibility, passability and environmental perception of the automatic inspection robot.
[0007] To solve the above problems, the technical solution adopted by the present invention is as follows:
[0008] An automatic inspection robot based on multimodal technology, comprising:
[0009] The steering wheel chassis adopts a stable tic-tac-toe frame and is equipped with a guide wheel;
[0010] The upper patrol gimbal is arranged on the steering wheel chassis through a gimbal frame and includes: an industrial control center, a laser radar, a speaker and a high-resolution camera;
[0011] A wheel module is arranged in the steering wheel chassis, comprising a steering motor, a speed motor and a steering wheel, wherein the steering motor is used to control the steering direction of the steering wheel, and the speed motor is used to control the steering direction of the steering wheel;
[0012] The development board is installed in the industrial control center, equipped with STM32 series chips and providing interfaces supporting various external sensors and modules;
[0013] Among them, the yaw axis of the upper patrol gimbal adopts a cross roller bearing, and the power transmission of the pitch axis of the upper patrol gimbal adopts a parallelogram connecting rod.
[0014] As a preferred embodiment of the present invention, the crossed roller bearing is a RA9008 crossed roller bearing, the yaw axis and pitch axis of the upper patrol gimbal are both provided with GM6020 motors, the transmission structure of the GM6020 motor of the yaw axis adopts a mosquito coil board structure, the outer ring of the RA9008 crossed roller bearing is fixed to the bottom plate of the upper patrol gimbal, the inner ring of the RA9008 crossed roller bearing is fixed to the upper patrol gimbal and the synchronous belt, and the lower part of the inner ring of the RA9008 crossed roller bearing adopts a thrust bearing to balance the force.
[0015] As a preferred embodiment of the present invention, the gimbal is distributed along the yaw axis, the GM6020 motor of the pitch axis is built into the gimbal, the industrial control center is placed on the GM6020 motor of the pitch axis, and thrust bearings are installed between the parallelogram connecting rods.
[0016] As a preferred embodiment of the present invention, the number of the steering wheels is 4, all of which adopt independent suspension with oil pressure damping shock absorption system, and the wheel hubs of the steering wheels are all coated with polyurethane; the number of the steering motors and the speed motors is 4, each corresponding to one steering wheel.
[0017] As a preferred embodiment of the present invention, the development board performs motion calculation on the steering wheel chassis, decomposes the target angle and target speed into different angles and different speeds, and gives them to each steering wheel.
[0018] As a preferred embodiment of the present invention, the development board, when performing motion solving, includes:
[0019] Assuming that the forward speed of the automatic inspection robot is 1, the translation speed is 1, and the rotation speed is 1, the three-axis speeds of the four steering wheels are obtained according to the three-axis speeds of the automatic inspection robot, as shown in Formula 1, Formula 2, and Formula 3:
[0020] Vx1=Vx2=Vx3=Vx4=1(1);
[0021] Vy1=Vy2=Vy3=Vy4=1(2);
[0022] Vw1=Vw2=Vw3=Vw4=1xR=R(3);
[0023] The speed components of each steering wheel are synthesized to obtain the magnitude and direction of the resultant speed, as shown in Formulas 4 to 11:
[0024]
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
[0031]
[0032] The three-axis speeds are the angular speeds of the rolling, pitching and yaw axes.
[0033] As a preferred embodiment of the present invention, the steering motor is a GM6020 motor, and the GM6020 motor uses a cascade PID to control the steering direction of the steering wheel, including:
[0034] The target angle obtained by motion calculation and the current angle fed back by the GM6020 motor are used as the target value and current value of the angle loop;
[0035] By adjusting the proportional coefficient, differential coefficient and integral coefficient, the target speed value of the speed loop is obtained;
[0036] Obtain the speed value fed back by the GM6020 motor and adjust the set proportional coefficient, differential coefficient and integral coefficient to obtain the current value required by the GM6020 motor;
[0037] The current angle and speed value of the GM6020 motor are obtained through the IMU, and the IMU has a built-in gyroscope and accelerometer, and the gyroscope is used to detect the 3-axis speed.
[0038] As a preferred embodiment of the present invention, the speed motor is a GM3508 motor, and the GM3508 motor controls the steering of the steering wheel through a single speed loop, including:
[0039] The target speed obtained by motion calculation and the speed value fed back by the GM3508 motor are used as the target value and current value of the speed loop;
[0040] By adjusting and setting the proportional coefficient, differential coefficient and integral coefficient, the current value required by the GM3508 motor is obtained;
[0041] The speed value of the GM3508 motor is obtained through the IMU, and the IMU has a built-in gyroscope and an accelerometer, and the gyroscope is used to detect the 3-axis speed.
[0042] As a preferred embodiment of the present invention, the GM6020 motors of the yaw axis and pitch axis of the upper patrol pan / tilt are controlled by fuzzy PID, including:
[0043] Determine a domain, map the error value to the domain, establish a membership function, substitute the mapped error value into the membership function, and obtain the values of the proportional coefficient, integral coefficient and differential coefficient in the domain by comparing with the rule table, and then obtain the output according to the interval mapping relationship.
[0044] As a preferred embodiment of the present invention, the laser radar is a LIVOXMID-360 laser radar;
[0045] The automatic inspection robot uses the high-resolution camera and the LIVOXMID-360 laser radar to perform real-time three-dimensional environment modeling and obstacle detection, including:
[0046] Based on point cloud obstacle segmentation and feature extraction, computer vision technology is used for target recognition, color segmentation and real-time image processing; convolutional neural network is used for target classification;
[0047] Using a simultaneous localization and mapping algorithm to construct a map in real time in an unknown environment and estimate the position of the automatic inspection robot;
[0048] Use deep learning technology to train obstacle avoidance models, combining real-time lidar information and visual information to dynamically avoid obstacles;
[0049] Based on the real-time constructed map and real-time perception information, the TEB path planning algorithm is adopted to ensure that the automatic inspection robot navigates along the optimal path;
[0050] Wherein, the high-resolution camera serves as a visual sensor.
[0051] Compared with the prior art, the present invention has the following beneficial effects:
[0052] (1) The automatic inspection robot chassis provided by the present invention uses a four-steering wheel structure, and each steering wheel group is driven by a combination of a GM3508 motor (steering) + GM6020 (rudder) + a hydraulic damping shock absorption system. It has high integration and strong adaptability. When used in conjunction with a servo system, it has higher accuracy and faster response. It also has a larger load and traction capacity, and is highly flexible and passable.
[0053] (2) The yaw axis design of the automatic inspection robot provided by the present invention adopts RA9008 cross roller bearings to achieve high rigidity and high precision, and is also easy to install; and the pitch axis power transmission adopts a parallelogram connecting rod design, and thrust bearings are installed between the connecting rods to ensure smooth and unobstructed rotation;
[0054] (3) The automatic patrol robot provided by the present invention uses a combination of LIVOXMID-360 laser radar and a high-resolution camera, which can perform three-dimensional environment modeling and obstacle detection in real time, and at the same time use computer vision technology to perform target recognition and color segmentation, thereby improving the comprehensiveness and accuracy of perception; the FAST_LIOSLAM algorithm is used to ensure that the automatic patrol robot navigates in a complex environment with the optimal path;
[0055] (4) The development board independently developed by the present invention is based on the STM32 series microcontroller chip, integrating advanced hardware design and innovative circuit layout. Its highly integrated design provides developers with excellent performance and flexibility, aiming to meet a wide range of embedded system development needs;
[0056] (5) The main control board of the present invention adopts a compact design. Through advanced three-dimensional packaging technology and compact layout design, the physical volume is successfully reduced, the applicability in space-constrained embedded systems is improved, and greater design flexibility is provided for developers. In terms of cost-effectiveness, by optimizing the selection of components and supply chain strategies, the overall manufacturing cost is effectively reduced, making the development board more competitive and adaptable to cost-sensitive projects without compromising performance and reliability. At the same time, by improving power management, clock system and peripheral interfaces, the performance and reliability of the development board are further improved to ensure stable operation under various environmental conditions, providing developers with a more reliable hardware platform. This solves the problem that the existing development boards are physically large, which causes certain layout and packaging problems in space-constrained embedded systems, and imposes certain restrictions on some application scenarios with high requirements for compact design and miniaturization.
[0057] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 This is a first overall diagram of the automatic inspection robot based on multimodal technology provided by the present invention;
[0059] Figure 2 is a second overall diagram of the automatic inspection robot based on multimodal technology provided by the present invention;
[0060] Figure 3 is a third overall diagram of the automatic inspection robot based on multimodal technology provided by the present invention;
[0061] Figure 4 1. It is a structural diagram of a tic-tac-toe frame of a steering wheel chassis according to an embodiment of the present invention;
[0062] Figure 5 is a structural diagram of a steering wheel according to an embodiment of the present invention;
[0063] Figure 6 is a steering wheel solution diagram of an embodiment of the present invention;
[0064] Figure 7 is a PID control diagram of an embodiment of the present invention;
[0065] Figure 8 is a PID output diagram of an embodiment of the present invention;
[0066] Fig. 9 is a motion control step diagram of an embodiment of the present invention;
[0067] Fig.10 is a flow chart of the operation of the laser radar according to an embodiment of the present invention;
[0068] Fig.11is a YOLO-based depth recognition flow chart of an embodiment of the present invention;
[0069] Fig.12 It is an overall technical framework diagram of an automatic inspection robot based on multimodal technology according to an embodiment of the present invention;
[0070] Fig.13 is a structural diagram of a four-steering wheel chassis according to an embodiment of the present invention;
[0071] Fig.14 is a schematic diagram of a main control board according to an embodiment of the present invention;
[0072] Fig.15 This is a main control board PCB diagram of an embodiment of the present invention;
[0073] Fig.16 This is a rendering of a main control board according to an embodiment of the present invention.
[0074] Explanation of the figure numbers: 1. Steering wheel chassis; 2. Industrial control center; 3. Laser radar; 4. Speaker; 5. Guide wheel; 6. High-resolution camera; 7. Steering wheel; 8. Polyurethane rubber coating; 9. Independent suspension and oil pressure damping shock absorption system. DETAILED DESCRIPTION
[0075] The automatic inspection robot based on multimodal technology provided by the present invention is as follows: Figures 1 to 3 As shown, it includes: a steering wheel chassis 1, an upper patrol gimbal, a wheel module and a development board.
[0076] The steering wheel chassis 1 adopts a tic-tac-toe frame with a stable structure and is equipped with a guide wheel 5.
[0077] The upper patrol PTZ is arranged on the steering wheel chassis 1 through a PTZ frame, and includes: an industrial control center 2, a laser radar 3, a speaker 4 and a high-resolution camera 6.
[0078] The wheel module is arranged in the steering wheel chassis 1, and includes a steering motor, a speed motor and a steering wheel 7. The steering motor is used to control the steering direction of the steering wheel 7, and the speed motor is used to control the steering of the steering wheel 7.
[0079] The development board, installed in the industrial control center 2, is equipped with STM32 series chips and provides interfaces that support a variety of external sensors and modules.
[0080] Among them, the yaw axis of the upper patrol gimbal adopts a cross roller bearing, and the power transmission of the pitch axis of the upper patrol gimbal adopts a parallelogram connecting rod.
[0081] Furthermore, the crossed roller bearing is a RA9008 crossed roller bearing, the yaw axis and pitch axis of the upper patrol gimbal are both equipped with GM6020 motors, the transmission structure of the GM6020 motor of the yaw axis adopts a mosquito coil board structure, the outer ring of the RA9008 crossed roller bearing is fixed to the bottom plate of the upper patrol gimbal, the inner ring of the RA9008 crossed roller bearing is fixed to the upper patrol gimbal and the synchronous belt, and the lower part of the inner ring of the RA9008 crossed roller bearing adopts a thrust bearing to balance the force.
[0082] Furthermore, the gimbal is distributed along the yaw axis, the GM6020 motor of the pitch axis is built into the gimbal, the industrial control center 2 is placed on the GM6020 motor of the pitch axis, and thrust bearings are installed between the parallelogram connecting rods.
[0083] Furthermore, there are four steering wheels 7 , all of which adopt independent suspension with oil pressure damping shock absorption system, and the wheel hubs of the steering wheels 7 are all coated with polyurethane; there are four steering motors and speed motors, each corresponding to one steering wheel 7 .
[0084] Furthermore, the development board performs motion calculation on the steering wheel chassis 1 , decomposes the target angle and target speed into different angles and different speeds, and gives them to each steering wheel 7 .
[0085] Furthermore, when the development board performs motion solving, it includes:
[0086] Assuming that the forward speed of the automatic inspection robot is 1, the translation speed is 1, and the rotation speed is 1, the three-axis speeds of the four steering wheels 7 are obtained according to the three-axis speeds of the automatic inspection robot, as shown in Formula 1, Formula 2, and Formula 3:
[0087] Vx1=Vx2=Vx3=Vx4=1(1);
[0088] Vy1=Vy2=Vy3=Vy4=1(2);
[0089] Vw1=Vw2=Vw3=Vw4=1xR=R(3);
[0090] The 7-part speed of each steering wheel is synthesized to obtain the magnitude and direction of the resultant speed, as shown in Formula 4 to Formula 11:
[0091]
[0092]
[0093]
[0094]
[0095]
[0096]
[0097]
[0098]
[0099] Among them, the three-axis speed is the angular velocity of the roll, pitch, and yaw axes.
[0100] Preferably, the steering motor is a GM6020 motor, which uses a cascade PID to control the steering direction of the steering wheel 7, including:
[0101] The target angle obtained by motion calculation and the current angle fed back by the GM6020 motor are used as the target value and current value of the angle loop;
[0102] By adjusting the proportional coefficient, differential coefficient and integral coefficient, the target speed value of the speed loop is obtained;
[0103] Get the speed value fed back by the GM6020 motor and adjust the proportional coefficient, differential coefficient and integral coefficient to get the current value required by the GM6020 motor;
[0104] Among them, the current angle and speed value of the GM6020 motor are obtained through the IMU. The IMU has a built-in gyroscope and accelerometer. The gyroscope is used to detect the 3-axis speed.
[0105] Preferably, the speed motor is a GM3508 motor, which controls the steering of the steering wheel 7 through a single speed loop, including:
[0106] The target speed obtained by motion solution and the speed value fed back by the GM3508 motor are used as the target value and current value of the speed loop;
[0107] By adjusting the proportional coefficient, differential coefficient and integral coefficient, the current value required by the GM3508 motor can be obtained;
[0108] Among them, the speed value of the GM3508 motor is obtained through the IMU. The IMU has a built-in gyroscope and accelerometer. The gyroscope is used to detect the 3-axis speed.
[0109] Preferably, the GM6020 motors of the upper patrol gimbal yaw axis and pitch axis are controlled by fuzzy PID, including:
[0110] Determine a domain, map the error value to the domain, establish a membership function, substitute the mapped error value into the membership function, and obtain the values of the proportional coefficient, integral coefficient and differential coefficient in the domain by comparing with the rule table, and then obtain the output based on the interval mapping relationship.
[0111] Preferably, the laser radar 3 is a LIVOXMID-360 laser radar;
[0112] The autonomous inspection robot uses a high-resolution camera 6 and LIVOXMID-360 LiDAR for real-time 3D modeling of the environment and obstacle detection, including:
[0113] Based on point cloud obstacle segmentation and feature extraction, computer vision technology is used for target recognition, color segmentation and real-time image processing; convolutional neural network is used for target classification;
[0114] Use simultaneous localization and mapping algorithms to build maps and estimate the position of autonomous inspection robots in unknown environments in real time;
[0115] Use deep learning technology to train obstacle avoidance models, combining real-time lidar information and visual information to dynamically avoid obstacles;
[0116] Based on real-time constructed maps and real-time perception information, the TEB path planning algorithm is used to ensure that the automatic inspection robot navigates along the optimal path;
[0117] Among them, the high-resolution camera 6 serves as a visual sensor.
[0118] The following examples are provided to further illustrate the present invention, but the scope of the present invention is not limited thereto.
[0119] This embodiment provides an automatic inspection robot based on multimodal technology, which has functions such as automatic navigation, real-time image transmission and alarm.
[0120] Its main structure is divided into two parts: the steering wheel chassis 1 and the upper patrol platform.
[0121] 1. Mechanical: The whole automatic inspection robot consists of a steering wheel chassis 1 and an upper patrol platform. The steering wheel chassis 1 adopts a stable well-shaped frame. Figure 4 As shown, the main pipe is made of 6061 aluminum square tube and the plate is made of carbon fiber, which has high strength, strong load-bearing capacity and corrosion resistance.
[0122] The wheel module is driven by a steering wheel 7, the GM6020 motor is responsible for controlling the rudder direction, and the GM3508 motor is responsible for controlling the steering. It has fast and flexible movement speed and strong obstacle avoidance ability. It also adopts an independent suspension and oil pressure damping shock absorption system 9 to ensure that it has good shock absorption and passability. Figure 5As shown, the wheel hub is coated with polyurethane 8, which is wear-resistant and has strong grip, and can adapt to patrolling on various complex roads.
[0123] The outer protection frame is designed with larger rounded corners and is equipped with guide wheels 5, so that it can pass through some narrow passages.
[0124] The upper patrol gimbal yaw axis design adopts RA9008 cross roller bearings to achieve high rigidity and high precision, and is also easy to install. Since the yaw axis adopts a cross roller bearing solution with extremely small clearance, the transmission structure of the GM6020 motor adopts a mosquito coil board structure, which has better fault tolerance. The outer ring is fixed to the gimbal base plate, the inner ring is fixed to the gimbal and the synchronous belt, and the lower part of the inner ring adopts a thrust bearing to balance the force. The shaft system designed above can well ensure axial positioning and radial positioning.
[0125] At the same time, in order to reduce the rotational inertia of the yaw axis, the pan-tilt frame structure is distributed along the yaw axis, the GM6020 motor of the pitch axis is built into the pan-tilt frame, and the industrial control center 2 is placed, which reduces the overall center of mass and makes the center of mass closer to the axis of rotation. The pitch axis power transmission adopts a parallelogram connecting rod design, and thrust bearings are installed between the connecting rods to ensure smooth and unobstructed rotation, allowing more flexible patrols.
[0126] 2. Electronic control: The motion of the robot chassis is solved by decomposing the target angle and speed into different angles and speeds and giving them to each steering wheel 7. The specific algorithm is as follows Figure 6 shown.
[0127] Assuming that the forward speed of the vehicle is 1, the translation speed is 1, and the rotation speed is 1, the 3-axis speeds of the four wheels can be obtained from the 3-axis speeds of the vehicle:
[0128] Vx1=Vx2=Vx3=Vx4=1
[0129] Vy1=Vy2=Vy3=Vy4=1
[0130] Vw1=Vw2=Vw3=Vw4=1xR=R
[0131] The magnitude and direction of the resultant speed are obtained by synthesizing the speeds of each wheel group:
[0132]
[0133]
[0134]
[0135]
[0136]
[0137]
[0138]
[0139]
[0140] In this embodiment, the GM6020 motor is used for steering control, which has the advantages of high torque and high precision. The cascade PID is used to control its direction. The target angle obtained by motion solution and the current angle fed back by the motor are used as the target value and current value of the angle loop. Then, the target speed value of the speed loop is calculated by adjusting the proportional coefficient, differential coefficient and integral coefficient. Then, the speed value fed back by the motor is obtained and the above coefficients are adjusted and set to calculate the current value required by the motor. The GM3508 motor is used for steering control, which has the advantage of high speed. This embodiment uses a single speed loop control, and the target speed obtained by motion solution and the speed value fed back by the motor are used as the target value and current value of the speed loop. Then, the current value required by the motor is calculated by adjusting the proportional coefficient, differential coefficient and integral coefficient. Figure 7 Clearly shows how PID operates.
[0141] The upper patrol gimbal pitch axis and yaw axis both use GM6020 motors and are also controlled by cascade PID. However, in real life, if the two patrol points are far apart, the target error value is too large, and the adaptability of only using a set set of proportional coefficients, integral coefficients, and differential coefficients is not high. Therefore, this embodiment uses fuzzy PID control on the gimbal to include a variety of situations, making it more accurate and adaptable.
[0142] Fuzzy PID control: first determine a domain to map the error value into it, then establish the membership function, substitute the mapped error into the function, and calculate the values of the proportional coefficient, integral coefficient and differential coefficient in the domain according to the rule table, and then get the output according to the interval mapping relationship, such as Figure 8 shown.
[0143] The motion calculation algorithm for the chassis provided in this embodiment is calculated through the mechanical structure of the steering wheel chassis 1 and physical mechanics analysis. The motion calculation algorithm can not only control this type of steering wheel chassis, but also can be used for other different steering wheel chassis after slight modifications to the motion calculation algorithm after physical analysis.
[0144] The PID and fuzzy PID algorithms used in this embodiment are algorithms for accurately controlling motors, and in addition to being applicable to this structure, they can also be applied to a variety of structures.
[0145] Specific motion control steps, such as Fig. 9 As shown, including:
[0146] (1) Obtain real-time data of four steering motors and four speed motors through sensors;
[0147] (2) The direction of travel information calculated by the NUC laser radar is used as the input for the motion calculation of the steering wheel 7 to obtain the desired heading angle and speed of the four wheels;
[0148] (3) Substitute the desired heading angle and speed into the PID algorithm, calculate the error e(t), calculate the proportional, integral, and differential terms of the three PID terms based on the error, and calculate the sum of the three terms to get the output of the PID controller. The state of the controlled system will change due to the change in the controller output, which is fed back to the sensor for re-measurement as the PID output desired control data;
[0149] (4) The encoder, gyroscope, and IMU sense the execution status and realize state estimation; the gyroscope detects the angular velocity of the three axes of roll, pitch, and yaw, and obtains the angle estimation through integration. The IMU measures acceleration and angular velocity. The IMU has a built-in gyroscope and accelerometer, and uses Kalman filtering through filter fusion to eliminate noise errors;
[0150] (5) Abnormal state enters abnormal processing;
[0151] (6) The drive motor is controlled via CAN signals.
[0152] 3. Navigation: Use LIVOXMID-360 LiDAR sensor technology for real-time 3D modeling of the environment and obstacle detection. The algorithm includes obstacle segmentation and feature extraction based on point clouds. Use an integrated high-resolution camera as a visual sensor, and use computer vision technology for target recognition, color segmentation, and real-time image processing. Use convolutional neural network (CNN) for target classification. In terms of navigation algorithms, use the simultaneous localization and mapping (SLAM) algorithm FAST_LIO to build maps and estimate the robot's position in real time in unknown environments. Use deep learning technology to train obstacle avoidance models, combined with real-time LiDAR and visual information to dynamically avoid obstacles. Use deep reinforcement learning methods, deep Q network (DQN). Based on high-precision maps and real-time perception information, the TEB path planning algorithm is used to ensure that the robot navigates along the optimal path. It combines advanced sensing technology, navigation algorithms and decision-making modules to enable the robot to patrol tasks efficiently and safely in complex environments. At the same time, the built-in camera and speaker 4 can transmit images in real time and issue alarms.
[0153] The specific operation process of LiDAR 3 is as follows: Fig.10 As shown, including:
[0154] (1) Obtain sensor point cloud;
[0155] (2) Use the FAST_LIO algorithm for mapping and positioning, input the original point cloud data of the laser radar, the original data of the IMU, and the relevant parameters between the laser radar 3 and the IMU... to obtain the state estimation results, including the position, attitude, speed, deviation, etc. of the laser radar 3 and the IMU, and finally obtain the mapping and positioning results;
[0156] (3) Substitute the sensor point cloud into the point cloud segmentation 3DSegmentation to obtain the segmentation subset or cluster and its related information and obstacle point cloud;
[0157] (4) Use the BackgroundSubtraction algorithm based on the Gaussian mixture model to compare multiple point clouds, find the point cloud clusters that appear and disappear in the background, and extract the dynamic obstacle point cloud;
[0158] (5) Obtain the pose transformation relationship of the baselink coordinate frame relative to the world coordinate system from the robot's TF tree. Position the dynamic obstacle point cloud, determine its coordinate frame pose, transform the robot and obstacle coordinate frames to the world coordinate system, and calculate the position data between the robot and the dynamic obstacle point cloud frame;
[0159] (6) Pointcloud2toLasercan filters the point cloud, removes outliers, performs smoothing, projects the point cloud onto a two-dimensional scanning plane, and converts the segmented point cloud data into 2D laser scan data;
[0160] (7) Patrol according to the set patrol routes;
[0161] (8) The set patrol position data is passed to the TEB navigation for path planning. The TEB node subscribes to the robot's current position, speed, target position and other information, and receives obstacle data from the environment. The costmap is sampled along the trajectory, the cost on the trajectory is calculated, and finally a smooth minimum cost trajectory is output as a reference trajectory for speed control;
[0162] (9) The final calculated speed data and distance to the dynamic obstacle are transmitted to the 32 control layer;
[0163] (10)End.
[0164] YOLO-based deep recognition steps, such as Fig.11 As shown, including:
[0165] (1) Start;
[0166] (2) Input industrial camera video stream;
[0167] (3) Using a dynamic scaling method, the image is scaled adaptively according to the size of the input image and input into the YOLOv5 backbone network for detection;
[0168] (4) Using the attention mechanism variant CBAM algorithm, it can capture the dependencies between space and channels at the same time, enhance the expressiveness of features, reduce the number of parameters, and improve the detection effect;
[0169] (5) Use bilinear interpolation to sample the feature map of the previous step to obtain a more detailed feature map;
[0170] (6) Input the upsampled feature map into the target box encoding GIoU, calculate the IoU of the predicted box and the real box, the area of the minimum closure area between the two, the ratio of the area in the closure area that does not belong to the two, and subtract this ratio from the IoU to get the GIoU value, a real number between -1 and 1, indicating the overlap between the predicted box and the real box;
[0171] (7) Input the real label of the feature map and the GIoU overlap into the loss function FocalLoss to obtain the FocalLoss value L_fl as the optimization target, reduce the weight of simple samples, increase the weight of difficult samples, and improve the detection accuracy;
[0172] (8) Use the AdamW optimizer to adaptively adjust the learning rate and add a weight decay mechanism to prevent overfitting and improve the model's effectiveness.
[0173] (9) Output predicted target category and location information;
[0174] (10)End.
[0175] Hardware: The development board uses advanced multi-layer board technology to ensure signal integrity and electromagnetic compatibility, thereby providing reliable circuit performance. It is equipped with the STM32 series chip, a series of 32-bit microcontrollers with excellent performance, low power consumption and rich integration, providing developers with a wide range of peripherals and communication interfaces, including but not limited to GPIO, UART, SPI, I2C, etc. Its sophisticated clock system and stable power management module ensure the high stability and low power operation of the system. The development board also supports interfaces for a variety of external sensors and modules, providing developers with the possibility of flexible application in various application scenarios. In addition, the development board has a wealth of debugging and programming interfaces to make the development process more efficient. Supporting mainstream integrated development environments (IDEs), developers can easily write, debug and burn code. Overall, this self-developed development board based on the STM32 series chips, with its excellent performance, flexible scalability and reliable stability, provides first-class tools and platforms for embedded system development, meets the needs of a wide range of application fields, and can be adapted to a variety of structures.
[0176] In this embodiment, the overall technical framework of the automatic inspection robot based on multimodal technology is as follows: Fig.12 shown.
[0177] like Fig.13 As shown, the automatic inspection robot chassis provided in this embodiment uses a four-steering wheel structure, and each steering wheel group is driven by a combination of GM3508 motor (steering) + GM6020 (rudder) + hydraulic damping shock absorption system. It has high integration and strong adaptability. When used with the servo system, it has higher accuracy and faster response, and has a larger load and traction capacity, and high flexibility and passability.
[0178] The development board independently developed in this embodiment is based on the STM32 series microcontroller chip, integrating advanced hardware design and innovative circuit layout. Its highly integrated design provides developers with excellent performance and flexibility, aiming to meet a wide range of embedded system development needs.
[0179] like Fig.14 , Fig.15 as well as Fig.16As shown, the main control board of this embodiment adopts a compact design. Through advanced three-dimensional packaging technology and compact layout design, the physical volume is successfully reduced, the applicability in space-constrained embedded systems is improved, and greater design flexibility is provided for developers. In terms of cost-effectiveness, by optimizing the selection of components and the supply chain strategy, the overall manufacturing cost is effectively reduced, making the development board more competitive and adaptable to cost-sensitive projects without compromising performance and reliability. At the same time, by improving power management, clock system and peripheral interfaces, the performance and reliability of the development board are further improved to ensure stable operation under various environmental conditions, providing developers with a more reliable hardware platform. This solves the problem that the existing development board has a large physical volume, which causes certain layout and packaging problems in space-constrained embedded systems, and imposes certain restrictions on some application scenarios with high requirements for compact design and miniaturization.
[0180] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.
Claims
1. An automatic inspection robot based on multimodal technology, characterized in that: include: The steering wheel chassis adopts a stable tic-tac-toe frame and is equipped with a guide wheel; The upper patrol gimbal is arranged on the steering wheel chassis through a gimbal frame and includes: an industrial control center, a laser radar, a speaker and a high-resolution camera; A wheel module is arranged in the steering wheel chassis, comprising a steering motor, a speed motor and a steering wheel, wherein the steering motor is used to control the steering direction of the steering wheel, and the speed motor is used to control the steering direction of the steering wheel; The development board is installed in the industrial control center, equipped with STM32 series chips and providing interfaces supporting various external sensors and modules; Wherein, the yaw axis of the upper patrol gimbal adopts a cross roller bearing, and the power transmission of the pitch axis of the upper patrol gimbal adopts a parallelogram connecting rod; The laser radar is a LIVOX MID-360 laser radar; The automatic inspection robot uses the high-resolution camera and the LIVOX MID-360 laser radar to perform real-time environmental 3D modeling and obstacle detection, including: Based on point cloud obstacle segmentation and feature extraction, computer vision technology is used for target recognition, color segmentation and real-time image processing; convolutional neural network is used for target classification; Using a simultaneous localization and mapping algorithm to construct a map in real time in an unknown environment and estimate the position of the autonomous inspection robot; Use deep learning technology to train obstacle avoidance models, combining real-time lidar information and visual information to dynamically avoid obstacles; Based on the real-time constructed map and real-time perception information, the TEB path planning algorithm is adopted to ensure that the automatic inspection robot navigates along the optimal path; The process of using the high-resolution camera as a visual sensor and computer vision technology to perform target recognition includes: Input industrial camera video stream; A dynamic scaling method is used to adaptively scale the input image according to the size of the input image and input it into the YOLOv5 backbone network for detection. Use the attention mechanism variant CBAM algorithm to capture spatial and channel dependencies at the same time; Use bilinear interpolation to sample the feature map of the previous step to obtain a more detailed feature map; Input the upsampled feature map into the target box encoding GIoU, calculate the IoU of the predicted box and the real box, the area of the minimum closure area between the two, the ratio of the area in the closure area that does not belong to the two to the closure area, and subtract this ratio from the IoU to get the GIoU value, a real number between -1 and 1, which represents the overlap between the predicted box and the real box; Input the real label of the feature map and the GIoU overlap into the loss function FocalLoss to obtain the FocalLoss value L_fl as the optimization target; Use the AdamW optimizer to adaptively adjust the learning rate and add a weight decay mechanism; Output predicted target category and location information; The operation process of the laser radar includes: Get sensor point cloud; The FAST_LIO algorithm is used for mapping and positioning. The original point cloud data of the LiDAR, the original data of the IMU, and the related parameters between the LiDAR and the IMU are input to obtain the state estimation results, including the position, attitude, speed, and deviation of the LiDAR and the IMU, and finally the mapping and positioning results are obtained; Substitute the sensor point cloud into the point cloud segmentation 3DSegmentation to obtain the segmentation subset or cluster and its related information and obstacle point cloud; Use the BackgroundSubtraction algorithm based on the Gaussian mixture model to compare multiple point clouds, find point cloud clusters that appear and disappear in the background, and extract dynamic obstacle point clouds; Obtain the position conversion relationship between the baselink coordinate frame and the world coordinate system from the robot's TF tree, locate the dynamic obstacle point cloud, determine its coordinate frame position, transform the robot and obstacle coordinate frames to the world coordinate system, and calculate the position data between the robot and the dynamic obstacle point cloud frame; Pointcloud2toLasercan filters the point cloud, removes outliers, performs smoothing, projects the point cloud onto a two-dimensional scanning plane, and converts the segmented point cloud data into 2D laser scan data. Conduct patrols according to set patrol routes; The set patrol position data is passed to the TEB navigation for path planning. The TEB node subscribes to the robot's current position, speed, and target position information, receives the obstacle data of the environment, samples the costmap along the trajectory, calculates the cost on the trajectory, and finally outputs a smooth minimum cost trajectory as a reference trajectory for speed control; The final calculated speed data and distance from dynamic obstacles are transmitted to the control layer.
2. The automatic inspection robot based on multimodal technology according to claim 1 is characterized in that: The crossed roller bearing is a RA9008 crossed roller bearing. The yaw axis and pitch axis of the upper patrol gimbal are both provided with GM6020 motors. The transmission structure of the GM6020 motor of the yaw axis adopts a mosquito coil board structure. The outer ring of the RA9008 crossed roller bearing is fixed to the bottom plate of the upper patrol gimbal. The inner ring of the RA9008 crossed roller bearing is fixed to the upper patrol gimbal and the synchronous belt. The lower part of the inner ring of the RA9008 crossed roller bearing adopts a thrust bearing to balance the force.
3. The automatic inspection robot based on multimodal technology according to claim 2 is characterized in that: The pan-tilt frame is distributed along the yaw axis, the GM6020 motor of the pitch axis is built into the pan-tilt frame, the industrial control center is placed on the GM6020 motor of the pitch axis, and thrust bearings are installed between the parallelogram connecting rods.
4. The automatic inspection robot based on multimodal technology according to claim 1 is characterized in that: There are four steering wheels, all of which adopt independent suspension with oil pressure damping shock absorption system, and the wheel hubs of the steering wheels are all coated with polyurethane; there are four steering motors and four speed motors, each corresponding to one steering wheel.
5. The automatic inspection robot based on multimodal technology according to claim 4 is characterized in that: The development board performs motion calculation on the steering wheel chassis, decomposes the target angle and target speed into different angles and different speeds, and gives them to each steering wheel.
6. The automatic inspection robot based on multimodal technology according to claim 5 is characterized in that: When performing motion solving, the development board includes: Assuming that the forward speed of the automatic inspection robot is 1, the translation speed is 1, and the rotation speed is 1, the three-axis speeds of the four steering wheels are obtained according to the three-axis speeds of the automatic inspection robot, as shown in Formula 1, Formula 2, and Formula 3: Vx1=Vx2=Vx3=Vx4=1(1); Vy1=Vy2=Vy3=Yv4=1(2); Vw1=Vw2=Vw3=Vw4=1xR=R(3); The speed components of each steering wheel are synthesized to obtain the magnitude and direction of the resultant speed, as shown in Formulas 4 to 11: (4); (5); (6); (7); (8); (9); (10); (11); The three-axis speeds are the angular speeds of the rolling, pitching and yaw axes.
7. The automatic inspection robot based on multimodal technology according to claim 6 is characterized in that: The steering motor is a GM6020 motor, and the GM6020 motor uses a cascade PID to control the steering direction of the steering wheel, including: The target angle obtained by motion calculation and the current angle fed back by the GM6020 motor are used as the target value and current value of the angle loop; By adjusting the proportional coefficient, differential coefficient and integral coefficient, the target speed value of the speed loop is obtained; Obtain the speed value fed back by the GM6020 motor and adjust the set proportional coefficient, differential coefficient and integral coefficient to obtain the current value required by the GM6020 motor; The current angle and speed value of the GM6020 motor are obtained through the IMU, and the IMU has a built-in gyroscope and accelerometer, and the gyroscope is used to detect the 3-axis speed.
8. The automatic inspection robot based on multimodal technology according to claim 6, characterized in that: The speed motor is a GM3508 motor, and the GM3508 motor controls the steering of the steering wheel through a single speed loop, including: The target speed obtained by motion calculation and the speed value fed back by the GM3508 motor are used as the target value and current value of the speed loop; By adjusting and setting the proportional coefficient, differential coefficient and integral coefficient, the current value required by the GM3508 motor is obtained; The speed value of the GM3508 motor is obtained through the IMU, and the IMU has a built-in gyroscope and an accelerometer, and the gyroscope is used to detect the 3-axis speed.
9. The automatic inspection robot based on multimodal technology according to claim 2, characterized in that: The GM6020 motors of the upper patrol gimbal yaw axis and pitch axis are controlled by fuzzy PID, including: Determine a domain, map the error value to the domain, establish a membership function, substitute the mapped error value into the membership function, and obtain the values of the proportional coefficient, integral coefficient and differential coefficient in the domain by comparing with the rule table, and then obtain the output according to the interval mapping relationship.
