A mobile robot inspection and remote monitoring method in a complex indoor scene

Abstract

The application discloses a kind of complex indoor scene under mobile robot inspection and remote monitoring method, comprising: in the process of four-legged robot movement, host computer receives the lane line image of four-legged mobile robot two sides sent by industrial camera located four-legged mobile robot two sides in real time, lane line image is handled, and the straight line most fitted with actual lane line is obtained, and the straight line is used as the inspection route of four-legged mobile robot;Meanwhile, host computer detects the obstacle or wall located in front by laser radar located in the middle of the head of four-legged mobile robot, and according to the detection result, four-legged mobile robot is switched to obstacle avoidance mode or turning mode.The application can effectively overcome the interference of indoor yellow lane line wear and pollution, and can accurately adjust the motion posture at no lane line, and realize remote monitoring and control, and can be suitable for large indoor environment patrol and inspection.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, specifically relating to a method for mobile robot inspection and remote monitoring in complex indoor scenarios. Background Technology

[0002] Currently, robotics technology has made significant progress and is widely applied in various fields. Especially in inspection and monitoring, robots can effectively complete heavy, dangerous, or time-consuming tasks. However, existing robotic systems still face challenges in terms of mobility and stability under different terrain and environmental conditions. Researchers have proposed various algorithms and methods to address the path planning and navigation problems of robots during inspection processes. For example, autonomous navigation algorithms based on sensor data and map information use SLAM (Simultaneous Localization and Mapping) technology to achieve real-time map construction and location estimation, combined with planning algorithms to generate optimal paths. To achieve comprehensive inspection tasks, various sensors (such as cameras, LiDAR, and infrared sensors) are used to allow robots to acquire information about their surrounding environment and build accurate environmental models to support the execution of inspection tasks and decision-making.

[0003] In large indoor environments, AGVs (Automated Guided Vehicles) are widely used mobile robots. AGVs typically employ magnetic navigation technology, navigating indoors by attaching magnetic strips to the ground. However, this method is susceptible to mechanical damage from hard objects like metal passing through the loop, impacting navigation and making it suitable only for specific, singular environments. In large warehouses and other indoor scenarios, the accuracy of SLAM mapping is difficult to guarantee. Furthermore, the frequent movement of large goods, equipment, and personnel makes it difficult to maintain a fixed spatial structure, rendering SLAM mapping unsuitable for such environments. Additionally, using magnetic navigation in large settings requires laying numerous magnetic strips, disrupting existing operational activities within the environment.

[0004] Compared to common wheeled and tracked mobile robots, quadruped mobile robots have stronger mobility and are more suitable for inspection in large indoor environments. However, since quadruped mobile robots cannot use encoders to obtain mileage data, applying quadruped mobile robots to indoor inspection tasks can improve efficiency, but it is necessary to design appropriate strategies and methods to guide their movement based on the actual environment when encoders cannot be used.

[0005] Wheeled robots, such as cleaning robots and painting robots, can be guided by encoders calculating mileage. For example, patent publication number CN111538338B proposes a robot edge-following motion control system and method; patent publication number CN113863195B proposes an edge-cleaning method and cleaning vehicle; CN115993820A proposes maze robot path planning based on ROS system; and CN116175561A proposes a control method, chip, and robot for a painting robot. However, although these patents use LiDAR to detect objects such as walls, their functionality is limited, and they cannot switch between line following and wall-following. They are only suitable for fixed situations and are no longer applicable to complex indoor scenarios. Summary of the Invention

[0006] Technical problem solved: This invention discloses a method for mobile robot inspection and remote monitoring in complex indoor scenarios, which solves the problem of patrolling and inspecting quadrupedal mobile robots without encoders to obtain mileage. It can effectively overcome the interference of wear and pollution of indoor yellow lane lines, and can accurately adjust the movement posture in places without lane lines, and realize remote monitoring and control. It is applicable to patrolling and inspecting in large indoor environments.

[0007] Technical solution:

[0008] A method for mobile robot inspection and remote monitoring in complex indoor scenarios, comprising the following steps:

[0009] A LiDAR, industrial camera, and ultrasonic sensor are installed on the quadruped mobile robot. The robot is connected to a host computer via a USB expansion dock. The host computer receives control and task commands from the remote control terminal, switches the quadruped mobile robot to line-following mode, processes the sensor information using the ROS robot operating system to generate motion commands for the quadruped mobile robot, and sends them to the slave computer via UDP protocol. The slave computer then controls the quadruped mobile robot to move along the set inspection route according to the motion commands.

[0010] During the movement of the quadruped robot, the host computer receives real-time lane line images from industrial cameras located on both sides of the quadruped robot. The host computer processes the lane line images and fits the straight line that best matches the actual lane line. This straight line is used as the inspection route of the quadruped robot. At the same time, the host computer uses a LiDAR located in the center of the quadruped robot's head to detect obstacles or walls in front of it. Based on the detection results, the host computer switches the quadruped robot to obstacle avoidance mode or turning mode.

[0011] When the inspection task is completed and the quadruped mobile robot moves to the origin storage area, the quadruped mobile robot is switched to storage mode. An ultrasonic sensor located at the tail of the quadruped mobile robot is used to detect the distance between the tail of the quadruped mobile robot and the bottom baffle of the origin storage area. The quadruped mobile robot is controlled to move backward until the distance between the tail of the quadruped mobile robot and the bottom baffle of the origin storage area is less than the first preset storage distance threshold. Then, a lidar is used to detect the distance between the quadruped mobile robot and the side wall of the origin storage area. The quadruped mobile robot is controlled to move towards the side wall until the distance between the quadruped mobile robot and the side wall of the origin storage area is less than the second preset storage distance threshold.

[0012] Furthermore, the lidar is a two-dimensional lidar; the information obtained by the lidar scanning includes: a one-dimensional distance array [d0, d1, d2, ..., d...]. 359 ], where the element di in the array is the distance between the lidar and the obstacle, and i represents the angle of the lidar scan.

[0013] Furthermore, the host computer uses a lidar located in the center of the quadrupedal mobile robot's head to detect obstacles or walls in front of it. The process of switching the quadrupedal mobile robot to obstacle avoidance mode or turning mode based on the detection results includes:

[0014] During the movement of the mobile robot, [d] was extracted from 360 frames of distance data detected by LiDAR. 141 d 142 d 143 , ..., d 220 When 0.2 ≤ min[d] 141 d 142 d 143 , ..., d 220 When ]≤1, generate a stop command and publish the obstacle avoidance ROS topic;

[0015] The time is calculated to estimate whether the quadruped robot has reached the vicinity of the turning node. If it has reached the vicinity of the turning node, the distance data [d] is extracted from the 360 ​​frames of distance data detected by the LiDAR. 178 d 179 d 180 d 181 d 182 ], Predetermine the distance d between the turning node and the adjacent wall. x When min[d 178 d 179 d 180 d 181 d 182 ]≤d x At that time, a 90° turn command is generated, and the "Turn ROS" topic is published.

[0016] Furthermore, the distance between the quadruped robot and the side wall of the origin storage area is detected using lidar, and the quadruped robot is controlled to translate towards the side wall until the distance between the quadruped robot and the side wall of the origin storage area is less than the second preset storage distance threshold.

[0017] Take [d] from the 360 ​​frames of distance data from the lidar. 89 d 90 [Pre-set the distance d between the quadrupedal mobile robot and the right wall inside the warehouse.] z ,when At that moment, the quadruped robot stopped moving to the right, stood upright for 2 seconds, then lay down, waiting for the next task instruction.

[0018] Furthermore, the mobile robot inspection and remote monitoring method also includes the following steps:

[0019] During the movement of the quadruped robot, if the host computer cannot fit the straight line that best matches the actual lane line, the quadruped robot will switch to wall-following mode. It will use LiDAR to obtain the distance information between itself and the front wall and the side wall, determine whether to turn based on the distance information with the front wall, and adjust itself to be parallel to the side wall based on the distance information with the side wall.

[0020] Furthermore, the process of adjusting oneself to be parallel to the side wall based on the distance information includes the following steps:

[0021] The distances between the left and right sides of the quadruped robot and the wall were extracted from 360 frames of distance data detected by LiDAR. 60 d 61 d 62 d 63 ] and [d 119 d 120 d 121 d 122 ] Calculate the average value and And the difference ΔD=D1-D2, the adjustment speed of the quadruped robot is calculated using equation (1):

[0022]

[0023] In the formula, v r The actual adjustment speed is shown below, while the preset adjustment speed is shown below. A positive value for v0 indicates adjustment for left turns, and a negative value indicates adjustment for right turns.

[0024] Furthermore, if the host computer cannot fit a straight line that best matches the actual lane line, the quadruped mobile robot is switched to wall-following mode. It is then determined whether the current area belongs to the autonomously set wireless segment area. If so, the quadruped mobile robot is driven to move along the wall until it leaves the current area. If not, a timer is started while the quadruped mobile robot is moving along the wall. If the lane line is not detected after the preset waiting time, the quadruped mobile robot is stopped and a task termination command is sent to the remote terminal. If the lane line is detected again within the preset waiting time, the robot is switched back to line-following mode.

[0025] Furthermore, in line-following mode, the quadruped mobile robot pre-sets the view of either the left or right industrial camera for each route segment based on the relative position of the lane line and the quadruped mobile robot.

[0026] Further processing of the lane line image to fit the straight line that best matches the actual lane line includes the following steps:

[0027] The RGB image captured by the industrial camera is divided into regions to shield the field of view interference of the quadruped robot body, and the ROI region is extracted. The extracted ROI region is a three-channel RGB image, which is used as the input image. R, G, and B represent the pixel values ​​of the three channels, respectively. The corresponding H, S, and V channel pixel values ​​are calculated according to equations (2) to (4) to obtain the HSV image.

[0028]

[0029]

[0030] V = C max (4)

[0031] in, C max =max(R′,G′,B′);C min =min(R′, G′, B′); Δ=C max -C min ;

[0032] Binarize the HSV image according to formula (5) to output the original binary image:

[0033]

[0034] In the formula, dst(I) is the output pixel value, lowerb(I) is the lower threshold of the corresponding channel, upperb(I) is the upper threshold of the corresponding channel, and src(I) is the pixel value of the corresponding channel of the input image. When the pixel value of each channel is within the set upper and lower threshold range, dst(I) is 255; otherwise, it is 0.

[0035] Based on the wear and contamination of the lane lines on site, multiple sets of HSV color space thresholds for the ground lane lines are extracted. These thresholds are then fused according to formula (6) to generate a mask. The original binary image is used as input for color filtering to obtain the fused binary image, thus yielding the approximate lane line area.

[0036] dst = mask1|mask2|...|mask j (6)

[0037] In the formula, dst is the fused binary image, and mask is... j For masks processed under different color ranges, | indicates bitwise OR;

[0038] Opening and closing operations are performed on the fused binary image to remove minor contamination and wear interference, resulting in a complete lane line region. The opening operation first erodes the image to separate the fine, adhered parts and remove burrs and isolated points, and then dilates the image to maintain the original shape of the region. Specifically, when A is the initial image and B is an m×n structuring element, the erosion and dilation operations are shown in equations (7) and (8):

[0039]

[0040]

[0041] In the formula, z represents the translation of B onto A. Let Θ represent the expansion operator and Θ represent the erosion operator. Then the opening operation is as shown in equation (9):

[0042]

[0043] The closing operation eliminates small noise holes, smooths the lane line contour, and fills in breaks and gaps; the calculation process of the closing operation is shown in equation (10):

[0044]

[0045] Using the optimal binary image as input, Canny edge detection is performed. The main process includes: using a Gaussian distribution. Gaussian filtering is applied to pixel (x, y) to remove image noise; the Sobel operator S... x With S y Calculate the magnitude of the gradient at each pixel of the image I after Gaussian filtering. With direction Non-maximum suppression is performed based on the gradient direction and magnitude of each pixel to remove pixels that may not constitute an edge; a double threshold is set to retain strong edges and false edges, and the connection between strong edges and false edges is judged to retain the false edges connected to the strong edges;

[0046] The Hough line transform is applied to the edge-detected image to fit multiple sets of straight lines that are suspected to fit the actual lane lines. An angle limit θ is set, and tanθ is taken as the slope limit. Straight lines between -tanθ and tanθ are retained.

[0047] By determining the midpoint ordinates [y1, y2, ..., y] of multiple sets of straight lines parallel to the actual lane lines, a ], find y max =max(y1, y2, ..., y a ), y max The corresponding straight line is the straight line at the bottom of the ROI area. This straight line is used as the straight line that best fits the actual lane line and the standard for the quadruped mobile robot to follow the line.

[0048] Calculate the slope k and midpoint pixel coordinates (x, y) of the straight line that best fits the actual lane line at the bottom of the field of view. When k ≤ -0.016, adjust to the left; when k ≥ 0.016, adjust to the right. When using the left camera's field of view, translate to the right when y ≤ 400px and translate to the left when y ≥ 480px. When using the right camera's field of view, translate to the left when y ≤ 400px and translate to the right when y ≥ 480px. Ensure that the mobile robot is parallel to the lane line and approximately 0.3m away. Publish the "Line Following Adjustment ROS" topic.

[0049] The present invention also discloses a mobile robot inspection platform for complex indoor scenarios. The mobile robot inspection platform includes a lidar, a host computer, a slave computer, a left industrial camera, a right industrial camera, an ultrasonic sensor, a quadrupedal mobile robot body, joint motors, a USB expansion dock, a power supply, a head industrial camera, and a remote monitoring terminal.

[0050] A lidar is placed at the front end of the rectangular body surface of the quadruped mobile robot, perpendicular to the central axis of the cross section. A left industrial camera, a right industrial camera, and a head industrial camera are placed on the left and right sides of the cube frame above the rectangular body and in the head space of the quadruped mobile robot body, respectively. The central axis of the cube frame coincides with the central axis of the quadruped mobile robot body. An ultrasonic sensor is placed at the tail of the rectangular body of the quadruped mobile robot body.

[0051] Inside the quadrupedal mobile robot body are a host computer and a slave computer, which are connected by a network cable. The host computer includes a WiFi module.

[0052] The remote monitoring terminal is wirelessly connected to the host computer and uses the previously described method for mobile robot inspection and remote monitoring in complex indoor scenarios to control the quadrupedal mobile robot.

[0053] Beneficial effects:

[0054] The mobile robot inspection and remote monitoring method for complex indoor scenarios of the present invention enables quadrupedal mobile robots to complete patrol and inspection tasks in complex indoor environments without encoder-outputted mileage data. Attached Figure Description

[0055] Figure 1 This is a schematic diagram illustrating the principle framework of the mobile robot inspection and remote monitoring method in complex indoor scenarios according to an embodiment of the present invention.

[0056] Figure 2 This is a schematic diagram of the structure of a quadrupedal mobile robot according to an embodiment of the present invention;

[0057] Figure 3 This is a flowchart of a mobile robot inspection and remote monitoring method in a complex indoor scene according to an embodiment of the present invention. Detailed Implementation

[0058] The following embodiments are provided to enable those skilled in the art to more fully understand the present invention, but do not limit the invention in any way.

[0059] See Figure 3 This invention discloses a method for mobile robot inspection and remote monitoring in complex indoor scenarios, the method comprising the following steps:

[0060] A LiDAR, industrial camera, and ultrasonic sensor are installed on the quadruped mobile robot. The robot is connected to a host computer via a USB expansion dock. The host computer receives control and task commands from the remote control terminal, switches the quadruped mobile robot to line-following mode, processes the sensor information using the ROS robot operating system to generate motion commands for the quadruped mobile robot, and sends them to the slave computer via UDP protocol. The slave computer then controls the quadruped mobile robot to move along the set inspection route according to the motion commands.

[0061] During the movement of the quadruped robot, the host computer receives real-time lane line images from industrial cameras located on both sides of the quadruped robot. The host computer processes the lane line images and fits the straight line that best matches the actual lane line. This straight line is used as the inspection route of the quadruped robot. At the same time, the host computer uses a LiDAR located in the center of the quadruped robot's head to detect obstacles or walls in front of it. Based on the detection results, the host computer switches the quadruped robot to obstacle avoidance mode or turning mode.

[0062] When the inspection task is completed and the quadruped mobile robot moves to the origin storage area, the quadruped mobile robot is switched to storage mode. An ultrasonic sensor located at the tail of the quadruped mobile robot is used to detect the distance between the tail of the quadruped mobile robot and the bottom baffle of the origin storage area. The quadruped mobile robot is controlled to move backward until the distance between the tail of the quadruped mobile robot and the bottom baffle of the origin storage area is less than the first preset storage distance threshold. Then, a lidar is used to detect the distance between the quadruped mobile robot and the side wall of the origin storage area. The quadruped mobile robot is controlled to move towards the side wall until the distance between the quadruped mobile robot and the side wall of the origin storage area is less than the second preset storage distance threshold.

[0063] like Figure 1 and Figure 2 As shown, the present invention also discloses a mobile robot inspection platform for complex indoor scenarios. The mobile robot inspection platform includes a lidar, a host computer, a slave computer, a left industrial camera, a right industrial camera, an ultrasonic sensor, a quadrupedal mobile robot body, joint motors, a USB expansion dock, a power supply, a head industrial camera, and a remote monitoring terminal.

[0064] A lidar is placed at the front end of the rectangular body surface of the quadruped mobile robot, perpendicular to the central axis of the cross section. A left industrial camera, a right industrial camera, and a head industrial camera are placed on the left and right sides of the cube frame above the rectangular body and in the head space of the quadruped mobile robot body, respectively. The central axis of the cube frame coincides with the central axis of the quadruped mobile robot body. An ultrasonic sensor is placed at the tail of the rectangular body of the quadruped mobile robot body.

[0065] Inside the quadrupedal mobile robot body are a host computer and a slave computer, which are connected by a network cable. The host computer includes a WiFi module.

[0066] The remote monitoring terminal is wirelessly connected to the host computer and uses the previously described method for mobile robot inspection and remote monitoring in complex indoor scenarios to control the quadrupedal mobile robot.

[0067] The quadruped robot body 7 includes a rectangular body and four limbs. The rectangular body is a horizontal support structure, which houses the host computer 2, the slave computer 3, and the power supply 10. A LiDAR 1 is placed at the front of the central axis of the upper layer of the horizontal support, and is connected to a USB expansion dock 9 via a serial port. A cube frame is placed in the middle of the upper layer of the horizontal support, and an ultrasonic sensor 6 is placed at the rear, which is also connected to the USB expansion dock 9 via a serial port. The left industrial camera 4 and the right industrial camera 5 are placed at the midpoint of the top horizontal beams on the left and right sides of the cube frame, respectively. The left industrial camera 4 and the right industrial camera 5 are parallel to the central axis of the cube frame and are connected to the USB expansion dock 9. The USB expansion dock 9 is placed inside the cube frame, which also serves as space for wiring. The USB expansion dock 9 is connected to the host computer 2 via a USB cable. The quadruped robot body 7 includes a rectangular body with two joint motors 8 fixed on the left front end and two identical joint motors 8 symmetrically located on the right front end; the quadruped robot body 7 includes a rectangular body with two joint motors 8 fixed on the left rear end and two identical joint motors 8 symmetrically located on the right front end; each outermost joint motor 8 is connected to a leg structure, and there are a total of 4 leg structures. The 4 leg structures and the rectangular body together form the quadruped robot body 7.

[0068] LiDAR 1 has a range of 0.2m-16m and a 360° scanning range. The industrial cameras on the left (4), right (5), and head (all without distortion) have a 100° field of view and adjustable angles to the ground. The ultrasonic sensor 6 has a detection range of 0.03m-5m and uses linear measurement. LiDAR 1 and ultrasonic sensor 6 acquire distance information. The LiDAR coordinate system uses the robot's central axis as the polar axis and the front direction as the positive direction. LiDAR 1 is used to detect obstacles within a range of 0.2m-1m in front of the mobile robot during its inspection operation. LiDAR 1 also detects the distance between the mobile robot and the wall in front of it during the inspection operation. The detected information is transmitted to the host computer 2 via serial port, and this distance is used as a reference to determine whether to turn. Depending on the environment, when the mobile robot needs to move along the wall, LiDAR is used to detect the distance between the side of the mobile robot and the wall. The detected information is transmitted to the host computer 2 via serial port, and the difference in distance data within three angle ranges is calculated to determine the parallelism between the mobile robot and the wall. If they are not parallel, adjustments are made. Ultrasonic sensor 6 measures the straight-line distance from the tail of the mobile robot to the wall and baffle, which is used for the mobile robot to return to the origin. Industrial camera 4 on the left and industrial camera 5 on the right are fixed on the left and right sides of the mobile robot to detect lane lines on both sides of the road.

[0069] Both the host computer and the slave computer are microcomputers capable of running the Ubuntu system. The slave computer 3 receives and sends information to the host computer 2, processes the received information, determines the inspection task type, and sends action commands to the joint motor 8. It also sends start / stop commands and inspection task completion commands for sensors such as the lidar 1, ultrasonic sensor 6, left industrial camera 4, and right industrial camera 5 to the host computer 2 via UDP. The host computer 2 acquires the detection data from the lidar 1, ultrasonic sensor 6, left industrial camera 4, and right industrial camera 5, processes it, and sends the task commands and control commands to the slave computer 3. The host computer 2 communicates with the remote monitoring terminal.

[0070] This invention provides a method for mobile robot inspection and remote monitoring in complex indoor scenarios, based on the ROS operating system to achieve remote communication with intelligent mobile devices. Specifically, it includes: installing Ubuntu systems on both the host and slave computers and configuring the ROS environment; installing function packages on the host computer under the ROS operating system, including: publish / subscribe service function package, radar invocation function package, serial communication function package, Flask framework, and mjpg-streamer; running startup files on the host computer under the ROS operating system environment, including: web framework program, host computer-slave computer UDP communication program, lidar ranging program, line-following program, database entry program, and mjpg-streamer; connecting the host computer to the local area network, determining the host computer's IP address and port, and configuring connections between the host computer and the slave computer, and between the host computer and the remote terminal; connecting the slave computer to the local area network, determining the slave computer's IP address and port, and configuring the slave computer... The connection with the host computer involves writing the mobile robot's motion program on the slave computer, generating an executable file, writing a startup script, and setting it to start automatically upon power-on. After the host and slave computers are powered on, the programs described in steps three and four start automatically. The remote terminal connects to the local area network, and a browser is opened on the remote terminal, entering the address configured on the host computer to open the remote control terminal webpage. Upon entering the remote control terminal webpage, a task mode is selected, and the mobile robot patrols along the pre-set route, either along lane lines or walls. After reaching a designated location, it performs an inspection. After the inspection task is completed, the host computer reports the task completion result to the remote terminal webpage, and the mobile robot returns to its original position and lies down to wait. On the remote control terminal webpage, a control mode is selected, displaying real-time video, a virtual keyboard, and the running status. The mobile robot is controlled to move forward, backward, lateralize left, lateralize right, turn left, and turn right using the virtual buttons. The real-time image in front of the mobile robot can be viewed through the window.

[0071] The web application starts the Flask framework to configure the task interface, receives task requests from remote terminals, and transmits them to the UDP program via ROS topics. During the LiDAR ranging program, the host computer receives LiDAR distance detection information while the mobile robot is moving. Based on obstacle and wall detection, the program generates stop, left turn, and right turn control commands, which are then transmitted to the UDP program via ROS topics. The line-following program receives detection information from the left and right industrial cameras (4 and 5) while the mobile robot is moving. Based on the detected lane lines, the program generates left move, right move, left turn, and right turn control commands. Depending on the presence or absence of lane lines, it determines whether to switch to wall-following mode, which is also transmitted to the UDP program via ROS topics. During the warehousing process, the host computer receives detection information from the ultrasonic sensor 6 while the mobile robot is moving. When the mobile robot returns to the origin to perform the warehousing action, the program generates backward or forward control commands based on the distance from the tail of the mobile robot to the baffle in the origin storage area. These commands are then transmitted to the UDP sending program via a ROS topic. The host computer receives the ROS topic and generates UDP data frames from the task and control command set described in S31-S34. These UDP data frames are then sent to the slave computer, and the host computer simultaneously receives data frames sent by the slave computer via UDP. The head-mounted industrial camera 11 is invoked via mjpg-strider for remote video transmission within the local area network.

[0072] The LiDAR detects the distance between the front of the mobile robot and obstacles. During the robot's movement, when an object appears within a detection angle range of 140°-220° and a length range of 0.2m-1m, the program stops and publishes a ROS topic. The LiDAR also detects the distance between the front of the mobile robot and walls, with a detection angle range of 178°-182° and a length range adjusted according to the inspection environment. When the mobile robot is working in task mode and reaches a turn in its route, the LiDAR detection program detects the distance to the wall directly in front. When the distance reaches the set turning point, a turning command is generated and published in a ROS topic. The LiDAR also detects the distance between the right side of the mobile robot and walls. When the mobile robot needs to move along a wall in task mode, adjustment commands are published through a ROS topic.

[0073] Specifically, the lidar used in this invention is a two-dimensional lidar; the information obtained by the lidar scanning includes: a one-dimensional distance array [d0, d1, d2, ..., d...]. 359 ], where the element d in the array i The distance between the lidar and the obstacle is represented by d, where i represents the angle of the lidar scan. i The change in the value determines the distance required for the radar.

[0074] The lidar detection program enables obstacle avoidance, turning point determination, and wall detection.

[0075] LiDAR obstacle avoidance methods include: during the movement of the mobile robot, extracting [d] from 360 frames of distance data detected by the LiDAR. 141 d 142 d 143 , ...,d 220 When 0.2 ≤ min[d] 141 d 142 d 143 , ...,d 220 When ]≤1, generate a stop command and publish the obstacle avoidance ROS topic;

[0076] The LiDAR turning point determination method includes: estimating whether the quadruped robot has reached the vicinity of the turning point by calculating the time; if it has reached the vicinity of the turning point, then extracting [d] from the 360 ​​frames of distance data detected by the LiDAR. 178 d 179 d 180 d 181 d 182 [Predetermine the actual distance d between the turning node and the adjacent wall.] x When min[d 178 d 179 d 180 d 181 d 182 ]≤d x At that time, a 90° turn command is generated, the turn direction is set according to the needs of the site, and the turn ROS topic is published;

[0077] The lidar wall detection method includes: in areas without lane lines, switching to wall-following motion mode, and then extracting the distance between the right side of the quadruped robot and the wall from the 360 ​​frames of distance data detected by the lidar [d]. 60 d 61 d 62 d 63 ] and [d 119 d 120 d 121 d 122 ] Calculate the average value and And the difference ΔD = D1 - D2:

[0078]

[0079] In the formula, v rThe actual adjustment speed is shown in v0, which is the preset adjustment speed. A positive v0 indicates left turn adjustment, and a negative v0 indicates right turn adjustment. Publish the ROS topic for wall-following motion to keep the quadrupedal robot parallel to the wall.

[0080] When the mobile robot needs to move along the lane line in task mode, select the field of view of the left industrial camera 4 or the field of view of the right industrial camera 5 according to the relative position of the lane line and the mobile robot; ensure that the mobile robot is parallel to the lane line and about 0.3m away.

[0081] In task mode, if there are lane lines, the mobile robot is pre-set to use either the left industrial camera with 4 fields of view or the right industrial camera with 5 fields of view, based on the relative position of the lane lines and the mobile robot.

[0082] The RGB images captured by the industrial camera are divided into regions, the visual interference of the quadruped robot body 7 is shielded, and the ROI region is extracted.

[0083] The extracted ROI region is a three-channel RGB image, which is used as the input image. R, G, and B represent the pixel values ​​of the three channels, respectively. The corresponding H, S, and V channel pixel values ​​are calculated according to equations (2) to (4) to obtain the HSV image:

[0084]

[0085]

[0086] V = C max (4)

[0087] in, C max =max(R′,G′,B′);C min =min(R′, G′, B′); Δ=C max -C min .

[0088] Binarize the HSV image according to formula (5) to output the original binary image:

[0089]

[0090] In the formula, dst(I) is the output pixel value, lowerb(I) is the lower threshold of the corresponding channel, upperb(I) is the upper threshold of the corresponding channel, and src(I) is the pixel value of the corresponding channel of the input image. When the pixel value of each channel is within the set upper and lower threshold range, dst(I) is 255; otherwise, it is 0.

[0091] Based on the wear and contamination of the lane lines on site, multiple sets of HSV color space thresholds for the ground lane lines are extracted. These thresholds are then fused according to formula (6) to generate a mask. The original binary image is used as input for color filtering to obtain the fused binary image, thus yielding the approximate lane line area.

[0092] dst = mask1| mask2|…|mask i (6)

[0093] In the formula, dst is the fused binary image, and mask is... i For masks processed under different color ranges, | indicates bitwise OR;

[0094] Opening and closing operations are performed on the fused binary image to remove minor contamination and wear interference, resulting in a complete lane line region;

[0095] The opening operation first erodes the image to separate the fine, adhered areas and remove burrs and isolated points, and then dilates the image to maintain the original shape of the region. When A is the initial image and B is an m×n structuring element, the erosion and dilation operations are shown in equations (7) and (8):

[0096]

[0097]

[0098] In the formula, z represents the translation of B onto A. Let Θ represent the expansion operator and Θ represent the erosion operator. Then the opening operation is as shown in equation (9):

[0099]

[0100] The closing operation eliminates small noise holes, smooths lane line contours, and fills in breaks and gaps. The calculation process of the closing operation is shown in equation (10):

[0101]

[0102] Using the optimal binary image as input, Canny edge detection is performed. The main process includes: using a Gaussian distribution. Gaussian filtering is applied to pixel (x, y) to remove image noise; the Sobel operator S... x With S y Calculate the magnitude of the gradient at each pixel of the image I after Gaussian filtering. With direction Non-maximum suppression is performed based on the gradient direction and magnitude of each pixel to remove pixels that may not constitute an edge; a double threshold is set to retain strong edges and false edges, and the connection between strong edges and false edges is judged to retain the false edges connected to the strong edges;

[0103] The Hough line transform is applied to the edge-detected image to fit multiple sets of straight lines that are suspected to fit the actual lane lines. An angle limit θ is set, and tanθ is taken as the slope limit. Straight lines between -tanθ and tanθ are retained.

[0104] By determining the midpoint ordinates [y1, y2, ..., y] of multiple sets of straight lines parallel to the actual lane lines, i ], find y max =max(y1, y2, ..., y i ), y max The corresponding straight line is the straight line at the bottom of the ROI area. This straight line is the one that best matches the actual lane line. This straight line is used as the standard for the quadruped mobile robot to follow the lane.

[0105] Calculate the slope k and midpoint pixel coordinates (x, y) of the straight line that best fits the actual lane line at the bottom of the field of view. When k ≤ -0.016, adjust to the left; when k ≥ 0.016, adjust to the right. When using the left camera's field of view, translate to the right when y ≤ 400px and translate to the left when y ≥ 480px. When using the right camera's field of view, translate to the left when y ≤ 400px and translate to the right when y ≥ 480px. Ensure that the mobile robot is parallel to the lane line and approximately 0.3m away. Publish the "Line Following Adjustment ROS" topic.

[0106] When lane lines are present, the host computer processes video images of the lane lines captured by industrial cameras on the left and right sides. After image processing, it effectively overcomes interference from lane line wear and contamination, fits a straight line that closely matches the lane lines, and outputs adjustment commands based on the control strategy to keep the quadruped robot parallel to the lane lines. When there are no lane lines, the LiDAR detects the side walls, and outputs adjustment commands based on the control strategy to keep the quadruped robot parallel to the walls. The LiDAR obtains the distance to obstacles directly in front for obstacle avoidance judgment, and detects the distance between the turning node and the wall to determine the turning action. When the task is about to be completed, the robot adjusts its loading posture using LiDAR and ultrasonic sensors to complete the loading action. All turning nodes, task-designated points, line-following areas, and wall-moving areas are pre-determined. Areas without lane lines include: user-defined wireless segment areas and areas where the line-following program has not detected lane lines. For example, when moving on a long straight road, if the industrial camera fails to detect lane lines in 20 consecutive frames, it is determined that there are no lane lines. At this time, the state is switched, and the camera moves for 20 seconds by adjusting along the wall. If the lane lines are detected again during this process, the camera switches to the lane-following mode. If the lane lines are still not detected after 20 seconds, the camera stops moving and sends a task termination command to the remote terminal.

[0107] The characteristics of the data entry procedure include:

[0108] The time it takes for the quadrupedal mobile robot to move to the vicinity of the original warehouse area after completing the inspection task is estimated. At this time, it begins to decelerate, and the lidar makes a turning judgment using the turning point judgment method described in claim 9.

[0109] After completing the turn, control the quadruped robot to move backward and activate the ultrasonic sensor to detect the distance d between the tail and the bottom baffle of the storage area. y When d y When the distance is ≤168mm, stop moving backward and start moving to the right.

[0110] During the right translation process, the robot is kept parallel to the right wall using the wall-adjustment method described in claim 9. At this time, [d] in the 360-frame distance data from the lidar is taken. 89 d 90 [Pre-set the distance d between the quadrupedal mobile robot and the right wall inside the warehouse.] z ,when At that moment, the quadruped robot stopped moving to the right, stood upright for 2 seconds, then lay down, waiting for the next task instruction.

[0111] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for mobile robot inspection and remote monitoring in complex indoor scenarios, characterized in that, The mobile robot inspection and remote monitoring method includes the following steps: A LiDAR, industrial camera, and ultrasonic sensor are installed on the quadruped mobile robot. The robot is connected to a host computer via a USB expansion dock. The host computer receives control and task commands from the remote control terminal, switches the quadruped mobile robot to line-following mode, processes the sensor information using the ROS robot operating system to generate motion commands for the quadruped mobile robot, and sends them to the slave computer via UDP protocol. The slave computer then controls the quadruped mobile robot to move along the set inspection route according to the motion commands. During the movement of the quadruped robot, the host computer receives real-time lane line images from industrial cameras located on both sides of the quadruped robot. The host computer processes the lane line images and fits the straight line that best matches the actual lane line. This straight line is used as the inspection route of the quadruped robot. At the same time, the host computer uses a LiDAR located in the center of the quadruped robot's head to detect obstacles or walls in front of it. Based on the detection results, the host computer switches the quadruped robot to obstacle avoidance mode or turning mode. Specifically, the process of processing the lane line image and fitting it to obtain the straight line that best matches the actual lane line includes the following steps: The RGB image captured by the industrial camera is divided into regions to shield the field of view interference of the quadruped robot body, and the ROI region is extracted. The extracted ROI region is a three-channel RGB image, which is used as the input image. R, G, and B represent the pixel values ​​of the three channels, respectively. The corresponding H, S, and V channel pixel values ​​are calculated according to equations (1) to (3) to obtain the HSV image: ； （1） ； （2） ； （3） in, ; ; ; ； Binarize the HSV image according to formula (4) to output the original binary image: ； （4） In the formula, To output pixel values, This is the lower threshold for the corresponding channel. Here, is the upper threshold for the corresponding channel, and src(I) is the pixel value of the corresponding channel in the input image. The condition is a logical AND, indicating that the conditions for all three channels must be met simultaneously. When the pixel value of each channel is within the set upper and lower threshold range, dst(I) is 255; otherwise, it is 0. Based on the wear and contamination of the lane lines on site, multiple sets of HSV color space thresholds for the ground lane lines are extracted. These thresholds are then fused according to formula (5) to generate a mask. The original binary image is used as input for color filtering to obtain the fused binary image, thus yielding the approximate lane line area. ； （5） In the formula, dst is the fused binary image, and mask is... j For masks processed under different color ranges, | indicates bitwise OR; Opening and closing operations are performed on the fused binary image to remove minor contamination and wear interference, resulting in a complete lane line region. The opening operation first erodes the image to separate the fine adhesions and remove burrs and isolated points, and then dilates the image to maintain the original shape of the region. Specifically, when A is the initial image and B is an m×n structuring element, the erosion and dilation operations are shown in equations (6) and (7): ； （6） ； （7） In the formula, z represents the translation of B onto A. This represents the expansion operator expression. If the erosion operator is represented, then the opening operation is as shown in equation (8): ； （8） The closing operation eliminates small noise holes, smooths the lane line contour, and fills in breaks and gaps; the calculation process of the closing operation is shown in equation (9): ； （9） Using the optimal binary image as input, Canny edge detection is performed. The main process includes: using a Gaussian distribution. Gaussian filtering is applied to pixel (x, y) to remove image noise; the Sobel operator is used. and Calculate the magnitude of the gradient at each pixel of the image I after Gaussian filtering. With direction Non-maximum suppression is performed based on the gradient direction and magnitude of each pixel to remove pixels that may not constitute an edge; a double threshold is set to retain strong edges and virtual edges, and the connection between strong edges and virtual edges is judged to retain virtual edges connected to strong edges. The Hough line transform is applied to the edge-detected image to fit multiple sets of straight lines that appear to fit the actual lane lines, and the included angle limits are set. ,Pick As a slope boundary, retain The straight line between them; By determining the ordinates of the midpoints of multiple sets of straight lines parallel to the actual lane lines [ ],turn up , The corresponding straight line is the straight line at the bottom of the ROI area. This straight line is used as the straight line that best fits the actual lane line and the standard for line following of the quadruped mobile robot. Calculate the slope k of the straight line at the bottom of the field of view that best matches the actual lane line, and the midpoint pixel coordinates (x, y). Adjust left turn at the time. When turning right to adjust, when using the left-side camera view, Shift to the right. When shifting to the left, when using the right-side camera view, Shift to the left. Move the robot to the right to ensure it is parallel to the lane line and about 0.3m away, and post the "Line Inspection Adjustment ROS" topic. When the inspection task is completed and the quadruped mobile robot moves to the origin storage area, the quadruped mobile robot is switched to storage mode. An ultrasonic sensor located at the tail of the quadruped mobile robot is used to detect the distance between the tail of the quadruped mobile robot and the bottom baffle of the origin storage area. The quadruped mobile robot is controlled to move backward until the distance between the tail of the quadruped mobile robot and the bottom baffle of the origin storage area is less than the first preset storage distance threshold. Then, a lidar is used to detect the distance between the quadruped mobile robot and the side wall of the origin storage area. The quadruped mobile robot is controlled to move towards the side wall until the distance between the quadruped mobile robot and the side wall of the origin storage area is less than the second preset storage distance threshold.

2. The method for mobile robot inspection and remote monitoring in complex indoor scenarios according to claim 1, characterized in that, The lidar is a two-dimensional lidar; the information obtained by the lidar scanning includes: a one-dimensional distance array [d0, d1, d2, ..., d...]. 359 ], where the element d in the array i The distance between the lidar and the obstacle is denoted by , and i represents the angle of the lidar scan.

3. The method for mobile robot inspection and remote monitoring in complex indoor scenarios according to claim 2, characterized in that, The host computer uses a lidar located in the center of the quadruped robot's head to detect obstacles or walls in front of it. Based on the detection results, the process of switching the quadruped robot between obstacle avoidance and turning modes includes: During the movement of the mobile robot, distance data was extracted from 360 frames detected by LiDAR. ,when At that time, a stop command is generated, and the obstacle avoidance ROS topic is published; The time is calculated to estimate whether the quadruped robot has reached the vicinity of the turning point. If it has reached the vicinity of the turning point, the distance data is extracted from 360 frames of distance data detected by the LiDAR. Predetermine the distance between the turning point and the adjacent wall. ,when At that time, a 90° turn command is generated, and the "Turn ROS" topic is published.

4. The method for mobile robot inspection and remote monitoring in complex indoor scenarios according to claim 2, characterized in that, The distance between the quadruped robot and the side wall of the origin storage area is detected using LiDAR. The quadruped robot is then controlled to move towards the side wall until the distance between the quadruped robot and the side wall of the origin storage area is less than the second preset entry distance threshold. Take [d] from the 360 ​​frames of distance data from the lidar. 89 ,d 90 [Pre-set the distance between the quadrupedal mobile robot and the right wall inside the warehouse.] ,when At that time, the quadruped robot stopped moving to the right, stood for 2 seconds, then lay down and waited for the next task instruction.

5. The method for mobile robot inspection and remote monitoring in complex indoor scenarios according to claim 1, characterized in that, The mobile robot inspection and remote monitoring method also includes the following steps: During the movement of the quadruped robot, if the host computer cannot fit the straight line that best matches the actual lane line, the quadruped robot will switch to wall-following mode. It will use LiDAR to obtain the distance information between itself and the front wall and the side wall, determine whether to turn based on the distance information with the front wall, and adjust itself to be parallel to the side wall based on the distance information with the side wall.

6. The method for mobile robot inspection and remote monitoring in complex indoor scenarios according to claim 5, characterized in that, The process of adjusting oneself to be parallel to the side wall based on the distance information includes the following steps: The distances between the left and right sides of the quadruped robot and the wall were extracted from 360 frames of distance data detected by LiDAR. 60 ,d 61 ,d 62 ,d 63 ] and [d 119 ,d 120 ,d 121 ,d 122 ] Calculate the average value and and the difference The adjustment speed of the quadruped robot is calculated using equation (10): ； （10） In the formula, To adjust the speed in practice, To preset the adjustment speed, A positive value indicates a left turn adjustment. A negative value indicates a right turn adjustment.

7. The method for mobile robot inspection and remote monitoring in complex indoor scenarios according to claim 5, characterized in that, If the host computer cannot find a straight line that best matches the actual lane line, it switches the quadruped mobile robot to wall-following mode, determines whether the current area belongs to the autonomously set wireless segment area, and if so, drives the quadruped mobile robot to move along the wall until it leaves the current area; if not, it drives the quadruped mobile robot to move along the wall and starts a timer. If the lane line is not detected after the preset waiting time, the quadruped mobile robot stops moving and sends a task termination command to the remote terminal. If the lane line is detected again within the preset waiting time, it switches back to line-following mode.

8. The method for mobile robot inspection and remote monitoring in complex indoor scenarios according to claim 1, characterized in that, In line-following mode, the quadruped mobile robot is pre-set to use either the left-side or right-side industrial camera viewpoint for each route segment, based on the relative position of the lane line and the quadruped mobile robot.

9. A mobile robot inspection platform for complex indoor scenarios, characterized in that, The mobile robot inspection platform includes a lidar, a host computer, a slave computer, a left-side industrial camera, a right-side industrial camera, an ultrasonic sensor, a quadrupedal mobile robot body, articulated motors, a USB expansion dock, a power supply, a head-mounted industrial camera, and a remote monitoring terminal. A lidar is placed at the front end of the rectangular body surface of the quadruped mobile robot, perpendicular to the central axis of the cross section. A left industrial camera, a right industrial camera, and a head industrial camera are placed on the left and right sides of the cube frame above the rectangular body and in the head space of the quadruped mobile robot body, respectively. The central axis of the cube frame coincides with the central axis of the quadruped mobile robot body. An ultrasonic sensor is placed at the tail of the rectangular body of the quadruped mobile robot body. Inside the quadrupedal mobile robot body are a host computer and a slave computer, which are connected by a network cable. The host computer includes a WiFi module. The remote monitoring terminal is wirelessly connected to the host computer and uses the mobile robot inspection and remote monitoring method in complex indoor scenarios as described in any one of claims 1-8 to control the quadruped mobile robot.

Images