Robot rescue method, program product, storage medium, equipment and system

Abstract

The invention discloses a robot rescue method, a program product, a storage medium, equipment and a system. The method comprises the following steps: acquiring accurate positioning information of a robot and an unmanned aerial vehicle through an RTK technology; controlling the robot and the unmanned aerial vehicle to be located on the ground and in the air of the same position of the search-and-rescue area; acquiring a first video frame in front of the robot through a camera on the robot, and acquiring a second video frame below the robot through a camera on the unmanned aerial vehicle; performing image recognition processing on the first video frame and the second video frame, and constructing a three-dimensional model in front of the robot; and planning a front path of the robot according to the three-dimensional model. The accuracy of robot search and rescue path planning can be improved, the walking time of the robot is saved, and smooth search and rescue are ensured.

Description

Technical Field

[0001] This application relates to the field of search and rescue robot technology, specifically to a robot rescue method, program product, storage medium, device, and system. Background Technology

[0002] Currently, a common type of search and rescue robot is a multi-legged robot equipped with cameras and communication modules. It primarily uses cameras for obstacle recognition and path planning, and satellite communication modules for positioning. It is controlled by a fixed or mobile computer via a wireless communication module. However, due to the complex geological environment of search and rescue operations, existing multi-legged robots, relying solely on their onboard cameras, struggle to fully assess the surrounding environment. Furthermore, the positioning inaccuracies inherent in conventional satellite communication affect the accuracy of the robot's search path planning, thus hindering the successful execution of the search and rescue operation. Summary of the Invention

[0003] The main objective of this application is to provide a robot rescue method, program product, storage medium, device, and system that can improve the accuracy of robot search and rescue path planning and ensure the smooth progress of search and rescue. To achieve the above objective, this application adopts the following technical solution:

[0004] Firstly, this application provides a robot rescue method, including:

[0005] Accurate positioning information for robots and drones can be obtained through RTK technology;

[0006] The robot and drone are positioned on the ground and in the air, respectively, in the same location within the search and rescue area;

[0007] The robot's camera captures the first video frame in front of it, and the drone's camera captures the second video frame below it.

[0008] Image recognition processing is performed on the first and second video frames to construct a three-dimensional model in front of the robot;

[0009] The robot's forward path is planned based on the 3D model.

[0010] Secondly, this application provides a computer device, including a memory and a processor, wherein the memory stores computer program instructions; the computer program instructions are read and executed by the processor to perform the above-described robot rescue method.

[0011] Thirdly, this application provides a robot rescue system, including a drone, a robot, an RTK base station, and the aforementioned computer device; both the drone and the robot are equipped with cameras and RTK positioning modules; the computer device is connected to the drone, the robot, and the RTK base station via wireless communication.

[0012] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a computer to perform the above-described robot rescue method.

[0013] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the robot rescue method described above.

[0014] Compared with the prior art, this application achieves at least the following beneficial effects:

[0015] This application utilizes precise location measurement using RTK technology, combined with video frames from drones and search and rescue robots, and employs a highly trained recognition model to generate a 3D model of a certain range in front of the robot. Based on this 3D model, a suitable path for the robot with good obstacle avoidance and short distance is planned, thereby ensuring the accuracy of the robot's search and rescue path planning, saving the robot's travel time, and ensuring the smooth progress of the search and rescue operation. Attached Figure Description

[0016] One or more embodiments of this application will now be described by way of example only with reference to the accompanying drawings, in which:

[0017] Figure 1 A flowchart illustrating a robot rescue method provided in this application;

[0018] Figure 2 This is a flowchart illustrating the process of obtaining precise positioning information in an embodiment of this application.

[0019] Figure 3 This is a flowchart illustrating how to control a robot and a drone to the same location according to an embodiment of this application.

[0020] Figure 4 This is a flowchart illustrating the control of image recognition processing in an embodiment of this application;

[0021] Figure 5 This is a flowchart of the forward path planning method according to an embodiment of this application;

[0022] Figure 6 A structural block diagram of a computer device provided in an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of a robot rescue system provided in an embodiment of this application. Detailed Implementation

[0024] The present application will now be described in detail with reference to exemplary embodiments shown in the accompanying drawings. However, it should be understood that the present application may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. These embodiments are provided herein to make the content of the present application more complete and to fully convey the concept of the present application to those skilled in the art.

[0025] refer to Figure 1 This application provides a robot rescue method, including:

[0026] Accurate positioning information for robots and drones can be obtained through RTK technology;

[0027] The robot and drone are positioned on the ground and in the air, respectively, in the same location within the search and rescue area;

[0028] The robot's camera captures the first video frame in front of it, and the drone's camera captures the second video frame below it.

[0029] Image recognition processing is performed on the first and second video frames to construct a three-dimensional model in front of the robot;

[0030] The robot's forward path is planned based on the 3D model.

[0031] refer to Figure 2 In at least one embodiment, the process of obtaining precise positioning information according to this application includes:

[0032] Obtain pseudorange observations and station coordinates of the RTK base station, and generate differential data using the RTK base station as a reference point;

[0033] The differential data is transmitted to the RTK positioning modules on the robot and the drone respectively. The coordinates of the robot and the drone relative to the RTK base station are determined by the dynamic differential positioning method, and their instantaneous coordinates are calculated back based on the coordinates of the RTK base station, thereby obtaining the precise positioning information of the robot and the drone respectively.

[0034] RTK measurement utilizes carrier phase differential GPS technology for real-time positioning. It is precisely through these two measurement methods—differential correction and carrier phase ranging—that the accuracy of dynamic positioning can reach the centimeter level. Differential GPS technology uses the spatial correlation between the base station and the rover (i.e., robots and drones) for differential correction, thereby reducing positioning errors. The standard differential GPS principle involves setting up a base station on a high-precision known control point. The position coordinates of the measurement station are determined through single-point positioning of the base station. Then, the positioning error at the base station is determined by comparing the coordinates measured in real-time with the coordinates of the control point.

[0035] refer to Figure 3 In at least one embodiment, the process of controlling a robot and a drone to the same location according to this application includes:

[0036] Keep the robot docked at its starting position and obtain its precise positioning information.

[0037] Based on the precise positioning information of the starting position, control the drone to fly above that position;

[0038] Adjust the vertical distance between the drone and the robot to a fixed value;

[0039] Control the robot and drone to move forward in sync, while maintaining a constant vertical distance between them.

[0040] By controlling the robot and the drone to the same position and the same height difference, it is possible to distinguish that the video frames captured by the two have a strict correspondence and proportional relationship, which is more conducive to video frame analysis, image analysis and integration, thereby obtaining a more accurate 3D model.

[0041] refer to Figure 4 In at least one embodiment, the image recognition processing described in this application includes:

[0042] Process the first and second video frames to have the same pixel range;

[0043] The processed first and second video frames are labeled as the front view and top view, respectively.

[0044] The labeled first and second video frames are input into the trained recognition model;

[0045] The recognition model generates a three-dimensional model of a certain range in front of the robot.

[0046] The process first processes the first and second video frames to have the same pixel values, ensuring that their proportions and clarity are consistent. Then, it labels the view types, which helps reduce the amount of analysis and computation for the recognition model. The recognition model used is a commonly used image analysis model. After long-term and extensive image training, it can accurately understand and integrate images, thereby inferring and constructing a relatively reasonable and accurate 3D model from the main and top views.

[0047] refer to Figure 5 In at least one embodiment, the method for planning the forward path includes:

[0048] S1. Analyze whether there are obstacles in the A1 area with a width of L directly in front of the robot. If there are, control the robot to move forward from the A area. Otherwise, proceed to step S2.

[0049] S2. Analyze whether the robot can cross the obstacle based on its size. If yes, control the robot to move forward from area A. Otherwise, proceed to step S3.

[0050] S3. Analyze the A2l (corresponding to the left side of A1 area) and A2r (corresponding to the right side of A1 area) regions with widths of L on both sides of A1 area, and repeat steps S1 and S2 until a suitable area for the robot to move forward, either the Anl region or the Anr region, is analyzed. If only one area in the Anl region or the Anr region is suitable for passing through, control the robot to pass through that area. If both areas are suitable for passing through, proceed to step S4.

[0051] S4. Analyze whether the Bnl region behind the Anl region and the Bnr region behind the Anr region are suitable for the robot to pass through. The analysis method is the same as in steps S1 and S2. If both the Bnl and Bnr regions are suitable, select the Anl region and control the robot to pass through. If only one is suitable, select the region in front of the suitable region and control the robot to pass through.

[0052] The above path planning process adopts the method of first analyzing and selecting the intermediate path, and then analyzing and selecting the side path, which can obtain a relatively shorter path, thereby reducing the robot's walking time.

[0053] Since common search and rescue robots are multi-legged robots (such as hexapods), the width L is the maximum width that the robot's legs can occupy.

[0054] According to embodiments of this application, this application also provides a computer device, a robot rescue system, a readable storage medium, and a computer program product.

[0055] refer to Figure 6 In at least one embodiment of the computer device 100, the computer device 100 includes a memory 110 and a processor 120. The memory 110 stores computer program instructions, which are read and executed by the processor 120 to perform the management method described in the foregoing embodiments. Furthermore, for ease of interaction, the computer device 100 may also include a display 130.

[0056] The computer device described in this application is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, mainframes, and other suitable computers. The computer device can also represent various forms of mobile devices, such as personal digital processing devices, cellular phones, smartphones, wearable devices, and other similar computing devices. However, based on the requirement of being able to stably and securely process large-scale data, the computer device should be suitable in the form of a desktop computer, workstation, server, mainframe, etc. Furthermore, the components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein. Moreover, the memory, processor, and display included in the computer device are all components commonly found in the art, and the types and models included are conventional choices, which will not be elaborated upon here.

[0057] In contrast, since this application is mainly used outdoors, the computer device 100 is preferably in a lightweight and portable form, such as a laptop or a more integrated handheld device.

[0058] refer to Figure 7 In at least one embodiment of the robot rescue system, the system includes a drone 210, a robot 220, an RTK base station 230, and the aforementioned computer device 100; both the drone 210 and the robot 220 are equipped with cameras and RTK positioning modules; the computer device 100 is connected to the drone 210, the robot 220, and the RTK base station 230 via a wireless communication connection.

[0059] The typical workflow for setting up an RTK base station 230 at a known point is as follows: First, a GPS receiver is set up as a reference station at a known point with high accuracy and reliability. The area around the reference station should have a wide field of view and good observation conditions. A rover is then set up at the point to be measured. Both the reference station and the rover simultaneously observe satellites. After acquiring satellite signals, the reference station's receiver begins calculating its own position. The calculated result is then compared with the known coordinates to determine the error value. Based on the coordinate error, the positioning error for each satellite is calculated. Since the reference station's GPS receiver cannot know the number of satellites received by the rover's receiver, the reference station receiver locks onto all satellites in its field of view, calculates the positioning error for each satellite, and then encodes it into a standard code. This code is sent to the rover via the communication link. Upon receiving the code, the rover corrects for errors based on its own satellite observations to obtain an accurate positioning result.

[0060] When the RTK base station 230 is set up at an unknown point, it first performs single-point positioning to determine its coordinates, which are then used as the base station's known coordinates, replacing the high-precision known coordinates. At this point, because both the "high-precision known coordinates" determined by the base station and the coordinates obtained through real-time positioning are obtained through single-point positioning, there is a very small error; that is, the differential correction value is very small. This results in the rover's positioning accuracy only reaching the level of single-point positioning. The differential correction information of the base station is essentially the difference between the GPS single-point positioning result and the high-precision known control point. Therefore, the above purpose can also be achieved by using the rover to perform single-point correction on the known control point. At this point, the rover's differential correction value, also called the correction parameter, can be calculated. Then, the correction parameter is used to correct the base station's position to obtain the true high-precision known coordinates.

[0061] When the system is running, the RTK base station 230 works in conjunction with the mobile RTK positioning modules on the satellite 240, drone 210, and robot 220 to obtain the accurate positions of the drone 210 and robot 220. Based on the obtained positioning information and in conjunction with the cameras on the drone 210 and robot 220, the computer device 100 plans the path of robot 220 and controls robot 220 to move along the path.

[0062] In at least one embodiment, a readable storage medium stores a computer program that, when executed by a computer, performs the management method described in the foregoing embodiments.

[0063] The computer-readable medium described in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0064] In at least one embodiment of the computer program product, when the computer program product is executed by a processor, it implements the management method described in the foregoing embodiments.

[0065] The computer program product described in this application embodiment is stored in a specific readable storage medium and can be run by a computer device. This computer program product is not limited to running only on a particular operating system; such operating systems include Unix-like systems (e.g., Linux), Microsoft Windows, Apple macOS, Chrome OS, Android, etc. Furthermore, the programming language used in this computer program product is not limited to a specific programming language; such programming languages ​​include C++, Java, Python, JavaScript, Go, etc.

[0066] It should be understood that all the above embodiments are exemplary and not restrictive. Various modifications or variations made by those skilled in the art to the specific embodiments described above under the concept of this application should be within the protection scope of this application.

Claims

1. A robot rescue method, characterized in that, include: Accurate positioning information for robots and drones can be obtained through RTK technology; The robot and drone are positioned on the ground and in the air, respectively, in the same location within the search and rescue area; The robot's camera captures the first video frame in front of it, and the drone's camera captures the second video frame below it. Image recognition processing is performed on the first and second video frames to construct a three-dimensional model in front of the robot; The robot's forward path is planned based on the 3D model.

2. The robot rescue method as described in claim 1, characterized in that, The process of obtaining precise location information includes: Obtain pseudorange observations and station coordinates of the RTK base station, and generate differential data using the RTK base station as a reference point; The differential data is transmitted to the RTK positioning modules on the robot and the drone respectively. The coordinates of the robot and the drone relative to the RTK base station are determined by the dynamic differential positioning method, and their instantaneous coordinates are calculated back based on the coordinates of the RTK base station, thereby obtaining the precise positioning information of the robot and the drone respectively.

3. The robot rescue method as described in claim 1, characterized in that, The process of controlling the robot and the drone to the same location includes: Keep the robot docked at its starting position and obtain its precise positioning information. Based on the precise positioning information of the starting position, control the drone to fly above that position; Adjust the vertical distance between the drone and the robot to a fixed value; Control the robot and drone to move forward in sync, while maintaining a constant vertical distance between them.

4. The robot rescue method as described in claim 1, characterized in that, The image recognition processing includes: Process the first and second video frames to have the same pixel range; The processed first and second video frames are labeled as the front view and top view, respectively. The labeled first and second video frames are input into the trained recognition model; The recognition model generates a three-dimensional model of a certain range in front of the robot.

5. The robot rescue method as described in claim 1, characterized in that, The method for planning the forward path includes: S1. Analyze whether there are obstacles in the A1 area with a width of L directly in front of the robot. If there are, control the robot to move forward from the A area. Otherwise, proceed to step S2. S2. Analyze whether the robot can cross the obstacle based on its size. If yes, control the robot to move forward from area A. Otherwise, proceed to step S3. S3. Analyze regions A2l and A2r, both with a width of L on both sides of region A1. Repeat steps S1 and S2 until a suitable region for the robot to move forward, either region Anl or region Anr, is found. If only one region Anl or region Anr is suitable for passing through, control the robot to pass through that region. If both regions are suitable for passing through, proceed to step S4. S4. Analyze whether the Bnl region behind the Anl region and the Bnr region behind the Anr region are suitable for the robot to pass through. The analysis method is the same as in steps S1 and S2. If both the Bnl and Bnr regions are suitable, select the Anl region and control the robot to pass through. If only one is suitable, select the region in front of the suitable region and control the robot to pass through.

6. The robot rescue method as described in claim 1, characterized in that: The robot is a multi-legged robot, and the width L is the maximum width that the robot's legs can occupy.

7. A computer device comprising a memory and a processor, wherein the memory stores computer program instructions, characterized in that: When the computer program instructions are read and executed by the processor, the method described in any one of claims 1 to 6 is performed.

8. A robot rescue system, characterized in that: It includes a drone, a robot, an RTK base station, and the computer device as described in claim 7; both the drone and the robot are equipped with a camera and an RTK positioning module; the computer device is connected to the drone, the robot, and the RTK base station via a wireless communication connection.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a computer to perform the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that: When the computer program is executed by the processor, it implements the method of any one of claims 1 to 6.

Images