Over-the-horizon remote control mobile robot low-delay double-view-angle control system and method

Abstract

The invention relates to the technical field of over-the-horizon remote control mobile robots, in particular to an over-the-horizon remote control mobile robot low-delay double-view-angle control system and method, and the system comprises a mobile robot end and a remote control end, and the mobile robot end comprises a fisheye lens group and an image fusion processing module; the fisheye lens group captures a 360-degree panoramic image, the image fusion processing module preprocesses image data, generates a panoramic video stream in real time through an image fusion algorithm, and transmits the panoramic video stream to the remote control end through the communication module; and the remote control end performs selective double-view-angle display, and adjusts the observation view angle in real time based on the head action of the user or a manual instruction. The data transmission quantity is reduced, the video quality is ensured, the transmission efficiency is improved, the stable transmission of the panoramic video stream is ensured, the delay and packet loss phenomena are reduced, the time delay problem is solved, and the real-time performance and comfort of control are better.

Description

Technical Field

[0001] The present invention relates to the technical field of beyond-visual-range remote-controlled mobile robots, and in particular to a low-delay dual-viewing angle control system and method for beyond-visual-range remote-controlled mobile robots. Background Art

[0002] Beyond visual range remote control mobile robots are highly integrated and intelligent robot systems that can move and operate at long distances beyond the direct reach of human sight. This type of robot combines advanced mechanical structure, electrical control, sensor technology, computer vision and communication technology to achieve autonomous or remote control operations in complex environments.

[0003] In practical applications of existing beyond-visual-range remote-controlled mobile robots, the cumulative delay caused by factors such as image transmission, control signal processing and pan-tilt rotation seriously affects the real-time and comfort of control, especially in complex environments or high-speed moving scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a low-latency dual-view control system and method for a beyond-visual-range remote-controlled mobile robot, which solves the problem that in actual applications of existing beyond-visual-range remote-controlled mobile robots, the accumulated delay caused by factors such as image transmission, control signal processing and pan-tilt rotation seriously affects the real-time and comfort of control.

[0005] To achieve the above-mentioned object, the present invention provides a low-latency dual-view control system for a beyond-visual-range remote-controlled mobile robot, comprising a mobile robot end and a remote control end, wherein the mobile robot end is connected to the remote control end, the main body carried by the mobile robot end is a robot controlled by beyond-visual-range mobile control, the mobile robot end comprises a fisheye lens group and an image fusion processing module, and the fisheye lens group is connected to the image fusion processing module;

[0006] The fisheye lens group is used to capture a 360° panoramic image and upload the image data to the image fusion processing module;

[0007] The image fusion processing module is used to pre-process the image data, generate a panoramic video stream in real time through an image fusion algorithm, and transmit the panoramic video stream to the remote control terminal through the communication module;

[0008] The remote control terminal is used for selective dual-viewing angle display and real-time adjustment of the viewing angle based on the user's head movement or manual instructions.

[0009] The mobile robot terminal further comprises a high-definition image transmission communication module and a mobile driving module, wherein the high-definition image transmission communication module is connected to the image fusion processing module and the remote control terminal respectively, and the mobile driving module is connected to the remote control terminal;

[0010] The high-definition image transmission communication module is used to optimize the video compression algorithm and transmission protocol, and adopt the data packet scheduling and error recovery mechanism to transmit the panoramic video stream to the remote control terminal;

[0011] The mobile driving module is used to respond to the control instructions of the remote control terminal and control the movement of the robot.

[0012] Wherein, the remote control terminal includes a device module and a control module, the device module is connected to the high-definition image transmission communication module, and the control module is respectively connected to the fisheye lens group and the mobile drive module;

[0013] The device module includes a mobile phone, a PC or smart glasses, which is used to obtain a panoramic video stream and perform a selective dual-viewing display;

[0014] The control module is used to obtain user instructions, and control the fisheye lens group and the mobile driving module according to the user instructions to adjust the observation angle in real time.

[0015] Wherein, the remote control terminal further includes an inertial measurement module, and the inertial measurement module is connected to the control module;

[0016] The inertial measurement module is used to monitor the orientation changes of the user's head or the device in real time using an IMU sensor, generate corresponding user instructions, and upload them to the control module.

[0017] Wherein, the image fusion processing module includes a preprocessing unit and a fusion algorithm unit, the preprocessing unit is connected to the fisheye lens group, and the fusion algorithm unit is connected to the preprocessing unit;

[0018] The preprocessing unit is used to receive the image data from the fisheye lens group and perform preprocessing operations of correction, denoising, resizing and color correction on the received image;

[0019] The fusion algorithm unit is used to use an image fusion algorithm to perform fusion processing on the pre-processed images to generate a distortion-free panoramic image.

[0020] A low-latency dual-viewing angle control method for a beyond-visual-range remote-controlled mobile robot comprises the following steps:

[0021] Capture 360° panoramic images and pre-process the image data, including rectification, denoising, resizing, and color correction;

[0022] The pre-processed images are fused to generate a distortion-free panoramic image;

[0023] Transmit panoramic video stream to remote control terminal;

[0024] The remote control end receives the panoramic video stream and displays it in a selective dual-viewing angle. At the same time, the user adjusts the viewing angle in real time through head movements or manual commands.

[0025] The present invention discloses a low-latency dual-view control system and method for a beyond-visual-range remote-controlled mobile robot. The fisheye lens group at the mobile robot end captures a 360° panoramic image and uploads the image data to the image fusion processing module. The preprocessing unit of the image fusion processing module receives the image data and performs correction, denoising, resizing and color correction operations. The fusion algorithm unit of the image fusion processing module uses an image fusion algorithm to fuse the preprocessed image to generate a distortion-free panoramic image. The high-definition image transmission communication module receives the panoramic image and optimizes the video compression algorithm and transmission protocol. Using an efficient data packet scheduling and error recovery mechanism, the high-definition image transmission communication module transmits the panoramic video stream to the remote control end. The remote control end receives the panoramic video stream and performs selective dual-view display through the device module, such as a mobile phone, PC or smart glasses. The user can adjust the viewing angle in real time through head movements or manual instructions. The control module of the remote control end obtains user instructions and controls the fisheye lens group and the mobile drive module to achieve viewing angle adjustment and robot movement. It reduces the amount of data transmission while ensuring video quality, improves transmission efficiency, ensures stable transmission of panoramic video streams, reduces delays and packet loss, solves the latency problem, and makes control more real-time and comfortable. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0027] Figure 1 It is a principle block diagram of a low-latency dual-viewing angle control system for a beyond-visual-range remote-controlled mobile robot according to the first embodiment of the present invention.

[0028] Figure 2 It is a principle block diagram of the image fusion processing module and the high-definition image transmission communication module of the first embodiment of the present invention.

[0029] Figure 3 It is a flowchart of the steps of the low-latency dual-viewing angle control method of a beyond-visual-range remote-controlled mobile robot according to the second embodiment of the present invention.

[0030] In the figure: 101-mobile robot end, 102-remote control end, 103-fisheye lens group, 104-image fusion processing module, 105-high-definition image transmission communication module, 106-mobile drive module, 107-equipment module, 108-control module, 109-inertial measurement module, 110-preprocessing unit, 111-fusion algorithm unit, 114-transmission protocol unit, 115-data packet scheduling unit, 116-error recovery mechanism unit, 117-communication interface unit. DETAILED DESCRIPTION

[0031] Embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, but should not be construed as limiting the present invention.

[0032] The first embodiment of the present application is:

[0033] See also Figure 1 and Figure 2 ,in, Figure 1 It is a principle block diagram of a low-latency dual-viewing angle control system for a beyond-visual-range remote-controlled mobile robot according to the first embodiment of the present invention. Figure 2 It is a principle block diagram of the image fusion processing module 104 and the high-definition image transmission communication module 105 according to the first embodiment of the present invention.

[0034] The present invention provides a low-latency dual-view control system for a beyond-visual-range remote-controlled mobile robot, comprising a mobile robot end 101 and a remote control end 102, wherein the mobile robot end 101 comprises a fisheye lens group 103, an image fusion processing module 104, a high-definition image transmission communication module 105 and a mobile drive module 106, the remote control end 102 comprises a device module 107, a control module 108 and an inertial measurement module 109, the image fusion processing module 104 comprises a preprocessing unit 110 and a fusion algorithm unit 111, and the high-definition image transmission communication module 105 comprises a transmission protocol unit 114, a data packet scheduling unit 115, an error recovery mechanism unit 116 and a communication interface unit 117.

[0035] According to this specific implementation, the mobile robot end 101 is connected to the remote control end 102, the main body carried by the mobile robot end 101 is a robot controlled by beyond visual range movement, and the fisheye lens group 103 is connected to the image fusion processing module 104;

[0036] The fisheye lens group 103 is used to capture a 360° panoramic image and upload the image data to the image fusion processing module 104;

[0037] The image fusion processing module 104 is used to pre-process the image data, generate a panoramic video stream in real time through an image fusion algorithm, and transmit the panoramic video stream to the remote control terminal 102 through the communication module;

[0038] The remote control terminal 102 is used for selective dual-viewing angle display and real-time adjustment of viewing angle based on user head movements or manual instructions.

[0039] The number of the fisheye lens group 103 is at least two, and multiple fisheye lenses are used to capture 360° panoramic images to ensure coverage without blind spots. The image fusion processing module 104 uses a high-performance CPU or a dedicated image processing chip to fuse the images captured by multiple fisheye lenses in real time to generate a distortion-free panoramic video stream. The remote control terminal 102 supports devices such as smart phones, PCs or smart glasses. After acquiring the panoramic video stream, it can be displayed in split screens or split images as needed.

[0040] The high-definition image transmission communication module 105 is connected to the image fusion processing module 104 and the remote control terminal 102 respectively, and the mobile driving module 106 is connected to the remote control terminal 102;

[0041] The high-definition image transmission communication module 105 is used to optimize the video compression algorithm and transmission protocol, and adopt the data packet scheduling and error recovery mechanism to transmit the panoramic video stream to the remote control terminal 102;

[0042] The mobile driving module 106 is used to respond to the control instruction of the remote control terminal 102 and control the movement of the robot.

[0043] The high-definition image transmission communication module 105 adopts an optimized video compression algorithm and transmission protocol to ensure efficient and low-latency transmission of the panoramic video stream. The mobile drive unit controls the movement of the robot and responds to remote control instructions.

[0044] Secondly, the device module 107 is connected to the high-definition image transmission communication module 105, and the control module 108 is connected to the fisheye lens group 103 and the mobile driving module 106 respectively;

[0045] The device module 107 includes a mobile phone, a PC or smart glasses, which is used to obtain a panoramic video stream and perform a selective dual-viewing display;

[0046] The control module 108 is used to obtain user instructions, and control the fisheye lens group 103 and the mobile driving module 106 according to the user instructions to adjust the observation angle in real time.

[0047] The inertial measurement module 109 is connected to the control module 108;

[0048] The inertial measurement module 109 is used to monitor the orientation changes of the user's head or the device in real time using an IMU sensor, generate corresponding user instructions, and upload them to the control module 108 .

[0049] The device module 107 obtains real-time panoramic video images through devices such as smart phones, PCs or smart glasses, and can selectively display dual perspectives: driving perspective and observation perspective. The driving perspective is fixed, and the observation perspective can be adjusted through the IMU on the control side. The driving perspective is set to 0-90°, and the observation perspective is X+90° (the image of the observation perspective is actually a selective display of part of the panoramic video stream). The inertial measurement module 109 uses the IMU sensor of the control side device to monitor the changes in the orientation of the user's head or device in real time, and uploads it to the control module 108, automatically adjusting the observation perspective to the direction that the user is most concerned about, thereby enhancing immersion and ease of operation. The control module 108 adjusts the fisheye lens group 103 and the mobile drive module 106 in real time according to IMU data or manual instructions, updates the observation perspective screen, and continuously monitors and optimizes the data transmission quality during the movement to ensure low-latency and high-definition video transmission.

[0050] At the same time, the preprocessing unit 110 is connected to the fisheye lens group 103, and the fusion algorithm unit 111 is connected to the preprocessing unit 110;

[0051] The preprocessing unit 110 is used to receive the image data from the fisheye lens group 103, and perform preprocessing operations such as correction, denoising, resizing and color correction on the received image;

[0052] The fusion algorithm unit 111 is used to use an image fusion algorithm to fuse the pre-processed images to generate a distortion-free panoramic image.

[0053] The preprocessing unit 110 receives the original image data from the fisheye lens group 103, and performs preprocessing operations such as correction, denoising, resizing and color correction on the received original image to ensure the consistency of image quality. The fusion algorithm unit 111 adopts advanced image fusion algorithms, such as pyramid transformation, wavelet transformation or multi-scale geometric transformation, to fuse the preprocessed images to generate a distortion-free panoramic image.

[0054] In addition, the transmission protocol unit 114 is connected to the fusion algorithm unit 111;

[0055] The transmission protocol unit 114 is used to optimize the transmission protocol.

[0056] The data packet scheduling unit 115 is connected to the transmission protocol unit 114, the error recovery mechanism unit 116 is connected to the data packet scheduling unit 115, and the communication interface unit 117 is connected to the error recovery mechanism unit 116 and the remote control terminal 102 respectively;

[0057] The data packet scheduling unit 115 is used to intelligently schedule the sending order and priority of data packets according to network conditions and data transmission requirements;

[0058] The error recovery mechanism unit 116 is used to implement an error detection and recovery mechanism;

[0059] The communication interface unit 117 is used to provide a communication interface with the remote control terminal.

[0060] The transmission protocol unit 114 is responsible for implementing and optimizing transmission protocols, such as TCP, UDP combined with custom protocols, etc., to ensure reliable transmission of video data. The data packet scheduling unit 115 intelligently schedules the sending order and priority of data packets according to network conditions and data transmission requirements to reduce transmission delays. The error recovery mechanism unit 116 implements error detection and recovery mechanisms, such as retransmission mechanisms, forward error correction coding, etc., to reduce packet loss rate and improve transmission stability. The communication interface unit 117 provides communication interfaces with external devices, such as the remote control terminal, such as Wi-Fi, 5G, Ethernet, etc., to achieve remote transmission of video data.

[0061] A low-latency dual-view control system for a beyond-visual-range remote-controlled mobile robot according to the present embodiment is used. The multiple fisheye lens groups 103 of the mobile robot end 101 capture 360° panoramic images to ensure coverage without blind spots. The preprocessing unit 110 receives the image data of the fisheye lens group 103 and performs preprocessing operations such as correction, denoising, resizing and color correction. The fusion algorithm unit 111 performs fusion processing on the preprocessed images to generate a distortion-free panoramic image. The high-definition image transmission communication module 105 receives the panoramic image, optimizes the video compression algorithm and transmission protocol, and adopts data packet scheduling and error recovery mechanisms to transmit the panoramic video stream to the remote control end 102 with low latency and stability. The device module 107, such as a mobile phone, PC, or smart glasses, receives the panoramic video stream for selective dual-view display, namely, a driving view and an observation view. The user can adjust the viewing angle in real time through head movements or manual instructions. The control module 108 controls the fisheye lens group 103 and the mobile drive module 106 according to the user's instructions to achieve viewing angle adjustment and robot movement. The inertial measurement module 109 uses the IMU sensor to monitor the direction changes of the user's head or device in real time, generates corresponding instructions and uploads them to the control module 108, and automatically adjusts the viewing angle to the direction that the user is most concerned about. The fisheye lens group 103 provides a 360° panoramic view without blind spots, enhances the robot's environmental perception ability, and ensures the high quality and distortion-free of the transmitted video stream through preprocessing, fusion and encoding steps. The high-definition image transmission communication Module 105 uses optimized video compression algorithm and transmission protocol, combined with data packet scheduling and error recovery mechanism, to achieve low-latency transmission of video streams. The remote control terminal 102 supports selective dual-view display, and users can switch viewpoints freely according to their needs, which improves operational flexibility and immersion. The inertial measurement module 109 combines IMU sensors to achieve automatic adjustment of viewing angles, improve user experience and operational convenience. The remote control terminal 102 supports a variety of devices, such as smartphones, PCs, and smart glasses, to meet the needs of different users. The control module 108 continuously monitors and optimizes data transmission quality during movement to ensure low-latency and high-definition video transmission. The delay problem is solved, which makes the real-time and comfort of control better.

[0062] The second embodiment of the present application is:

[0063] Based on the first embodiment, please refer to Figure 3 ,in, Figure 3 The following is a flowchart of the steps of the low-latency dual-viewing angle control method for a beyond-visual-range remote-controlled mobile robot according to the second embodiment of the present invention. The low-latency dual-viewing angle control method for a beyond-visual-range remote-controlled mobile robot according to this embodiment includes the following steps:

[0064] S201: Capture a 360° panoramic image and pre-process the image data, including correction, denoising, resizing, and color correction;

[0065] S202: performing fusion processing on the pre-processed images to generate a distortion-free panoramic image;

[0066] S203: Transmitting the panoramic video stream to the remote control terminal 102;

[0067] S204: The remote control terminal 102 receives the panoramic video stream and performs a selective dual-viewing display. Meanwhile, the user adjusts the viewing angle in real time through head movements or manual instructions.

[0068] Specifically, the fisheye lens group 103 captures a 360° panoramic image and uploads the image data to the image fusion processing module 104. The preprocessing unit 110 receives the image data and performs correction, denoising, resizing and color correction operations. The fusion algorithm unit 111 uses an image fusion algorithm to fuse the preprocessed image to generate a distortion-free panoramic image. The high-definition image transmission communication module 105 receives the panoramic image and optimizes the video compression algorithm and transmission protocol. Using efficient data packet scheduling and error recovery mechanisms, the high-definition image transmission communication module 105 transmits the panoramic video stream to the remote control terminal 102. The remote control terminal 102 receives the panoramic video stream and performs selective dual-view display through the device module 107, such as a mobile phone, PC or smart glasses. The user can adjust the viewing angle in real time through head movements or manual instructions. The control module 108 obtains user instructions and controls the fisheye lens group 103 and the mobile drive module 106 to achieve viewing angle adjustment and robot movement. It reduces the amount of data transmission while ensuring video quality, improves transmission efficiency, ensures stable transmission of panoramic video streams, reduces delays and packet loss, solves the latency problem, and makes control more real-time and comfortable.

[0069] What is disclosed above is only one or more preferred embodiments of the present application, and cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A low-latency dual-view control system for a beyond-visual-range remote-controlled mobile robot, comprising a mobile robot end and a remote control end, wherein the mobile robot end is connected to the remote control end, and the main body carried by the mobile robot end is a robot controlled by beyond-visual-range mobile control, characterized in that: The mobile robot terminal includes a fisheye lens group and an image fusion processing module, and the fisheye lens group and the image fusion processing module are connected; The fisheye lens group is used to capture a 360° panoramic image and upload the image data to the image fusion processing module; The image fusion processing module is used to pre-process the image data, generate a panoramic video stream in real time through an image fusion algorithm, and transmit the panoramic video stream to the remote control terminal through the communication module; The remote control terminal is used for selective dual-viewing angle display and real-time adjustment of the viewing angle based on the user's head movement or manual instructions.

2. The low-latency dual-view control system for a beyond-visual-range remote-controlled mobile robot according to claim 1, characterized in that: The mobile robot terminal further comprises a high-definition image transmission communication module and a mobile driving module, wherein the high-definition image transmission communication module is connected to the image fusion processing module and the remote control terminal respectively, and the mobile driving module is connected to the remote control terminal; The high-definition image transmission communication module is used to optimize the video compression algorithm and transmission protocol, and adopt the data packet scheduling and error recovery mechanism to transmit the panoramic video stream to the remote control terminal; The mobile driving module is used to respond to the control instructions of the remote control terminal and control the movement of the robot.

3. The low-latency dual-view control system for a beyond-visual-range remote-controlled mobile robot as claimed in claim 2, characterized in that: The remote control terminal includes a device module and a control module, the device module is connected to the high-definition image transmission communication module, and the control module is respectively connected to the fisheye lens group and the mobile drive module; The device module includes a mobile phone, a PC or smart glasses, which is used to obtain a panoramic video stream and perform a selective dual-viewing display; The control module is used to obtain user instructions, and control the fisheye lens group and the mobile driving module according to the user instructions to adjust the observation angle in real time.

4. The low-latency dual-view control system for a beyond-visual-range remote-controlled mobile robot as claimed in claim 3, characterized in that: The remote control terminal further includes an inertial measurement module, and the inertial measurement module is connected to the control module; The inertial measurement module is used to monitor the orientation changes of the user's head or the device in real time using an IMU sensor, generate corresponding user instructions, and upload them to the control module.

5. The low-latency dual-view control system for a beyond-visual-range remote-controlled mobile robot according to claim 1, characterized in that: The image fusion processing module includes a preprocessing unit and a fusion algorithm unit, the preprocessing unit is connected to the fisheye lens group, and the fusion algorithm unit is connected to the preprocessing unit; The preprocessing unit is used to receive the image data from the fisheye lens group and perform preprocessing operations of correction, denoising, resizing and color correction on the received image; The fusion algorithm unit is used to use an image fusion algorithm to perform fusion processing on the pre-processed images to generate a distortion-free panoramic image.

6. A low-latency dual-viewing angle control method for a beyond-visual-range remote-controlled mobile robot, applicable to the low-latency dual-viewing angle control system for a beyond-visual-range remote-controlled mobile robot as claimed in claim 1, characterized in that: The following steps are involved: Capture 360° panoramic images and pre-process the image data, including rectification, denoising, resizing, and color correction; The pre-processed images are fused to generate a distortion-free panoramic image; Transmit panoramic video stream to remote control terminal; The remote control end receives the panoramic video stream and displays it in a selective dual-viewing angle. At the same time, the user adjusts the viewing angle in real time through head movements or manual commands.

Images