Mobile platform emergency remote control system based on environment monitoring video

Abstract

The invention discloses a mobile platform emergency remote control system based on an environment monitoring video, which belongs to the technical field of automation and computers and comprises a remote control server, a mobile platform, a remote control terminal and a monitoring platform. The remote control server comprises a data interaction and video transmission module. The mobile platform comprises a sensing module, a positioning module, a communication module, a control module and a task module. The remote control terminal comprises a communication module, a state monitoring module and a remote control module. The monitoring platform comprises a monitoring platform server and a plurality of monitoring cameras. And when the camera of the mobile platform is invalid, the remote control terminal obtains the video of the monitoring camera near the platform from the monitoring server, so that the operator can continue to complete the remote control task. According to the invention, an emergency solution assisted by a monitoring system is provided for remote control of the mobile platform, the adaptive capacity of the unmanned mobile platform is improved, the operation continuity is guaranteed, and the operation risk of the mobile platform is reduced.

Description

Technical Field

[0001] The invention belongs to the field of automation and computer technology, and in particular relates to a mobile platform emergency remote control system and method based on environmental monitoring video. Background Art

[0002] Remote control of mobile platforms can be used as a solution to automatic mobile platform failure or abnormality, improving the safety and efficiency of mobile platform operations. Through remote control, operators can better cope with dangerous environments without being exposed to dangerous conditions, which not only helps personnel safety, but also improves production efficiency and reduces operating costs.

[0003] In existing remote control technologies, sensors such as cameras are usually relied upon to obtain environmental data, and these data are transmitted to remote operators in real time through wireless communication technology. The operators remotely control the mobile platform through display screens and control devices.

[0004] Chinese Patent Application: A remote driving method and remote control device based on an autonomous driving system (Application No.: 202410501027.2) discloses a remote driving method, which aims to solve the problem of how to ensure the safe control and safe driving of the autonomous driving vehicle when there are problems with the cockpit network communication or the driver himself. However, the patent does not solve the problem after the camera fails during the remote driving process.

[0005] Chinese Patent Application: Remote Control Driving System for Cooperative Human-Machine Co-driving with Vehicle-Road-Cloud (Application No.: 202410793474.X) discloses a remote control driving system for cooperative human-machine co-driving with vehicle-road-cloud, which aims to solve the problem that the remote human-machine co-driving solution cannot be used in the event of a system failure. In this patent, remote control is achieved by directly communicating between the cloud cockpit and the vehicle, which cannot effectively support remote control based on a wide area network.

[0006] Remote control plays an important role in the field of unmanned mobile platforms. The above-mentioned documents have certain limitations in different aspects, lack the ability to adapt to the environment and faults, and it is difficult to ensure the continuity of operations. Therefore, the remote control technology of unmanned mobile platforms needs to be improved. Summary of the invention

[0007] The present invention aims to solve the above problems of the prior art. A mobile platform emergency remote control system based on environmental monitoring video is proposed. The technical solution of the present invention is as follows:

[0008] A mobile platform emergency remote control system based on environmental monitoring video, comprising:

[0009] Remote control server: It consists of two parts: data interaction module and video transmission module. The data interaction module is used to process the requests from the mobile platform and the remote control terminal, and forward the data messages from the mobile platform and the remote control terminal. The video transmission module is used to process and transmit video data to ensure that the video stream is transmitted to the remote control terminal in real time.

[0010] Mobile platform: including perception module, positioning module, mobile platform communication module, control module and task module; the perception module is responsible for collecting environmental data around the mobile platform and real-time video from the camera; the positioning module is responsible for obtaining the positioning information of the mobile platform; the platform communication module is responsible for data communication with the remote control server to achieve real-time data transmission; the task module is responsible for sending control instructions to the control module according to the preset operation tasks; the control module is responsible for controlling the mobile platform according to the received data and instructions;

[0011] Remote control terminal: including remote control module, remote control terminal communication module and status monitoring module; the remote control module is responsible for collecting data of control equipment for remote control by the operator; the remote control terminal communication module is used for data communication with the remote control server to ensure that the operator's control instructions can be transmitted to the mobile platform in real time and obtain real-time mobile platform camera video; the status monitoring module is used to display real-time video and mobile platform status information obtained from the mobile platform and monitoring platform;

[0012] Monitoring platform: includes monitoring platform server and multiple monitoring cameras. The video of each monitoring camera is transmitted to the monitoring platform.

[0013] Furthermore, when the mobile platform is in normal working state, it does not need the support of monitoring platform, and its operation steps are as follows:

[0014] 2.1 When the mobile platform is in normal working state, it is in automatic working mode and performs operations according to a given path. During the operation, the vehicle status information and camera images with target detection results are continuously uploaded to the data interaction and video transmission module of the remote control server through the communication module;

[0015] 2.2 When the remote control terminal is in normal working state, during the operation of the mobile platform, the remote control terminal will request the data interaction module and video transmission module of the remote control server to obtain the status information and camera image of the specified mobile platform, and display the obtained data visually on the screen;

[0016] 2.3 When the mobile platform encounters an abnormal environment during the automatic operation and the operation task cannot be performed, it stops the operation and requests manual takeover from the remote control server; the remote control server forwards the manual takeover request to the remote control terminal;

[0017] 2.4 After receiving the manual takeover request from the mobile platform, the remote control terminal requests remote takeover from the remote control server. After the request is approved, the operation mode of the mobile platform is switched from automatic mode to collaborative mode or remote control mode; after the operator remotely controls the mobile platform to the designated area, the operation mode of the mobile platform is switched to automatic mode.

[0018] Furthermore, in collaborative mode, when the mobile platform sensor detects a collision risk, it will reject the remote control command and return an alarm message; in remote control mode, when the mobile platform sensor detects a collision risk, it will still operate according to the remote control command.

[0019] Furthermore, when the mobile platform is in emergency working state, the monitoring platform needs to support it. The operation steps are as follows:

[0020] 3.1 The mobile platform stops working immediately after detecting a camera failure, and sends a fault message to the remote control server data interaction module through the communication module, and requests remote takeover;

[0021] 3.2 After receiving the mobile platform fault message through the remote control server, the remote control terminal analyzes the received fault information; based on the positioning information provided by the mobile platform, it communicates with the monitoring platform through the communication module to obtain the videos of several monitoring cameras closest to the mobile platform;

[0022] 3.3 The remote control terminal performs a bird's-eye view BEV transformation on multiple mobile platform environment monitoring videos obtained from the monitoring platform;

[0023] 3.4 The remote control terminal performs splicing processing on the multiple monitoring images transformed by the BEV; after the splicing processing is completed, a complete view of the surrounding environment of the mobile platform is presented on the display of the remote control terminal;

[0024] 3.5 The remote control operator controls the mobile platform to the designated destination based on the surrounding environment view of the mobile platform and the auxiliary information provided by the system;

[0025] 3.6 Repeat steps 3.2-3.5 until the mobile platform is remotely controlled to the designated end point and wait for subsequent processing.

[0026] Furthermore, the BEV transformation and splicing processing steps are as follows:

[0027] 4.1BEV transformation: First, the camera’s intrinsic parameter matrix and distortion coefficients are used to correct image distortion. Then, the camera’s extrinsic parameters are used to transform the image of each camera to a unified bird’s-eye view through the perspective transformation matrix. Finally, the camera image is geometrically transformed through the perspective transformation matrix to transform it from the perspective around the mobile platform to a bird’s-eye view.

[0028] 4.2 Stitching: First, by using feature point matching and image registration algorithms, find the overlapping areas and align them accurately; then fuse these overlapping areas; finally, adjust the image color through color correction and color smoothing algorithms to make it transition naturally at the stitching.

[0029] Furthermore, the feature point matching and image registration algorithm specifically includes:

[0030] First, feature extraction is performed on the two images to obtain feature points. Then, matching feature point pairs are found by performing similarity measurement. Then, image space coordinate transformation parameters are obtained through the matching feature point pairs. Finally, image registration is performed based on the coordinate transformation parameters.

[0031] Furthermore, the color correction and color smoothing algorithm specifically includes:

[0032] First, the color distribution of adjacent images is adjusted through color histogram matching to reduce inconsistency. Second, independent color balance of RGB channels is performed to optimize contrast and brightness to make the colors more natural. Next, weighted averaging or multi-resolution fusion technology is used to process overlapping areas to smooth transitions and reduce color differences. Subsequently, Gaussian blur and other filters are used to smooth the stitching edges to eliminate abruptness. Finally, local brightness and global hue adjustments are made to ensure that the brightness and hue of the entire stitched image are consistent.

[0033] The advantages and beneficial effects of the present invention are as follows:

[0034] The present invention provides a mobile platform emergency remote control system and method based on environmental monitoring video. The system is divided into two working state modes: normal working state and emergency working state, ensuring that emergency situations of unmanned mobile platforms can be handled safely and efficiently under different situations.

[0035] The present invention realizes the safe and flexible operation of the unmanned mobile platform in emergency situations through the combination of remote control and environmental monitoring video, as well as the dual-mode switching between normal working state and emergency working state. When a fault or emergency is detected, the system automatically switches to the emergency state, triggering remote operation. The operator can control the surrounding environment of the platform in real time through the video and make precise control.

[0036] Whether in normal working conditions or in emergency situations, the system can respond flexibly, through multi-mode switching and remote control, to ensure that the unmanned mobile platform can adapt to various complex environments and ensure operation continuity and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The present invention provides a preferred embodiment of a mobile platform emergency remote control system and method based on environmental monitoring video.

[0038] Figure 2 This is a schematic diagram of the operation mode of the mobile platform provided by the present invention.

[0039] Figure 3 This is a flow chart of the normal working state of the mobile platform provided by the present invention.

[0040] Figure 4 This is a flow chart of the emergency working state operation of the mobile platform provided by the present invention. DETAILED DESCRIPTION

[0041] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0042] The technical solution of the present invention to solve the above technical problems is:

[0043] Figure 1 The present invention shows a mobile platform emergency remote control system and method based on environmental monitoring video. The system includes four parts: a remote control server, a mobile platform, a remote control terminal and a monitoring platform, wherein:

[0044] Remote control server: It includes two parts: data interaction and video transmission module. The data interaction module is responsible for processing the requests from the mobile platform and the remote control terminal, and forwarding the data messages from the mobile platform and the remote control terminal. The video transmission module is responsible for processing and transmitting video data, ensuring that the video stream is transmitted to the remote control terminal in real time.

[0045] Mobile platform: includes perception, positioning, communication, control and task modules. The perception module is responsible for collecting environmental data around the mobile platform and real-time video from the camera. The positioning module is responsible for obtaining the positioning information of the mobile platform. The communication module is responsible for data communication with the remote control server to achieve real-time data transmission. The task module is responsible for sending control instructions to the control module according to the preset operation tasks. The control module is responsible for controlling the mobile platform according to the received data and instructions.

[0046] Remote control terminal: includes remote control, communication and status monitoring modules. The remote control module is responsible for collecting data from the control equipment for the operator to perform remote control. The communication module is used to communicate data with the remote control server to ensure that the operator's control instructions can be transmitted to the mobile platform in real time and obtain real-time mobile platform camera video. The status monitoring module is responsible for displaying real-time video and mobile platform status information obtained from the mobile platform and monitoring platform.

[0047] Monitoring platform: includes monitoring platform server and multiple monitoring cameras. The video of each monitoring camera is transmitted to the monitoring platform.

[0048] Figure 2 Shown are three mobile platform operation modes of a mobile platform emergency remote control system and method based on environmental monitoring video of the present invention, namely automatic mode, collaborative mode and remote control mode.

[0049] (1) Automatic mode:

[0050] Status description: In automatic mode, all actions of the mobile platform are completely determined and instructed by the mobile platform based on the data of the surrounding environment. The remote control terminal cannot perform other operations except remote takeover.

[0051] Instruction calculation: The instructions of the mobile platform have the highest authority, namely:

[0052] Cmd final =Cmd move

[0053] (2) Remote mode:

[0054] Status description: In remote mode, all actions of the mobile platform are completely controlled by the remote operator. Even if the mobile platform detects an obstacle in front of it, it will only remind the operator and will not stop the action. In this mode, the remote operator has the highest operating authority.

[0055] Command calculation: The remote control operator's command has the highest authority, namely:

[0056] Cmd final =Cmd remote

[0057] (3) Collaborative model:

[0058] Status description: In collaborative mode, the action of the mobile platform is determined by the mobile platform's comprehensive data on the surrounding environment and the remote control operator's instructions. For example, when the mobile platform detects an obstacle ahead and needs to stop, if the remote control operator gives the instruction to move forward, the instruction will not be executed to ensure safety.

[0059] Command calculation: Integrate the commands from the mobile platform and the remote operator:

[0060]

[0061] Among them, flag threat =1 means the mobile platform detects an obstacle that threatens collision. threat =0 means no.

[0062] Figure 3This is a specific flow chart of the normal working state in the two operating states of the system of the present invention. The mobile platform operation process is as follows:

[0063] (1) The mobile platform starts and initializes all systems and sensors.

[0064] (2) The mobile platform is connected to the remote control server through the communication module. If the connection fails, a reconnection attempt is made. If the connection is successful, the process proceeds to step (3).

[0065] (3) The mobile platform enters automatic mode and begins to operate according to the predetermined route and tasks.

[0066] (4) Use the perception module for environment perception and target detection.

[0067] (5) Continue uploading the status information of the mobile platform to the remote control server data interaction module. If a camera image request is received from the remote control server, proceed to step (6); otherwise, proceed to step (7).

[0068] (6) Upload the camera image to the remote control server video transmission module.

[0069] (7) Make a judgment based on the target detection result. If a potential danger (such as an obstacle or a pedestrian) is detected, stop the action immediately and go to step (8). Otherwise, go to step (4).

[0070] (8) During the automatic operation of the mobile platform, if the sensor detects an abnormal environment that makes the operation task unable to be performed, the task will be stopped immediately, and the current status information and camera images will be uploaded to the remote control server through the communication module, and a manual takeover request will be sent at the same time.

[0071] (9) After the remote control terminal takes over, the operation mode of the mobile platform is switched from automatic mode to collaborative mode or remote control mode.

[0072] Secondly, the operation process of the remote control terminal is as follows:

[0073] (1) Start the remote control terminal and initialize all necessary hardware and software components.

[0074] (2) The remote control terminal attempts to connect to the remote control server through the communication module. If the connection is successful, proceed to step (3), otherwise repeat step (2).

[0075] (3) Obtain a list of currently active mobile platforms from the remote control server and connect to the specified mobile platform.

[0076] (4) Send a request to obtain the status information and camera image of the specified mobile platform.

[0077] (5) Visually display the data obtained from the remote control server on the terminal screen, including the mobile platform status information and camera images. If a remote takeover request is received from the mobile platform, proceed to step (6); otherwise, proceed to step (4).

[0078] (6) When the remote control terminal receives the remote takeover request from the mobile platform, it sends a remote takeover request to the remote control server and waits for the server's confirmation and authorization.

[0079] (7) After the takeover request is authorized, the operator remotely controls the mobile platform through the remote control terminal, uses the status information and camera images uploaded by the mobile platform to avoid obstacles, and safely controls the mobile platform to the designated safe area.

[0080] (8) After the mobile platform is safely remotely controlled to the designated area, the operator changes the operating mode of the mobile platform back to the automatic mode. The mobile platform re-enters the automatic mode and continues to perform its mission.

[0081] Figure 4 This is a specific flow chart of the emergency working state of the system of the present invention.

[0082] The mobile platform operation process is as follows:

[0083] (1) When a camera fails during the operation of the mobile platform, an emergency response is triggered and the mobile platform immediately stops the mission.

[0084] (2) The mobile platform uploads the current status information and environmental data to the remote control server and requests remote takeover from the remote control server.

[0085] (3) After receiving the control information from the remote control terminal, the operation mode is changed to the remote control mode.

[0086] The operation process of the remote control terminal is as follows:

[0087] (1) After receiving the remote takeover request, the monitoring platform server is requested to obtain the video data of multiple monitoring cameras closest to the faulty mobile platform based on the positioning information of the mobile platform.

[0088] (2) Perform bird's-eye view (BEV) transformation on multiple surveillance videos. First, the camera's intrinsic parameter matrix and distortion coefficient are used to correct image distortion. Then, the camera's extrinsic parameters are used to transform the image of each camera to a unified bird's-eye view through a perspective transformation matrix. Finally, the camera image is geometrically transformed through the perspective transformation matrix to transform it from the perspective around the mobile platform to a bird's-eye view.

[0089] (3) After BEV transformation, multiple surveillance video images are stitched together. First, feature point matching and image registration algorithms are used to find overlapping areas and align them accurately. These overlapping areas are then fused. Finally, color correction and color smoothing algorithms are used to adjust the image colors so that they have a natural transition at the stitching points.

[0090] (4) After the stitching process is completed, a complete view of the surrounding environment is presented on the display of the remote control terminal.

[0091] (5) The operator remotely controls the mobile platform to a nearby safe area based on the surveillance video and stops the action to wait for the staff to handle the situation.

[0092] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions.

[0093] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0094] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A mobile platform emergency remote control system based on environmental monitoring video, characterized in that: include: Remote control server: It includes two parts: data interaction module and video transmission module. The data interaction module is used to process the requests from the mobile platform and the remote control terminal, and forward the data messages from the mobile platform and the remote control terminal. The video transmission module is used for processing and transmitting video data to ensure that the video stream is transmitted to the remote control terminal in real time; Mobile platform: including perception module, positioning module, mobile platform communication module, control module and task module; the perception module is responsible for collecting environmental data around the mobile platform and real-time video from the camera; the positioning module is responsible for obtaining the positioning information of the mobile platform; the platform communication module is responsible for data communication with the remote control server to achieve real-time data transmission; the task module is responsible for sending control instructions to the control module according to the preset operation tasks; the control module is responsible for controlling the mobile platform according to the received data and instructions; Remote control terminal: including remote control module, remote control terminal communication module and status monitoring module; the remote control module is responsible for collecting data of control equipment for remote control by the operator; the remote control terminal communication module is used for data communication with the remote control server to ensure that the operator's control instructions can be transmitted to the mobile platform in real time and obtain real-time mobile platform camera video; the status monitoring module is used to display real-time video and mobile platform status information obtained from the mobile platform and monitoring platform; Monitoring platform: includes monitoring platform server and multiple monitoring cameras. The video of each monitoring camera is transmitted to the monitoring platform.

2. According to claim 1, a mobile platform emergency remote control system based on environmental monitoring video is characterized in that: When the mobile platform is in normal working state, it does not need the support of monitoring platform, and its operation steps are as follows: 2.1 When the mobile platform is in normal working state, it is in automatic working mode and performs operations according to a given path. During the operation, the vehicle status information and camera images with target detection results are continuously uploaded to the data interaction and video transmission module of the remote control server through the communication module; 2.2 When the remote control terminal is in normal working state, during the operation of the mobile platform, the remote control terminal will request the data interaction module and video transmission module of the remote control server to obtain the status information and camera image of the specified mobile platform, and display the obtained data visually on the screen; 2.3 When the mobile platform encounters an abnormal environment during the automatic operation and the operation task cannot be performed, it stops the operation and requests manual takeover from the remote control server; the remote control server forwards the manual takeover request to the remote control terminal; 2.4 After receiving the manual takeover request from the mobile platform, the remote control terminal requests remote takeover from the remote control server. After the request is approved, the operation mode of the mobile platform is switched from automatic mode to collaborative mode or remote control mode; after the operator remotely controls the mobile platform to the designated area, the operation mode of the mobile platform is switched to automatic mode.

3. The mobile platform emergency remote control system based on environmental monitoring video according to claim 2 is characterized in that: In collaborative mode, when the mobile platform sensor detects a collision risk, it will reject the remote control command and return an alarm message; in remote control mode, when the mobile platform sensor detects a collision risk, it will still operate according to the remote control command.

4. The mobile platform emergency remote control system based on environmental monitoring video according to claim 1 is characterized in that: When the mobile platform is in emergency working state, the monitoring platform needs support. The operation steps are as follows: 3.1 The mobile platform stops working immediately after detecting a camera failure, and sends a fault message to the remote control server data interaction module through the communication module, and requests remote takeover; 3.2 After receiving the mobile platform fault message through the remote control server, the remote control terminal analyzes the received fault information; based on the positioning information provided by the mobile platform, it communicates with the monitoring platform through the communication module to obtain the videos of several monitoring cameras closest to the mobile platform; 3.3 The remote control terminal performs a bird's-eye view BEV transformation on multiple mobile platform environment monitoring videos obtained from the monitoring platform; 3.4 The remote control terminal performs splicing processing on the multiple monitoring images transformed by the BEV; after the splicing processing is completed, a complete view of the surrounding environment of the mobile platform is presented on the display of the remote control terminal; 3.5 The remote control operator controls the mobile platform to the designated destination based on the surrounding environment view of the mobile platform and the auxiliary information provided by the system; 3.6 Repeat steps 3.2-3.5 until the mobile platform is remotely controlled to the designated end point and wait for subsequent processing.

5. The mobile platform emergency remote control system based on environmental monitoring video according to claim 4 is characterized in that: The BEV transformation and splicing processing steps are as follows: 4.1BEV transformation: First, the camera’s intrinsic parameter matrix and distortion coefficients are used to correct image distortion. Then, the camera’s extrinsic parameters are used to transform the image of each camera to a unified bird’s-eye view through the perspective transformation matrix. Finally, the camera image is geometrically transformed through the perspective transformation matrix to transform it from the perspective around the mobile platform to a bird’s-eye view. 4.2 Stitching: First, by using feature point matching and image registration algorithms, find the overlapping areas and align them accurately; then fuse these overlapping areas; finally, adjust the image color through color correction and color smoothing algorithms to make it transition naturally at the stitching.

Images