Remote driving system and method based on low speed state

Abstract

The application discloses a remote driving system and method based on a low-speed state, belongs to the remote driving technology, and aims to solve the technical problem of how to assist intelligent driving when the intelligent driving is in a fault state, reduce the labor intensity, and guarantee the safety of information and driving. The technical scheme is as follows: the system comprises a remote robot end, a robot management server, a remote driving client and a push stream server, the remote robot end pushes video streams to the robot management server after processing pictures of multiple cameras, the remote driving client receives video pictures, and the remote driving client sends instructions to control the remote robot end according to the video pictures; the remote robot end is equipped with a SIM card and a wifi module, so that the remote robot end can communicate through various networks; the robot management server is used for recording the states of all activated remote robot ends and remote robots and positioning information within a set time.

Description

Technical Field

[0001] This invention relates to the field of remote driving technology, specifically a remote driving system and method based on low-speed conditions. Background Technology

[0002] Currently, intelligent driving delivery robots may malfunction under certain extreme working conditions. To address this, initially, a large number of personnel from various locations may need to travel to the scene for rescue, which is time-consuming and labor-intensive. To reduce the workload of humans, remote driving applications have emerged. Remote customer service personnel can first attempt to drive the robot to extricate itself from the predicament. Only for extrication tasks that cannot be completed can they be handed over to humans for rescue, effectively reducing the number of times personnel need to be deployed.

[0003] Therefore, how to assist intelligent driving when it malfunctions, reduce the intensity of manual labor, and at the same time ensure the security of information and driving safety are urgent technical problems that need to be solved. Summary of the Invention

[0004] The technical objective of this invention is to provide a remote driving system and method based on low-speed conditions to address the problem of how to assist intelligent driving when it malfunctions, reduce manual labor intensity, and simultaneously ensure information security and driving safety.

[0005] The technical task of the present invention is achieved in the following manner: a remote driving system based on low speed, the system includes a remote robot terminal, a robot management server, a remote driving client and a streaming server. The remote robot terminal processes the images from multiple cameras and pushes the video stream to the robot management server. The remote driving client then receives the video images and sends instructions to control the remote robot terminal based on the video images.

[0006] The remote robot is equipped with a SIM card and a Wi-Fi module to ensure communication using multiple networks. This invention does not heavily rely on 5G technology, but using 5G will further improve security and reduce the difficulty of use.

[0007] The robot management server is used to record the status (normal status, abnormal status, or remote takeover status) of all activated remote robot terminals and remote robots, as well as their location information within a set time period.

[0008] The remote driving client is used to log in to the robot management server and obtain token information from the robot management server to communicate with the robot management server;

[0009] The streaming server is used to receive data from the streaming client, convert data protocols, and push video data.

[0010] Preferably, the streaming server is used in conjunction with a video CDN server, which is used to maintain low latency and stability of data and reduce packet loss and latency.

[0011] Preferably, the streaming server is built using an SRS real-time video server and configured to receive RTMP protocol data, convert it into RTC data, and then forward it. In actual tests, this process takes very little time.

[0012] Preferably, after receiving new data from all cameras, the remote robot uses OpenCV to synthesize the image and then uses FFmpeg to push a frame of the image via the RTMP protocol. The purpose is to wait for the data to be updated before pushing the stream, thus saving bandwidth consumption.

[0013] Preferably, the remote driving client monitors the images transmitted from the remote robot through a web player in the remote driving client. The images have been converted to the WebRTC protocol by the streaming server, so they can be played using a web player.

[0014] More preferably, the robot management server communicates with the remote robot terminal via encrypted communication, requesting the remote robot terminal to switch to remote driving mode and turn on the remote robot's camera. At the same time, it starts image processing and push functions. The remote robot terminal stitches and compresses multiple camera images using OpenCV, adds auxiliary lines, and then pushes them out.

[0015] More preferably, OpenCV splicing is as follows:

[0016] For the left rear camera, crop out the 480 pixels on the left.

[0017] For the rear right camera, crop out the 480 pixels on the right.

[0018] Left front right splicing;

[0019] The first row of images is horizontally stitched together, and the three images (left, center, and right) are output to hImage.

[0020] Output an image of size (720, 240);

[0021] For the rear camera, crop the top 240 to make it (1280, 720) for easier stitching below;

[0022] Right front, rear, and left front splicing;

[0023] The images are overlaid from the front right, rear, and front left cameras.

[0024] More preferably, when the remote robot terminal switches to remote driving mode, it sends a remote control symbol topic. After receiving the remote control symbol topic, the arbitration node switches control of the chassis to the remote driving client.

[0025] More ideally, the following situations may occur during the operation of a remote robot:

[0026] When the video stream stutters, reduce the forward gear and steering parameters on the remote robot to perform precise full-speed operation; at the same time, increase the command sending frequency to make command control smoother.

[0027] When a network disconnection occurs, the remote robot will brake within 40ms if it does not receive any messages. The remote robot will not move forward directly when the network is disconnected.

[0028] When the remote robot leaves the faulty area, the AI ​​takes over control of the remote robot and switches to autonomous driving mode. If the AI ​​can continue to control the remote robot to perform tasks, the remote driving client can request the release of the remote robot from the robot management server. The robot management server returns the release result to the remote driving client, modifies the remote robot to the normal state, and notifies the remote robot to close the streaming program. Releasing the remote robot will send a topic with AI control symbols, and chassis control will switch to AI.

[0029] A remote driving method based on low-speed conditions, the method is as follows:

[0030] S1. The remote robot terminal is equipped with a SIM card and a Wi-Fi module, and can use multiple networks at the same time. Currently, most parks do not have Wi-Fi and 5G configurations, and most need to be implemented in a 4G environment.

[0031] S2. The remote robot terminal is powered on, automatically connects to the robot management server, and uploads the status of the remote robot terminal in real time;

[0032] S3. The remote driving client logs into the robot management server and obtains token information from the robot management server to communicate with the robot management server;

[0033] S4. When the remote driving client requests to control any remote robot, the robot management server updates the remote robot to the controlled state and returns the result. The robot management server communicates with the remote robot through encrypted communication, requesting the remote robot to switch to remote driving state and turn on the camera, start the image processing and streaming program, and push out the image after stitching and compressing multiple camera images and adding auxiliary lines.

[0034] S5. The streaming server is located close to both the park and the remote driving client. If the distance is too far or multiple parks need to be served, the video CDN server provided by the service provider can be used to maintain low latency and stability of the data.

[0035] S6. The remote driving client sees the image transmitted from the remote robot control terminal. Even with RTMP protocol for pushing, WebRTC protocol for pulling, and CDN server assistance, there is still a certain delay in image transmission. Therefore, the driver needs to make a prediction and leave a sufficient safe distance. Given that the target of this invention is a low-speed target, the safety requirement is met.

[0036] S7. In situations where the network environment in the park is poor, the video stream is prone to stuttering. Adjust the forward speed and the steering parameters to perform fine-grained slow control; at the same time, you can increase the command sending frequency to make command control smoother.

[0037] S8. In step S6, if a prolonged network disconnection occurs, the remote robot will brake within 40ms of not receiving a message, so the remote robot will not move forward directly when the network is disconnected.

[0038] S9. Once the faulty area is cleared, the intelligent driving system takes over the remote robot. The remote robot switches to autonomous driving mode. If the AI ​​can continue to control the remote robot to perform tasks, the remote driving client can request the robot management server to release the robot.

[0039] S10. The robot management server returns the release result to the remote driving client, modifies the remote robot to the normal state, and notifies the remote robot to close the streaming program.

[0040] S11. This remote driving mission has ended.

[0041] The remote driving system and method based on low-speed conditions of the present invention have the following advantages:

[0042] (a) This invention aims to assist intelligent driving. When intelligent driving falls into a malfunction, this invention can restore intelligent driving while ensuring the security of information and the safety of driving.

[0043] (ii) The remote robot terminal of the present invention can switch between multiple states, including autonomous driving state, remote driving state, and forced stop state; at the same time, the remote robot terminal of the present invention can operate under Wi-Fi network or cellular network, which can meet the needs of remote driving at low speeds, and the effect is even better under 5G network.

[0044] (III) This invention assists intelligent driving when it falls into a fault state, reduces the intensity of manual labor, and ensures the security of information and driving safety at the same time; it also ensures the two-way security of data transmission and reception, and sets up an emergency mechanism that is activated when the network is stuck or lost, so that the remote robot can stop quickly and ensure the safety of remote driving.

[0045] (iv) In the event that intelligent driving is in trouble or when necessary, the remote driving client requests the robot management server to take over the selected robot, and ensures the security of two-way information exchange during the process to prevent the robot from being hijacked by others and causing accidents.

[0046] (v) In this invention, there are many areas in the park with poor network conditions. In these areas, both the remote driving client and the remote robot have emergency mechanisms that can automatically brake when the control signal is lost, preventing the remote robot from continuing to execute the previous instructions and causing an accident.

[0047] (vi) This invention stitches and compresses multiple camera images before streaming, reducing bandwidth consumption and data latency. It can also add vehicle body auxiliary lines to the stitched images to prevent collisions caused by not being able to see the width of the vehicle body.

[0048] (vii) This invention differs from traditional live streaming, which uses only one camera. This invention uses 4-8 cameras and requires processing of the data from all cameras before pushing frames. Therefore, the time-based frame pushing is changed to callback-based frame pushing to prevent delays. The image stitching method is bound to a callback function, and the image is pushed out immediately after stitching, instead of setting a frame rate and pushing one frame at intervals. Otherwise, old frames may be pushed out continuously, affecting the driver's judgment. Attached Figure Description

[0049] The invention will be further described below with reference to the accompanying drawings.

[0050] Appendix Figure 1 This is a block diagram of a remote driving system based on low-speed conditions. Detailed Implementation

[0051] The remote driving system and method based on low-speed conditions of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example 1:

[0053] As attached Figure 1As shown, this embodiment provides a remote driving system based on low-speed conditions. The system includes a remote robot terminal, a robot management server, a remote driving client, and a streaming server. The remote robot terminal processes the images from multiple cameras and pushes the video stream to the robot management server. The remote driving client then receives the video images and sends commands to control the remote robot terminal based on the video images.

[0054] The remote robot is equipped with a SIM card and a Wi-Fi module to ensure communication using multiple networks;

[0055] The robot management server is used to record the status (normal status, abnormal status, or remote takeover status) of all activated remote robot terminals and remote robots, as well as their location information within a set time period.

[0056] The remote driving client is used to log in to the robot management server and obtain token information from the robot management server to communicate with the robot management server;

[0057] The streaming server is used to receive data from the streaming client, convert data protocols, and push video data.

[0058] In this embodiment, the streaming server is used in conjunction with a video CDN server. The key code is as follows:

[0059] export CANDIDATE="192.168.1.10"

[0060] docker run--rm--env CANDIDATE=$CANDIDATE\

[0061] -p 1935:1935-p 8080:8080-p 1985:1985-p 8000:8000 / udp\

[0062] registry.cn-hangzhou.aliyuncs.com / ossrs / srs:4\

[0063] objs / srs-c conf / rtmp2rtc.conf.

[0064] Video CDN servers are used to maintain low latency and stability of data, reducing packet loss and latency.

[0065] In this embodiment, the streaming server is built using an SRS real-time video server and configured to receive RTMP protocol data, convert it into RTC data, and then forward it. In actual testing, this process takes very little time.

[0066] In this embodiment, after receiving new data from all cameras, the remote robot uses OpenCV to synthesize the image and then uses FFmpeg to push a frame of the image via the RTMP protocol. The purpose is to wait for the data to be updated before pushing the stream, thus saving bandwidth consumption.

[0067] In this embodiment, the remote driving client monitors the images transmitted from the remote robot through a web player in the remote driving client. The images have been converted to the WebRTC protocol by the streaming server, so they can be played using a web player.

[0068] In this embodiment, the robot management server and the remote robot client communicate via encrypted communication. The server requests the remote robot client to switch to remote driving mode and activate its camera. Simultaneously, it initiates image processing and push functions. The remote robot client uses OpenCV to stitch and compress multiple camera feeds, adds auxiliary lines, and then pushes the feed out. Stitching and compression are performed to save bandwidth and to create a visual representation of a human driving scenario, including the main interface, left and right rearview mirrors, and a front rearview mirror. Auxiliary lines are added because the vehicle body is not visible in the camera lens, making obstacle avoidance difficult; these lines address the challenge of navigating narrow roads and obstacles. When the remote robot client switches to remote driving mode, it sends a remote control symbol topic. Upon receiving this topic, the arbitration node transfers chassis control to the driving client. OpenCV clipping and stitching code:

[0069]

[0070]

[0071] In this embodiment, when the remote robot switches to remote driving mode, it sends a remote control symbol topic. After receiving the remote control symbol topic, the arbitration node switches control of the chassis to the remote driving client.

[0072] The following situations may occur during the operation of the remote robot in this embodiment:

[0073] When the video stream stutters, reduce the forward gear and steering parameters on the remote robot to perform precise full-speed operation; at the same time, increase the command sending frequency to make command control smoother.

[0074] When a network disconnection occurs, the remote robot will brake within 40ms if it does not receive any messages. The remote robot will not move forward directly when the network is disconnected.

[0075] When the remote robot leaves the faulty area, the AI ​​takes over control of the remote robot and switches to autonomous driving mode. If the AI ​​can continue to control the remote robot to perform tasks, the remote driving client can request the release of the remote robot from the robot management server. The robot management server returns the release result to the remote driving client, modifies the remote robot to the normal state, and notifies the remote robot to close the streaming program. Releasing the remote robot will send a topic with AI control symbols, and chassis control will switch to AI.

[0076] Example 2:

[0077] This embodiment provides a remote driving method based on low-speed conditions, the method is as follows:

[0078] S1. The remote robot terminal is equipped with a SIM card and a Wi-Fi module, and can use multiple networks at the same time. Currently, most parks do not have Wi-Fi and 5G configurations, and most need to be implemented in a 4G environment.

[0079] S2. When the remote robot is powered on, it automatically connects to the robot management server and uploads the status of the remote robot in real time. When a new remote robot is activated, it will automatically connect to the robot management server set in the firmware. The robot management server records all activated remote robot terminals and their status (normal, abnormal, or under remote takeover), as well as basic data such as location over a period of time.

[0080] S3. The remote driving client logs into the robot management server and obtains token information from the robot management server to communicate with the robot management server;

[0081] S4. When a remote driving client requests control of any remote robot, the robot management server updates the remote robot to a controlled state and returns the result. The robot management server communicates with the remote robot via encrypted communication, requesting the remote robot to switch to remote driving mode and activate its camera. It then initiates image processing and streaming, stitching and compressing multiple camera feeds, adding guide lines, and pushing the image out. The stitching and compression are done to save bandwidth and to create a visual representation of a human driving scenario, including the main interface, left and right rearview mirrors, and a front rearview mirror. Guide lines are added because the vehicle body is not visible in the camera lens, making obstacle avoidance difficult; these lines address the challenge of navigating narrow roads and obstacles. Once the remote robot switches to remote driving mode, it sends a remote control symbol topic. Upon receiving this topic, the arbitration node transfers control of the chassis to the remote client. OpenCV clipping and stitching code:

[0082]

[0083]

[0084] S5. The streaming server is located close to both the park and the remote driving client. If the distance is too far or multiple parks need to be served, the video CDN server provided by the service provider can be used to maintain low latency and stability of the data.

[0085] The function of the streaming server is to receive data from the streaming client, convert data protocols, and push video data. It can be used in conjunction with a video CDN server to reduce packet loss and latency. The streaming server needs to be set up in a location relatively close to both the campus and the remote driving terminal. If the distance is too far or multiple campuses need to be served, a CDN server (Content Delivery Network) provided by the service provider can be used to address the issues of high latency and packet loss due to long distances; it is generally not needed in most cases. This maintains low latency and stability of the data. The streaming server is set up using an SRS real-time video server; key code is as follows:

[0086] export CANDIDATE="192.168.1.10"

[0087] docker run--rm--env CANDIDATE=$CANDIDATE\

[0088] -p 1935:1935-p 8080:8080-p 1985:1985-p 8000:8000 / udp\registry.cn-hangzhou.aliyuncs.com / ossrs / srs:4\

[0089] objs / srs-c conf / rtmp2rtc.conf

[0090] Configure the server so that the robot management server receives RTMP protocol data, converts it into RTC data, and then forwards it. In actual testing, this process takes very little time.

[0091] After receiving new data from all cameras, the remote robot uses ffmpeg to stream one frame via the RTMP protocol. This is to save bandwidth by waiting for data updates before streaming. Below is the encoder configuration code:

[0092]

[0093] S6. The remote driving client can view the video feed from the remote robot control terminal via a web player within the client. The video feed has been converted to the WebRTC protocol by the streaming server, so it can be played using a web player. Even with RTMP streaming and WebRTC streaming with CDN server assistance, there is still a certain latency in image transmission. Therefore, the driver needs to anticipate this and maintain a sufficient safety distance. Given that this solution targets low-speed objects, the safety requirements are met.

[0094] S7. In situations where the network environment in the park is poor, the video stream is prone to stuttering. Adjust the forward speed and the steering parameters to perform fine-grained slow control; at the same time, you can increase the command sending frequency to make command control smoother.

[0095] S8. In step S6, if a prolonged network disconnection occurs, the remote robot will brake within 40ms of not receiving a message, so the remote robot will not move forward directly when the network is disconnected.

[0096] S9. Once the faulty area is cleared, the intelligent driving system takes over the remote robot. The remote robot switches to autonomous driving mode. If the AI ​​can continue to control the remote robot to perform tasks, the remote driving client can request the robot management server to release the robot.

[0097] S10. The robot management server returns the release result to the remote driving client, modifies the remote robot to the normal state, and notifies the remote robot to close the streaming program.

[0098] S11. This remote driving mission has ended.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A remote driving system based on low-speed conditions, characterized in that, The system includes a remote robot terminal, a robot management server, a remote driving client, and a streaming server. The remote robot terminal processes the images from multiple cameras and pushes the video stream to the streaming server. The remote driving client then receives the video images and sends commands to control the remote robot terminal based on the video images. The remote robot is equipped with a SIM card and a Wi-Fi module to ensure communication using multiple networks; The robot management server is used to record all activated remote robot terminals, the status of remote robots, and their location information within a set time period; The remote driving client is used to log in to the robot management server and obtain token information from the robot management server to communicate with the robot management server; The streaming server is used to receive data from the streaming client, convert data protocols, and push video data. The streaming server is used in conjunction with a video CDN server, which is used to maintain low latency and stability of the data. The streaming server is set up using an SRS real-time video server, and configured to receive RTMP protocol data, convert it into RTC data, and then forward it. After receiving new data from all cameras, the remote robot uses OpenCV to composite the image and then uses FFmpeg to stream one frame via the RTMP protocol. The purpose is to wait for data updates before streaming. The image stitching method is bound to a callback function so that the image is pushed out immediately after stitching. The remote driving client monitors the video feed transmitted from the remote robot via a web player within the client. The video feed has been converted to the WebRTC protocol by the streaming server. The robot management server communicates with the remote robot terminal via encrypted communication, requesting the remote robot terminal to switch to remote driving mode and turn on the remote robot's camera. At the same time, it starts image processing and push functions. The remote robot terminal stitches and compresses multiple camera images using OpenCV, adds auxiliary lines, and then pushes them out. When the remote robot switches to remote driving mode, it sends a remote control symbol topic. After receiving the remote control symbol topic, the arbitration node switches chassis control to the remote driving client. The following situations may occur during the operation of a remote robot: When the video stream stutters, reduce the forward gear and steering parameters on the remote robot to perform precise full-speed operation; at the same time, increase the command sending frequency to make command control smoother. When a network disconnection occurs, the remote robot will brake within 40ms if it does not receive any messages. The remote robot will not move forward directly when the network is disconnected. When the remote robot leaves the faulty area, the AI ​​takes over control of the remote robot and switches to autonomous driving mode. If the AI ​​can continue to control the remote robot to perform tasks, the remote driving client can request the release of the remote robot from the robot management server. The robot management server returns the release result to the remote driving client, modifies the remote robot to the normal state, and notifies the remote robot to close the streaming program. Releasing the remote robot will send a topic with AI control symbols, and chassis control will switch to AI.

2. A remote driving method based on low-speed conditions, characterized in that, This method is based on the remote driving system under low-speed conditions as described in claim 1, and is as follows: S1. The remote robot terminal is equipped with a SIM card and a Wi-Fi module, and can use multiple networks simultaneously; S2. The remote robot terminal is powered on, automatically connects to the robot management server, and uploads the status of the remote robot terminal in real time; S3. The remote driving client logs into the robot management server and obtains token information from the robot management server to communicate with the robot management server; S4. When a remote driving client requests to control any remote robot, the robot management server updates the remote robot to the "under control" status and returns the result. The robot management server communicates with the remote robot terminal via encrypted communication, requesting the remote robot terminal to switch to remote driving mode and turn on the camera, start the image processing and streaming program, and push out the image after stitching and compressing multiple camera images and adding auxiliary lines. S5. The streaming server is located close to the park and the remote driving client. If the distance is far or multiple parks need to be served, a video CDN server is used to maintain low latency and stability of the data. S6. The remote driving client sees the screen transmitted from the remote robot control terminal; S7. In situations where the network environment in the park is poor, the video stream is prone to stuttering. Adjust the forward speed and the steering parameters to perform fine-grained slow control; at the same time, you can increase the command sending frequency to make command control smoother. S8. In step S6, if the network connection is lost, the remote robot will brake within 40ms if it does not receive any messages. Therefore, the remote robot will not move forward directly when the network is lost. S9. Once the faulty area is cleared, the intelligent driving system takes over the remote robot. The remote robot switches to autonomous driving mode. If the AI ​​can continue to control the remote robot to perform tasks, the remote driving client can request the robot management server to release the robot. S10. The robot management server returns the release result to the remote driving client, modifies the remote robot to the normal state, and notifies the remote robot to close the streaming program. S11. This remote driving mission has ended.

Images