Remote driving and remote monitoring method and system based on unmanned platform

Abstract

The invention provides a remote driving and remote monitoring method and system based on an unmanned platform, and relates to the technical field of unmanned vehicles, and the method comprises the steps: obtaining steering wheel data and environment image data of an unmanned sanitation vehicle; determining an operation mode of the unmanned sanitation vehicle based on the steering wheel data; when the operation mode of the unmanned sanitation vehicle is automatic driving, the unmanned sanitation vehicle is controlled to automatically drive by using the optimal driving route; when the operation mode of the unmanned sanitation vehicle is remote driving, the unmanned platform is used for processing steering wheel data and environment image data to obtain a monitoring video of the unmanned sanitation vehicle, and the speed and direction are controlled; based on the monitoring video, the control speed and the direction of the unmanned sanitation vehicle, a remote driving instruction is obtained, remote driving of the unmanned sanitation vehicle is carried out, and remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform are completed. The problem that the unmanned sanitation vehicle is difficult to operate safely and efficiently in a complex environment is solved.

Description

Technical Field

[0001] This specification relates to the technical field of driverless vehicles, and particularly to a remote driving and remote monitoring method and system based on an unmanned platform. Background Art

[0002] In recent years, with the rapid development of autonomous driving technology and communication technology, all walks of life have been exploring how to apply these new technologies to improve efficiency and safety, and the sanitation industry is no exception. Traditional sanitation operations usually require a large amount of manual labor, which is not only costly, but also workers need to work in complex and sometimes dangerous environments. Unmanned platforms, including automatic cleaning vehicles, can perform tasks such as sweeping, vacuuming, and washing without human intervention. The emergence of unmanned platforms provides an automated solution for the sanitation industry, which can significantly improve operation efficiency and safety. However, when operating in the complex road environment of the city, unmanned platforms face many challenges, such as high-density traffic flow, pedestrians, various fixed and moving obstacles, etc. These complex situations pose high requirements on the navigation, perception, and real-time response capabilities of unmanned sanitation vehicles.

[0003] Most existing driverless sanitation vehicle platforms rely on traditional wireless communication technologies, such as Wi-Fi or 4G; most existing technologies use a single-view camera for video monitoring, and these technologies have deficiencies in terms of transmission stability and real-time performance in complex road environments; most existing technologies use a single-view camera for video monitoring, and it is difficult to comprehensively grasp the vehicle's surrounding environment in case of emergencies, posing potential safety hazards; some existing models will use UDP or TCP protocols for communication operations within a local area network. Although the transmission stability and real-time performance are greatly increased, the remote operation range is greatly reduced, making it difficult to meet actual needs. Summary of the Invention

[0004] Aiming at the above deficiencies in the prior art, a remote driving and remote monitoring method and system based on an unmanned platform provided by the present invention solves the problem that it is difficult for unmanned sanitation vehicles to operate safely and efficiently in complex environments.

[0005] To achieve the above invention objective, the technical solution adopted by the present invention is: A remote driving and remote monitoring method based on an unmanned platform, comprising: S1: Obtain the steering wheel data and environmental image data of the unmanned sanitation vehicle; S2: Based on the steering wheel data, determine the operation mode of the unmanned sanitation vehicle; the operation mode includes autonomous driving and remote driving; S3: When the operation mode of the unmanned sanitation vehicle is autonomous driving, use the grid map algorithm and dynamic obstacle avoidance algorithm to calculate the driving route, obtain the optimal driving route, and control the unmanned sanitation vehicle to drive autonomously; S4: When the operation mode of the unmanned sanitation vehicle is remote driving, based on the image stitching technology and color space conversion algorithm, the unmanned platform is used to process the steering wheel data and environmental image data to obtain the monitoring video, control speed and direction of the unmanned sanitation vehicle; S5: Based on the monitoring video, control speed and direction of the unmanned sanitation vehicle, obtain remote driving instructions, and remotely drive the unmanned sanitation vehicle to complete the remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

[0006] The beneficial effects of the present invention are as follows: The processor uses the steering wheel data and environmental image data of the unmanned sanitation vehicle to realize the automatic driving of the unmanned sanitation vehicle through the grid map algorithm and dynamic obstacle avoidance algorithm, and realizes the remote driving and remote monitoring of the unmanned sanitation vehicle through the image stitching technology and color space conversion algorithm. (1) By using the 5G communication technology, using the cloud server for data transfer, adopting the WebRTC audio and video live broadcast technology for real-time monitoring video display, and controlling the unmanned sanitation vehicle by remotely operating the steering wheel through the Socket technology, it is possible to achieve remote driving control with low latency regardless of distance and stable, smooth and high-definition remote monitoring, thus ensuring the safe driving of the vehicle in various complex road scenarios; (2) By acquiring the images of multiple cameras on the unmanned sanitation vehicle, processing and stitching each frame, a complete panoramic stitching BEV bird's-eye view can be obtained, and a more complete and clear monitoring image can be obtained; (3) By converting the color of each frame image of the cameras in different color spaces, the color space is unified to achieve a good color presentation effect; (4) In this way, the problems of communication delay, poor video quality, single perspective and insufficient environmental adaptability in the prior art are effectively solved.

[0007] Further, the S1 includes: Using multiple cameras in different directions covered by the unmanned sanitation vehicle to obtain multi-channel environmental image data; Using the steering wheel interface of the unmanned sanitation vehicle to obtain steering wheel data.

[0008] In this way, the optimal driving path can be calculated, collisions and path deviations can be avoided, and a safe and efficient driving path can be planned in a complex road environment, improving the sanitation operation efficiency.

[0009] Further, the S3 includes: When the operation mode of the unmanned sanitation vehicle is automatic driving, using the grid map algorithm to rasterize the working area in the environmental image data to obtain rasterized data; Set the starting point and ending point of the rasterized data, create open and closed lists, use the evaluation function to calculate and select the optimal raster, and continuously expand the current raster until the end point is found or no path can be found, to obtain the initial driving route; Identify obstacles through detection and predict their motion trajectories to obtain obstacle-related data; Analyze the data related to the unmanned sanitation vehicle and obstacles based on the dynamic obstacle avoidance algorithm to obtain an obstacle avoidance decision. When the original path is infeasible due to obstacles, re-plan the route to obtain the optimal driving route and control the autonomous driving of the unmanned sanitation vehicle.

[0010] Through image data processing, it can provide comprehensive and accurate environmental visual information, improving driving safety and monitoring comprehensiveness; calculating the steering wheel data to achieve precise speed control and ensuring the stable and controllable driving of the vehicle.

[0011] Further, the S4 includes: S410: Based on the environmental image data and camera parameters, obtain the equivalent refraction angles of images from different perspectives through fisheye distortion correction; S420: Based on the environmental image data with the equivalent refraction angles, perform perspective transformation processing to obtain a mapping matrix; S430: Perform transformation processing on the mapping matrix to obtain a complete panoramic image; S440: Use the unmanned platform to splice the multiple panoramic images in chronological order to obtain the monitoring video; S450: Use the unmanned platform to calculate the steering wheel data to obtain the control speed and direction of the unmanned sanitation vehicle.

[0012] In this way, the overlapping area can be eliminated to obtain a wide-angle and large-field-of-view video image, which can provide a 360-degree panoramic view and improve the environmental perception ability and driving safety.

[0013] Further, the S410 includes: Based on the environmental image data and camera parameters, use the color space conversion algorithm to convert the conversion matrix of different color spaces of the environmental image data into the relationship between the color data of each dimension in the original camera color space and the color data of each dimension in the RGB color space, and obtain the environmental image data in a unified color space; For the environmental image data in the unified color space, obtain the equivalent refraction angles of images from different perspectives through fisheye distortion correction.

[0014] In this way, it can ensure that all images are unified into the same color space, thereby achieving a consistent color presentation effect. This processing method can effectively eliminate the color difference between different cameras and improve the overall visual perception of the spliced images.

[0015] Further, the S450 includes: Use the unmanned platform to initialize the steering wheel data to obtain a speed variable; Parse the data received by the server to obtain angle-related values, forward-related values, and stop-related values; Based on the angle-related values, the forward-related values, and the stop-related values, calculate to obtain a target speed value; Utilize a smoothing control algorithm, with the target speed value as the standard, optimize the speed variable to obtain the control speed and direction of the unmanned sanitation vehicle.

[0016] Furthermore, the expression for the equivalent refraction angle of the environmental image data is: ; The expression for the mapping matrix is: ; Wherein, represents the equivalent refraction angle after fisheye distortion, represents the incident angle, , , and all represent the distortion coefficients of the fisheye camera, represents the abscissa of the transformed image, represents the ordinate of the transformed image, represents the height coordinate of the transformed image, represents the camera internal parameter matrix, represents the abscissa of the original image, represents the ordinate of the original image.

[0017] Furthermore, the S430 includes: S431: Perform matrix transformation on the mapping matrix to obtain the transformed matrix; S432: Perform image segmentation on the transformed matrix, extract the overlapping region in the image, and obtain the segmented image; S433: Generate a fusion weight based on the segmented image and the overlapping region in the image to obtain the fused image; S434: Stitch the fused image, input the stitched image into S431 for reprocessing until a complete panoramic view image is obtained.

[0018] A remote driving and remote monitoring system based on an unmanned platform, comprising: An acquisition module for acquiring the steering wheel data and environmental image data of the unmanned sanitation vehicle; An operation module for determining the operation mode of the unmanned sanitation vehicle based on the steering wheel data; the operation mode includes autonomous driving and remote driving; An autonomous driving module, which is used to calculate the driving route using the grid map algorithm and the dynamic obstacle avoidance algorithm to obtain the optimal driving route and control the autonomous driving of the unmanned sanitation vehicle when the operation mode of the unmanned sanitation vehicle is autonomous driving; A monitoring module, which is used to process the steering wheel data and the environmental image data based on the unmanned platform using the image stitching technology and the color space conversion algorithm to obtain the monitoring video, control speed and direction of the unmanned sanitation vehicle when the operation mode of the unmanned sanitation vehicle is remote driving; A remote driving module, which is used to obtain remote driving instructions based on the monitoring video, control speed and direction of the unmanned sanitation vehicle, and perform remote driving on the unmanned sanitation vehicle to complete the remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

[0019] Further, the obtaining module includes: An environmental data obtaining unit, which is used for a user to obtain multi-channel environmental image data by using cameras at multiple different orientations covered by the unmanned sanitation vehicle; A steering wheel data obtaining unit, which is used to obtain the steering wheel data by using the steering wheel interface of the unmanned sanitation vehicle.

[0020] In this way, more accurate and comprehensive environmental information can be provided to ensure that the unmanned sanitation vehicle can identify and avoid obstacles in real time, further improving the safety of the unmanned sanitation vehicle. Brief Description of the Drawings

[0021] This specification will further illustrate in the form of exemplary embodiments, and these exemplary embodiments will be described in detail through the drawings. These embodiments are not restrictive. In these embodiments, the same numbers represent the same structures, where: Figure 1 is a schematic diagram of the modules of a remote driving and remote monitoring system based on an unmanned platform shown in some embodiments of this specification; Figure 2 is an exemplary flowchart of a remote driving and remote monitoring method based on an unmanned platform shown in some embodiments of this specification. Detailed Embodiments

[0022] The following describes the specific embodiments of the present invention to facilitate those skilled in the art of this technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art of this technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions created using the concept of the present invention are within the scope of protection.

[0023] Embodiment 1 Figure 1It is a schematic diagram of modules of a remote driving and remote monitoring system based on an unmanned platform shown in some embodiments of this specification.

[0024] In some embodiments, the remote driving and remote monitoring system based on an unmanned platform may include an acquisition module, an operation module, an autonomous driving module, a monitoring module, and a remote driving module.

[0025] The acquisition module is used to acquire the steering wheel data and environmental image data of the unmanned sanitation vehicle. For more details about the steering wheel data and environmental image data, reference can be made to Figure 2 and its related description.

[0026] In some embodiments, the acquisition module includes an environmental data acquisition unit and a steering wheel data acquisition unit.

[0027] The environmental data acquisition unit uses cameras at multiple different orientations covered by the unmanned sanitation vehicle to acquire multiple channels of environmental image data.

[0028] The steering wheel data acquisition unit is used to acquire the steering wheel data by using the steering wheel interface of the unmanned sanitation vehicle.

[0029] In some embodiments, the acquisition module can acquire and parse the steering wheel data through the API interface of the steering wheel, send the data to the monitoring module through the cloud server by creating a UDP socket and binding the local address and port.

[0030] The cloud server is a relay server established based on 5G communication technology, which is used to receive and parse the data sent by the client by creating a UDP socket and binding the address and port, and forward the corresponding data to the monitoring module.

[0031] In some embodiments, the acquisition module further includes an optical monitoring unit, an acoustic wave monitoring unit, and a data fusion unit.

[0032] The optical monitoring unit includes a lidar for acquiring optical environmental data.

[0033] The acoustic wave monitoring unit includes ultrasonic sensors for acquiring acoustic environmental data.

[0034] The data fusion unit is used to fuse the optical environmental data, the acoustic environmental data, and the multiple channels of environmental image data through multi-sensor data synchronization, spatial coordinate transformation, and obstacle detection algorithms to obtain optimized multiple channels of environmental image data.

[0035] In some embodiments, the acquisition module can use the optical monitoring unit and the acoustic wave monitoring unit to implement the environmental perception and obstacle detection functions.

[0036] In this way, more accurate and comprehensive environmental information can be provided to ensure that the driverless sanitation vehicle can identify and avoid obstacles in real time, further improving the safety of the driverless sanitation vehicle.

[0037] An operation module, configured to determine the operation mode of the driverless sanitation vehicle based on the steering wheel data; the operation mode includes autonomous driving and remote driving. For more details about the operation mode, please refer to Figure 2 and its related description.

[0038] An autonomous driving module, when the operation mode of the driverless sanitation vehicle is autonomous driving, is configured to calculate the driving route using the grid map algorithm and the dynamic obstacle avoidance algorithm to obtain the optimal driving route, and control the driverless sanitation vehicle to drive autonomously. For more details about the optimal driving route, please refer to Figure 2 and its related description.

[0039] A monitoring module, when the operation mode of the driverless sanitation vehicle is remote driving, is configured to process the steering wheel data and the environmental image data based on the unmanned platform using the image stitching technology and the color space conversion algorithm to obtain the monitoring video, control speed and direction of the driverless sanitation vehicle. For more details about the monitoring video, control speed and direction, please refer to Figure 2 and its related description.

[0040] In some embodiments, the acquisition module can transmit the steering wheel data and the environmental image data to the monitoring module through the WebRTC protocol. For example, the acquisition module can add the stitched and color-converted real-time video stream to the WebRTC channel, and perform preliminary processing and compression; the monitoring module creates a UDP socket, binds the local address and port, initializes the local media information, prepares to receive the video stream, and listens to the WebRTC channel; through the cloud server relay, the monitoring module initiates a WebRTC remote call and establishes a connection with the acquisition module; after the monitoring module receives the video stream sent by the acquisition module, it decodes the video stream and renders it on the local display screen to ensure that the operator can monitor the working environment and status of the driverless sanitation vehicle in real time.

[0041] The video stream is video data reflecting the operation of the driverless sanitation vehicle and the changes in the surrounding environment. For example, the video stream can include the steering wheel data and the environmental image data at multiple consecutive times, etc.

[0042] A remote driving module, configured to obtain a remote driving instruction based on the monitoring video, control speed and direction of the driverless sanitation vehicle, and perform remote driving on the driverless sanitation vehicle to complete remote driving and remote monitoring of the driverless sanitation vehicle based on the unmanned platform. For more details about the remote driving instruction, please refer to Figure 2 and its related description.

[0043] In some embodiments, the remote driving module can encapsulate control messages into the VehicleCmd topic and publish them according to the message format of the ROS (Robot Operating System) software of the driverless sanitation vehicle, driving the driverless sanitation vehicle to travel according to the received instructions.

[0044] In some embodiments, a remote driving and remote monitoring system based on an unmanned platform can be used to execute a remote driving and remote monitoring method based on an unmanned platform, including: S1: Obtain the steering wheel data and environmental image data of the driverless sanitation vehicle; S2: Based on the steering wheel data, determine the operating mode of the driverless sanitation vehicle; the operating mode includes autonomous driving and remote driving; S3: When the operating mode of the driverless sanitation vehicle is autonomous driving, use the grid map algorithm and the dynamic obstacle avoidance algorithm to calculate the driving route to obtain the optimal driving route, and control the driverless sanitation vehicle to drive autonomously; S4: When the operating mode of the driverless sanitation vehicle is remote driving, based on the image stitching technology and the color space conversion algorithm, use the unmanned platform to process the steering wheel data and environmental image data to obtain the monitoring video, control speed and direction of the driverless sanitation vehicle; S5: Based on the monitoring video, control speed and direction of the driverless sanitation vehicle, obtain remote driving instructions, and remotely drive the driverless sanitation vehicle to complete the remote driving and remote monitoring of the driverless sanitation vehicle based on the unmanned platform.

[0045] In some embodiments of this specification, the processor uses the remote driving and remote monitoring system based on the unmanned platform to execute the remote driving and remote monitoring method based on the unmanned platform. (1) By using the 5G communication technology, using the cloud server for data transfer, adopting the WebRTC audio and video live broadcast technology for real-time monitoring video display, and controlling the driverless sanitation vehicle by remotely operating the steering wheel through the Socket technology, it is possible to achieve remote driving control with low latency regardless of distance and stable, smooth and high-definition remote monitoring, thus ensuring the safe driving of the vehicle in various complex road scenarios; (2) By obtaining the images of multiple cameras on the driverless sanitation vehicle, processing and stitching each frame, a complete panoramic stitching BEV bird's-eye view can be obtained to get a more complete and clear monitoring image; (3) By converting the color of each frame image of cameras in different color spaces, the color space can be unified to achieve a good color presentation effect; (4) In this way, the problems of communication delay, poor video quality, single perspective and insufficient environmental adaptability in the prior art are effectively solved.

[0046] Embodiment 2 Figure 2 is an exemplary flowchart of a remote driving and remote monitoring method based on an unmanned platform shown in some embodiments of this specification. As Figure 2As shown, the process includes the following steps. In some embodiments, the process can be executed by a processor.

[0047] S1: Obtain the steering wheel data and environmental image data of the unmanned sanitation vehicle.

[0048] An unmanned sanitation vehicle is an unmanned vehicle used for sanitation operations.

[0049] The steering wheel data is data reflecting the running condition of the steering wheel of the unmanned sanitation vehicle. For example, the steering wheel data may include the steering angle, rotation speed, and running mode of the steering wheel, etc.

[0050] The environmental image data is data reflecting the surrounding environment condition of the unmanned sanitation vehicle. For example, the environmental image data may include optical environmental data, acoustic environmental data, and multi-channel environmental image data, etc.

[0051] In some embodiments, the processor can use cameras at multiple different orientations covered by the unmanned sanitation vehicle to obtain multi-channel environmental image data; and use the steering wheel interface of the unmanned sanitation vehicle to obtain the steering wheel data.

[0052] In some embodiments, the processor can install a camera in each of the front, rear, left, and right directions of the unmanned sanitation vehicle to obtain environmental image data in the front, rear, left, and right directions.

[0053] S2: Based on the steering wheel data, determine the running mode of the unmanned sanitation vehicle.

[0054] The running mode is a mode for determining the current control situation of the unmanned sanitation vehicle. For example, the running mode may include autonomous driving and remote driving.

[0055] Autonomous driving is a mode in which the unmanned sanitation vehicle automatically drives according to the road conditions.

[0056] Remote driving is a mode in which a driver remotely controls the driving of the unmanned sanitation vehicle.

[0057] In some embodiments, the processor can extract the running mode based on the steering wheel data of the unmanned sanitation vehicle.

[0058] S3: When the running mode of the unmanned sanitation vehicle is autonomous driving, use the grid map algorithm and the dynamic obstacle avoidance algorithm to calculate the driving route, obtain the optimal driving route, and control the unmanned sanitation vehicle to drive autonomously.

[0059] The optimal driving route is a route that combines the lowest collision risk and the shortest driving distance.

[0060] In some embodiments, the processor may implement S3 based on the following steps: When the operation mode of the unmanned sanitation vehicle is autonomous driving, use the grid map algorithm to rasterize the working area in the environmental image data to obtain rasterized data; set the starting point and ending point of the rasterized data, create open and closed lists, use the evaluation function to calculate and select the optimal grid, continuously expand the current grid until the end point is found or no path can be found, to obtain the initial driving route; detect and identify obstacles and predict their movement trajectories to obtain obstacle-related data; analyze the unmanned sanitation vehicle and the obstacle-related data based on the dynamic obstacle avoidance algorithm to obtain an obstacle avoidance decision, and re-plan the route when the original path is infeasible due to obstacles, to obtain the optimal driving route, and control the unmanned sanitation vehicle to drive autonomously.

[0061] The rasterized data is the data obtained by rasterizing the working area in the environmental image data.

[0062] The initial driving route is the driving route obtained by calculating and selecting the optimal grid based on the evaluation function.

[0063] The obstacle-related data is the data related to the size and movement trajectory of the obstacle.

[0064] S4: When the operation mode of the unmanned sanitation vehicle is remote driving, based on the image stitching technology and color space conversion algorithm, use the unmanned platform to process the steering wheel data and environmental image data to obtain the monitoring video, control speed and direction of the unmanned sanitation vehicle.

[0065] The monitoring video is the monitoring video of the surrounding environment of the unmanned sanitation vehicle with a wide angle and large field of view.

[0066] The control speed and direction are the data reflecting the current running speed and direction of the unmanned sanitation vehicle.

[0067] In some embodiments, the processor may implement S4 based on the following steps.

[0068] S410: Based on the environmental image data and camera parameters, obtain the equivalent refraction angle of images from different perspectives through fisheye distortion correction.

[0069] The camera parameters are the inherent parameters of the camera used for monitoring. For example, the camera parameters may include internal parameters and external parameters.

[0070] The equivalent refraction angle is the transformation parameter for converting images with different incident angles into the same perspective.

[0071] In some embodiments, the processor may utilize a color space conversion algorithm based on the environmental image data and camera parameters to convert the conversion matrix of different color spaces of the environmental image data into the relationship between the color data of each dimension in the original camera color space and the color data of each dimension in the RGB color space, obtaining environmental image data in a unified color space; for the environmental image data in the unified color space, through fisheye distortion correction, the equivalent refraction angle of images with different perspectives is obtained.

[0072] In some embodiments, the expression of the equivalent refraction angle of the environmental image data may be: ; where represents the equivalent refraction angle after fisheye distortion, represents the incident angle, , , and all represent the distortion coefficients of the fisheye camera.

[0073] S420: Based on the environmental image data of the equivalent refraction angle, perform perspective transformation processing to obtain a mapping matrix.

[0074] The mapping matrix is a matrix that transforms images with different perspectives to the bird's-eye view perspective.

[0075] In some embodiments, the expression of the mapping matrix may be: ; where represents the abscissa of the transformed image, represents the ordinate of the transformed image, represents the height coordinate of the transformed image, represents the camera internal parameter matrix, represents the abscissa of the original image, represents the ordinate of the original image.

[0076] S430: Perform transformation processing on the mapping matrix to obtain a complete panoramic image.

[0077] The complete panoramic image is an image that reflects the complete environmental situation around the unmanned sanitation vehicle.

[0078] In some embodiments, the processor may implement S430 based on the following steps.

[0079] S431: Perform matrix transformation on the mapping matrix to obtain a transformed matrix.

[0080] In some embodiments, the processor may perform rotation and translation transformation on the mapping matrix to obtain a transformed matrix.

[0081] S432: Segment the transformed matrix to extract the overlapping region in the image, obtaining a segmented image.

[0082] The segmented image is an image obtained by segmenting the transformed matrix based on the overlapping region in the image.

[0083] S433: Generate a fusion weight based on the segmented image and the overlapping region in the image, obtaining a fused image.

[0084] The fused image is an image in which the splicing gaps are eliminated after multiple segmented images are fused.

[0085] In some embodiments, the processor may generate a fusion weight mask according to the overlapping region, perform a fusion process on the segmented image, and obtain a fused image.

[0086] S434: Splice the fused image, input the spliced image into S431 for reprocessing until a complete panoramic view image is obtained.

[0087] S440: Use the unmanned platform to splice the multiple panoramic view images in chronological order to obtain the monitoring video.

[0088] S450: Use the unmanned platform to calculate the steering wheel data to obtain the control speed and direction of the unmanned sanitation vehicle.

[0089] In some embodiments, the processor may implement S450 based on the following steps: initialize the steering wheel data using the unmanned platform to obtain a speed variable; parse the data received by the server to obtain angle-related values, forward-related values, and stop-related values; based on the angle-related values, the forward-related values, and the stop-related values, calculate to obtain a target speed value; use a smoothing control algorithm to optimize the speed variable with the target speed value as the standard to obtain the control speed and direction of the unmanned sanitation vehicle.

[0090] The speed variable is a variable that reflects the speed and its changes of the unmanned sanitation vehicle.

[0091] The target speed value is the optimal speed value for the current movement.

[0092] In some embodiments, the processor may initialize the ROS node and the velocity topic publisher, and define multiple velocity-related variables. Then, it receives data from the server through a UDP socket, parses the received data into values related to angles, forward movement, and stopping. Based on these values, it calculates the target velocity (including forward and backward movement, steering, and lateral movement), and then, through a smoothing control algorithm, makes the actual control velocity gradually approach the target velocity. Finally, it assigns the calculated velocity value to the ROS velocity topic variable and publishes the topic, thereby obtaining the control velocity and direction of the unmanned sanitation vehicle.

[0093] S5: Based on the monitoring video, control velocity, and direction of the unmanned sanitation vehicle, obtain a remote driving instruction, and perform remote driving on the unmanned sanitation vehicle to complete the remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

[0094] The remote driving instruction is a remote instruction for controlling the driving of the unmanned sanitation vehicle.

[0095] In some embodiments, the processor may obtain a remote driving instruction remotely input by the driver based on the monitoring video, control velocity, and direction of the unmanned sanitation vehicle, and perform remote driving on the unmanned sanitation vehicle to complete the remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

[0096] In some embodiments of this specification, the processor uses the steering wheel data and environmental image data of the unmanned sanitation vehicle to achieve the autonomous driving of the unmanned sanitation vehicle through the grid map algorithm and the dynamic obstacle avoidance algorithm, and realizes the remote driving and remote monitoring of the unmanned sanitation vehicle through the image stitching technology and the color space conversion algorithm. (1) By using the 5G communication technology, using the cloud server for data transfer, adopting the WebRTC audio and video live broadcast technology for real-time monitoring video display, and controlling the unmanned sanitation vehicle by remotely operating the steering wheel through the Socket technology, remote driving control with low latency regardless of distance and stable, smooth, and high-definition remote monitoring can be achieved, thereby ensuring the safe driving of the vehicle in various complex road scenarios; (2) By acquiring the images of multiple cameras on the unmanned sanitation vehicle, processing and stitching each frame, a complete panoramic stitching BEV bird's-eye view can be obtained, and a more complete and clear monitoring image can be obtained; (3) By performing color conversion on each frame image of the cameras in different color spaces, the color space can be unified to achieve a good color presentation effect; (4) In this way, the problems of communication latency, poor video quality, single perspective, and insufficient environmental adaptability in the prior art are effectively solved.

Claims

1. A remote driving and remote monitoring method based on an unmanned platform, characterized in that, Including: S1: Obtain the steering wheel data and environmental image data of the unmanned sanitation vehicle; S2: Based on the steering wheel data, determine the operation mode of the unmanned sanitation vehicle; the operation mode includes autonomous driving and remote driving; S3: When the operation mode of the unmanned sanitation vehicle is autonomous driving, use the grid map algorithm and dynamic obstacle avoidance algorithm to calculate the driving route, obtain the optimal driving route, and control the unmanned sanitation vehicle to drive autonomously; S4: When the operation mode of the unmanned sanitation vehicle is remote driving, based on the image stitching technology and color space conversion algorithm, use the unmanned platform to process the steering wheel data and environmental image data to obtain the monitoring video, control speed and direction of the unmanned sanitation vehicle; S5: Based on the monitoring video, control speed and direction of the unmanned sanitation vehicle, obtain remote driving instructions, and remotely drive the unmanned sanitation vehicle to complete the remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

2. The remote driving and remote monitoring method based on an unmanned platform according to claim 1, wherein The S1 includes: Use cameras at multiple different orientations covered by the unmanned sanitation vehicle to obtain multi-channel environmental image data; Use the steering wheel interface of the unmanned sanitation vehicle to obtain the steering wheel data.

3. The remote driving and remote monitoring method based on an unmanned platform according to claim 1, wherein, The S3 includes: When the operation mode of the unmanned sanitation vehicle is autonomous driving, use the grid map algorithm to rasterize the working area in the environmental image data to obtain rasterized data; Set the starting point and ending point of the rasterized data, create open and closed lists, use the evaluation function to calculate and select the optimal raster, continuously expand the current raster until the end point is found or no path can be found, and obtain the initial driving route; Detect and identify obstacles and predict their movement trajectories to obtain obstacle-related data; Based on the dynamic obstacle avoidance algorithm, analyze the unmanned sanitation vehicle and the obstacle-related data to obtain an obstacle avoidance decision. When the original path is infeasible due to obstacles, re-plan the route to obtain the optimal driving route, and control the unmanned sanitation vehicle to drive autonomously.

4. The remote driving and remote monitoring method based on an unmanned platform according to claim 1, wherein, The S4 includes: S410: Based on the environmental image data and camera parameters, through fisheye distortion correction, obtain the equivalent refraction angles of images from different perspectives; S420: Based on the environmental image data with the equivalent refraction angles, perform perspective transformation processing to obtain a mapping matrix; S430: Perform transformation processing on the mapping matrix to obtain a complete panoramic image; S440: Use the unmanned platform to stitch the multiple panoramic images in chronological order to obtain the monitoring video; S450: Use the unmanned platform to calculate the steering wheel data to obtain the control speed and direction of the unmanned sanitation vehicle.

5. The remote driving and remote monitoring method based on an unmanned platform according to claim 4, characterized in that, The S410 includes: Based on the environmental image data and camera parameters, use the color space conversion algorithm to convert the conversion matrix of different color spaces of the environmental image data into the relationship between the color data of each dimension in the original camera color space and the color data of each dimension in the RGB color space, and obtain the environmental image data in a unified color space; For the environmental image data in the unified color space, through fisheye distortion correction, obtain the equivalent refraction angles of images from different perspectives.

6. The remote driving and remote monitoring method based on an unmanned platform according to claim 4, characterized in that The S450 includes: Use the unmanned platform to initialize the steering wheel data to obtain a speed variable; Parse the data received by the server to obtain angle-related values, forward-related values, and stop-related values; Based on the angle-related values, the forward-related values, and the stop-related values, calculate to obtain a target speed value; Use a smoothing control algorithm to optimize the speed variable with the target speed value as the standard, to obtain the control speed and direction of the unmanned sanitation vehicle.

7. The remote driving and remote monitoring method based on an unmanned platform according to claim 4, characterized in that The expression for the equivalent refraction angle of the environmental image data is: ； The expression for the mapping matrix is: ； Among them, represents the equivalent refraction angle after fish-eye distortion, represents the incident angle, , , and all represent the distortion coefficients of the fish-eye camera, represents the abscissa of the transformed image, represents the ordinate of the transformed image, represents the height coordinate of the transformed image, represents the camera internal parameter matrix, represents the abscissa of the original image, represents the ordinate of the original image.

8. The remote driving and remote monitoring method based on an unmanned platform according to claim 4, wherein The S430 includes: S431: Perform matrix transformation on the mapping matrix to obtain a transformed matrix; S432: Perform image segmentation on the transformed matrix, extract the overlapping region in the image, to obtain a segmented image; S433: Generate a fusion weight based on the segmented image and the overlapping region in the image, to obtain a fused image; S434: Stitch the fused image, input the stitched image into S431 for reprocessing until a complete panoramic view image is obtained.

9. A remote driving and remote monitoring system based on an unmanned platform, characterized in that, Includes: An acquisition module, used to acquire the steering wheel data and environmental image data of the unmanned sanitation vehicle; An operation module, used to determine the operation mode of the unmanned sanitation vehicle based on the steering wheel data; the operation mode includes autonomous driving and remote driving; An autonomous driving module, when the operation mode of the unmanned sanitation vehicle is autonomous driving, used to calculate the driving route using a grid map algorithm and a dynamic obstacle avoidance algorithm to obtain an optimal driving route, and control the unmanned sanitation vehicle to drive autonomously; A monitoring module, when the operation mode of the unmanned sanitation vehicle is remote driving, used to process the steering wheel data and environmental image data based on an image stitching technology and a color space conversion algorithm on an unmanned platform to obtain a monitoring video, control speed, and direction of the unmanned sanitation vehicle; A remote driving module, used to obtain a remote driving instruction based on the monitoring video, control speed, and direction of the unmanned sanitation vehicle, and perform remote driving on the unmanned sanitation vehicle to complete remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

10. The remote driving and remote monitoring system based on an unmanned platform according to claim 9, characterized in that, The acquisition module includes: An environmental data acquisition unit, which uses cameras at multiple different orientations covered by the unmanned sanitation vehicle to acquire multiple channels of environmental image data; A steering wheel data acquisition unit, used to acquire steering wheel data using the steering wheel interface of the unmanned sanitation vehicle.

Images