A method and system for remote driving and remote monitoring based on unmanned platform

Abstract

The application provides a kind of remote driving and remote monitoring method and system based on unmanned platform, it is related to unmanned vehicle technical field, the method is to obtain the steering wheel data and environmental image data of unmanned sanitation vehicle;Based on steering wheel data, determine the operation mode of unmanned sanitation vehicle;When the operation mode of unmanned sanitation vehicle is automatic driving, utilize the optimal driving route, control unmanned sanitation vehicle automatic driving;When the operation mode of unmanned sanitation vehicle is remote driving, utilize unmanned platform to process steering wheel data and environmental image data, obtain the monitoring video, control speed and direction of unmanned sanitation vehicle;Based on the monitoring video, control speed and direction of unmanned sanitation vehicle, obtain remote driving instruction, remote driving is carried out to unmanned sanitation vehicle, complete remote driving and remote monitoring based on unmanned platform to unmanned sanitation vehicle.The application solves the problem that unmanned sanitation vehicle is difficult to run safely and efficiently in complex environment.

Description

Technical Field

[0001] This specification relates to the field of unmanned vehicle technology, and in particular to a remote driving and remote monitoring method and system based on an unmanned platform. Background Art

[0002] With the rapid development of autonomous driving and communication technologies in recent years, various industries are exploring how to apply these new technologies to improve efficiency and safety, and the sanitation industry is no exception. Traditional sanitation operations typically require extensive manual labor, which is not only costly but also requires workers to work in complex and sometimes dangerous environments. Unmanned platforms, including autonomous cleaning vehicles, are capable of performing tasks such as sweeping, vacuuming, and cleaning without human intervention. The emergence of unmanned platforms has provided the sanitation industry with automated solutions that can significantly improve operational efficiency and safety. However, operating in the complex urban road environment, unmanned platforms face numerous challenges, such as high-density traffic, pedestrians, and various fixed and moving obstacles. These complex situations place high demands on the navigation, perception, and real-time response capabilities of unmanned sanitation vehicles.

[0003] Most existing unmanned sanitation vehicle platforms rely on traditional wireless communication technologies, such as Wi-Fi or 4G. Existing technologies mostly use single-view cameras for video surveillance, which have deficiencies in transmission stability and real-time performance in complex road environments. Existing technologies mostly use single-view cameras for video surveillance, which makes it difficult to fully grasp the vehicle's surroundings in emergency situations, posing a safety hazard. Some existing models use UDP or TCP-based protocols to communicate within the local area network. Although this greatly increases transmission stability and real-time performance, the remote operation range is greatly reduced, making it difficult to meet actual needs. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the present invention provides a remote driving and remote monitoring method and system based on an unmanned platform, which solves the problem that unmanned sanitation vehicles are difficult to operate safely and efficiently in complex environments.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a remote driving and remote monitoring method based on an unmanned platform, comprising:

[0006] S1: Acquire the steering wheel data and environmental image data of the unmanned sanitation vehicle;

[0007] S2: Determine an operating mode of the unmanned sanitation vehicle based on the steering wheel data; the operating mode includes automatic driving and remote driving;

[0008] S3: When the unmanned sanitation vehicle is in automatic driving mode, the grid map algorithm and dynamic obstacle avoidance algorithm are used to calculate the driving route, obtain the optimal driving route, and control the automatic driving of the unmanned sanitation vehicle;

[0009] S4: When the unmanned sanitation vehicle is in remote driving mode, the steering wheel data and the environmental image data are processed by the unmanned platform based on image stitching technology and color space conversion algorithm to obtain a monitoring video of the unmanned sanitation vehicle and control the speed and direction;

[0010] S5: Based on the monitoring video of the unmanned sanitation vehicle, control the speed and direction, obtain remote driving instructions, remotely drive the unmanned sanitation vehicle, and complete remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

[0011] The beneficial effects of the present invention are as follows: the processor uses the steering wheel data and environmental image data of the unmanned sanitation vehicle to realize the automatic driving of the unmanned sanitation vehicle through the grid map algorithm and the dynamic obstacle avoidance algorithm, and realizes the remote driving and remote monitoring of the unmanned sanitation vehicle through the image stitching technology and the color space conversion algorithm. (1) By utilizing 5G communication technology, using cloud servers for data transfer, adopting WebRTC audio and video live broadcast technology for real-time monitoring video display, and remotely operating the steering wheel to control the unmanned sanitation vehicle through Socket technology, it is possible to realize low-latency remote driving control at unlimited distance and stable, smooth and high-definition remote monitoring, thereby ensuring the safe driving of the vehicle in various complex road scenes; (2) by obtaining images from multiple cameras on the unmanned sanitation vehicle, processing and stitching each frame, a complete surround stitching BEV bird's-eye view can be obtained, and a more complete and clear monitoring image can be obtained; (3) by performing color conversion on each frame of the camera image in different color spaces, the color space is unified to achieve a good color rendering effect; (4) in this way, the problems of communication delay, poor video quality, single perspective and insufficient environmental adaptability in the existing technology are effectively solved.

[0012] Furthermore, the S1 includes:

[0013] Utilize multiple cameras in different locations covered by unmanned sanitation vehicles to obtain multi-channel environmental image data;

[0014] Use the steering wheel interface of the unmanned sanitation vehicle to obtain steering wheel data.

[0015] In this way, the optimal driving path can be calculated to avoid collisions and path deviations, and a safe and efficient driving path can be planned in complex road environments, thereby improving the efficiency of sanitation operations.

[0016] Furthermore, the S3 includes:

[0017] When the unmanned sanitation vehicle is in the automatic driving mode, the working area in the environmental image data is rasterized using a raster map algorithm to obtain rasterized data;

[0018] Setting the starting point and end point of the rasterized data, creating open and closed lists, applying an evaluation function to calculate and select the optimal grid, and continuously expanding the current grid until the end point is found or no path can be found, thereby obtaining an initial driving route;

[0019] Obstacle-related data is obtained by detecting and identifying obstacles and predicting their movement trajectories;

[0020] Based on the dynamic obstacle avoidance algorithm, the data related to the unmanned sanitation vehicle and obstacles are analyzed to obtain obstacle avoidance decisions. When the original path is not feasible due to obstacles, the route is replanned to obtain the optimal driving route and control the automatic driving of the unmanned sanitation vehicle.

[0021] Through image data processing, it can provide comprehensive and accurate environmental visual information, improve driving safety and comprehensive monitoring; calculate steering wheel data to achieve precise speed control and ensure stable and controllable vehicle driving.

[0022] Furthermore, the S4 includes:

[0023] S410: Obtaining equivalent refraction angles of images at different viewing angles through fisheye distortion correction based on the environmental image data and camera parameters;

[0024] S420: Performing perspective transformation processing based on the environmental image data of the equivalent refraction angle to obtain a mapping matrix;

[0025] S430: performing transformation processing on the mapping matrix to obtain a complete surround view image;

[0026] S440: Using the unmanned platform, stitching the multiple surround view images in chronological order to obtain the monitoring video;

[0027] S450: Utilize the unmanned platform to calculate the steering wheel data to obtain the control speed and direction of the unmanned sanitation vehicle.

[0028] In this way, overlapping areas can be eliminated and video images with a wide angle and large field of view can be obtained, which can provide a 360-degree surround view and improve environmental perception and driving safety.

[0029] Furthermore, the S410 includes:

[0030] Based on the environmental image data and camera parameters, using a color space conversion algorithm, converting the conversion matrix of the different color spaces of the environmental image data into the relationship between the color data of each dimension in the original camera color space and the color data of each dimension in the RGB color space, thereby obtaining environmental image data in a unified color space;

[0031] The environmental image data in the unified color space is subjected to fisheye distortion correction to obtain equivalent refraction angles of images at different viewing angles.

[0032] This ensures that all images are aligned to the same color space, resulting in consistent color rendering. This processing effectively eliminates color differences between different cameras and improves the overall appearance of the stitched image.

[0033] Furthermore, the S450 includes:

[0034] Initializing the steering wheel data using an unmanned platform to obtain a speed variable;

[0035] Parse the data received from the server to obtain angle-related values, forward-related values, and stop-related values;

[0036] Obtaining a target speed value by calculation based on the angle-related value, the forward-related value, and the stop-related value;

[0037] By using a smoothing control algorithm and taking the target speed value as a standard, the speed variable is optimized to obtain the control speed and direction of the unmanned sanitation vehicle.

[0038] Furthermore, the expression of the equivalent refraction angle of the environmental image data is:

[0039] ;

[0040] The expression of the mapping matrix is:

[0041] ;

[0042] in, represents the equivalent refraction angle after fisheye distortion, represents the angle of incidence, 、 、 and Both represent the distortion coefficient of the fisheye camera, represents the horizontal coordinate of the transformed image, represents the ordinate of the transformed image, Represents the height coordinate of the transformed image, represents the camera intrinsic parameter matrix, represents the horizontal coordinate of the original image, Indicates the vertical coordinate of the original image.

[0043] Furthermore, the S430 includes:

[0044] S431: Performing matrix transformation on the mapping matrix to obtain a transformed matrix;

[0045] S432: performing image segmentation on the transformed matrix, extracting overlapping areas in the image, and obtaining a segmented image;

[0046] S433: generating a fusion weight based on the segmented image and the overlapping area in the image to obtain a fused image;

[0047] S434: stitching the fused images, and inputting the stitched images into S431 for further processing until a complete surround view image is obtained.

[0048] A remote driving and remote monitoring system based on an unmanned platform, comprising:

[0049] An acquisition module is used to obtain steering wheel data and environmental image data of the unmanned sanitation vehicle;

[0050] An operation module, configured to determine an operation mode of the unmanned sanitation vehicle based on the steering wheel data; the operation mode includes automatic driving and remote driving;

[0051] The automatic driving module is used to calculate the driving route using the grid map algorithm and dynamic obstacle avoidance algorithm when the unmanned sanitation vehicle is in automatic driving mode, obtain the optimal driving route, and control the automatic driving of the unmanned sanitation vehicle;

[0052] A monitoring module, when the unmanned sanitation vehicle is in remote driving mode, is used to process the steering wheel data and environmental image data based on the unmanned platform based on image stitching technology and color space conversion algorithm to obtain monitoring video of the unmanned sanitation vehicle and control speed and direction;

[0053] The remote driving module is used to obtain remote driving instructions based on the monitoring video of the unmanned sanitation vehicle, control the speed and direction, remotely drive the unmanned sanitation vehicle, and complete remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

[0054] Furthermore, the acquisition module includes:

[0055] Environmental data acquisition unit: users use multiple cameras in different directions covered by unmanned sanitation vehicles to obtain multi-channel environmental image data;

[0056] The steering wheel data acquisition unit is used to obtain steering wheel data using the steering wheel interface of the unmanned sanitation vehicle.

[0057] In this way, more accurate and comprehensive environmental information can be provided, ensuring that unmanned sanitation vehicles can identify and avoid obstacles in real time, further improving the safety of unmanned sanitation vehicles. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] This specification will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, like numbers represent like structures, wherein:

[0059] Figure 1 is a module diagram of a remote driving and remote monitoring system based on an unmanned platform according to some embodiments of this specification;

[0060] Figure 2 This is an exemplary flow chart of a remote driving and remote monitoring method based on an unmanned platform according to some embodiments of this specification. DETAILED DESCRIPTION

[0061] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0062] Example 1

[0063] Figure 1 This is a module diagram of a remote driving and remote monitoring system based on an unmanned platform according to some embodiments of this specification.

[0064] In some embodiments, the remote driving and remote monitoring system based on an unmanned platform may include an acquisition module, an operation module, an automatic driving module, a monitoring module and a remote driving module.

[0065] The acquisition module is used to obtain the steering wheel data and environmental image data of the unmanned sanitation vehicle. For more details about the steering wheel data and environmental image data, please refer to Figure 2 and its related descriptions.

[0066] In some embodiments, the acquisition module includes an environment data acquisition unit and a steering wheel data acquisition unit.

[0067] Environmental data acquisition unit: users use multiple cameras in different directions covered by unmanned sanitation vehicles to obtain multi-channel environmental image data.

[0068] The steering wheel data acquisition unit is used to obtain steering wheel data using the steering wheel interface of the unmanned sanitation vehicle.

[0069] In some embodiments, the acquisition module can obtain and parse the steering wheel data through the API interface of the steering wheel, create a UDP socket, bind the local address and port, and send the data to the monitoring module through the cloud server.

[0070] The cloud server is a transit server built based on 5G communication technology. It is used to receive and parse the data sent by the client by creating a UDP socket and binding the address and port, and forward the corresponding data to the monitoring module.

[0071] In some embodiments, the acquisition module further includes an optical monitoring unit, an acoustic wave monitoring unit, and a data fusion unit.

[0072] The optical monitoring unit includes a lidar, which is used to obtain optical environment data.

[0073] The acoustic wave monitoring unit includes an ultrasonic sensor for acquiring acoustic environment data.

[0074] The data fusion unit is used to fuse the optical environment data, the acoustic environment data and the multi-channel environment image data through multi-sensor data synchronization, spatial coordinate conversion and obstacle detection algorithm to obtain optimized multi-channel environment image data.

[0075] In some embodiments, the acquisition module can utilize an optical monitoring unit and an acoustic wave monitoring unit to implement environmental perception and obstacle detection functions.

[0076] In this way, more accurate and comprehensive environmental information can be provided, ensuring that unmanned sanitation vehicles can identify and avoid obstacles in real time, further improving the safety of unmanned sanitation vehicles.

[0077] The operation module is used to determine the operation mode of the unmanned sanitation vehicle based on the steering wheel data; the operation mode includes automatic driving and remote driving. For more details about the operation mode, please refer to Figure 2 and its related descriptions.

[0078] The automatic driving module is used to calculate the driving route using the grid map algorithm and dynamic obstacle avoidance algorithm when the unmanned sanitation vehicle is in automatic driving mode, obtain the optimal driving route, and control the automatic driving of the unmanned sanitation vehicle. For more details about the optimal driving route, please refer to Figure 2 and its related descriptions.

[0079] The monitoring module, when the unmanned sanitation vehicle is in remote driving mode, is used to process the steering wheel data and environmental image data based on the unmanned platform based on image stitching technology and color space conversion algorithm to obtain the monitoring video of the unmanned sanitation vehicle and control the speed and direction. For more details about monitoring video, controlling speed and direction, please refer to Figure 2 and its related descriptions.

[0080] In some embodiments, the acquisition module can transmit the steering wheel data and environmental image data to the monitoring module via the WebRTC protocol. For example, the acquisition module can add the spliced ​​and color-converted real-time video stream to the WebRTC channel and perform preliminary processing and compression; the monitoring module creates a UDP socket, binds the local address and port, initializes local media information, prepares to receive the video stream, and monitors the WebRTC channel; through the cloud server, the monitoring module initiates a WebRTC remote call and establishes a connection with the acquisition module; after receiving the video stream sent by the acquisition module, the monitoring module decodes the video stream and renders it on the local display screen, ensuring that the operator can monitor the working environment and status of the unmanned sanitation vehicle in real time.

[0081] Video streams are video data that reflects the operation of an unmanned sanitation vehicle and changes in the surrounding environment. For example, a video stream can include multiple continuous images of steering wheel data and environmental images.

[0082] The remote driving module is used to control the speed and direction of the unmanned sanitation vehicle based on the monitoring video, obtain remote driving instructions, remotely drive the unmanned sanitation vehicle, and complete remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform. For more details about remote driving instructions, please refer to Figure 2 and its related descriptions.

[0083] In some embodiments, the remote driving module can encapsulate the control message into the VehicleCmd topic and publish it according to the message format of the unmanned sanitation vehicle ROS (Robot Operating System) software, and drive the unmanned sanitation vehicle to drive according to the received instructions.

[0084] In some embodiments, a remote driving and remote monitoring system based on an unmanned platform can be used to execute a remote driving and remote monitoring method based on an unmanned platform, including: S1: obtaining steering wheel data and environmental image data of an unmanned sanitation vehicle; S2: determining the operating mode of the unmanned sanitation vehicle based on the steering wheel data; the operating modes include automatic driving and remote driving; S3: when the operating mode of the unmanned sanitation vehicle is automatic driving, the driving route is calculated using a grid map algorithm and a dynamic obstacle avoidance algorithm to obtain the optimal driving route and control the automatic driving of the unmanned sanitation vehicle; S4: when the operating mode of the unmanned sanitation vehicle is remote driving, based on image stitching technology and color space conversion algorithm, the steering wheel data and environmental image data are processed using an unmanned platform to obtain a monitoring video of the unmanned sanitation vehicle, and control speed and direction; S5: based on the monitoring video, control speed and direction of the unmanned sanitation vehicle, remote driving instructions are obtained, and the unmanned sanitation vehicle is remotely driven, completing remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

[0085] In some embodiments of the present specification, the processor utilizes an unmanned platform-based remote driving and remote monitoring system to execute an unmanned platform-based remote driving and remote monitoring method. (1) By utilizing 5G communication technology, using a cloud server for data transfer, adopting WebRTC audio and video live broadcast technology for real-time monitoring video display, and remotely operating the steering wheel to control the unmanned sanitation vehicle through Socket technology, low-latency remote driving control with unlimited distance and stable, smooth, and high-definition remote monitoring can be achieved, thereby ensuring the safe driving of the vehicle in various complex road scenes; (2) By acquiring images from multiple cameras on the unmanned sanitation vehicle, processing and splicing each frame, a complete surround view stitching BEV bird's-eye view can be obtained, resulting in a more complete and clear monitoring image; (3) By performing color conversion on each frame of the camera image in different color spaces, the color space is unified to achieve a good color rendering effect; (4) In this way, the problems of communication delay, poor video quality, single perspective, and insufficient environmental adaptability in the existing technology are effectively solved.

[0086] Example 2

[0087] Figure 2 This is an exemplary flow chart of a remote driving and remote monitoring method based on an unmanned platform according to some embodiments of this specification. Figure 2 As shown, the process includes the following steps. In some embodiments, the process can be executed by a processor.

[0088] S1: Obtain steering wheel data and environmental image data of the unmanned sanitation vehicle.

[0089] Unmanned sanitation vehicles are unmanned vehicles used for sanitation operations.

[0090] Steering wheel data is data that reflects the operation of the steering wheel of an unmanned sanitation vehicle. For example, steering wheel data can include steering wheel rotation angle, rotation speed, and operating mode.

[0091] Environmental image data is data that reflects the environment around the unmanned sanitation vehicle. For example, environmental image data can include optical environmental data, acoustic environmental data, and multi-channel environmental image data.

[0092] In some embodiments, the processor can use multiple cameras in different directions covered by the unmanned sanitation vehicle to obtain multi-channel environmental image data; and use the steering wheel interface of the unmanned sanitation vehicle to obtain steering wheel data.

[0093] In some embodiments, the processor can install a camera in the front, rear, left, and right directions of the unmanned sanitation vehicle to obtain environmental image data in the four directions.

[0094] S2: Determine the operating mode of the unmanned sanitation vehicle based on the steering wheel data.

[0095] The operating mode is the mode that determines the current control status of the unmanned sanitation vehicle. For example, the operating mode can include automatic driving and remote driving.

[0096] Autonomous driving is a mode in which unmanned sanitation vehicles drive automatically according to road conditions.

[0097] Remote driving is a mode in which the driver remotely controls the driving of the unmanned sanitation vehicle.

[0098] In some embodiments, the processor can extract the operating mode based on the steering wheel data of the unmanned sanitation vehicle.

[0099] S3: When the unmanned sanitation vehicle is in automatic driving mode, the grid map algorithm and dynamic obstacle avoidance algorithm are used to calculate the driving route, obtain the optimal driving route, and control the automatic driving of the unmanned sanitation vehicle.

[0100] The optimal driving route is the route that combines the lowest collision risk and the shortest driving distance.

[0101] In some embodiments, the processor can implement S3 based on the following steps: when the operating mode of the unmanned sanitation vehicle is automatic driving, the working area in the environmental image data is rasterized using a grid map algorithm to obtain rasterized data; the starting point and end point of the rasterized data are set, and by creating open and closed lists, the evaluation function is used to calculate and select the optimal grid, and the current grid is continuously expanded until the end point is found or the path cannot be found, thereby obtaining an initial driving route; obstacle-related data is obtained by detecting and identifying obstacles and predicting their motion trajectories; obstacle-related data of the unmanned sanitation vehicle is analyzed based on a dynamic obstacle avoidance algorithm to obtain obstacle avoidance decisions, and when the original path is not feasible due to obstacles, the route is replanned to obtain the optimal driving route, and the unmanned sanitation vehicle is controlled to drive automatically.

[0102] The rasterized data is data obtained by rasterizing the working area in the environment image data.

[0103] The initial driving route is a driving route obtained by calculating the evaluation function and selecting the optimal grid.

[0104] The obstacle-related data is data related to the obstacle size and movement trajectory.

[0105] S4: When the operation mode of the unmanned sanitation vehicle is remote driving, the steering wheel data and environmental image data are processed by the unmanned platform based on image stitching technology and color space conversion algorithm to obtain the monitoring video of the unmanned sanitation vehicle and control the speed and direction.

[0106] The monitoring video is a wide-angle, large-field-of-view video of the surrounding environment of the unmanned sanitation vehicle.

[0107] The control speed and direction are data that reflect the current operating speed and direction of the unmanned sanitation vehicle.

[0108] In some embodiments, the processor may implement S4 based on the following steps.

[0109] S410: Based on the environmental image data and camera parameters, obtain equivalent refraction angles of images with different viewing angles through fisheye distortion correction.

[0110] Camera parameters are intrinsic parameters of the camera used for monitoring. For example, camera parameters can include intrinsic parameters and extrinsic parameters.

[0111] The equivalent refraction angle is a transformation parameter that converts images with different incident angles into the same viewing angle.

[0112] In some embodiments, the processor can use a color space conversion algorithm based on the environmental image data and camera parameters to convert the conversion matrix of different color spaces of the environmental image data into the relationship between the color data of each dimension in the original camera color space and the color data of each dimension in the RGB color space, thereby obtaining environmental image data in a unified color space; and for the environmental image data in the unified color space, fisheye distortion correction is performed to obtain the equivalent refraction angles of images of different perspectives.

[0113] In some embodiments, the expression for the equivalent refraction angle of the environmental image data may be:

[0114] ;

[0115] in, represents the equivalent refraction angle after fisheye distortion, represents the angle of incidence, 、 、 and Both represent the distortion coefficient of the fisheye camera.

[0116] S420: Perform perspective transformation processing based on the environmental image data of the equivalent refraction angle to obtain a mapping matrix.

[0117] The mapping matrix is ​​a matrix that transforms images from different perspectives to a bird's-eye view.

[0118] In some embodiments, the mapping matrix may be expressed as:

[0119] ;

[0120] in, represents the horizontal coordinate of the transformed image, represents the ordinate of the transformed image, Represents the height coordinate of the transformed image, represents the camera intrinsic parameter matrix, represents the horizontal coordinate of the original image, Indicates the vertical coordinate of the original image.

[0121] S430: performing transformation processing on the mapping matrix to obtain a complete surround view image.

[0122] The complete surround view image is an image that reflects the complete environment around the unmanned sanitation vehicle.

[0123] In some embodiments, the processor may implement S430 based on the following steps.

[0124] S431: Perform matrix transformation on the mapping matrix to obtain a transformed matrix.

[0125] In some embodiments, the processor may perform rotation and translation transformation on the mapping matrix to obtain a transformed matrix.

[0126] S432: performing image segmentation on the transformed matrix, extracting overlapping areas in the image, and obtaining a segmented image.

[0127] The segmented image is an image obtained by segmenting the transformed matrix based on the overlapping areas in the image.

[0128] S433: Generate fusion weights based on the segmented image and the overlapping area in the image to obtain a fused image.

[0129] The fused image is an image in which multiple segmented images are fused together to eliminate the gaps between them.

[0130] In some embodiments, the processor may generate a fusion weight mask based on the overlapping areas, and perform fusion processing on the segmented images to obtain a fused image.

[0131] S434: stitching the fused images, and inputting the stitched images into S431 for further processing until a complete surround view image is obtained.

[0132] S440: Using the unmanned platform, the plurality of surround view images are spliced ​​in chronological order to obtain the monitoring video.

[0133] S450: Utilize the unmanned platform to calculate the steering wheel data to obtain the control speed and direction of the unmanned sanitation vehicle.

[0134] In some embodiments, the processor can implement S450 based on the following steps: using the unmanned platform to initialize the steering wheel data to obtain a speed variable; parsing the data received from the server to obtain angle-related values, forward-related values, and stop-related values; based on the angle-related values, the forward-related values, and the stop-related values, obtaining a target speed value through calculation; using a smoothing control algorithm, taking the target speed value as a standard, optimizing the speed variable to obtain the control speed and direction of the unmanned sanitation vehicle.

[0135] The speed variable is a variable that reflects the speed and changes of the unmanned sanitation vehicle.

[0136] The target speed value is the optimal speed value for the current motion.

[0137] In some embodiments, the processor can initialize a ROS node and a speed topic publisher, defining multiple speed-related variables. It then receives data from a server via a UDP socket and parses the data into angle, forward, and stop values. Based on these values, it calculates the target speed (including forward, reverse, steering, and lateral movement). Then, using a smoothing control algorithm, it gradually approaches the target speed. Finally, it assigns the calculated speed value to a ROS speed topic variable and publishes the topic, thereby obtaining the control speed and direction of the unmanned sanitation vehicle.

[0138] S5: Based on the monitoring video of the unmanned sanitation vehicle, control the speed and direction, obtain remote driving instructions, remotely drive the unmanned sanitation vehicle, and complete remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

[0139] Remote driving instructions are remote instructions for controlling the driving of unmanned sanitation vehicles.

[0140] In some embodiments, the processor can obtain remote driving instructions input remotely by the driver based on the monitoring video, control speed and direction of the unmanned sanitation vehicle, remotely drive the unmanned sanitation vehicle, and complete remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

[0141] In some embodiments of this specification, the processor uses the steering wheel data and environmental image data of the unmanned sanitation vehicle to realize the automatic driving of the unmanned sanitation vehicle through the grid map algorithm and the dynamic obstacle avoidance algorithm, and realizes the remote driving and remote monitoring of the unmanned sanitation vehicle through the image stitching technology and the color space conversion algorithm. (1) By utilizing 5G communication technology, using cloud servers for data transfer, adopting WebRTC audio and video live broadcast technology for real-time monitoring video display, and remotely operating the steering wheel to control the unmanned sanitation vehicle through Socket technology, it is possible to realize low-latency remote driving control at unlimited distance and stable, smooth and high-definition remote monitoring, thereby ensuring the safe driving of the vehicle in various complex road scenes; (2) By obtaining images from multiple cameras on the unmanned sanitation vehicle, processing and stitching each frame, a complete surround stitching BEV bird's-eye view can be obtained to obtain a more complete and clear monitoring image; (3) By performing color conversion on each frame of the camera image in different color spaces, the color space is unified to achieve a good color rendering effect; (4) In this way, the problems of communication delay, poor video quality, single perspective and insufficient environmental adaptability in the existing technology are effectively solved.

Claims

1. A remote driving and remote monitoring method based on an unmanned platform, characterized in that: include: S1: Acquire steering wheel data and environmental image data of the unmanned sanitation vehicle; wherein the steering wheel data is data reflecting the operation of the steering wheel of the unmanned sanitation vehicle, and the environmental image data is data reflecting the environmental conditions surrounding the unmanned sanitation vehicle; S2: Determine an operating mode of the unmanned sanitation vehicle based on the steering wheel data; the operating mode includes automatic driving and remote driving; S3: When the unmanned sanitation vehicle is in automatic driving mode, the grid map algorithm and dynamic obstacle avoidance algorithm are used to calculate the driving route, obtain the optimal driving route, and control the automatic driving of the unmanned sanitation vehicle; S4: When the unmanned sanitation vehicle is in remote driving mode, the steering wheel data and environmental image data are processed by the unmanned platform based on image stitching technology and color space conversion algorithm to obtain monitoring video of the unmanned sanitation vehicle and control speed and direction; including: S410: Obtaining equivalent refraction angles of images at different viewing angles through fisheye distortion correction based on the environmental image data and camera parameters; S420: Performing perspective transformation processing based on the environmental image data of the equivalent refraction angle to obtain a mapping matrix; S430: performing transformation processing on the mapping matrix to obtain a complete surround view image; S440: Using the unmanned platform, stitching the multiple surround view images in chronological order to obtain the monitoring video; S450: Calculating the steering wheel data using the unmanned platform to obtain a control speed and direction of the unmanned sanitation vehicle; The expression of the equivalent refraction angle is: ； The expression of the mapping matrix is: ； in, represents the equivalent refraction angle after fisheye distortion, represents the angle of incidence, 、 、 and Both represent the distortion coefficient of the fisheye camera, represents the horizontal coordinate of the transformed image, represents the ordinate of the transformed image, Represents the height coordinate of the transformed image, represents the camera intrinsic parameter matrix, represents the horizontal coordinate of the original image, Indicates the vertical coordinate of the original image; S5: Based on the monitoring video of the unmanned sanitation vehicle, control the speed and direction, obtain remote driving instructions, remotely drive the unmanned sanitation vehicle, and complete remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

2. The remote driving and remote monitoring method based on an unmanned platform according to claim 1, characterized in that: Said S1 comprises: Utilize multiple cameras in different locations covered by unmanned sanitation vehicles to obtain multi-channel environmental image data; Use the steering wheel interface of the unmanned sanitation vehicle to obtain steering wheel data.

3. The remote driving and remote monitoring method based on an unmanned platform according to claim 1, characterized in that: The S3 includes: When the unmanned sanitation vehicle is in the automatic driving mode, the working area in the environmental image data is rasterized using a raster map algorithm to obtain rasterized data; Setting the starting point and end point of the rasterized data, creating open and closed lists, applying an evaluation function to calculate and select the optimal grid, and continuously expanding the current grid until the end point is found or no path can be found, thereby obtaining an initial driving route; Obstacle-related data is obtained by detecting and identifying obstacles and predicting their movement trajectories; Based on the dynamic obstacle avoidance algorithm, the data related to the unmanned sanitation vehicle and obstacles are analyzed to obtain obstacle avoidance decisions. When the original path is not feasible due to obstacles, the route is replanned to obtain the optimal driving route and control the automatic driving of the unmanned sanitation vehicle.

4. The remote driving and remote monitoring method based on an unmanned platform according to claim 1, characterized in that: The S410 includes: Based on the environmental image data and camera parameters, using a color space conversion algorithm, converting the conversion matrix of the different color spaces of the environmental image data into the relationship between the color data of each dimension in the original camera color space and the color data of each dimension in the RGB color space, thereby obtaining environmental image data in a unified color space; The environmental image data in the unified color space is subjected to fisheye distortion correction to obtain equivalent refraction angles of images at different viewing angles.

5. The remote driving and remote monitoring method based on an unmanned platform according to claim 1, characterized in that: The S450 includes: Initializing the steering wheel data using an unmanned platform to obtain a speed variable; Parse the data received from the server to obtain angle-related values, forward-related values, and stop-related values; Obtaining a target speed value by calculation based on the angle-related value, the forward-related value, and the stop-related value; By using a smoothing control algorithm and taking the target speed value as a standard, the speed variable is optimized to obtain the control speed and direction of the unmanned sanitation vehicle.

6. The remote driving and remote monitoring method based on an unmanned platform according to claim 1, characterized in that: The S430 includes: S431: Performing matrix transformation on the mapping matrix to obtain a transformed matrix; S432: performing image segmentation on the transformed matrix, extracting overlapping areas in the image, and obtaining a segmented image; S433: generating a fusion weight based on the segmented image and the overlapping area in the image to obtain a fused image; S434: stitching the fused images, and inputting the stitched images into S431 for further processing until a complete surround view image is obtained.

7. A remote driving and remote monitoring system based on an unmanned platform, used to execute the remote driving and remote monitoring method based on an unmanned platform according to any one of claims 1 to 6, characterized in that: include: An acquisition module is used to obtain steering wheel data and environmental image data of the unmanned sanitation vehicle; An operation module, configured to determine an operation mode of the unmanned sanitation vehicle based on the steering wheel data; the operation mode includes automatic driving and remote driving; The automatic driving module is used to calculate the driving route using the grid map algorithm and dynamic obstacle avoidance algorithm when the unmanned sanitation vehicle is in automatic driving mode, obtain the optimal driving route, and control the automatic driving of the unmanned sanitation vehicle; A monitoring module, when the unmanned sanitation vehicle is in remote driving mode, is used to process the steering wheel data and environmental image data based on the unmanned platform based on image stitching technology and color space conversion algorithm to obtain monitoring video of the unmanned sanitation vehicle and control speed and direction; The remote driving module is used to obtain remote driving instructions based on the monitoring video of the unmanned sanitation vehicle, control the speed and direction, remotely drive the unmanned sanitation vehicle, and complete remote driving and remote monitoring of the unmanned sanitation vehicle based on the unmanned platform.

8. The remote driving and remote monitoring system based on an unmanned platform according to claim 7 is characterized in that: The acquisition module includes: Environmental data acquisition unit: users use multiple cameras in different directions covered by unmanned sanitation vehicles to obtain multi-channel environmental image data; The steering wheel data acquisition unit is used to obtain steering wheel data using the steering wheel interface of the unmanned sanitation vehicle.

Images