A remote control driving method based on multi-viewing angles

Abstract

The present application discloses a remote control driving method based on multiple perspectives, which is used in an unmanned driving system, and involves unmanned driving. The unmanned driving system includes: a vehicle-side remote Handler, a cockpit HMI, a remote control driving system module, a vehicle-side CloudHandler, and a cloud platform. In view of the problem that unmanned driving in the prior art cannot cope with complex environments, the present application establishes a communication link and a control process between the cloud platform, the remote control driving system module, the cockpit HMI, and the vehicle-side remote Handler, thereby realizing the remote control takeover of the vehicle in multiple modes. In each link of initiating, establishing, and executing remote control driving, the remote control driving conditions such as cockpit environmental parameters and communication status are detected, as well as the remote control request is manually confirmed and verified in the cloud, so as to ensure the safety and reliability of the remote control driving from the link and process. At the same time, the automatic remote control takeover and remote monitoring mode in the event of a fault further improve the fault tolerance and stability of the system.

Description

Technical Field

[0001] The present application relates to the field of unmanned driving, and in particular to a remote control driving method based on multiple perspectives. Background Art

[0002] In recent years, with the development of emerging technologies such as artificial intelligence, 5G communications, and cloud computing, unmanned driving technology has made great progress. Unmanned driving systems have been widely used in many fields such as smart transportation, smart logistics, and smart mining, effectively improving operational efficiency and safety and reducing labor costs. Especially in some complex and dangerous working environments, unmanned driving systems can replace manual tasks.

[0003] Since the real-world road environment is complex and changeable, with various structured and unstructured obstacles, it is difficult for unmanned vehicles to cope with it solely by relying on their own perception and decision-making capabilities. Especially when operating in complex scenes such as canyons, mines, and construction sites, the environment layout is irregular, there are many obstacles, and the road conditions are difficult to predict, which puts extremely high demands on the unmanned driving system. How to give unmanned vehicles stronger environmental adaptability has become the focus of industry research.

[0004] In response to the above challenges faced by unmanned driving, the industry has conducted a lot of research and proposed a variety of solutions. Remote control driving is an effective auxiliary means. By introducing human intelligence to command the driving of unmanned vehicles through the human-in-the-loop method, the shortcomings of unmanned driving systems in complex environments can be compensated. At present, there are some unmanned vehicle remote control driving systems for complex scene applications, but there are still problems such as imperfect functions and unsatisfactory experience, which are mainly reflected in: most remote control driving systems only support active control by the driver, and lack auxiliary modes such as fault response and task management. When the system is abnormal, it cannot automatically switch to remote takeover, which poses a safety hazard. At the same time, in the scenario of multi-vehicle collaborative operation, it is difficult for dispatchers to uniformly control and dispatch tasks for all vehicles, which is a heavy manpower burden and inefficient. Summary of the invention

[0005] In response to the problem that unmanned driving in the existing technology cannot cope with complex environments, the present application provides a remote control driving method based on multiple perspectives, which realizes vehicle remote control takeover in multiple modes by establishing a communication link and control process between the cloud platform, the remote control driving system module, the cockpit HMI and the vehicle-side remoteHandler.

[0006] The purpose of this application is achieved through the following technical solutions.

[0007] The present application provides a remote control driving method based on multiple perspectives, which is used for an unmanned driving system, including: the unmanned driving system includes: a vehicle-side remote Handler, a cockpit HMI, a remote control driving system module, a vehicle-side CloudHandler and a cloud platform; wherein the cloud platform includes a communication gateway, a large screen, a device management module and an alarm management module; when the cockpit HMI generates vehicle selection data E402 and a start remote control instruction E401, the cockpit HMI sends a remote control application P1 to the cloud platform through the remote control driving system module, and after the cloud platform generates a control instruction C1, it issues a remote control instruction F401, establishes a connection between the cockpit HMI and the vehicle-side remoteHandler, and remotely controls the vehicle by receiving vehicle-side video data V1 and sending a remote control instruction E1; when the vehicle-side CloudHandler detects a vehicle failure, the vehicle-side CloudHandl er reports the fault data D to the alarm module; the alarm module combines with the device management module and the communication gateway, and decides whether to send the control instruction C1 to the remote control driving system module according to the fault data D and the driver confirmation instruction E8, so as to realize the remote control takeover when the vehicle fails; when the large screen generates the vehicle and cockpit selection data and sends the remote control takeover instruction, the device management module decides whether to send the control instruction C1 to the remote control driving system module through the communication gateway according to the vehicle and cockpit status and the driver confirmation instruction E8, so as to realize the remote control takeover of the vehicle initiated by the cloud platform; when the cockpit HMI generates the vehicle selection data E402 and the start monitoring instruction, the remote control driving system module sends the monitoring instruction E5 through the communication gateway after confirming that the vehicle status meets the preset conditions, and establishes the connection between the vehicle-side remote Handler and the cockpit HMI, so as to realize the remote monitoring of the vehicle by the cockpit HMI.

[0008] Among them, the vehicle-side remote Handler: a remote processing module installed on the unmanned vehicle, responsible for communicating with the cockpit and the cloud platform, receiving remote control commands and converting them into vehicle control commands, while collecting on-board sensor data and driving videos, and transmitting them back to the cockpit or the cloud. Cockpit HMI: Human-Machine Interface, refers to the interface and equipment located in the remote cockpit for the driver to interact with the human machine. Through the HMI, the driver can obtain vehicle driving information, view on-board videos, and issue remote control commands. Remote control driving system module: a software module deployed in the cockpit, connecting the HMI and the communication gateway, responsible for transmitting remote control commands and on-board data between the driver and the vehicle. This module also has functions such as connection management and command verification to ensure the reliability of the remote control process. Vehicle-side CloudHandler: a communication module installed on the unmanned vehicle, responsible for data interaction between the vehicle and the cloud platform. When a vehicle fault is detected, the module will immediately report the fault information to the cloud alarm management module. Device management module: a functional module on the cloud platform, responsible for maintaining the status information of all connected vehicles and cockpits. Based on the vehicle status and cockpit request, the device management module decides whether to approve and establish a remote control connection. Alarm management module: a functional module on the cloud platform, responsible for receiving and processing various fault information reported by the vehicle. When a vehicle fails, the alarm management module will promptly notify the cockpit or arrange for remote control takeover to avoid accidents. Vehicle selection data E402: generated by the cockpit HMI, used to inform the remote control driving system module of the target vehicle ID that the driver wants to remotely control or monitor, thereby establishing a connection with a specific vehicle. Start remote control command E401: generated by the cockpit HMI, instructing the remote control driving system module to start the remote control connection with the vehicle and start taking over the driving task of the vehicle. Remote control command F401: a command issued by the cloud platform to the vehicle-side remote Handler and the remote control driving system module, instructing both parties to establish the communication connection required for vehicle remote control and prepare for remote control takeover.

[0009] Furthermore, the vehicle is remotely controlled by receiving the vehicle-side video data V1 and sending the remote control command E1, including: when the cockpit HMI generates the vehicle selection data E402 and the start remote control command E401, it detects whether the remote control driving system module meets the preset conditions. If so, the remote control driving system module sends a remote control application P1 to the cloud platform; otherwise, the remote control driving system module sends a message prompt to the cockpit HMI; after the cloud platform receives the remote control application P1, it determines whether to take over remote control. If so, the cloud platform sequentially sends the start streaming command E3 and the start remote control command E401 to the vehicle-side remote Handler, and sends the start remote control command E401 to the cockpit HMI; otherwise, the cloud platform sends an alarm message M1 to the remote control driving system module; after the vehicle-side remote Handler receives the start streaming command E3 and the start remote control command E401, it establishes a connection with the cockpit HMI and starts streaming, and at the same time receives the remote control command E1 from the cockpit HMI.

[0010] Among them, Start Streaming: It is the process of starting the vehicle camera to collect video and transmit video stream data to the cockpit in real time after the vehicle-side remote Handler module receives the start streaming instruction E3 issued by the cloud platform. Specifically, when the vehicle-side remote Handler receives the start streaming instruction E3, it will perform the following operations: Activate the vehicle camera: The remote Handler sends a wake-up and working mode setting instruction to the vehicle camera to put it into the video acquisition state. Establish a video encoder: The remote Handler creates a video encoder instance to encode and compress the collected original video frames, usually using efficient compression formats such as H.264 or H.265 to reduce transmission bandwidth. Initiate video stream transmission: The remote Handler establishes a dedicated data channel (such as RTMP, RTSP, WebRTC, etc.) with the cockpit HMI to continuously push the encoded video stream data to the HMI. Dynamically adjust the streaming parameters: The remote Handler adjusts the video encoding bit rate, frame rate, resolution and other parameters in real time according to the stability of the communication link and the bandwidth status, while ensuring smooth and clear pictures and reducing transmission delay as much as possible.

[0011] Furthermore, detecting whether the remote control driving system module meets the preset conditions includes: collecting cockpit environmental data through the remote control driving system module, the environmental data including ambient temperature, humidity and light intensity, and judging whether the collected environmental data is within a preset range; obtaining hardware parameters of the remote control driving system module, the hardware parameters including power supply voltage, CPU usage and memory usage, and judging whether the hardware parameters meet the preset remote control driving conditions; obtaining the communication link status with the vehicle-side remoteHandler, and judging whether the current communication delay and packet loss rate meet the preset remote control driving conditions.

[0012] Specifically, the remote control driving system module collects environmental parameter data in the cockpit in real time through the integrated environmental sensor. Among them: the temperature and humidity sensor (such as DHT11, DHT22, etc.) is used to collect the ambient temperature and relative humidity in the cockpit. The sensor is connected to the MCU (microcontroller) of the remote control driving system module through a single bus. The MCU triggers the sensor to sample at a certain frequency (such as 1Hz) and reads the temperature and humidity data through the I2C interface. The light sensor (such as BH1750, etc.) is used to collect the ambient light intensity in the cockpit. The sensor is connected to the MCU through the I2C interface. The MCU triggers the sensor to sample at a certain frequency (such as 1Hz) and reads the current light intensity value (unit: Lux). The MCU packages the collected temperature, humidity, and light intensity data and sends them to the host computer of the remote control driving system module periodically (such as 1Hz) through the UART interface. The host computer parses the sensor data packet and saves the parsing result (floating point format) in the memory for subsequent judgment. The host computer compares the collected temperature, humidity, and light intensity values ​​with the preset threshold range (which can be read from the configuration file). If any parameter exceeds the preset range (such as temperature>35℃, humidity>70%, light<100Lux, etc.), the cockpit environment is considered to not meet the remote control driving conditions, and the host computer immediately sends an environmental abnormality prompt message to the cockpit HMI.

[0013] Get the hardware parameters of the remote control driving system module. The host computer of the remote control driving system module periodically obtains the current hardware operating parameters through the operating system API or third-party library: Power supply voltage: The host computer reads the current system power supply voltage value through the system API (such as the voltage file in the / sys / class / power_supply / directory) to determine whether it is within the preset range (such as 4.75~5.25V). If the voltage is abnormal, the host computer immediately sends a power supply abnormality prompt to the HMI. CPU usage: The host computer reads the / proc / stat file and periodically (such as 1 second) calculates the CPU time ratio in each state to obtain the CPU usage percentage. If the CPU usage exceeds the preset threshold (such as 90%), the host computer immediately sends a CPU overload prompt to the HMI. Memory usage: The host computer reads the / proc / meminfo file and periodically (such as 1 second) calculates the current physical memory and swap space usage. If the memory usage exceeds the preset threshold (such as 80%), the host computer immediately sends an insufficient memory prompt to the HMI.

[0014] Get the communication link status with the remote Handler on the vehicle side. Maintain a fixed heartbeat channel between the host computer of the remote control driving system module and the remote Handler module on the vehicle side. Both parties periodically (such as 1 second) send heartbeat packets to each other, and count the round-trip delay and packet loss at the application layer: Communication delay: When the host computer sends a heartbeat packet, it carries the sending timestamp T1 (milliseconds) in the packet header. After receiving the heartbeat packet, the remote Handler extracts T1 and immediately replies with a response packet with T1. The host computer receives the response packet, extracts the response timestamp T2, and then calculates the round-trip delay RTT = T2 - T1. Packet loss rate: The host computer maintains a heartbeat packet sending window of a fixed size (such as 100), numbers each sent heartbeat packet (1~100), and records its sending time. In the response packet, the remote Handler needs to carry the corresponding heartbeat packet number. The host computer marks the corresponding heartbeat packet as answered in the window according to the response packet number. Periodically (e.g., 5 seconds), the host computer counts the number of unanswered heartbeat packets in the window and divides it by the window size to get the packet loss percentage. Then, the round-trip delays of all heartbeat packets in the window are accumulated and averaged to get the average communication delay. The host computer compares the calculated average communication delay and packet loss rate with the preset threshold (which can be read from the configuration file). If the communication delay exceeds the preset value (e.g., 200 milliseconds) or the packet loss rate exceeds the preset value (e.g., 5%), the host computer immediately sends a communication abnormality prompt to the HMI.

[0015] Further, the remote control takeover when the vehicle fails includes: when the vehicle-side CloudHandler detects a vehicle failure, it generates fault data D and reports the fault data D to the alarm management module; after the alarm management module receives the fault data D, it makes a judgment through the fault analysis algorithm and sends the judgment result to the device management module; after receiving the judgment result, the device management module obtains the idle cockpit data and sends the remote control instruction F401 to the remote control driving system module through the communication gateway; after receiving the remote control instruction F401, the remote control driving system module sends a manual takeover confirmation request E6 to the cockpit HMI; after receiving the manual takeover confirmation request E6, the cockpit HMI displays a remote control takeover prompt message for the driver to confirm whether to agree to the remote control takeover; the driver generates a manual takeover confirmation instruction E8 through the cockpit HMI, and after receiving the manual takeover confirmation instruction E8, the remote control driving system module generates a remote control confirmation instruction E7 and sends the remote control confirmation instruction E7 to the communication gateway; after receiving the remote control confirmation instruction E7, the device management module determines whether to agree to the remote control takeover through the decision algorithm, and if agreed, the device management module sends the control instruction C1 to the vehicle-side remote through the communication gateway. Handler and remote control driving system module; after receiving the control command C1, the vehicle-side remoteHandler establishes a connection with the cockpit HMI for interaction, and sets the state of the cockpit HMI to remote control.

[0016] Among them, the fault analysis algorithm is a data processing program deployed in the cloud alarm management module, which is used to quickly and accurately analyze and judge the fault data reported by the vehicle, identify the fault type and severity, and automatically generate disposal suggestions. In this application, the fault data D reported by the vehicle is formatted and parsed, and key fields such as fault code, fault occurrence time, and vehicle status parameters are extracted, and stored in a structured database or memory data object for subsequent analysis. According to the predefined fault type dictionary or fault tree model, the extracted fault codes are classified and queried to identify specific fault types, such as power system faults, sensor faults, network faults, etc. According to the evaluation rules of different fault types, the vehicle speed, acceleration, steering angle and other state parameters at the time of the fault are comprehensively analyzed, and the dangerousness of the fault is automatically judged by referring to the historical experience of handling similar faults, such as "negligible", "minor", "serious", "critical", etc. According to the fault type and severity, the optimal solution is matched from the preset disposal strategy set, and suggestions for handling the fault are automatically generated, such as "limiting vehicle speed", "switching redundant system", "driver remote takeover", "safe parking", etc.

[0017] Among them, the decision-making algorithm is a strategy analysis program deployed in the cloud device management module, which is used to review and make decisions on remote control takeover requests, judge the feasibility and rationality of the current remote control takeover plan, and balance various factors to give a final conclusion on whether to approve the takeover. In this application, the remote control confirmation instruction E7 sent by the remote control driving system module is parsed to extract key information such as the ID of the vehicle to be remotely controlled, the fault type, the driver ID, and the cockpit ID. According to the vehicle ID, the Internet of Vehicles platform database is queried to obtain the vehicle's real-time location, speed, power, fault status and other driving data to determine whether the vehicle is in a remote control state. According to the driver ID, the personnel management system database is queried to obtain the driver's license type, remote control driving level, accumulated remote control mileage and other qualification data to determine whether he meets the requirements for taking over the vehicle. According to the cockpit ID, the performance parameters such as the hardware configuration, network bandwidth, and system load of the cockpit are obtained, and the historical remote control records are referred to determine whether the cockpit can provide stable and reliable remote control services. According to the evaluation results of the takeover plan, a decision to approve or reject the remote control application is made to generate a control instruction C1. If approved, the target vehicle and designated cockpit are noted in C1; if rejected, the reason for rejection is stated in C1.

[0018] Furthermore, the alarm management module makes a judgment after receiving the fault data D, and sends the judgment result to the equipment management module, including: after the alarm management module receives the fault data D reported by the vehicle-side CloudHandler, it extracts the fault component type feature D1 and the fault location D2 from the fault data D; uses the extracted fault component type feature D1 and the fault location D2 as input, and uses the pre-trained decision tree model to calculate the severity of the current fault data; when the calculated severity is greater than the set threshold, generates a remote control takeover judgment result.

[0019] Specifically, the vehicle-mounted CloudHandler module monitors the working status of each component of the vehicle (such as the power system, brake system, steering system, etc.) in real time. Once an abnormality is detected, the fault-related data is immediately packaged and reported to the remote alarm management module. The fault data D includes at least: fault component type feature D1 (such as power battery, brake actuator, etc.) and fault location D2 (such as GPS location, lane number, etc.). The alarm management module receives the fault data D reported by the vehicle-mounted CloudHandler in real time based on the socket communication mechanism. After receiving the fault data D, the alarm management module first parses it and extracts key features from the original data packet, including the fault component type feature D1 and the fault location D2. The extraction process can be achieved by regular expression matching. The alarm management module loads the decision tree model that has been pre-trained offline, and uses the extracted fault component type feature D1 and the fault location D2 as input (i.e., the root node of the decision tree), and traverses the decision tree layer by layer. Among them, each internal node represents the judgment condition of the input feature, and the leaf node represents the output result of the fault severity. Based on the comparison results of the feature values ​​and the judgment conditions, the decision tree model will eventually give the severity of the current fault, such as "mild", "moderate", "serious" and other different levels.

[0020] The alarm management module has a severity threshold to determine whether manual remote control is needed. When the severity calculated by the decision tree model is greater than the threshold (such as reaching the "serious" level), the alarm management module generates a remote control takeover judgment result, believing that the current vehicle cannot continue to operate autonomously and needs to be switched to manual remote control. The alarm management module pushes the remote control takeover judgment result to the device management module in real time via WebSocket to notify it to start remote control. At the same time, the alarm management module also stores information such as fault data D, fault severity, and remote control takeover judgment result in the fault log database to facilitate post-event root cause analysis and model iteration optimization.

[0021] Furthermore, the fault component type feature D1 includes: vehicle identification number, fault occurrence time, fault code and fault parameters; the fault occurrence location D2 includes: underground partition number, section number and coordinate data; wherein, the underground partition number represents the area number of the underground mining area, and the section number represents the unique number of the section in the corresponding area.

[0022] Specifically, the fault component type feature D1 is encapsulated in a structured JSON format and contains the following fields: Vehicle identification number (vehicleID): used to uniquely identify each unmanned vehicle, and the vehicle VIN code or custom encoding rules can be used. Fault occurrence time (faultTime): Use the ISO8601 standard format (such as "2023-06-01T12:30:00+08:00") to record the timestamp of the fault occurrence, accurate to seconds, and including time zone information. Fault code (faultCode): Refer to the fault diagnostic code (DTC) defined in the OBD-II standard, and use a combination of letters + numbers (such as "P0001") to indicate the fault type. The letters represent the major fault categories (such as power system, body system, etc.), and the numbers represent the specific fault items. Fault parameters (faultParams): In the form of key-value pairs, record the quantitative parameters related to the current fault (such as voltage value, speed, etc.) for a detailed description of the fault status for subsequent cause analysis.

[0023] The fault location D2 is represented by hierarchical coding, from coarse to fine, it is underground partition, road section, and coordinates: Underground partition number (zoneID): Use two digits (such as "01", "02", etc.) to identify different areas of the underground mining area. The partition division should take into account factors such as tunnel layout, ventilation system, power supply, etc., so as to be independent of each other and easy to manage. Section number (roadID): On the basis of the partition number, use three digits (such as "001", "002", etc.) to identify the independent road section within each partition. The section division should have clear starting and ending points and moderate length, which is conducive to the rapid locking of the fault location. Coordinate data (coordinate): Use the underground coordinate system (such as defining the X axis along the tunnel direction, the Y axis vertical to the tunnel direction, and the Z axis vertical to the shaft direction) to accurately record the spatial position of the unmanned vehicle when the fault occurs, in meters. GPS signals may not be able to penetrate underground, and the high-precision positioning module of the unmanned vehicle itself needs to provide location information.

[0024] Furthermore, the cloud platform actively initiates the remote control takeover of the vehicle, including: the vehicle-side remote Handler receives the remote control command E1 and the start remote control command E401 from the cockpit HMI or the cloud platform; the remote control driving system module receives the start remote control command E401 sent by the cockpit HMI, and detects whether the remote control driving system module meets the preset conditions, and if so, sends a remote control application P1 to the communication gateway; the communication gateway receives and sends the remote control application P1 to the device management module; the device management module verifies and judges the remote control application P1, and if the remote control takeover is agreed, a control command C1 is generated, and the control command C1 is sent to the vehicle-side remote Handler and the remote control driving system module respectively through the communication gateway; otherwise, a takeover refusal message M1 is generated, and the takeover refusal message M1 is sent to the alarm management module; the vehicle-side remote After receiving the control instruction C1, the Handler sends the start streaming instruction E3, the start remote control instruction E401 and the start monitoring instruction E5 to the remote control driving system module; after receiving the control instruction C1, the remote control driving system module sends a takeover confirmation request E6 to the cockpit HMI; the cockpit HMI receives the takeover confirmation request E6 and prompts the driver to confirm; the driver generates a manual takeover confirmation instruction E8 through the HMI, and after receiving the manual takeover confirmation instruction E8, the remote control driving system module generates a remote control confirmation instruction E7 and sends it to the communication gateway; the communication gateway sends the remote control confirmation instruction E7 to the device management module for reconfirmation; if the device management module agrees to the remote control takeover, the device management module generates a remote control permission instruction C2, and the communication gateway sends the remote control permission instruction C2 to the vehicle-side remoteHandler and the remote control driving system module respectively; after receiving the remote control permission instruction C2, the vehicle-side remote Handler establishes a connection with the cockpit HMI, sets the vehicle-side status to remote control, and collects video stream data V1 through the on-board camera and sends it to the cockpit HMI.

[0025] Furthermore, it also includes: when the on-board terminal determines that the vehicle needs to stop, controlling the on-board camera to collect video stream data V1 of the vehicle's surrounding environment, and sending the video stream data V1 to the cloud platform through the wireless network; receiving the video stream data V1 of the vehicle's surrounding environment sent by the on-board terminal through the wireless network; wherein, the video stream data V1 is collected by the on-board camera when the on-board terminal determines that the vehicle needs to stop; performing image processing on the received video stream data V1 to obtain parking position information; the image processing includes: performing parking space target detection on the key frames in the video stream data V1, using the YOLO algorithm based on the convolutional neural network to segment and identify the key frame images, and obtaining the pixel coordinate position and size information of the parking space in the image; wherein, the key frame is dynamically selected according to the frame rate of the video stream data V1 and the driving speed of the vehicle; The method comprises the following steps: obtaining the GPS coordinates of the current position of the vehicle; converting the pixel coordinate information of the parking space into the three-dimensional space coordinates in the camera coordinate system according to the internal and external parameter matrix of the vehicle-mounted camera; converting the parking space coordinates in the camera coordinate system to the vehicle coordinate system according to the heading angle and pitch angle of the vehicle; converting the parking space coordinates in the vehicle coordinate system to the world coordinate system according to the GPS coordinates and IMU data of the vehicle to obtain the world coordinates of the parking space; calculating the relative distance and angle between the current position of the vehicle and the parking space according to the GPS coordinates of the vehicle and the world coordinates of the parking space through a triangulation algorithm to obtain the accurate position coordinates of the parking space relative to the vehicle; sending the world coordinate information and relative position information of the parking space to the vehicle-mounted terminal through a wireless network for trajectory planning and motion control of the vehicle.

[0026] Receive the parking position coordinate information sent by the cloud platform through the wireless network, including the world coordinate information of the parking position and the position information relative to the vehicle; obtain the real-time position, speed and heading angle information of the vehicle; among which, the real-time position information comes from the vehicle-mounted GPS module, the speed information comes from the wheel speed sensor, and the heading angle information comes from the IMU inertial measurement unit; estimate the position and attitude of the vehicle through a multi-sensor information fusion algorithm; specifically include: using the Kalman filter algorithm to fuse the GPS position information, IMU attitude information and wheel speed information to obtain the optimal estimated values ​​of the vehicle's position, speed and attitude; using the particle filter algorithm, combined with the vehicle-mounted laser radar SLAM positioning results, to correct the estimated value of the Kalman filter to further improve the accuracy and robustness of the vehicle's position and attitude estimation; using the Model Predictive Control (MPC) model predictive control algorithm to generate the optimal parking trajectory from the current position to the center of the parking space; specifically include: using the relative position between the vehicle and the parking space as a constraint and the shortest distance to the center of the parking space as the goal, establish an MPC optimization problem; integrate the vehicle's dynamic model, kinematic model, and the constraints of the control quantity and state quantity into the MPC optimization problem;

[0027] Through iterative solution of quadratic functional, the optimal prediction sequence of vehicle state and control quantity in the future period, i.e. parking trajectory, is obtained; based on the position point and heading angle information in the parking trajectory, the Pure Pursuit algorithm is used to calculate the steering wheel angle required for tracking the trajectory in real time; specifically, it includes: setting a preview distance on the parking trajectory, and calculating the lateral deviation and heading deviation of the preview point relative to the center of the rear axle of the vehicle in real time; using geometric relationship to calculate the steering wheel angle based on the preview distance, lateral deviation and heading deviation, and sending the angle information to the vehicle steering system; using PID control algorithm to calculate the acceleration and braking force control quantity of the vehicle in real time; specifically, it includes: taking the speed of each position point on the parking trajectory as the target, calculating the deviation between the current speed of the vehicle and the target speed in real time; using PID control law, according to the speed deviation and its integral and differential, calculating the control quantity of throttle opening and brake pressure, and sending the control command to the vehicle's drive system and brake system; according to the feedback information of the steering system, drive system and brake system, the vehicle's motion state is corrected in real time to ensure that the vehicle accurately drives along the parking trajectory and finally stops smoothly in the center of the parking space to complete the vehicle's automatic parking process.

[0028] Furthermore, the cockpit HMI remotely monitors the vehicle, including: the cockpit HMI sends a vehicle selection instruction E402 to the remote control driving system module to start the remote control process; after the remote control driving system module receives the vehicle selection instruction E402, it detects whether the remote control driving system module meets the preset conditions, and if so, sends a remote control application P1 to the communication gateway, otherwise it prompts the driver through the cockpit HMI that the remote control driving system module does not meet the preset conditions; the communication gateway receives the remote control application P1, forwards it to the device management module for evaluation, and determines whether the vehicle status and the remote control driving system module meet the remote control takeover; if the verification passes, the device management module generates a control instruction F401, which is sent to the vehicle-side remote Handler and the remote control driving system module respectively through the communication gateway; if the verification fails, the device management module generates a takeover rejection message M1, which is sent to the alarm management module and prompts the driver through the communication gateway; the vehicle-side remote The Handler receives the control instruction F401, and sends the start streaming instruction E3, the start remote control instruction E401, and the start monitoring instruction E5 to the remote control driving system module; after receiving the control instruction F401, the remote control driving system module sends a manual takeover confirmation request E6 to the cockpit HMI; the cockpit HMI receives the manual takeover confirmation request E6, and prompts the driver to confirm; the driver generates a manual takeover confirmation instruction E8, and the cockpit HMI sends the manual takeover confirmation instruction E8 to the remote control driving system module; after receiving the manual takeover confirmation instruction E8, the remote control driving system module generates a remote control confirmation instruction E7 and sends it to the communication gateway; the communication gateway forwards the remote control confirmation instruction E7 to the device management module for secondary confirmation; the device management module confirms the remote control confirmation instruction E7, supervises the establishment of the remote control session, and generates a remote control permission instruction C2 that is sent to the vehicle-side remote Handler and the remote control driving system module through the communication gateway respectively; the vehicle-side remote After the Handler receives the remote control permission command 2, it establishes a data communication connection with the cockpit HMI, sets the vehicle-side status to remote control, collects the vehicle's real-time video data V1 through the vehicle-mounted camera and sends it to the cockpit HMI; after the remote control driving system module receives the remote control permission command C2, it sets the cockpit status to remote control; the cockpit HMI receives the real-time video data V1 transmitted by the vehicle-side through the data communication connection established with the vehicle-side remote Handler, displays the vehicle's real-time picture through the streaming software, and realizes remote monitoring of the vehicle.

[0029] Compared with the prior art, the advantages of this application are:

[0030] By establishing communication links and control processes between the cloud platform, remote control driving system module, cockpit HMI and vehicle-side remote handler, the remote control takeover of vehicles in multiple modes is realized, including driver initiative, automatic takeover when vehicle fails, cloud platform initiative, remote monitoring and other modes. The remote control method can be flexibly selected according to actual needs, which improves the system's adaptability to complex working conditions.

[0031] In the initiation, establishment, and execution of remote control driving, the safety and reliability of remote control driving are guaranteed from the link and process through the detection of remote control driving conditions such as cockpit environmental parameters and communication status, as well as manual confirmation and cloud verification of remote control requests. At the same time, the automatic remote control takeover and remote monitoring mode in the event of a fault further improve the system's fault tolerance and stability.

[0032] The vehicle uploads information such as video data to the cloud platform, which can aggregate and analyze data from multiple vehicles to mine the value of the data. For example, in an unmanned parking scenario, the cloud platform detects and locates parking spaces based on the video images uploaded by the vehicle, and combines the positioning of the vehicle itself to obtain accurate parking control instructions, which can realize automatic parking of the vehicle. Combining vehicle-side perception with cloud-based intelligence expands the functions of the unmanned driving system.

[0033] The remote control driving system module and cloud platform are deployed remotely, which can realize multi-vehicle sharing and reduce the configuration of single vehicle hardware. At the same time, the functions such as fault monitoring and alarm reporting are centralized in the cloud management, which can timely discover and handle vehicle faults. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present application will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0035] Figure 1 is an exemplary fluid diagram of a cockpit actively applying for remote control takeover according to some embodiments of the present application;

[0036] Figure 2 is an exemplary fluid diagram of a vehicle fault-triggered remote control takeover according to some embodiments of the present application;

[0037] Figure 3 This is an exemplary flow diagram of the cloud actively initiating remote control takeover according to some embodiments of the present application;

[0038] Figure 4 This is an exemplary fluid diagram of cloud-based remote monitoring according to some embodiments of the present application. DETAILED DESCRIPTION

[0039] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0040] At present, the intelligent transformation of mines is in full swing. For example, in underground mining scenarios, with the advancement of intelligent mine construction, unmanned driving operations are gradually being implemented. When the mine truck safety officer gets off the truck, the unmanned driving system may encounter problems in actual application and cannot continue unmanned driving, and cannot accurately stop at the corresponding position. At this time, a measure is needed to achieve human intervention, and remote control driving products based on multi-perspectives are an important measure to solve the need for human intervention to take over the vehicle during the application of the unmanned driving system.

[0041] The entire remote control driving system consists of three parts: remote control cockpit, vehicle and cloud control platform. It is mainly divided into two technical functions: remote control takeover and remote monitoring. The remote control takeover is divided into three situations: the cockpit actively applies for remote control takeover, the vehicle fault triggers remote control takeover and the cloud actively initiates remote control takeover. The unmanned driving system includes: vehicle-side remoteHandler, cockpit HMI, remote control driving system module, vehicle-side CloudHandler and cloud platform; the cloud platform includes communication gateway, large screen, equipment management module and alarm management module.

[0042] When the cockpit HMI generates vehicle selection data E402 and starts remote control command E401, the cockpit HMI sends a remote control application P1 to the cloud platform through the remote control driving system module. After the cloud platform generates a control command C1, it sends a remote control command F401 to establish a connection between the cockpit HMI and the vehicle-side remote handler, and remotely controls the vehicle by receiving the vehicle-side video data V1 and sending the remote control command E1.

[0043] When the vehicle-side CloudHandler detects a vehicle fault, the vehicle-side CloudHandler reports the fault data D to the alarm module; the alarm module combines the device management module and the communication gateway, and decides whether to send a control instruction C1 to the remote control driving system module based on the fault data D and the driver's confirmation instruction E8, so as to realize remote control takeover when the vehicle fails;

[0044] When the large screen generates vehicle and cockpit selection data and issues remote control takeover instructions, the device management module decides whether to issue control instructions C1 to the remote control driving system module through the communication gateway according to the vehicle and cockpit status and the driver's confirmation instruction E8, thereby realizing the vehicle remote control takeover initiated by the cloud platform;

[0045] When the cockpit HMI generates vehicle selection data E402 and starts monitoring instructions, the remote control driving system module sends monitoring instructions E5 through the communication gateway after confirming that the vehicle status meets the preset conditions, and establishes a connection between the vehicle-side remote Handler and the cockpit HMI, thereby realizing remote monitoring of the vehicle by the cockpit HMI.

[0046] like Figure 1 As shown, remote control driving: the cockpit actively applies for remote control takeover, and the human-machine interface (HMI) of the cockpit generates vehicle selection data and a remote control start command. The vehicle selection data is used to identify the target vehicle to be remotely controlled, and the remote control start command triggers the start of the remote control driving process. After receiving the remote control command from the HMI, the remote control driving system module first performs a series of self-checks to ensure that the system meets the necessary conditions for remote control driving.

[0047] The self-test content includes: collecting environmental parameters in the cockpit through sensors, such as temperature, humidity, light intensity, etc., to determine whether they are within the preset appropriate range. Abnormal environmental parameters may affect equipment performance and driver status, and early warning is required. Read the working parameters of key hardware of the remote control driving system, such as power supply voltage, CPU usage, memory usage, etc., to evaluate whether the hardware is running at full capacity and whether there are any abnormalities or hidden faults. Detect the quality of the communication link with the on-board remote control module (remoteHandler), obtain the current communication delay and packet loss rate, and determine whether it meets the requirements for real-time and reliable transmission of control instructions and status information.

[0048] If the self-check finds that any indicator does not meet the preset conditions, the remote control driving system module sends a prompt message to the HMI, indicating the problems with the current remote control environment. If all checks pass, the system module sends a remote control application to the cloud platform, requesting control of the target vehicle. After receiving the remote control application, the cloud platform decides whether to approve the remote control takeover based on factors such as the current vehicle status, traffic environment, and network conditions. If the takeover is agreed, the cloud platform will send the start streaming command and the start remote control command to the on-board remote Handler to inform it to prepare to receive control commands; at the same time, the start remote control command is also sent to the cockpit HMI to put it into remote control mode. If the takeover is refused, the cloud platform will send an alarm message to the remote control driving system module, indicating the reason for the refusal.

[0049] After receiving the start streaming command and start remote control command from the cloud platform, the onboard remote handler immediately establishes a data connection with the cockpit HMI. On the one hand, the remote handler pushes the collected vehicle environment perception information (such as video images, lidar point clouds, etc.) to the HMI in real time to provide environmental cognition; on the other hand, the remote handler is also ready to receive remote control commands and convert the driver's control actions into vehicle execution actions.

[0050] After the cockpit HMI enters the remote control mode and is normally connected to the remote Handler, the driver can control the vehicle movement by operating the equipment (such as the steering wheel, accelerator, brake, etc.). The HMI collects the driver's control instructions and sends them to the vehicle module in real time through the communication link with the remote Handler to complete the remote control driving process of the vehicle. During the remote control process, the driver understands the surrounding environment of the vehicle through the vehicle video, radar and other information displayed on the HMI, and adjusts the control instructions according to driving needs.

[0051] like Figure 2 As shown in the figure, vehicle failure triggers remote control takeover. The vehicle-mounted CloudHandler module generates corresponding fault data when it finds a vehicle failure through real-time monitoring of the vehicle's systems and components. The fault data contains key information such as the time, location, fault code, and related parameters of the fault. CloudHandler reports the generated fault data to the alarm management module in the cloud for processing.

[0052] After the alarm management module receives the fault data reported by the vehicle, it first parses the data and extracts features. Among them, the extracted features are mainly divided into two categories: Fault component type features: including the vehicle's unique identification number, the timestamp of the fault, the fault code (such as the OBD code), and the parameter values ​​related to the fault. These features are used to determine the type and severity of the fault. Fault location features: including the underground partition number, section number, and specific coordinate data. Among them, the underground partition number represents the logical partition within the mining area, and the section number is the unique identifier of the road in each partition. Combined with the coordinate data, the location of the faulty vehicle can be accurately located.

[0053] The alarm management module inputs the extracted fault features into the pre-trained decision tree model to evaluate the severity of the current fault. The decision tree model establishes a correspondence between fault features and severity by learning from historical fault data. When the calculated severity exceeds the preset threshold, the alarm management module generates a remote control takeover judgment result and sends it to the equipment management module. Based on the judgment result of the alarm management module, the equipment management module confirms that the faulty vehicle needs to be remotely taken over. To this end, the equipment management module first queries the currently idle cockpit information and selects an available cockpit to perform the remote control task. Then, the equipment management module sends a remote control command to the remote control driving system module of the selected cockpit through the communication gateway, requesting it to prepare to take over the control of the vehicle.

[0054] After the remote control driving system module in the cockpit receives the remote control command from the equipment management module, it immediately sends a manual takeover confirmation request to the human-machine interface (HMI) of the cabin. After receiving the request, the HMI will pop up a remote control takeover prompt to inform the driver that there is a faulty vehicle that needs to be remotely controlled. The driver can choose to agree or refuse to take over. If the driver confirms through the HMI that he agrees to the remote control takeover, the command will be sent to the remote control driving system module. The system module generates a remote control confirmation command and sends it to the device management module in the cloud through the communication gateway, indicating that the driver is ready to start remote control. After receiving the remote control confirmation command from the cockpit, the device management module will make another judgment, comprehensively considering factors such as road conditions and communication quality, and decide whether to approve this remote control takeover request. If the takeover is agreed, the device management module will simultaneously send control commands to the remote handler module of the faulty vehicle and the remote control driving system module in the cockpit, commanding the two to establish a communication connection and start the remote control process.

[0055] After receiving the remote control command, the remote Handler module of the faulty vehicle will actively establish a data transmission channel with the cockpit HMI. On the one hand, the remote Handler pushes the vehicle environment information collected by the on-board sensor to the HMI in real time, making it convenient for the driver to remotely perceive the surrounding conditions of the vehicle; on the other hand, the remote Handler is always ready to receive remote control commands from the HMI and control the vehicle to perform corresponding actions according to the commands. At the same time, the remote Handler will set the vehicle's status parameters to "remote control" for monitoring by other modules.

[0056] After the cockpit HMI is properly connected to the remote handler, it will reconstruct the environment around the vehicle on the display screen based on the vehicle video, radar and other information. The driver can control the movement of the faulty vehicle by operating the physical buttons and touch screen of the HMI, such as direction, accelerator, brake, etc., and drive it away from the fault point or move it to a safe location until the fault is eliminated or manual rescue arrives. During the remote control takeover process, the driver can monitor the vehicle status in real time and adjust the control strategy at any time as needed.

[0057] The three-level coordination of on-board fault detection, cloud-based intelligent judgment, and manual cockpit takeover enables rapid response and safe control of faulty vehicles. The fault severity is assessed through fault data analysis, and remote control takeover is automatically requested; factors such as road conditions and communication quality are comprehensively considered to reasonably dispatch cockpit resources; vehicle-side perception, manual control, and real-time feedback are used to ultimately control the faulty vehicle within a safe range.

[0058] like Figure 3 As shown in the figure, the cloud actively initiates remote control takeover. In certain specific situations, such as bad weather, traffic accidents, etc., when it is difficult for the vehicle to drive autonomously, the cloud can actively initiate remote control takeover of the vehicle. At this time, the device management module will generate a remote control command and a start remote control command, and send them to the on-board remote Handler module and the remote control driving system module in the cockpit through the communication gateway, respectively, to request control of the vehicle. After receiving the remote control command and the start remote control command from the cloud, the on-board remote Handler module enters the standby state and is ready to establish a communication connection with the cockpit at any time. At the same time, the remote control driving system module in the cockpit also receives the start remote control command and starts a series of self-checks to ensure that the necessary conditions for remote control driving are met. The self-check content includes: whether the environmental parameters of the cockpit are normal, whether the key hardware is running at full load, and whether the communication link with the on-board module is stable.

[0059] If the remote control driving system module detects that all conditions meet the preset requirements, it will send a remote control application to the communication gateway, requesting permission for the cockpit to take over the target vehicle. The communication gateway forwards the application to the device management module in the cloud. If any condition is not met, the remote control driving system module will cancel the remote control process and will not send the remote control application.

[0060] After receiving the remote control request from the cockpit, the device management module starts to verify and judge. Taking into account the current vehicle status, environmental conditions, communication quality, driver identity and other factors, the device management module decides whether to approve the cockpit to take over the vehicle. If the takeover is approved, the device management module generates a control instruction and sends it to the on-board remote handler and remote control driving system module through the communication gateway, notifying both to start remote control. If the takeover is refused, the device management module will generate a refusal to take over message and send it to the alarm management module to explain the reason for the refusal. The alarm management module can adjust the remote control strategy or notify manual processing accordingly.

[0061] After receiving the control command from the device management module, the on-board remote handler will send three commands to the remote control driving system module in the cockpit: start streaming command, start remote control command, and start monitoring command. These three commands notify the cockpit to prepare to receive the on-board video stream, send remote control commands, and monitor the vehicle status. At the same time, after receiving the control command, the remote control driving system module will also send a takeover confirmation request to the human-machine interface (HMI) of the cabin, asking the driver whether to confirm taking over the vehicle.

[0062] After the cockpit HMI receives the takeover confirmation request from the remote control driving system module, a prompt message will pop up, asking the driver to confirm. The driver can decide whether to accept the remote control task based on his own conditions and judgment of the vehicle environment. If the driver confirms the takeover through the HMI, the remote control driving system module will generate a remote control confirmation instruction and send it to the device management module in the cloud through the communication gateway.

[0063] After receiving the remote control confirmation command, the device management module will conduct a second confirmation. This step is mainly to prevent the driver from misoperating or causing the command to be out of sync due to communication delays. If the device management module agrees to take over the remote control again, it will generate a remote control permission command, which will be sent by the communication gateway to the vehicle remote handler and the remote control driving system module to formally approve this remote control behavior. After receiving the remote control permission command, the vehicle remote handler immediately establishes a dedicated data transmission channel with the cockpit HMI and sets the vehicle's status parameter to "remote control", indicating that the vehicle enters the controlled state. At the same time, the remote handler turns on the vehicle camera, starts to collect video images of the road in front of the vehicle, and transmits the video stream data to the cockpit HMI in real time for the driver to observe and judge. Subsequently, the driver can issue control commands through the HMI based on the video screen, which will be executed by the remote handler to complete the remote control driving task of the vehicle. The cloud actively requests remote control based on the road environment, and the system can perform condition checks based on the actual conditions of the cockpit and the vehicle. The driver and the system confirm the remote control separately. Multiple judgments and verifications effectively improve the safety and reliability of remote control. At the same time, by reasonably dividing the responsibilities of the cloud, terminal, and people, and using technical means such as real-time video transmission and on-board execution, the timeliness of control instructions and the consistency of vehicle responses are guaranteed, thereby achieving smooth and precise remote control.

[0064] like Figure 4 As shown, for remote monitoring, the driver selects the target vehicle to be monitored through the human-machine interface (HMI) in the cockpit and sends a vehicle selection instruction to the remote control driving system module, officially starting the remote monitoring process.

[0065] After receiving the vehicle selection command, the remote control system module first performs a series of condition checks to ensure that the remote control system meets the requirements of remote monitoring. These conditions may include: the stability of the communication link, the resolution of the display device, the data processing capability, etc. If all conditions meet the preset standards, the remote control system module will send a remote control application to the communication gateway to request monitoring rights for the target vehicle. If any condition is not met, the remote control system module will regard the cockpit as not having remote control capabilities, refuse to send the application, and prompt the driver with the specific reason through the HMI.

[0066] After receiving the remote control application from the remote control driving system module, the communication gateway forwards it to the device management module in the cloud. The device management module comprehensively evaluates the current state of the target vehicle and the performance of the remote control driving system to determine whether to agree to this remote monitoring request. Among them, the evaluation of the vehicle status mainly focuses on the environmental conditions and driving status of the vehicle to ensure that the remote control does not bring additional risks to the vehicle's driving. The evaluation of the remote control driving system mainly focuses on the completeness, stability, reliability and other indicators of the system's hardware and software. Only when both evaluations are passed, the device management module will generate control instructions, which will be sent to the on-board remote Handler module and the remote control driving system module in the cockpit through the communication gateway, approving the two to start establishing a monitoring connection.

[0067] If any of the assessments fail, the device management module will generate a takeover refusal message and send it to the alarm management module, detailing the reasons for the refusal to monitor. At the same time, the device management module displays a prompt message on the cockpit HMI through the communication gateway, informing the driver that remote monitoring is not possible. After receiving the control command issued by the device management module, the on-board remote handler module will send three commands to the remote control driving system module in the cockpit: start streaming command, start remote control command, and start monitoring command. These three commands prompt the cockpit to be ready to receive on-board video, send control commands (if necessary), and monitor the vehicle status. At the same time, after receiving the control command, the remote control driving system module will also send a manual takeover confirmation request to the driver through the HMI. The purpose of this request is to reconfirm whether the driver is ready to implement remote monitoring.

[0068] After the driver receives the takeover confirmation request from the remote control driving system through the cockpit HMI, manual confirmation is required. This confirmation behavior indicates that the driver voluntarily undertakes this remote monitoring task and is ready. After the confirmation instruction is generated, it is sent to the remote control driving system module by the HMI. After the remote control driving system module receives the manual takeover confirmation instruction, it will generate a remote control confirmation instruction and send it to the device management module in the cloud through the communication gateway. This instruction indicates that the cockpit has completed the remote control preparation and can formally establish a monitoring connection with the vehicle. After receiving the remote control confirmation instruction, the device management module will confirm the instruction again. After supervising that the cockpit and the vehicle have successfully established a remote control session, the device management module will generate a remote control permission instruction, which will be sent by the communication gateway to the on-board remote Handler module and the cockpit remote control driving system respectively, approving the two to start video transmission and monitoring operations.

[0069] After receiving the remote control permission command, the on-board remote Handler module immediately establishes a stable data transmission channel with the cockpit HMI. At the same time, the remote Handler sets the vehicle's operating status to "remote control monitoring mode", indicating that the vehicle is in a normal driving state under external monitoring. In order to allow the driver to view the road conditions ahead of the vehicle in real time, the remoteHandler module continuously collects high-definition video images through the on-board camera, and transmits the video stream data to the cockpit in real time through the data channel with the HMI. At the same time, the remote control driving system module will also set its own status to "remote control", prompting the driver that the monitoring process has begun.

[0070] The cockpit HMI continuously receives real-time video streams from the on-board remote handler module through a dedicated data communication connection. Using video decoding and image processing technology, the HMI can restore the video stream to a clear and stable driving picture, and display it to the driver through a large-size display screen, so that the driver can obtain an immersive observation effect. With this perspective, the driver can monitor the vehicle and its surrounding environment in real time and keep abreast of the vehicle's dynamics at any time.

[0071] The present application also includes: when the vehicle terminal determines through sensor data analysis that the vehicle needs to stop, the vehicle camera is controlled to start collecting video data of the vehicle's surrounding environment. The camera records a video stream V1 with a resolution of 1920x1080 at a frame rate of 30 frames per second. The vehicle terminal transmits the video stream V1 to the cloud platform in real time for processing through wireless networks such as 4G / 5G. Considering the limited bandwidth of the vehicle network, the video can be first encoded and compressed in H.264 and then uploaded to the cloud. After the cloud platform receives the video stream data V1 sent by the vehicle terminal, it decodes it and restores it to the original image sequence. The cloud platform selects key frames from the video stream for parking space detection, and the frequency of key frame selection is 1 frame per second. Parking space detection uses the YOLO (You Only Look Once) algorithm based on convolutional neural network. The algorithm first divides the input image into a 7x7 grid, predicts 2 bounding boxes for each grid, and each bounding box contains 5 prediction values: target confidence, bounding box center point coordinates (x, y) and bounding box width and height (w, h). Then the prediction box is threshold filtered and non-maximum suppressed to obtain the final detection result.

[0072] The pixel coordinates of the center point of a parking space bounding box output by the YOLO network are: , with a width of w' pixels and a height of h' pixels. In order to realize the automatic parking of the vehicle, it is also necessary to convert it into distance information in the vehicle coordinate system. First, according to the imaging principle, the mapping relationship between the pixel coordinate system and the camera coordinate system can be established: ; ;in, is the principal point coordinate of the camera, is the focal length of the camera, and Z is the depth distance from the camera to the center of the parking space. Z can be measured by binocular vision or laser radar. Then, the coordinates in the camera coordinate system are transformed to the vehicle coordinate system through the rigid body transformation matrix T: ; The T matrix can be obtained in advance through internal and external parameter calibration. The external parameter is the installation posture of the camera relative to the vehicle. Finally, combined with the longitude and latitude coordinates (lat, lon) of the vehicle's current position measured by the vehicle's GPS, according to the conversion formula from geodetic coordinates to local coordinates, the distance (dx, dy) from the center of the parking space to the vehicle and the yaw angle theta can be obtained: ; ; ; Where R is the radius of the earth, yaw is the heading angle of the vehicle, It is the GPS coordinate of the parking space, which can be obtained by querying the high-precision map.

[0073] The cloud platform obtains the world coordinates of the parking space by analyzing the vehicle video stream. and the relative position of the vehicle After that, the information is packaged into JSON format and sent to the vehicle terminal in real time through lightweight publish / subscribe message protocols such as MQTT. The vehicle terminal subscribes to the message push from the cloud, and after receiving the parking location information, it extracts the world coordinates and relative position data as the input for generating the parking trajectory.

[0074] At the same time, the vehicle terminal estimates the vehicle's own status in real time through multi-sensor fusion. First, the vehicle's current position is obtained through the vehicle's RTK-GPS. and heading angle , calculate the vehicle speed v and acceleration a based on the wheel speed meter and IMU data. Then, use the Kalman filter algorithm to fuse multiple sources of information such as GPS position, IMU attitude, wheel speed, etc. to obtain the state estimation values ​​including vehicle position, speed, acceleration, and heading angle. Kalman filtering is performed recursively through two steps: prediction and update, to dynamically estimate the system state.

[0075] Prediction phase: state vector ; Observation vector ; Status prediction: ; Covariance prediction: ; Update phase: Kalman gain: ; Status update: ; Covariance update: ; A is the state transfer matrix, H is the observation matrix, Q and R are the covariances of process noise and observation noise respectively.

[0076] After obtaining the vehicle state estimate and parking position, the vehicle terminal uses the MPC algorithm to plan the optimal trajectory from the current position to the center of the parking space. MPC solves an optimal control problem in a finite time domain in a rolling manner. At each sampling moment, it uses the current state as the initial condition, predicts the future system state, solves a series of optimal control sequences, and applies the first element of the sequence to the system, and then repeats the process at the next sampling moment. This rolling optimization strategy ensures closed-loop control performance.

[0077] Define the state variables as vehicle position and heading angle , the control variables are vehicle speed and front wheel steering angle , the objective function is: , The vehicle heading indicating the target position at the center of the parking space; The multiplication factor representing the rate of change of the control variable; The kinematic model of the vehicle: ; Constraints on state and control variables: ; .

[0078] This problem can be transformed into a quadratic programming (QP) solution: ; Constraints: ; ; ; Where H and f are the coefficient matrices of the quadratic and linear terms of the objective function, A and b are the coefficient matrices and boundaries of the inequality constraints, , is the coefficient matrix and bounds of the equality constraints, and The vehicle terminal uses a QP solver, such as OSQP, CVXGEN, etc., to find the optimal control sequence for the next N steps. Take the first element of the sequence As the control instruction at the current moment, it is executed by the chassis controller to control the vehicle to drive along the planned parking trajectory.

[0079] In summary, after the vehicle terminal receives the parking position information sent from the cloud, it estimates the vehicle's own state through Kalman filtering, and uses the vehicle state and parking position as constraints to solve the optimal control problem using the MPC algorithm, generating the shortest distance trajectory from the current position to the center of the parking space, and then discretizing the trajectory into a series of control instructions for execution, and finally completing the automatic parking of the vehicle. Compared with simply tracking the pre-planned reference trajectory, MPC can dynamically adjust the control strategy according to the real-time status of the vehicle and the environment, has stronger robustness and adaptability, and is more suitable for trajectory planning and tracking control in complex parking scenarios.

[0080] The vehicle terminal obtains a series of trajectory points and corresponding reference heading angles from the current position to the center of the parking space based on the optimal control sequence solved by the MPC algorithm. In order to achieve accurate tracking and control of the planned trajectory by the vehicle, the trajectory information needs to be converted into control commands such as steering, throttle, and brake that can be executed by the vehicle chassis.

[0081] First, the Pure Pursuit algorithm is used to calculate the front wheel steering angle required for the vehicle to follow the trajectory based on the trajectory points and the reference heading angle. Pure Pursuit is essentially a geometric method that calculates the steering curvature from the current position of the vehicle to the forward point on the trajectory to obtain the steering wheel control amount. The algorithm flow is as follows: The trajectory point set is , the vehicle's current position is , the look-ahead distance is Traverse the track points and calculate the distance from the vehicle's current position to each track point : Find satisfaction The nearest trajectory point , as the target tracking point. The coordinates of the target point in the vehicle coordinate system Convert to distance and yaw angle : ; ; ; ; Calculate the front wheel angle based on the geometric relationship : ; Where L is the front and rear wheelbase of the vehicle. The selection of affects the tracking performance and is generally taken as 1 to 3 m. The larger the value, the smoother the trajectory tracking, but the less sensitive it is.

[0082] Next, we need to control the speed of the vehicle so that it can decelerate smoothly to 0 when it reaches the parking space. Here, we use the PID control algorithm to calculate the control amount of the accelerator and brake according to the difference between the current speed of the vehicle and the target speed. Let the speed of the vehicle at the kth moment be , the target speed is , define the velocity error as: ; then the throttle / brake control amount for: ;in, They are proportional, integral, and differential coefficients, respectively, and can be determined by empirical tuning methods such as Ziegler-Nichols. The on-board terminal converts the steering control amount delta and the throttle / brake control amount u into specific actuator instructions, such as steering wheel angle, electronic throttle opening, brake pressure, etc., and sends them to the ECUs of the steering system, power system, and brake system through the CAN bus. The ECU parses the instructions and controls the actuators to complete the corresponding actions, so that the vehicle tracks the planned parking trajectory. As the vehicle approaches the parking space, the Pure Pursuit algorithm dynamically adjusts the front wheel angle, and the PID adjusts the vehicle speed, and finally the vehicle is parked smoothly in the center of the parking space. The on-board terminal continuously monitors the vehicle speed. When it detects that the vehicle is completely stationary, it considers that the parking is completed, sends a parking command, cuts off the power output, and automatically pulls up the electronic handbrake to prevent the vehicle from sliding down the slope. At this point, the entire automatic parking process is over.

Claims

1. A remote control driving method based on multiple perspectives, used in an unmanned driving system, characterized in that: include: The unmanned driving system includes: vehicle-side remote handler, cockpit HMI, remote control driving system module, vehicle-side CloudHandler and cloud platform; the cloud platform includes communication gateway, large screen, equipment management module and alarm management module; When the cockpit HMI generates vehicle selection data E402 and remote control command E401, the cockpit HMI sends a remote control application P1 to the cloud platform through the remote control driving system module. After the cloud platform generates a control command C1, it sends a remote control command F401 to establish a connection between the cockpit HMI and the vehicle-side remote handler, and remotely controls the vehicle by receiving the vehicle-side video data V1 and sending the remote control command E1. When the vehicle-side CloudHandler detects a vehicle fault, the vehicle-side CloudHandler reports the fault data D to the alarm module; the alarm module combines the device management module and the communication gateway, and decides whether to send a control instruction C1 to the remote control driving system module based on the fault data D and the driver's confirmation instruction E8, so as to realize remote control takeover when the vehicle fails; When the large screen generates vehicle and cockpit selection data and issues remote control takeover instructions, the device management module decides whether to issue control instructions C1 to the remote control driving system module through the communication gateway according to the vehicle and cockpit status and the driver's confirmation instruction E8, thereby realizing the vehicle remote control takeover initiated by the cloud platform; When the cockpit HMI generates vehicle selection data E402 and starts monitoring instructions, the remote control driving system module sends monitoring instructions E5 through the communication gateway after confirming that the vehicle status meets the preset conditions, and establishes a connection between the vehicle remote Handler and the cockpit HMI, thereby realizing remote monitoring of the vehicle by the cockpit HMI; Remote control takeover in case of vehicle failure, including: When the vehicle-side CloudHandler detects a vehicle fault, it generates fault data D and reports the fault data D to the alarm management module; After receiving the fault data D, the alarm management module makes a judgment through the fault analysis algorithm and sends the judgment result to the equipment management module; After receiving the judgment result, the equipment management module obtains the idle cockpit data and sends the remote control command F401 to the remote control driving system module through the communication gateway; After receiving the remote control command F401, the remote control system module sends a manual takeover confirmation request E6 to the cockpit HMI; After receiving the manual takeover confirmation request E6, the cockpit HMI displays a remote control takeover prompt message for the pilot to confirm whether he agrees to the remote control takeover; The driver generates a manual takeover confirmation instruction E8 through the cockpit HMI. After receiving the manual takeover confirmation instruction E8, the remote control driving system module generates a remote control confirmation instruction E7 and sends the remote control confirmation instruction E7 to the communication gateway; After receiving the remote control confirmation instruction E7, the device management module determines whether to agree to the remote control takeover through the decision algorithm. If it agrees, the device management module sends the control instruction C1 to the vehicle-side remote handler and the remote control driving system module through the communication gateway; After receiving the control command C1, the vehicle-side remote handler establishes a connection with the cockpit HMI for interaction, and sets the status of the cockpit HMI to remote control; After receiving the fault data D, the alarm management module makes a judgment through the fault analysis algorithm and sends the judgment result to the equipment management module, including: After receiving the fault data D reported by the vehicle-side CloudHandler, the alarm management module extracts the fault component type feature D1 and the fault location D2 from the fault data D; The extracted fault component type feature D1 and fault location D2 are used as input, and the severity of the current fault data is calculated using the pre-trained decision tree model; When the calculated severity is greater than the set threshold, a remote control takeover judgment result is generated; The fault component type feature D1 includes: vehicle identification number, fault occurrence time, fault code and fault parameters; The fault location D2 includes: an underground partition number, a section number and coordinate data; wherein the underground partition number indicates the area number of the underground mining area, and the section number indicates the unique number of the section in the corresponding area.

2. The multi-view remote control driving method according to claim 1, characterized in that: The vehicle is remotely controlled by receiving the vehicle-side video data V1 and sending the remote control command E1, including: When the cockpit HMI generates vehicle selection data E402 and starts remote control command E401, it detects whether the remote control driving system module meets the preset conditions. If so, the remote control driving system module sends a remote control application P1 to the cloud platform; otherwise, the remote control driving system module sends a message prompt to the cockpit HMI; After receiving the remote control application P1, the cloud platform determines whether to take over the remote control. If yes, the cloud platform sends the start streaming command E3 and the start remote control command E401 to the vehicle-side remote handler, and sends the start remote control command E401 to the cockpit HMI. Otherwise, the cloud platform sends the warning information M1 to the remote control driving system module. After receiving the start streaming command E3 and the start remote control command E401, the vehicle-side remote handler establishes a connection with the cockpit HMI and starts streaming, while receiving the remote control command E1 from the cockpit HMI.

3. The multi-view remote control driving method according to claim 2, characterized in that: Check whether the remote control driving system module meets the preset conditions, including: Collect cockpit environmental data through the remote control driving system module, the environmental data includes ambient temperature, humidity and light intensity, and determine whether the collected environmental data is within the preset range; Obtain the hardware parameters of the remote control driving system module, including the power supply voltage, CPU usage and memory usage, and determine whether the hardware parameters meet the preset remote control driving conditions; Obtain the communication link status with the vehicle-side remote handler and determine whether the current communication delay and packet loss rate meet the preset remote control driving conditions.

4. The multi-view remote control driving method according to any one of claims 2 or 3, characterized in that: The cloud platform actively initiates the vehicle remote control takeover, including: The vehicle-side remote handler receives the remote control command E1 and the start remote control command E401 from the cockpit HMI or cloud platform; The remote control system module receives the start remote control command E401 sent by the cockpit HMI, and detects whether the remote control system module meets the preset conditions. If so, it sends a remote control application P1 to the communication gateway; The communication gateway receives and sends the remote control application P1 to the device management module; The device management module verifies and judges the remote control application P1. If the remote control takeover is approved, a control instruction C1 is generated and sent to the vehicle-side remote handler and the remote control driving system module through the communication gateway. Otherwise, a takeover refusal message M1 is generated and sent to the alarm management module. After receiving the control command C1, the vehicle-side remote handler sends the start streaming command E3, the start remote control command E401 and the start monitoring command E5 to the remote control driving system module; after receiving the control command C1, the remote control driving system module sends a takeover confirmation request E6 to the cockpit HMI; The cockpit HMI receives the takeover confirmation request E6 and prompts the driver to confirm; the driver generates a manual takeover confirmation instruction E8 through the HMI, and after the remote control driving system module receives the manual takeover confirmation instruction E8, it generates a remote control confirmation instruction E7 and sends it to the communication gateway; The communication gateway sends the remote control confirmation command E7 to the device management module for reconfirmation; if the device management module agrees to the remote control takeover, the device management module generates a remote control permission command C2, and the communication gateway sends the remote control permission command C2 to the vehicle-side remote handler and the remote control driving system module respectively; After receiving the remote control permission command C2, the vehicle-side remote handler establishes a connection with the cockpit HMI, sets the vehicle-side status to remote control, and collects video stream data V1 through the vehicle-mounted camera and sends it to the cockpit HMI.

5. The multi-view remote control driving method according to claim 4, characterized in that: Also includes: When the vehicle needs to stop, the video stream data V1 is sent to the cloud platform; The cloud platform uses machine algorithms to process the video stream data V1, obtain the parking position, and control the parking of the vehicle according to the parking position.

6. The multi-view remote control driving method according to claim 5, characterized in that: Cockpit HMI remote monitoring of the vehicle, including: The cockpit HMI sends the vehicle selection command E402 to the remote control system module to start the remote control process; After receiving the vehicle selection instruction E402, the remote control driving system module detects whether the remote control driving system module meets the preset conditions. If so, it sends a remote control application P1 to the communication gateway. Otherwise, it prompts the driver through the cockpit HMI that the remote control driving system module does not meet the preset conditions. The communication gateway receives the remote control application P1 and forwards it to the device management module for evaluation to determine whether the vehicle status and the remote control driving system module meet the requirements for remote control takeover. If the verification is successful, the device management module generates a control instruction F401 and sends it to the vehicle-side remote handler and the remote control driving system module through the communication gateway. If the verification is unsuccessful, the device management module generates a takeover rejection message M1 and sends it to the alarm management module and prompts the driver through the communication gateway. The vehicle-side remote handler receives the control command F401 and sends the start streaming command E3, the start remote control command E401 and the start monitoring command E5 to the remote control driving system module; after receiving the control command F401, the remote control driving system module sends a manual takeover confirmation request E6 to the cockpit HMI; The cockpit HMI receives the manual takeover confirmation request E6 and prompts the driver to confirm; the driver generates a manual takeover confirmation instruction E8, and the cockpit HMI sends the manual takeover confirmation instruction E8 to the remote control driving system module; After receiving the manual takeover confirmation instruction E8, the remote control driving system module generates a remote control confirmation instruction E7 and sends it to the communication gateway; the communication gateway forwards the remote control confirmation instruction E7 to the device management module for secondary confirmation; The device management module confirms the remote control confirmation command E7, supervises the establishment of the remote control session, and generates a remote control permission command C2 which is sent to the vehicle-side remote handler and the remote control driving system module through the communication gateway; After receiving the remote control permission command C2, the vehicle-side remote handler establishes a data communication connection with the cockpit HMI, sets the vehicle-side status to remote control, collects the vehicle's real-time video data V1 through the vehicle-mounted camera and sends it to the cockpit HMI; after receiving the remote control permission command C2, the remote control driving system module sets the cockpit status to remote control; The cockpit HMI establishes a data communication connection with the remote handler on the vehicle side, receives the real-time video data V1 transmitted from the vehicle side, displays the real-time image of the vehicle through the streaming software, and realizes remote monitoring of the vehicle.

Images