Vehicle remote control driving system and method integrating AR virtual path projection

Abstract

The invention provides a vehicle remote control driving system and method integrating AR virtual path projection, and relates to the technical field of vehicle remote control. The system constructs a high-precision three-dimensional environment model through cooperative work of an environment sensing module and a data processing and decision module; the AR path generation unit generates a collision-free virtual path and a vehicle dynamic rehearsal picture, and the collision-free virtual path and the vehicle dynamic rehearsal picture are visually presented in an augmented reality mode through the remote control terminal outside the vehicle, so that a driver can obtain comprehensive situation awareness beyond a physical visual field outside the vehicle; and after the virtual path is confirmed or adjusted in combination with the interaction control, the vehicle control strategy unit automatically generates a cooperative control command adapting to the road condition, and an execution mechanism is driven to perform accurate execution. According to the system, the safety and operation confidence of remote control driving are improved, the blind area collision risk is effectively avoided, the trafficability of the vehicle in a complex environment is enhanced, meanwhile, the operation difficulty is greatly reduced through AR visual interaction, and intelligent and user-friendly vehicle exterior remote control driving experience is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle remote control technology, and in particular to a vehicle remote control driving system and method integrated with AR virtual path projection. BACKGROUND

[0002] With the development of automobile intelligence, vehicle remote control driving technology has been gradually applied to narrow parking space access and other scenarios. The existing technology mainly falls into two categories: one is to realize vehicle straight forward or backward movement in a very short distance through an intelligent key; the other is to remotely control through a cellular network connected mobile phone APP. However, these technologies have significant defects when dealing with complex or narrow environments: first, environmental perception relies on the naked eye observation of the driver, the field of view is limited, there are a lot of blind spots, it is difficult to accurately judge the distance between the vehicle and the obstacle, and it is easy to cause collision or scratching; second, the vehicle lacks adaptive cooperative control capability, the remote control command is only a simple action execution, and it cannot be automatically adjusted according to the road conditions, so the driver still needs to return to the vehicle to operate, which violates the original intention of convenient remote control; third, the human-computer interaction mode is simple, lacks intuitive path preview and risk warning mechanism, and the operation process has high uncertainty. SUMMARY

[0003] The purpose of the present application is to provide a vehicle remote control driving system and method integrated with AR virtual path projection, to solve one or more technical problems existing in the prior art, and at least provide a beneficial choice or create conditions.

[0004] The solution to the technical problem of the present application is: the present application provides a vehicle remote control driving system integrated with AR virtual path projection, comprising an environment perception module, a data processing and decision module, an out-of-vehicle remote control terminal and an execution mechanism; The environment perception module is configured to collect environmental data and vehicle state data around the vehicle; The data processing and decision module comprises the following units: An environment modeling unit configured to fuse the environmental data and vehicle state data to generate a three-dimensional environment model including obstacles, terrain features and passable areas; An AR path generation unit configured to generate a collision-free virtual path and a vehicle virtual model according to the three-dimensional environment model, simulate the dynamic travel process of the vehicle virtual model along the virtual path, and generate a dynamic preview picture; A vehicle control strategy unit configured to automatically generate a cooperative control command according to the driver's input confirmation instruction, the three-dimensional environment model and the virtual path; The out-of-vehicle remote control terminal is configured to display real-time images of the three-dimensional environment model, superimpose the dynamic preview picture in an AR manner, and provide interactive controls to receive the driver's input confirmation instruction. The execution mechanism is configured to receive and execute the cooperative control command to control the vehicle to automatically travel along the virtual path.

[0005] Further, the terrain features in the three-dimensional environment model include slope, slope direction and ground adhesion coefficient; the environment perception module includes a visual sensor, a laser radar, an ultrasonic radar and a vehicle state sensor; the vehicle state sensor includes an inclination sensor, a wheel speed sensor and a height sensor; The visual sensor includes at least four wide-angle cameras for stitching to generate a panoramic bird's eye view around the vehicle; the laser radar is configured to construct a three-dimensional point cloud map around the vehicle to identify obstacle profiles, slopes and pothole depths.

[0006] Further, the environment modeling unit is further configured to: calibrate the visual data, laser radar point cloud data, ultrasonic ranging data and vehicle state data in space-time synchronization through a multi-sensor fusion algorithm to generate a three-dimensional environment model with centimeter-level precision.

[0007] Further, the virtual path generated by the AR path generation unit satisfies the following conditions: the curvature radius of the virtual path matches the minimum turning radius of the vehicle, and the distance between the virtual model of the vehicle and the obstacles in the environment model is not less than a preset safety threshold.

[0008] Further, the cooperative control command automatically generated by the vehicle control strategy unit in the off-road scene includes: locking the differential lock, raising the suspension and adjusting the power distribution ratio; the cooperative control command automatically generated in the urban narrow scene includes: limiting the vehicle speed to within 3km / h, activating the panoramic image and optimizing the steering smoothness.

[0009] Further, the data processing and decision module has an AI algorithm model built in, and the core function is processed locally at the vehicle end.

[0010] Further, in the remote control terminal outside the vehicle, the interaction control includes a confirmation execution control and a re-planning control; The confirmation execution control is used to receive the confirmation instruction of the virtual path input by the driver; The re-planning control is used to receive the adjustment instruction of the virtual path input by the driver; After the re-planning control is triggered, the AR path generation unit re-generates the virtual path and the dynamic preview picture according to the intention control instruction adjusted by the driver.

[0011] Further, the display interface of the off-vehicle remote control terminal is further configured to display the distance between the vehicle and the obstacle and a risk warning mark in real time, and when the distance is lower than a preset safety threshold, an alarm is sent through sound, vibration or screen flickering.

[0012] Further, the execution mechanism includes a drive motor controller, a steering controller, a brake controller, a suspension controller and a differential lock controller, and the cooperative control command is transmitted to each controller through a CAN bus.

[0013] In another aspect, the application provides a vehicle remote driving method integrated with AR virtual path projection, applied to the vehicle remote driving system integrated with AR virtual path projection, including the following steps: Collecting environmental data around the vehicle and vehicle state data; Based on the environmental data and vehicle state data, a three-dimensional environmental model containing obstacles, terrain features and passable areas is generated; According to the three-dimensional environmental model, a collision-free virtual path and a vehicle virtual model are generated, and a dynamic preview picture of the virtual path and the vehicle virtual model is displayed in an AR manner through an off-vehicle remote control terminal; the dynamic preview picture is used to simulate the driving process of the vehicle virtual model along the virtual path; The off-vehicle remote control terminal receives a confirmation instruction or a re-planning instruction of the driver on the virtual path, if a re-planning instruction is received, the virtual path and the dynamic preview picture are re-generated, and if a confirmation instruction is received, a cooperative control command is generated according to the three-dimensional environmental model; The execution mechanism executes the cooperative control command to control the vehicle to automatically drive along the virtual path.

[0014] The application provides a vehicle remote driving system integrated with AR virtual path projection, through the cooperative work of the environmental perception module and the data processing and decision module, a high-precision three-dimensional environmental model is constructed, and a collision-free virtual path and a vehicle dynamic preview picture are generated by an AR path generation unit, which are intuitively presented in an augmented reality manner through an off-vehicle remote control terminal, so that the driver can obtain a comprehensive situation awareness beyond the physical field of view outside the vehicle; after confirming or adjusting the virtual path through the interactive control, the vehicle control strategy unit automatically generates a cooperative control command adapted to the road conditions to drive the execution mechanism to execute accurately. The system improves the safety and operation confidence of remote driving, effectively avoids the risk of blind area collision, enhances the passing ability of the vehicle in complex environment, and greatly reduces the operation difficulty through AR visual interaction, realizing a high-precision, intelligent and user-friendly off-vehicle remote driving experience. The application also provides a corresponding method, and the beneficial effects of the method are the same as those of the above method, which will not be repeated here.

[0015] Other features and advantages of the present application will be set forth in the description that follows, and in part will be apparent from the description, or can be learned by practice of the application. The purposes and other advantages of the present application will be realized and attained by the structure particularly pointed out in the description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0016] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and are intended to serve as an exemplification of the application, together with the description, and are not intended to limit the application in its broader aspects.

[0017] Figure 1 is a structural diagram of a vehicle remote driving system integrated with AR virtual path projection provided by the present application; Figure 2 is a flowchart of a vehicle remote driving method integrated with AR virtual path projection provided by the present application. DETAILED DESCRIPTION

[0018] In order to make the purposes, technical solutions and advantages of the present application more clear, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0019] The present application will be further described below in combination with the drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0020] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0022] With the rapid development of intelligent and networked technologies for automobiles, remote control driving, as a function to improve user convenience and vehicle control flexibility, has gradually moved from a concept to practical application. Currently, remote control driving technology mainly serves the needs of low-speed movement of vehicles in specific scenarios, such as automatic parking in or out of narrow parking spaces, precise vehicle movement during loading and unloading, or assisting drivers in completing high-difficulty driving operations in complex terrains. However, existing remote control driving technology is still in its early stages, with basic function implementation relying mainly on simple wireless signal transmission and passive response of vehicle actuators, lacking active environmental perception, intelligent decision-making, and human-machine collaborative control capabilities, making it difficult to meet safety and reliability requirements in complex, high-risk, or high-precision scenarios.

[0023] Currently, mainstream remote control driving technologies can be divided into two categories: the first category is short-distance remote control technology based on intelligent keys. This technology usually uses low-frequency wireless signals (such as 315 MHz / 433 MHz) to achieve straight forward or backward control of vehicles within a very short distance (usually less than 5 meters), commonly found in the "off-car parking" or "out-of-garage assistance" functions of some high-end vehicles. Its advantages include fast response speed, low delay, and no need to rely on external networks, but the functionality is extremely limited, only supporting simple movement in a preset direction, unable to achieve steering, obstacle avoidance, or path planning, and the driver must manually operate the remote control near the vehicle, with severely limited visibility, making it extremely easy to cause scratches or collisions due to blind spots.

[0024] The second category is remote control driving technology based on mobile terminals (such as smartphone APPs) and implemented through cellular networks (4G / 5G). This type of technology allows users to send control instructions from a distance by connecting the vehicle to a cloud platform through a mobile phone, and the vehicle executes the corresponding actions after receiving the instructions. The advantage of this solution is that it breaks through the distance limitation and can theoretically achieve cross-region remote control. However, its application is subject to many limitations: first, it highly depends on stable, low-latency high-speed network connections, and in environments with weak or no network signal coverage, such as underground parking lots, remote mountain areas, and tunnels, communication latency is high or even interrupted, leading to remote control failure or serious safety hazards; second, most systems do not integrate real-time environmental perception and autonomous decision-making capabilities, with the vehicle only serving as an "execution terminal" and unable to dynamically adjust driving strategies based on the surrounding environment, and the driver cannot obtain real-time panoramic information of the vehicle's surroundings, making the operation like "blind driving" with high risk; third, the human-machine interaction interface is simple, usually only providing direction buttons or virtual joysticks, lacking visualization of key information such as vehicle posture, path prediction, and obstacle distance, making it difficult for users to make accurate judgments.

[0025] The above prior art exposes a series of fundamental defects in practical application, which seriously restricts the popularization and deepening of remote driving technology: environmental perception completely depends on the naked eye observation of the driver, the vehicle itself sensor is not effectively utilized, resulting in limited field of view and numerous blind spots when operating outside the vehicle, making it difficult to accurately determine the relative position of the vehicle and obstacles; the vehicle system lacks intelligent decision-making and cooperative control capability, the remote control process is only simple instruction execution, which cannot automatically adjust the differential lock, suspension height, power distribution or limit the vehicle speed according to the road conditions, and the driver still needs to frequently return to the vehicle to operate, which violates the original intention of convenient remote control; the human-computer interaction mode is simple, lacks intuitive path preview, distance prompt and risk warning mechanism, and the operation process has high uncertainty and heavy psychological burden; some schemes rely on cloud processing and cellular network, which is prone to delay, interruption or even failure in areas with no signal or weak signal (such as underground garage, mountainous area), and has poor reliability; and the existing technology is mostly limited to flat and open scenes, and is difficult to adapt to complex working conditions such as off-road, extreme narrow road and mechanical parking lot, so the application scenarios are extremely limited. These problems make the current remote driving technology have low safety, poor user experience and insufficient practicality, and it is far from meeting the application requirements of intelligence, high reliability and full scene.

[0026] To solve the above technical problems, the application provides a vehicle remote driving system and method integrated with AR virtual path projection, the core of which is to build an intelligent closed loop of "perception-decision-execution-interaction". The system constructs a three-dimensional environment model with centimeter-level precision in real time through multi-sensor fusion (vision, laser radar, ultrasonic wave, vehicle state sensor), generates a collision-free virtual path by a vehicle-mounted local AI algorithm and simulates the dynamic driving process of the vehicle, superimposes the virtual path and the vehicle model on the real scene in an augmented reality (AR) way through an off-vehicle remote control terminal to preview, the driver can intuitively confirm or re-plan the path before execution, realizing a safe interaction paradigm of "preview-confirmation-execution"; at the same time, the vehicle control strategy unit can automatically control the differential lock, suspension, power distribution, vehicle speed and other systems according to the scene without human intervention; the whole system operates based on local communication and does not rely on external network, ensuring the reliability in areas without signal, and significantly improving the safety, accuracy and user experience of remote driving in complex environments.

[0027] First, the vehicle remote driving system integrated with AR virtual path projection provided by the embodiments of the application will be described in detail below with reference to the accompanying drawings.

[0028] With reference to Figure 1 The vehicle remote driving system integrated with AR virtual path projection provided by the embodiments of the application comprises an environment perception module, a data processing and decision-making module, an off-vehicle remote control terminal and an execution mechanism.

[0029] In some embodiments of the present application, the environment perception module is configured to collect environment data around the vehicle and vehicle state data. The environment perception module includes a vision sensor, a laser radar, an ultrasonic radar, and a vehicle state sensor, and the vehicle state sensor includes a tilt angle sensor, a wheel speed sensor, and a height sensor.

[0030] Specifically, the environment perception module is responsible for comprehensively collecting environment data around the vehicle and its own state information, providing a high-precision data basis for subsequent environment modeling, path planning, and cooperative control. In the out-of-vehicle remote control scenario, the driver's field of view is limited, and there are a large number of visual blind spots. This module compensates for the shortcomings of human eye observation through active perception, can obtain the position of obstacles around the vehicle, terrain features, and key information such as vehicle body posture and motion state in real time, and ensure that the system has an environment recognition capability that exceeds human perception in complex environments (such as off-road sections and narrow parking spaces), fundamentally improving the safety and reliability of remote control driving.

[0031] The module adopts a multi-sensor fusion architecture, including a vision sensor, a laser radar, an ultrasonic radar, and a vehicle state sensor, to achieve complementary advantages and redundant backup. The vision sensor provides rich texture and semantic information, the laser radar constructs a high-precision three-dimensional point cloud map to identify obstacle contours and terrain undulations, and the ultrasonic radar enhances the ability to detect close-range obstacles. The vehicle state sensor is further subdivided into a tilt angle sensor, a wheel speed sensor, and a height sensor, which are used to monitor the vehicle body pitch and roll angle, wheel speed and slip trend, and suspension travel and ground clearance, respectively. These data not only support three-dimensional environment modeling, but also directly participate in vehicle cooperative control decisions, enabling the system to automatically adjust control strategies based on actual road conditions and vehicle state, achieving intelligent and adaptive remote control driving experience.

[0032] In some embodiments of the present application, the vision sensor includes at least four wide-angle cameras for stitching to generate a panoramic bird's eye view of the vehicle.

[0033] The visual sensor is used to achieve 360-degree dead-angle-free visual coverage of the vehicle's surrounding environment. Each camera captures image information of the vehicle's near-field area with its wide-angle characteristics, especially capable of clearly identifying low obstacles (such as curbs, fire hydrants, low piles, other vehicles' bumpers, etc.) in close proximity to the vehicle body, which are often blind spots that cannot be observed by traditional driving perspectives and rearview mirrors. Through image stitching, distortion correction, and perspective transformation algorithms, the system fuses the original images of the four cameras into a continuous, seamless panoramic bird's eye view, enabling the driver to intuitively see the spatial relationship between the vehicle as a whole and the surrounding environment on the remote control terminal outside the vehicle. This panoramic view not only greatly enhances the driver's situational awareness, but also provides a real background picture for the subsequent superimposed display of the augmented reality (AR) virtual path, ensuring the precise alignment of the virtual path with the real environment, and is the key visual basis for achieving high-precision, visual remote control operation.

[0034] In some embodiments of the present application, the laser radar is configured to construct a three-dimensional point cloud map of the vehicle's surroundings to identify obstacle profiles, slopes, and pothole depths.

[0035] The laser radar is used to provide millimeter to centimeter level distance measurement accuracy, generating dense three-dimensional point cloud data to accurately restore the geometric structure of the vehicle's surrounding environment. Unlike visual sensors that rely on texture and lighting, laser radar can stably detect the profiles, heights, and distances of obstacles by actively emitting laser beams and receiving reflected signals, regardless of lighting conditions (such as night, backlight, and shadows). More importantly, laser radar can accurately identify the three-dimensional morphology of the ground, including the inclination angle of the slope, the depth and range of the pothole, and the height of the protruding rock, which is crucial for determining the passability of the terrain. For example, in off-road scenarios, the system can determine whether the vehicle will bottom out or roll over based on the point cloud data, thereby planning a safe driving path. This three-dimensional point cloud map not only provides the basis for the environment modeling unit to generate drivable areas, but also serves as a geometric reference for AR virtual path projection, ensuring that the virtual vehicle model's position, attitude, and motion trajectory in the real scene are highly realistic, significantly improving the accuracy and credibility of path preview, and is an indispensable technical support for safe and intelligent remote driving.

[0036] In some embodiments of the present application, the data processing and decision-making module includes an environment modeling unit, an AR path generation unit, and a vehicle control strategy unit.

[0037] In some embodiments of the present application, the environment modeling unit is configured to fuse environmental data and vehicle state data to generate a three-dimensional environment model including obstacles, terrain features, and drivable areas. Among them, the terrain features in the three-dimensional environment model include slope, slope direction, and ground adhesion coefficient.

[0038] The environment modeling unit is further configured to perform spatio-temporal synchronization calibration on the visual data, the lidar point cloud data, the ultrasonic ranging data, and the vehicle state data by a multi-sensor fusion algorithm to generate a three-dimensional environment model with centimeter-level precision.

[0039] Specifically, the environment modeling unit plays a fundamental role in the data processing and decision-making module, and its main task is to fuse multi-source data from visual sensors, lidar, ultrasonic radar, and vehicle state sensors to generate a comprehensive and accurate three-dimensional environment model. This model not only contains the position and shape of obstacles, but also describes in detail the terrain features such as slope, slope direction, and ground adhesion coefficient, etc. key information, providing detailed environmental background for subsequent path planning. By applying advanced multi-sensor fusion algorithms, the environment modeling unit can achieve spatio-temporal synchronization calibration of data from different sources, ensuring that the three-dimensional environment model constructed reaches centimeter-level precision. This step is crucial for accurately identifying passable areas and avoiding potential dangers, and also provides a reliable environmental reference framework for the AR path generation unit, ensuring the feasibility and safety of virtual path planning.

[0040] In some embodiments of the present application, the AR path generation unit is configured to generate a collision-free virtual path and a vehicle virtual model according to the three-dimensional environment model, and simulate the dynamic travel process of the vehicle virtual model along the virtual path to generate a dynamic preview picture.

[0041] The virtual path generated by the AR path generation unit satisfies the following conditions: the curvature radius of the virtual path matches the minimum turning radius of the vehicle, and the distance between the vehicle virtual model and the obstacles in the environment model is not less than a preset safety threshold.

[0042] Specifically, the AR path generation unit is responsible for calculating an optimal driving path with no collision risk based on the high-precision three-dimensional environment model provided by the environment modeling unit, and creating a corresponding vehicle virtual model to simulate its dynamic travel process along this path. The dynamic preview picture generated by this process not only visually demonstrates how the vehicle avoids obstacles and adapts to terrain changes, but also allows the driver to visually confirm or adjust before execution. In particular, the AR path generation unit will ensure that the planned virtual path meets the physical constraints of the vehicle, such as the turning radius cannot be smaller than the minimum turning radius of the vehicle, and ensure that the vehicle virtual model maintains a certain safety distance from any obstacles. This path preview mechanism based on augmented reality technology greatly improves the safety and efficiency of remote driving, allowing the driver to make more intelligent and confident decisions in complex environments.

[0043] The vehicle control strategy unit is configured to automatically generate cooperative control commands according to the confirmation instruction input by the driver, the three-dimensional environment model and the virtual path. The cooperative control commands automatically generated by the vehicle control strategy unit in the off-road scene include locking the differential lock, raising the suspension and adjusting the power distribution ratio. In the urban narrow scene, the cooperative control commands automatically generated include limiting the vehicle speed to within 3 km / h, activating the panoramic image and optimizing the steering smoothness.

[0044] Specifically, the vehicle control strategy unit automatically generates a series of cooperative control commands to guide the specific action of the vehicle according to the confirmation instruction input by the driver through the off-vehicle remote control terminal, in combination with the three-dimensional environment model and the generated virtual path. These control commands cover multiple aspects from power distribution to suspension adjustment. In the off-road scene, the differential lock may be automatically locked, the suspension may be raised, or the power distribution ratio may be adjusted. In the urban narrow scene, the vehicle speed may be limited to within 3 km / h, the panoramic image system may be activated, or the steering smoothness may be optimized. In this way, the vehicle control strategy unit not only enhances the passing ability of the vehicle in various complex road conditions, but also simplifies the operation burden of the driver, so that precise and safe remote driving of the vehicle can be achieved without manual intervention.

[0045] In some embodiments of the present application, the data processing and decision-making module is built-in with an AI algorithm model, and the core function is processed locally at the vehicle end. In addition, the off-vehicle remote control terminal (such as a mobile phone APP) and the vehicle perform instruction and image data transmission through low-latency and high-reliability local direct connection communication (such as Wi-Fi 6 / Bluetooth 5.0 UWB technology). The AI algorithm model is pre-installed in the vehicle-mounted computing unit, and all perception and decision-making are completed at the vehicle end without relying on external networks.

[0046] Specifically, the present application builds an intelligent remote driving system completely independent of external networks by building-in an AI algorithm model in the data processing and decision-making module and deploying the core function in the vehicle-mounted computing unit for local processing. All environmental perception, three-dimensional modeling, path planning and control decision-making are completed in a closed loop at the vehicle end, avoiding the delay, network interruption risk and data security risks caused by cloud transmission, significantly improving the real-time performance, reliability and privacy protection capability of the system. At the same time, the vehicle and the mobile phone APP use low-latency and high-reliability local direct connection communication technologies such as Wi-Fi 6, Bluetooth 5.0 or UWB to ensure real-time transmission of high-definition panoramic images, AR dynamic preview pictures and other large-bandwidth data, as well as precise response of control instructions. This architecture not only guarantees the functionality in environments without cellular networks such as underground garages, mountainous areas and tunnels, but also achieves high-precision, low-latency and all-weather off-vehicle remote control through edge intelligence, providing solid technical support for safe driving in complex scenarios.

[0047] In some embodiments of the present application, the off-board remote terminal is configured to display a real-time image of the three-dimensional environment model, superimpose the dynamic preview picture in an AR manner, and provide interactive controls to receive the confirmation instructions input by the driver. Further, the interactive controls include a confirmation execution control for receiving the confirmation instructions input by the driver on the virtual path, and a re-planning control for receiving the adjustment instructions input by the driver on the virtual path. After the re-planning control is triggered, the AR path generation unit re-generates the virtual path and the dynamic preview picture according to the intention control instructions adjusted by the driver.

[0048] As the core interactive interface between the driver and the vehicle, the off-board remote terminal displays the three-dimensional environment model and the panoramic image generated by the environment perception module in real time, and superimposes the virtual path planned by the system and the dynamic driving process of the vehicle in the real scene through augmented reality (AR) technology, forming an intuitive visual preview picture. This "virtual-real fusion" display method enables the driver to fully understand the spatial relationship between the vehicle and the surrounding obstacles outside the vehicle, clearly predict the driving trajectory, passability and potential risks, greatly improving the situational awareness capability and effectively solving the problems of limited field of view and multiple operation blind areas in traditional remote driving. At the same time, the terminal provides interactive controls to build a safe closed loop of "preview-confirmation-execution", ensuring that all automatic driving instructions are carried out under the premise of being visible, predictable and controllable by the driver, significantly enhancing the safety and user confidence of the operation.

[0049] Among them, the interactive controls include two key functions of "confirmation execution" and "re-planning". The "confirmation execution" control is used to receive the approval instructions of the driver on the virtual path, embodying the safety design concept of "man-in-the-loop", i.e. the system only starts automatic driving after obtaining explicit authorization from the user, preventing accidents caused by misoperation. The "re-planning" control gives the driver the ability to actively intervene in path planning. When the path automatically generated by the system does not meet the expectations, the driver can trigger this function and input new control intentions, and the system will re-calculate the optimal path and generate a new AR preview picture accordingly. This mechanism realizes efficient iterative interaction between man and machine, which not only takes advantage of AI intelligent planning, but also retains the flexibility of human experience judgment, especially suitable for complex and high-risk scenarios such as off-road, narrow lanes, and mechanical parking spaces, significantly improving the adaptability, safety and user experience of the system.

[0050] In some embodiments of the present application, the display interface of the off-board remote terminal is further configured to display the distance between the vehicle and the obstacle and the risk warning mark in real time, and issue an alarm through sound, vibration or screen flashing when the distance is below a preset safety threshold.

[0051] The display interface of the off-vehicle remote control terminal displays the accurate distance information of the vehicle and the surrounding obstacles in real time, and presents risk warning marks in a graphical manner, significantly improving the spatial perception and situation judgment ability of the driver during off-vehicle remote control. In the absence of in-vehicle visual and auditory feedback, the system dynamically displays the distance between the vehicle in each direction and the nearest obstacle in the form of digital labels, distance bars, color blocks, and other forms, and visually prompts in a prominent color, icon, or highlighted border when approaching a dangerous area, helping the driver quickly identify potential collision points. This intuitive and quantitative information presentation method effectively compensates for the perception blind spot of off-vehicle operation, especially when moving a car in a narrow space, extreme parking, or driving in complex terrain, significantly reducing the risk of scratches due to distance misjudgment and enhancing the accuracy and controllability of the operation.

[0052] When the distance between the vehicle and the obstacle is less than the preset safety threshold, the system triggers a multi-modal warning mechanism, issuing a warning to the driver through sound prompts (such as a beeping sound or a voice warning), device vibration (vibration of the mobile phone or remote control), and screen flashing. This multi-modal warning method combines hearing, touch, and vision to ensure that the warning information is perceived in time under different environmental conditions (such as noise, distraction, or visual deviation from the screen), effectively improving the reliability and timeliness of the response. This real-time warning function complements the AR path preview, forming a full-link safety protection system that confirms in advance, dynamically monitors in the process, and immediately warns of risks, further ensuring the safety and user confidence of the remote control driving process.

[0053] In some embodiments of the present application, the execution mechanism is configured to receive and execute a cooperative control command to control the vehicle to automatically travel along the virtual path. The execution mechanism includes a drive motor controller, a steering controller, a brake controller, a suspension controller, and a differential lock controller, and the cooperative control command is transmitted to each controller through a CAN bus.

[0054] The execution mechanism, as the final execution unit of the remote control driving system, is responsible for receiving and executing the cooperative control command generated by the vehicle control strategy unit to drive the vehicle to automatically and accurately travel along the virtual path planned by AR. It covers key components such as drive motor controllers, steering controllers, brake controllers, suspension controllers, and differential lock controllers, and can comprehensively control the power output, steering angle, braking state, vehicle height, and drive wheel torque distribution of the vehicle. Through the cooperative work of these actuators, the vehicle can automatically complete low-speed moving, path tracking, stable driving, and safe parking operations without the intervention of the driver, especially in complex off-road or narrow urban environments, achieving intelligent responses such as automatic speed limiting, optimizing steering smoothness, raising the suspension, or locking the differential lock, and truly converting "perception - decision" into safe and reliable "execution" actions, which is the core guarantee for realizing full-scene intelligent remote control driving.

[0055] All cooperative control commands are transmitted to each execution controller through the vehicle CAN bus, making full use of the mature and highly reliable communication architecture in the automotive industry. The CAN bus has low latency, strong anti-interference and high fault tolerance, and can stably and in real time deliver control instructions in a complex electromagnetic environment, ensuring that each subsystem responds synchronously and avoiding safety hazards caused by asynchronous execution. This communication method not only ensures the accuracy and real-time performance of instruction transmission during remote driving, but also enhances the compatibility and integration of the system with the original electronic control system of the vehicle, without the need for additional wiring or dedicated networks, reducing development and deployment costs. Efficient communication through the CAN bus makes the entire remote driving closed loop highly reliable and engineering realizable, providing a solid technical foundation for the practical application of the system on actual vehicles.

[0056] The following further illustrates the system in combination with an embodiment.

[0057] First, after the driver gets off the vehicle, the "off-road remote control" mode is activated through the mobile phone APP. The APP immediately displays the panoramic image generated by the vehicle-mounted camera stitching and intelligently identifies environmental risks, warning that the right side of the vehicle is only 30 cm away from the edge of the ravine and reminding the driver to be careful. Then, the driver inputs the initial instruction "turn left by 10 degrees and straight back 5 meters" on the APP interface. The system immediately switches to the AR projection mode, which captures the real scene around the vehicle in real time through the mobile phone camera and displays a blue non-collision virtual path and a semi-transparent vehicle virtual model on the screen, dynamically simulating the entire process of reversing the vehicle.

[0058] Then, the driver observes the preview screen and finds that when the virtual vehicle is about 3 meters away, the distance between the left front corner of the virtual vehicle and the cliff is too close. The distance bar on the APP interface turns red and triggers visual and audio alarms, indicating that there is a risk of scratching. To avoid danger, the driver immediately clicks the "replan" button and modifies the instruction to "straight back". Based on the updated user intention and the latest three-dimensional environment model, the system quickly recalculates the path. The new AR projection shows that the vehicle will back up along a straight path that is more to the right but still maintains a safe distance, with no collision risk throughout the journey. After confirming that the new path is safe and feasible, the driver clicks "confirm execution". The vehicle control strategy unit is then started, automatically executing cooperative control according to the off-road scene: maintaining the suspension height to improve passability, locking the rear differential lock to enhance driving stability, and accurately limiting the vehicle speed to within 3 km / h to ensure safe operation.

[0059] Then, the vehicle starts to automatically and smoothly back along the AR rehearsal path, and the real-time state data during the execution process is continuously returned to the APP interface. When the ultrasonic radar monitors that the right rear wheel is close to the 20cm warning line from the edge of the ditch, the APP again issues a sound warning to remind the driver, but the system is still within the safe range, and the task continues to be executed. Finally, the vehicle accurately and smoothly completes the reversing action, and automatically stops after reaching the predetermined position. The entire remote control process is safe, intelligent and efficient, fully embodying the technical advantages of the combination of AR path rehearsal, man-machine collaborative decision-making and vehicle system adaptive control.

[0060] Secondly, with reference to Figure 2 The embodiment of the application provides a vehicle remote driving method integrated with AR virtual path projection, which is applied to the vehicle remote driving system integrated with AR virtual path projection and comprises the following steps.

[0061] In step S110, the environmental data around the vehicle and the vehicle state data are collected.

[0062] In step S110, the original information of the external environment and the state of the vehicle is comprehensively acquired through the multi-sensor system carried by the vehicle. According to the content of the system embodiment, the environmental data includes the information such as the position, contour, distance and terrain undulation of the obstacles collected by the visual sensor, the laser radar and the ultrasonic radar; and the vehicle state data covers the dynamic parameters such as the vehicle body posture, movement speed and ground clearance provided by the inclination sensor, the wheel speed sensor and the height sensor. The high-frequency and high-precision collection of these data provides a reliable basis for subsequent environment modeling and path planning. In the outdoor remote control scene, the driver cannot rely on the traditional driving perspective, so the ability of the system to actively perceive the environment is crucial. This step constructs a 360-degree omnidirectional perception coverage around the vehicle through multi-source sensor fusion, effectively eliminates the visual blind area, and lays a data foundation for realizing safe and intelligent remote control.

[0063] In step S120, a three-dimensional environment model containing obstacles, terrain features and passable areas is generated based on the environmental data and the vehicle state data.

[0064] In step S120, after the data collection is completed, the heterogeneous data is processed by a multi-sensor fusion algorithm for spatio-temporal synchronization calibration and fusion, and a high-precision, centimeter-level three-dimensional environment model is constructed. The model not only accurately restores the spatial position and geometric shape of static obstacles, but also contains key terrain features such as slope, slope direction, and ground adhesion coefficient, and identifies areas that the vehicle can safely pass through. This process is a prerequisite for intelligent path planning, and the system determines which areas are drivable and which areas have the risk of bottoming out or rollover through the model. The construction of the three-dimensional environment model realizes the transition from raw data to structured environment cognition, enabling the vehicle to "understand" complex road conditions, providing an accurate geographical and physical reference framework for subsequent AR path generation and cooperative control, and is a core link connecting perception and decision-making.

[0065] In step S130, a collision-free virtual path and a vehicle virtual model are generated based on the three-dimensional environment model, and a dynamic preview picture of the virtual path and the vehicle virtual model is displayed in an AR manner through an off-vehicle remote control terminal.

[0066] The dynamic preview picture is used to simulate the progress of the vehicle virtual model along the virtual path.

[0067] In step S130, the system automatically plans an optimal driving path that is collision-free and executable based on the three-dimensional environment model and vehicle dynamics constraints (such as minimum turning radius), and generates a three-dimensional virtual model consistent with the real vehicle. The off-vehicle remote control terminal captures the real scene in real time through the camera, and the system displays the virtual path and the vehicle model in an AR manner, dynamically simulating the entire process of the vehicle driving along the path. This "virtual test drive" mechanism allows the driver to intuitively predict the final pose of the vehicle, the distance relationship with obstacles, and the feasibility of passing before the command is executed, greatly improving the predictability and safety of the operation. AR preview not only reduces psychological burden, but also provides a basis for decision-making, realizing the transition from "blind operation" to "visual remote control".

[0068] In step S140, the off-vehicle remote control terminal receives the driver's confirmation instruction or re-planning instruction for the virtual path. If a re-planning instruction is received, the virtual path and the dynamic preview picture are regenerated, and if a confirmation instruction is received, a cooperative control command is generated based on the three-dimensional environment model.

[0069] In step S140, the system does not enforce the automatic path, but passes the preview result to the driver for evaluation. If the driver is satisfied with the current path, he can send an instruction through the "confirm execution" control to trigger the subsequent control process; if he believes that the path has risks or does not meet expectations, he can click "replan" and input the adjustment intention (such as offset direction, change end point), and the system will recalculate the path and generate a new AR preview picture accordingly. This interactive mode realizes efficient iteration between man and machine, both retaining the efficiency of AI intelligent planning and respecting the experience and judgment of the driver, and is particularly suitable for high-risk or personalized demand scenarios. This step ensures that all automatic driving behaviors are carried out with the user's knowledge and control, significantly improving the safety and user trust of the system.

[0070] In step S150, the control execution mechanism executes the cooperative control command to control the vehicle to automatically drive along the virtual path.

[0071] In step S150, after receiving the confirmation instruction of the driver, the vehicle control strategy unit generates a cooperative control command containing the coordinated actions of power, steering, braking, suspension, differential lock and other systems, and issues it to each execution controller through the CAN bus. The execution mechanism accurately responds to the instruction to drive the vehicle automatically, smoothly and safely along the virtual path of the AR preview, realizing centimeter-level precision trajectory tracking. During the entire execution process, the system continuously monitors environmental changes and vehicle state, and has real-time adjustment capability. This step completes the closed loop from "virtual preview" to "real execution", enabling the vehicle to achieve high passability and high safety in complex environments, and truly realizing the "what you see is what you get" intelligent remote control driving experience.

[0072] In summary, the vehicle remote driving system and method integrated with AR virtual path projection provided by the embodiments of the present application have the following technical effects.

[0073] The present application realizes a comprehensive breakthrough in safety, intelligence and user experience of remote control driving by deeply integrating environment perception, intelligent decision-making, augmented reality interaction and vehicle cooperative control technology. The system uses multi-sensor fusion technology to build a high-precision three-dimensional environment model, and combines AR technology to superimpose the virtual path and vehicle dynamic preview picture on the real scene, so that the driver can obtain global situational awareness beyond physical vision outside the vehicle, effectively eliminating blind spots and significantly reducing collision risks caused by distance misjudgment. Through the "preview-confirmation-execution" human-machine cooperation mechanism, the driver can intuitively evaluate the feasibility of the path before operation, and trigger the re-planning if necessary, realizing efficient complementarity of intelligent algorithms and human experience and improving the decision-making reliability in complex scenarios.

[0074] Further, the vehicle control strategy unit automatically calls differential lock, suspension, power distribution and other systems according to the environment and the path, realizes the whole vehicle collaborative control without manual intervention, and truly plays the convenient advantage of off-car remote control. The whole system is based on localized AI processing and vehicle-level direct communication, does not depend on external network, and ensures stable operation in signal-free areas. Finally, the system accurately controls the actuator through the CAN bus, realizes high-precision automatic driving of the vehicle along the virtual path, and realizes real-time feedback of the state and risk warning in the process, and builds a full-link safety closed loop from perception, decision, interaction to execution. The scheme is not only suitable for narrow parking spaces, mechanical garages and other urban scenes, but also can be expanded to off-road escape, extreme passage and other high-difficulty working conditions, greatly expanding the application boundary of remote control driving, and providing users with a safe, intelligent, intuitive and reliable new control experience.

[0075] It should be noted that, in various specific embodiments of the present application, the data processing activities related to user information, user behavior data, historical data and location information, etc. related to the user's identity or personal characteristics will be carried out in accordance with the principles of legality, legitimacy and necessity. The collection, use, storage and processing of all data are subject to the applicable national and regional laws and regulations and industry standards, and the informed consent of the user is obtained through clear and explicit means before processing. For the processing of sensitive personal information, the user's separate consent will be obtained through pop-up prompts, independent confirmation pages and other conspicuous means. If there is a conflict between the relevant processing and laws and regulations, the laws and regulations will be followed, and only within the scope allowed by laws and regulations, the necessary data processing will be carried out, and all data-based applications, analysis and technical implementation will be carried out within the scope allowed by laws and regulations.

[0076] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation schematic. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, with the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of a larger operation are independently executed.

[0077] Furthermore, although the present application is described in the context of functional modules, it is understood that one or more of the functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary to an understanding of the present application. Rather, the properties, functions and internal relationships of the various functional modules disclosed in the devices shown herein are deemed to be illustrative of the principles of the present application. Accordingly, the present application is not limited to the specific implementations disclosed herein. Rather, the skilled artisan will appreciate that the present application can be practiced using other implementations that fall within the scope of the claims attached hereto. It is also understood that the specific concepts disclosed herein are merely illustrative of the principles of the present application. For example, although the present application is described in the context of a mobile device, the present application is not limited to this context. Rather, the present application can be applied to any device that is capable of receiving and processing data. Therefore, the scope of the present application is not intended to be limited to the particular examples disclosed.

[0078] If the functions are implemented in software, the functions can be stored in or implemented as one or more computer program products, which can be incorporated into a computer-readable medium for use by or in connection with an apparatus, method, or system as described herein. The computer-readable medium can be a computer- only medium, a partially software and partially hardware medium, or a medium that contains both software and hardware. The computer-readable medium can be any available medium or a combination thereof that is accessible by a general purpose or special purpose computer system. The computer-readable medium can be, for example, a computer storage medium. By way of example, the computer storage medium can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of computer-executable instructions or data structures and that can be accessed by a general purpose or special purpose computer system. Combinations of the above should also be included within the scope of the computer-readable medium.

[0079] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable program units, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instruction from the instruction execution system, apparatus, or device and execute the instruction. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or apparatus or device. More specific examples (a nonexhaustive list) of the computer-readable medium include an electrical connection, hard-wired ROM, random access memory (RAM),

[0080] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0081] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above described embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable

[0082] In the above description of the present application, reference has been made to descriptive terms such as "one embodiment / scheme", "another embodiment / scheme" or "some embodiments / schemes" etc. which can mean that a particular feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. The illustrative examples described above do not necessarily all refer to the same embodiment or example of the application. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0083] While the embodiments of the present application have been shown and described, it is to be understood that various modifications, substitutions, combinations, and variations can be made therein without departing from the spirit and scope of the present application, which is defined by the appended claims and their equivalents.

[0084] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are included in the scope defined by the claims of the present application.

Claims

1. A vehicle remote control driving system integrating AR virtual path projection, characterized in that, It includes an environmental perception module, a data processing and decision-making module, an external remote control terminal, and an actuator; The environmental perception module is configured to collect environmental data around the vehicle and vehicle status data. The data processing and decision-making module includes the following units: The environment modeling unit is configured to fuse the environmental data and vehicle status data to generate a three-dimensional environment model including obstacles, terrain features and passable areas. The AR path generation unit is configured to generate a collision-free virtual path and a vehicle virtual model based on the three-dimensional environment model, and to simulate the dynamic movement of the vehicle virtual model along the virtual path to generate a dynamic pre-show. The vehicle control strategy unit is configured to automatically generate cooperative control commands based on the driver's confirmation command, the three-dimensional environment model, and the virtual path; The vehicle-mounted remote control terminal is configured to display real-time images of the three-dimensional environment model, overlay the dynamic pre-show screen in AR mode, and provide interactive controls to receive confirmation commands input by the driver. The actuator is configured to receive and execute the cooperative control command to control the vehicle to automatically travel along the virtual path.

2. The vehicle remote control driving system integrating AR virtual path projection according to claim 1, characterized in that, The terrain features in the three-dimensional environment model include slope, aspect, and ground adhesion coefficient; the environment perception module includes a visual sensor, lidar, ultrasonic radar, and vehicle status sensor; the vehicle status sensor includes a tilt sensor, wheel speed sensor, and height sensor. The visual sensor includes at least four wide-angle cameras for stitching together to generate a panoramic bird's-eye view of the vehicle's surroundings; the lidar is configured to construct a three-dimensional point cloud map around the vehicle to identify obstacle outlines, slopes, and pothole depths.

3. The vehicle remote control driving system integrating AR virtual path projection according to claim 1, characterized in that, The environment modeling unit is also configured to perform spatiotemporal synchronous calibration of visual data, lidar point cloud data, ultrasonic ranging data and vehicle status data through a multi-sensor fusion algorithm to generate a three-dimensional environment model with centimeter-level accuracy.

4. The vehicle remote control driving system integrating AR virtual path projection according to claim 1, characterized in that, The virtual path generated by the AR path generation unit satisfies the following conditions: the radius of curvature of the virtual path matches the minimum turning radius of the vehicle, and the distance between the vehicle virtual model and obstacles in the environment model is not less than a preset safety threshold.

5. The vehicle remote control driving system integrating AR virtual path projection according to claim 1, characterized in that, The vehicle control strategy unit automatically generates the following collaborative control commands in off-road scenarios: locking the differential lock, raising the suspension, and adjusting the power distribution ratio; and automatically generates the following collaborative control commands in narrow urban scenarios: limiting the vehicle speed to within 3 km / h, activating the panoramic image, and optimizing steering smoothness.

6. The vehicle remote control driving system integrating AR virtual path projection according to claim 1, characterized in that, The data processing and decision-making module has a built-in AI algorithm model, and its core functions are processed locally on the vehicle side.

7. The vehicle remote control driving system integrating AR virtual path projection according to claim 1, characterized in that, In the vehicle-mounted remote control terminal, the interactive controls include a confirmation execution control and a replanning control; The confirmation execution control is used to receive confirmation instructions for the virtual route input by the driver; The replanning control is used to receive adjustment instructions for the virtual route input by the driver; After the replanning control is triggered, the AR path generation unit regenerates the virtual path and dynamic pre-show screen according to the driver's adjusted intention control command.

8. The vehicle remote control driving system integrating AR virtual path projection according to claim 1, characterized in that, The display interface of the vehicle-mounted remote control terminal is also configured to: display the distance between the vehicle and obstacles and risk warning signs in real time; when the distance is lower than a preset safety threshold, an alarm is issued by sound, vibration or screen flashing.

9. The vehicle remote control driving system integrating AR virtual path projection according to claim 1, characterized in that, The actuators include a drive motor controller, a steering controller, a brake controller, a suspension controller, and a differential lock controller. The coordinated control commands are transmitted to each controller via a CAN bus.

10. A method for remote-controlled vehicle driving integrating AR virtual path projection, characterized in that, The vehicle remote driving system with integrated AR virtual path projection as described in any one of claims 1 to 9 includes the following steps: Collect environmental data around the vehicle and vehicle status data; Based on the environmental data and vehicle status data, a three-dimensional environmental model including obstacles, terrain features and passable areas is generated. Based on the three-dimensional environment model, a collision-free virtual path and a vehicle virtual model are generated, and a dynamic pre-show of the virtual path and vehicle virtual model is overlaid and displayed in AR mode via an external remote control terminal; the dynamic pre-show is used to simulate the movement of the vehicle virtual model along the virtual path. The vehicle remote control terminal receives the driver's confirmation command or replanning command for the virtual path. If a replanning command is received, the virtual path and dynamic pre-show screen are regenerated. If a confirmation command is received, a collaborative control command is generated based on the three-dimensional environment model. The control actuator executes the cooperative control command to control the vehicle to automatically travel along the virtual path.

Images