Remote driving control method, virtual reality terminal, vehicle terminal, server, and storage medium

Abstract

The application relates to the technical field of remote driving, in particular to a remote driving control method, a virtual reality terminal, a vehicle terminal, a server and a readable storage medium, and aims to solve the technical problems of large resource consumption, large data transmission volume and time delay of a remote driving control system. For this purpose, the application obtains an environment sensing result of an environment where a vehicle terminal is located based on an environment sensing device of the vehicle terminal; obtains a dynamic target segmentation result of the environment where the vehicle terminal is located according to the environment sensing result and map information of the environment where the vehicle terminal is located; encodes the dynamic target segmentation result to obtain a dynamic target coding result, so that a virtual reality device obtains a virtual reality image rendering result according to the dynamic target coding result and position information of the vehicle terminal, thereby realizing remote driving control of the vehicle terminal by the virtual reality device, effectively reducing the transmission data volume and the computing power requirement of the environment sensing data, and reducing the time delay of data transmission.

Description

Technical Field

[0001] This application relates to the field of remote driving technology, specifically to a remote driving control method, a virtual reality terminal, a vehicle-mounted device, a server, and a readable storage medium. Background Technology

[0002] In existing technologies, virtual reality (VR) technology has been introduced into autonomous taxis to support remote human intervention. By incorporating VR technology, remote operators can take over vehicle control as if driving a real person. This technology not only improves the efficiency and accuracy of remote operation but also enables operators to better understand and respond to various situations encountered while the vehicle is in motion. This demonstrates the feasibility of VR remote driving technology.

[0003] However, existing technologies based on virtual reality for remote driving control often suffer from problems such as high system resource consumption, large data transmission volume, and increased system latency due to the need for real-time 3D reconstruction of road conditions.

[0004] Accordingly, there is a need in the field for a new remote driving control solution to address the aforementioned problems. Summary of the Invention

[0005] In order to overcome the above-mentioned defects, this application is made to solve or at least partially solve the technical problems of high resource consumption, large data transmission volume and time delay in the prior art of remote driving control systems.

[0006] In a first aspect, a remote driving control method is provided, the method being applied to a vehicle; the vehicle is communicatively connected to a virtual reality terminal; the method includes: Based on the vehicle-mounted environmental perception device, obtain the environmental perception results of the environment in which the vehicle is located; Based on the environmental perception results and the map information of the environment where the vehicle is located, obtain the dynamic target segmentation results of the environment where the vehicle is located; The dynamic target segmentation result is encoded to obtain a dynamic target encoding result, so that the virtual reality device can obtain a virtual reality image rendering result based on the dynamic target encoding result and the vehicle's position information, thereby realizing remote driving control of the vehicle.

[0007] In one technical solution of the aforementioned remote driving control method, obtaining the dynamic target segmentation result of the vehicle's environment based on the environmental perception result and the map information of the intelligent environment includes: Based on the map information, obtain static road element data of the environment in which the vehicle is located; Based on the environmental perception results and the static road element data, and using a dynamic object detection algorithm, the dynamic targets in the environment where the vehicle is located are obtained, and the dynamic targets are segmented to obtain the dynamic target segmentation results.

[0008] In one technical solution of the aforementioned remote driving control method, encoding the dynamic target segmentation result to obtain a dynamic target encoding result includes: The dynamic target segmentation results corresponding to multiple environmental sensing devices are transformed to the same coordinate system; The dynamic target segmentation results corresponding to the multiple environmental sensing devices after conversion are fused to obtain a dynamic target fusion result; The dynamic target fusion result is encoded to obtain the dynamic target encoding result.

[0009] In one technical solution of the aforementioned remote driving control method, the step of segmenting the dynamic target to obtain the dynamic target segmentation result includes: For each environmental sensing device, the dynamic target is segmented from the environmental sensing result of the environmental sensing device, and the position coordinates of each dynamic target are obtained. The segmented image corresponding to the dynamic target and the position coordinates are used as the dynamic target segmentation result.

[0010] In one technical solution of the above-mentioned remote driving control method, the step of enabling the virtual reality device to obtain a virtual reality image rendering result based on the dynamic target encoding result and the vehicle's position information includes: The dynamic target encoding result and the vehicle's location information are transmitted to the virtual reality device, so that the virtual reality device performs the following steps: Three-dimensional reconstruction is performed based on the dynamic target encoding results to obtain the dynamic target three-dimensional reconstruction results; Based on the vehicle's location information, the virtual static road conditions of the environment in which the vehicle is located are obtained, and based on the virtual static road conditions, the three-dimensional reconstruction results of the static road conditions are obtained. Based on the dynamic target 3D reconstruction results and the static road condition 3D reconstruction results, obtain the virtual reality image rendering results; or, The dynamic target encoding result is transmitted to the cloud server, so that the cloud server performs the following steps: Three-dimensional reconstruction is performed based on the dynamic target encoding results to obtain the dynamic target three-dimensional reconstruction results; Based on the location information, the virtual static road conditions of the environment where the vehicle is located are obtained, and based on the virtual static road conditions, the three-dimensional reconstruction results of the static road conditions are obtained. Based on the dynamic target 3D reconstruction results and the static road condition 3D reconstruction results, the virtual reality image rendering results are obtained; The virtual reality image rendering result is transmitted to the virtual reality device.

[0011] In one technical solution of the above-mentioned remote driving control method, the method further includes: Based on the environmental perception results of multiple environmental perception devices on the vehicle, multi-source data fusion is performed to obtain multi-source perception fusion results; Based on the environmental perception results from multiple environmental perception devices on the vehicle, feature extraction is performed to obtain feature extraction results; Based on the multi-source perception fusion result and the feature extraction result, the trajectory of the vehicle is predicted to obtain the trajectory prediction result; The trajectory prediction result is sent to the virtual reality device so that the virtual reality device can display the trajectory prediction result, thereby achieving compensation for the remote driving control of the vehicle.

[0012] In one technical solution of the aforementioned remote driving control method, the step of extracting features based on the environmental perception results from multiple environmental perception devices on the vehicle to obtain feature extraction results includes: Obtain static road element data of the environment in which the vehicle is located; Based on the static road element data and the environmental perception results, feature extraction is performed to obtain the road constraint information of the environment in which the vehicle is located and the motion trend information of the vehicle, which are used as the feature extraction results.

[0013] In one technical solution of the above-mentioned remote driving control method, the method further includes: Based on the environmental perception results and the vehicle-road cooperative data, the future road condition perception results of the vehicle are obtained; Based on the future road condition perception results and the vehicle's operating status, obtain the vehicle's behavior prediction results; The behavior prediction result is sent to the virtual reality device so that the virtual reality device can provide multi-sensory warnings based on the behavior prediction result.

[0014] In one technical solution of the aforementioned remote driving control method, obtaining the future road condition perception result of the vehicle based on the environmental perception result and the vehicle-road cooperative data includes: Based on the environmental perception results and the static map information of the environment where the vehicle is located, the obstacle trajectory prediction results of the environment where the vehicle is located are obtained. Based on the vehicle-road cooperative data, obtain the road condition prediction results of the environment in which the vehicle is located; The future road condition perception result is obtained based on the obstacle trajectory prediction result and the road condition prediction result.

[0015] In one technical solution of the above-mentioned remote driving control method, the method further includes: Obtain the type of requirement generated by the vehicle; Based on the demand type, a demand prompt is sent to the virtual reality device so that the virtual reality device can respond to the demand prompt.

[0016] In one technical solution of the above-mentioned remote driving control method, sending a demand prompt to the virtual reality device according to the demand type includes: The method of requesting information is determined based on the type of request and the current driving scenario on the vehicle. Based on the aforementioned request prompting method, the request prompt is sent to the virtual reality device.

[0017] In a second aspect, a remote driving control method is provided, the method being applied to a virtual reality device; the virtual reality device is communicatively connected to a vehicle; the method includes: Obtain virtual reality image rendering results; wherein the virtual reality image rendering results are obtained based on the dynamic target encoding results generated by the vehicle and the location information of the vehicle, wherein the dynamic target encoding results are obtained in the following manner: based on the environmental perception device of the vehicle, obtain the environmental perception results of the environment in which the vehicle is located; based on the environmental perception results and the map information of the environment in which the vehicle is located, obtain the dynamic target segmentation results of the environment in which the vehicle is located; encode the dynamic target segmentation results. Based on the virtual reality image rendering results, remote driving control is performed on the vehicle.

[0018] In one technical solution of the above-mentioned remote driving control method, obtaining the virtual reality image rendering result includes: The generated dynamic target encoding result and the vehicle's location information are obtained from the vehicle terminal. Three-dimensional reconstruction is performed based on the dynamic target encoding results to obtain the dynamic target three-dimensional reconstruction results; Based on the location information, the virtual static road conditions of the environment where the vehicle is located are obtained, and based on the virtual static road conditions, the three-dimensional reconstruction results of the static road conditions are obtained. The virtual reality image rendering result is obtained based on the dynamic target 3D reconstruction result and the static road condition 3D reconstruction result.

[0019] In one technical solution of the above-mentioned remote driving control method, obtaining the virtual reality image rendering result includes: Receive virtual reality image rendering results from a cloud server, wherein the virtual reality image rendering results are obtained by the cloud server in the following manner: Obtain the dynamic target encoding result generated by the vehicle and the location information of the vehicle; Three-dimensional reconstruction is performed based on the dynamic target encoding results to obtain the dynamic target three-dimensional reconstruction results; Based on the location information, the virtual static road conditions of the environment where the vehicle is located are obtained, and based on the virtual static road conditions, the three-dimensional reconstruction results of the static road conditions are obtained. The virtual reality image rendering result is obtained based on the dynamic target 3D reconstruction result and the static road condition 3D reconstruction result.

[0020] In one technical solution of the above-mentioned remote driving control method, the method further includes: Obtain the trajectory prediction results generated by the vehicle; Based on the trajectory prediction results, compensation is made for the remote driving control of the vehicle.

[0021] In one technical solution of the above-mentioned remote driving control method, the method further includes: Obtain the behavior prediction results from the vehicle end; Based on the predicted behavior, a multi-sensory early warning system is implemented.

[0022] In one technical solution of the above-mentioned remote driving control method, the method further includes: Obtain the requirement prompt generated by the vehicle; wherein the requirement prompt is generated based on the requirement type of the vehicle. Based on the aforementioned requirement prompts, the vehicle-side requirement response is performed.

[0023] In a third aspect, a remote driving control method is provided, the method being applied to a cloud server; the cloud server is communicatively connected to both a virtual reality device and a vehicle; the method includes: Obtain the dynamic target encoding result generated by the vehicle and the location information of the vehicle; Based on the dynamic target encoding result and the vehicle's location information, a virtual reality image rendering result is obtained, and the virtual reality image rendering result is sent to the virtual reality device so that the virtual reality device can remotely control the vehicle based on the virtual reality image rendering result. The dynamic target encoding result is obtained as follows: based on the vehicle's environmental perception device, the environmental perception result of the environment in which the vehicle is located is obtained; based on the environmental perception result and the map information of the environment in which the vehicle is located, the dynamic target segmentation result of the environment in which the vehicle is located is obtained; and the dynamic target segmentation result is encoded.

[0024] In one technical solution of the above-mentioned remote driving control method, obtaining the virtual reality image rendering result based on the dynamic target encoding result includes: Three-dimensional reconstruction is performed based on the dynamic target encoding results to obtain the dynamic target three-dimensional reconstruction results; Obtain the location information of the vehicle; based on the location information, obtain the virtual static road conditions of the environment in which the vehicle is located, and based on the virtual static road conditions, obtain the three-dimensional reconstruction results of the static road conditions; The virtual reality image rendering result is obtained based on the dynamic target 3D reconstruction result and the static road condition 3D reconstruction result.

[0025] In a fourth aspect, a vehicle terminal is provided, the vehicle terminal including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above-described technical solutions of the remote driving control method.

[0026] In a fifth aspect, a virtual reality device is provided, the virtual reality device including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above-described remote driving control methods.

[0027] In a sixth aspect, a cloud server is provided, the cloud server including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above-described remote driving control methods.

[0028] In a seventh aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the method described in any of the above-described remote driving control methods.

[0029] The above-described technical solutions of this application have at least one or more of the following beneficial effects: In implementing the remote driving control method provided in this application, the application acquires environmental perception results of the vehicle's environment based on the vehicle's environmental perception device; obtains dynamic target segmentation results of the vehicle's environment based on the environmental perception results and map information of the vehicle's environment; encodes the dynamic target segmentation results to obtain dynamic target encoding results, enabling the virtual reality device to obtain virtual reality image rendering results based on the dynamic target encoding results and the vehicle's location information, thereby realizing remote driving control of the vehicle by the virtual reality device. Through the above configuration, this application can acquire dynamic target segmentation results of the vehicle's environment using the vehicle's environmental perception device, and the vehicle transmits only the dynamic target segmentation results for virtual reality image rendering. When performing 3D reconstruction on a cloud server or virtual reality device, 3D reconstruction of dynamic targets can be performed based on the dynamic target encoding results, and 3D reconstruction of virtual static road conditions can be performed based on the vehicle's location information. This eliminates the need for the vehicle to transmit environmentally perceived road condition information to the cloud server or virtual reality device, effectively reducing the amount of environmental perception data transmitted, reducing the computational power requirements of the remote driving control process, reducing the system cost of remote driving control, and reducing data transmission latency. Attached Figure Description

[0030] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein: Figure 1 This is a schematic diagram of a scenario for remote driving control in existing technologies; Figure 2 This is a schematic diagram of the control process for remote driving control in existing technologies; Figure 3 This is a schematic flowchart of the main steps of a remote driving control method according to an embodiment of this application; Figure 4 This is a schematic flowchart of the main steps of remote driving control according to one embodiment of the present application. Figure 5 This is a flowchart illustrating the main steps of a static road condition 3D reconstruction process according to one embodiment of this application. Figure 6 This is a schematic diagram illustrating the transmission process of dynamic target segmentation results according to a specific example of an embodiment of this application; Figure 7 This is a schematic flowchart of the main steps of vehicle trajectory prediction according to one embodiment of the present application. Figure 8This is a schematic diagram of the main steps for multi-sensory early warning based on the behavior prediction results of the vehicle terminal according to one embodiment of the present application. Figure 9 This is a flowchart illustrating the main steps of responding to vehicle-side requirements according to one embodiment of this application. Detailed Implementation

[0031] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.

[0032] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, and memory, and may also include software components, such as program code, or a combination of software and hardware. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular forms of the terms "a" and "this" can also include plural forms.

[0033] Driverless vehicles or manned intelligent vehicles generally possess autonomous driving systems, enabling them to drive autonomously in place of humans. However, due to technological or policy reasons, non-human driving technologies still pose safety risks in complex road conditions or certain specific scenarios. This necessitates human intervention, and virtual reality (VR) remote driving control technology offers a solution for such scenarios. For example... Figure 1 As shown, in the existing technology, VR remote driving technology can support both manned and unmanned intelligent vehicles.

[0034] like Figure 1 As shown in the image above, for manned intelligent driving vehicles, the intelligent driving system or cloud-based large-scale model can determine whether the driver poses a safety risk (such as fatigue, sudden illness, etc.) and road conditions, and thus decide whether to activate remote driving mode. Of course, the driver can also choose the remote driving option independently.

[0035] like Figure 1 As shown in the image below, for autonomous vehicles, the vehicle's own intelligent driving system or a cloud-based large model can detect road condition information, determine whether there are safety issues under the road conditions, and then decide whether to activate VR remote driving.

[0036] While VR remote driving technology can safeguard intelligent vehicles, it requires real-time 3D road condition reconstruction, which consumes significant system resources. Furthermore, the massive amount of data increases system latency. Specifically, for example... Figure 2 As shown, existing remote driving control technologies fuse information from multiple sensors (LiDAR, cameras, millimeter-wave radar, etc.) on a remote vehicle and transmit it to the cloud via low-latency networks such as 5G. The cloud first reconstructs the road conditions based on the multi-sensor information, then renders and encodes the left and right eye images based on the VR terminal's posture data, and pushes the encoded images to the VR system for display. In the above solutions, real-time 3D road condition reconstruction is the most time-consuming and resource-intensive.

[0037] Based on the aforementioned problems in the existing technology, this application proposes a new remote driving control scheme.

[0038] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a remote driving control method according to an embodiment of this application. Figure 1 As shown, the remote driving control method in this application embodiment mainly includes the following steps S101 to S105.

[0039] Step S101: Based on the vehicle-side environmental perception device, obtain the environmental perception results of the environment in which the vehicle is located.

[0040] In this embodiment, environmental perception can be performed based on the vehicle-mounted environmental perception device to obtain environmental perception results.

[0041] In one implementation, the vehicle can be equipped with multiple environmental sensing devices, such as cameras, LiDAR, and millimeter-wave radar. For cameras, multiple cameras can be installed on the vehicle to capture images of the surrounding environment, thereby achieving environmental perception.

[0042] In one implementation, the vehicle can be an intelligent driving vehicle, an autonomous vehicle, or the like.

[0043] Step S102: The vehicle-side obtains the dynamic target segmentation results of the environment it is in based on the environmental perception results and the map information of the environment in which the vehicle-side is located.

[0044] In this embodiment, the environmental perception results and the map information of the vehicle's environment can be combined to perform dynamic target segmentation on the environmental perception results, thereby obtaining dynamic target segmentation results.

[0045] In one embodiment, step S102 may further include steps S1021 and S1022: Step S1021: Based on the map information, obtain static road element data of the environment where the vehicle is located.

[0046] In this embodiment, to effectively reduce the resource consumption of the 3D reconstruction process, static road condition information can be recorded using map information (e.g., high-precision map information) during virtual reality rendering. When inputting a route on the vehicle, road element data (e.g., lane lines, slope, curvature, etc.) contained in the map information can be pre-loaded. Based on the map information, static road element data of the vehicle's environment can be obtained.

[0047] Step S1022: Based on the environmental perception results and the static road element data, and using the dynamic object detection algorithm, obtain the dynamic targets in the environment where the vehicle is located, and segment the dynamic targets to obtain the dynamic target segmentation results.

[0048] In this embodiment, static road element data can be combined with a dynamic object detection algorithm to distinguish moving people / objects and other dynamic targets from the environmental perception results, and the dynamic targets can be segmented to obtain dynamic target segmentation results.

[0049] In one implementation, for a vehicle containing multiple environmental sensing devices, for each environmental sensing device, dynamic targets can be segmented from the environmental sensing results of the environmental sensing devices, and the position coordinates of each dynamic target can be obtained. The segmented image and position coordinates corresponding to the dynamic target are used as the dynamic target segmentation result.

[0050] In this embodiment, such as Figure 6 As shown, for multiple environmental perception devices (e.g., cameras) on the vehicle, dynamic targets in each camera's view can be segmented, and the top-left corner coordinates of each dynamic target in the original image can be marked as its position coordinates. Based on the segmented view and position coordinates, the dynamic target segmentation result is obtained. For overlapping dynamic targets, during data transmission, the dynamic targets of each camera can be transmitted separately according to a pre-agreed camera order. For example, the dynamic targets of camera 1, including four dynamic targets (A, B, C, and D), can be transmitted first. After transmitting the dynamic targets according to their top-left corner coordinates in the original image, the dynamic targets of camera 2 can be transmitted, and so on.

[0051] Step S103: Encode the dynamic target segmentation result to obtain the dynamic target encoding result.

[0052] In this embodiment, the dynamic target segmentation result can be encoded to obtain a dynamic target encoding result. Commonly used encoding methods in the art can be used to encode the dynamic target segmentation result, and this application does not limit this method.

[0053] In one implementation, the dynamic target segmentation results corresponding to multiple environmental sensing devices can be converted to the same coordinate system; the converted dynamic target segmentation results corresponding to multiple environmental sensing devices can be fused to obtain a dynamic target fusion result; and the dynamic target fusion result can be encoded to obtain a dynamic target encoding result.

[0054] In this embodiment, the dynamic target segmentation results corresponding to multiple environmental perception devices on the vehicle can be transformed to the same coordinate system. This same coordinate system can be the global coordinate system, the vehicle-side coordinate system, or other commonly used coordinate systems. The dynamic target segmentation results corresponding to the environmental perception devices in the same coordinate system are then fused to obtain a dynamic target fusion result. Commonly used fusion algorithms in the art can be used to achieve the fusion of dynamic target segmentation results, thereby ensuring that when multiple environmental perception devices detect the same dynamic target or when dynamic targets overlap, the dynamic target is accurately and effectively identified.

[0055] Step S104: Obtain the virtual reality image rendering result.

[0056] In this embodiment, virtual reality image rendering can be performed based on a cloud server, using dynamic target encoding results and vehicle location information, to obtain a virtual reality image rendering result. The cloud server can communicate with both the virtual reality device and the vehicle. Alternatively, virtual reality image rendering can be performed based on the virtual reality device, using dynamic target encoding results and vehicle location information, to obtain a virtual reality image rendering result. The virtual reality device can communicate with the vehicle.

[0057] In one implementation, the communication connection can be a 5G (5th Generation Mobile Communication Technology) network communication connection.

[0058] In one implementation, the process of rendering virtual reality images based on a virtual reality device mainly includes: The vehicle transmits the dynamic target encoding result and the vehicle's location information to the virtual reality device. The virtual reality device performs 3D reconstruction based on the dynamic target encoding result to obtain the dynamic target 3D reconstruction result. Based on the vehicle's location information, it obtains the virtual static road conditions of the environment in which the vehicle is located, and obtains the static road condition 3D reconstruction result based on the virtual static road conditions. Based on the dynamic target 3D reconstruction result and the static road condition 3D reconstruction result, the virtual reality image rendering result is obtained.

[0059] In one implementation, the process of rendering virtual reality images based on a cloud server mainly includes: The vehicle transmits the dynamic target encoding results to the cloud server; the cloud server performs 3D reconstruction based on the dynamic target encoding results to obtain the dynamic target 3D reconstruction results, and obtains the virtual static road conditions of the vehicle's environment based on the vehicle's location information, and obtains the static road condition 3D reconstruction results based on the virtual static road conditions; based on the dynamic target 3D reconstruction results and the static road condition 3D reconstruction results, obtains the virtual reality image rendering results; and transmits the virtual reality image rendering results to the virtual reality device.

[0060] In one embodiment, the process of obtaining the virtual static road conditions of the vehicle's environment based on the vehicle's location information may include: looking up a table in a preset model library based on the vehicle's location information to obtain the static virtual road conditions of the vehicle's environment. The preset model library can be a model library obtained by pre-modeling and classifying virtual road conditions.

[0061] In one implementation, during the process of obtaining virtual reality image rendering results, virtual reality image rendering can be achieved by combining the posture information of the virtual reality device, the dynamic target encoding results, and the virtual static road conditions.

[0062] Step S105: The virtual reality device performs remote driving control on the vehicle based on the virtual reality image rendering results.

[0063] In this embodiment, the virtual reality device can render left and right eye images in a three-dimensional road condition based on the virtual reality rendered image results and the virtual reality device's posture data, thereby enabling remote driving control of the vehicle based on the rendered left and right eye images.

[0064] Based on the methods described in steps S101 to S105 above, this embodiment of the application acquires the environmental perception results of the vehicle's environment using the vehicle's environmental perception device; acquires the dynamic target segmentation results of the vehicle's environment based on the environmental perception results and map information of the vehicle's environment; encodes the dynamic target segmentation results to obtain dynamic target encoding results, so that the virtual reality device can obtain virtual reality image rendering results based on the dynamic target encoding results and the vehicle's location information, thereby realizing remote driving control of the vehicle by the virtual reality device. Through the above configuration, this embodiment of the application can acquire the dynamic target segmentation results of the vehicle's environment using the vehicle's environmental perception device, and the vehicle transmits only the dynamic target segmentation results for virtual reality image rendering. When performing 3D reconstruction on the cloud server or virtual reality device, 3D reconstruction of dynamic targets can be performed based on the dynamic target encoding results, and 3D reconstruction of virtual static road conditions can be performed based on the vehicle's location information. This eliminates the need for the vehicle to transmit the environmental perception road condition information to the cloud server or virtual reality device, effectively reducing the amount of environmental perception data transmitted, reducing the computing power requirements of the remote driving control process, reducing the system cost of remote driving control, and reducing data transmission latency.

[0065] The following is combined Figure 4 and Figure 5 The remote driving control method and road condition reconstruction process of the embodiments of this application will be further described.

[0066] like Figure 4 As shown, multi-sensor information can be acquired based on the vehicle terminal (i.e., the vehicle itself), and combined with map information of the vehicle's environment to perform dynamic target segmentation. The dynamically encoded target results obtained from this segmentation are then sent to the cloud via a 5G grid. Similarly, the VR terminal (i.e., the virtual reality device) can send its posture information to the cloud. The cloud server can combine the vehicle's location information to obtain virtual static road conditions of the vehicle's environment, achieving road condition reconstruction. Furthermore, it combines the virtual static road conditions, the encoded stream (i.e., the dynamic target encoding results), and the posture data of the virtual reality device to perform binocular image rendering, obtaining binocular images (i.e., virtual reality image rendering results), which are then displayed on the VR terminal, thereby enabling remote driving control of the vehicle based on the VR terminal. Figure 4 The dashed box in the figure marks the main difference between the remote driving control method of this application embodiment and the prior art. That is, when the vehicle terminal transmits data, it only transmits the dynamic target encoding result. For static road conditions, it uses high-precision maps to record static road condition information. When performing three-dimensional reconstruction of road conditions, it directly looks up the current virtual static road conditions in the model library based on the vehicle terminal's location information, which reduces the amount of data transmission for three-dimensional reconstruction of road conditions.

[0067] like Figure 5 As shown, the road condition reconstruction process in this embodiment includes: based on multiple cameras (camera 1, camera 2, ..., camera n) and map information on the vehicle, dynamic and static detection of the vehicle's environmental perception results is achieved, and dynamic image encoding and transmission are performed to realize the transmission of the dynamic target encoding results. Then, a cloud server or virtual reality device can perform dynamic image 3D reconstruction based on the dynamic target encoding results, construct virtual static 3D road conditions based on vehicle information, and merge the dynamic and static road condition reconstruction models to achieve VR image rendering.

[0068] In one implementation, such as Figure 7 As shown, vehicle trajectory prediction can be performed on the vehicle side, and then compensation can be made for the remote driving control process based on the trajectory prediction results. Specifically, compensation for the remote driving control process based on the vehicle-side trajectory prediction results can be achieved based on the following steps S201 to S205: Step S201: Based on the environmental perception results of multiple environmental perception devices on the vehicle, perform multi-source data fusion to obtain multi-source perception fusion results.

[0069] In this embodiment, environmental perception can be performed based on multiple environmental perception devices on the vehicle, and multi-source data fusion can be carried out to obtain a multi-source perception fusion result. The multi-source environmental data can include latitude and longitude / altitude, angular velocity / acceleration from the IMU (Inertial Measurement Unit), point cloud data from the LiDAR, image frames from the visual camera, and vehicle speed data from the wheel speed sensor. The environmental perception data from all environmental perception devices can be unified under the same time reference to ensure timestamp alignment of the multi-source environmental perception data. Kalman filtering can be used to remove GPS (Global Positioning System) signal jitter, IMU drift errors, etc., to achieve noise reduction filtering. The multi-source data fusion process can employ an extended Kalman filter algorithm, integrating the advantages of each environmental perception device. With GPS / IMU data as the core, LiDAR point cloud data assists in correcting position deviations, wheel speed data calibrates driving speed, and the output is a multi-source perception fusion result of vehicle "position (x, y, z) + attitude (roll angle, pitch angle, yaw angle) + motion parameters (vehicle speed, acceleration)".

[0070] Step S202: Based on the environmental perception results of multiple environmental perception devices on the vehicle, perform feature extraction to obtain the feature extraction results.

[0071] In this embodiment, static road element data of the vehicle's environment can be acquired; based on the static road element data and the environmental perception results, feature extraction is performed to obtain road constraint information of the vehicle's environment and motion trend information of the vehicle, which are used as feature extraction results.

[0072] Specifically, static road elements are mainly used to extract road information, which can be pre-loaded directly in the cloud. Based on the fused motion parameters, the vehicle speed change rate, steering angle, and acceleration trend can be extracted to capture vehicle driving patterns and finally output a structured feature vector containing road constraint information and vehicle motion trend information.

[0073] Step S203: Based on the multi-source perception fusion results and feature extraction results, perform trajectory prediction on the vehicle side to obtain the trajectory prediction results.

[0074] In this implementation, a Transformer model can be used, taking "multi-source perception fusion results + feature extraction results" as input, to learn the association between historical trajectories and environmental and motion features, thereby achieving vehicle trajectory prediction. During trajectory prediction, the prediction time window can be adaptively adjusted according to real-time network transmission latency (e.g., a 150ms delay can predict 300ms, and a 50ms delay can predict 150ms). Continuous vehicle trajectory points within the next 100-500ms are predicted (one point every 10ms). The trajectory prediction result can include trajectory confidence (e.g., if the confidence is below 80% in a sharp turn scenario, a redundant model is triggered to fill in the gap). During trajectory prediction, the "predicted trajectory" can be compared in real-time with the "vehicle position measured by LiDAR / vision," and the deviation value can be calculated. Simultaneously, road constraints (e.g., vehicles cannot exceed lane lines) are considered to determine the reasonableness of the prediction, and the deviation value is passed back to the prediction model to adjust the model parameters. Predicted trajectories exceeding road constraints are forcibly corrected to a reasonable range.

[0075] Step S204: Send the trajectory prediction results to the virtual reality device.

[0076] In this embodiment, the vehicle can send the trajectory prediction results to a virtual reality device.

[0077] Step S205: The virtual reality device displays the trajectory prediction results, thereby achieving compensation for the remote driving control at the vehicle end.

[0078] In this embodiment, the virtual reality device can convert the trajectory prediction results from the vehicle to a coordinate system recognizable by the remote driving cabin (virtual reality device), and output the trajectory prediction results in a visual format, marking them on the driving screen, thereby achieving compensation for the remote driving control process.

[0079] In one implementation, such as Figure 8 As shown, multi-sensory warnings for virtual reality devices can be generated based on the vehicle's perception of future road conditions, specifically including the following steps S301 to S303: Step S301: The vehicle obtains the future road condition perception results based on the environmental perception results and vehicle-road cooperative data.

[0080] In this embodiment, the vehicle-mounted device can obtain obstacle trajectory prediction results for its environment based on environmental perception results and static map information of the environment in which the vehicle is located. Based on vehicle-road cooperative data, it can obtain road condition prediction results for the environment in which the vehicle is located. Based on the obstacle trajectory prediction results and road condition prediction results, it can obtain future road condition perception results.

[0081] Specifically, the vehicle-side system can combine its environmental perception devices with static map information of the surrounding environment to analyze lane curvature, slope, and obstacles (such as pedestrians suddenly crossing the lane) in real time, predicting the trajectory of obstacles in the next 1-3 seconds (e.g., whether a pedestrian will enter the lane), as the obstacle trajectory prediction result. Based on vehicle-road cooperative data, it can obtain road conditions 500 meters to 1 kilometer ahead, as the road condition prediction result. Then, based on cloud-based traffic big data, it can integrate real-time traffic flow and historical congestion patterns to predict the traffic efficiency of the upcoming intersection, the speed of vehicles in front and behind, and other information. Based on the obstacle trajectory prediction result and the road condition prediction result, the vehicle-side system can obtain the future road condition perception result.

[0082] Step S302: The vehicle-mounted device obtains the behavior prediction results based on the future road condition perception results and the vehicle-mounted device's operating status, and sends the behavior prediction results to the virtual reality device.

[0083] In this implementation, the vehicle-side combines future road condition perception results with its own operational status to predict vehicle behavior. Specifically, the vehicle-side can predict "natural behavior without road condition interference" (such as constant speed in a straight line and gentle turning) based on a vehicle dynamics model (combining information such as vehicle speed, steering angle, and braking force). Then, it overlays future road condition constraints (such as "a 5° curvature curve 300 meters ahead") and uses a "Transformer + reinforcement learning" model to learn the remote driver's operating habits (such as aggressive lane changes and smooth braking), outputting a precise "trajectory prediction for the next 1-3 seconds" + "behavioral label" (such as "turn left in 1.2 seconds"). When the prediction confidence level is <85% (such as sudden extreme road conditions), it can automatically switch to "shortest safe path prediction" and trigger a high-intensity warning. When the predicted distance to the vehicles in front and behind enters a dangerous distance, it can mark it in advance in the virtual reality device and remind the remote driver.

[0084] Step S303: The virtual reality device provides multi-sensory warnings based on the behavior prediction results from the vehicle.

[0085] In this embodiment, the virtual reality device can achieve multi-sensory early warning based on the predicted behavior of the vehicle. Specifically, the core principles of the multi-sensory early warning process are: "early warning 0.3-0.5 seconds in advance" + "multi-sensory collaboration without disturbance," avoiding the abruptness of a single visual cue and allowing the remote driver to "naturally perceive and predict." Figure 8 As shown, the warning includes four warning modes, which can be selected according to the driver's comfort level.

[0086] - Visual warning methods: 1) [Spatiotemporal trajectory light strip]: A semi-transparent blue light strip is superimposed on the VR screen to render the vehicle's driving path 1 second in advance (e.g., the light strip bends to the left when turning left). The brightness of the light strip changes with the confidence level of the prediction (high confidence level → bright, low confidence level → flashing); 2) "Road condition label advance rendering": Future road condition risks (e.g., construction, sharp bends) are displayed in the VR screen 0.5 seconds in advance (e.g., construction areas are marked with a red semi-transparent frame, which gradually becomes clearer as the distance is zoomed in); 3) "Viewpoint linkage": When predicting lane changes / turns, the VR screen makes a slight angle shift 50ms in advance (e.g., the viewpoint shifts slightly to the left when turning left) to simulate the "centrifugal force prediction" of a real vehicle.

[0087] - Auditory warning: 1) "Directional sound effect + rhythm matching": When changing lanes, the earphone on the same side emits a "gradual beeping sound" (e.g., when turning left → the left earphone emits a sound, and the sound frequency increases as the lane change approaches); when decelerating, a "low and continuous sound" comes from the front (the volume is positively correlated with the deceleration); 2) "Semantic minimalist prompts": complex behaviors (such as "avoiding pedestrians + slowing down") are only expressed in 3 words or less (e.g., "avoid to the left" "slow down ahead") to avoid speech interference.

[0088] -Haptic warning: 1) "VR seat directional vibration": anticipate left turn → vibrate on the left side of the seat, right turn → vibrate on the right side, deceleration → vibrate in front of the seat. The vibration intensity = the "impact" of the predicted behavior (e.g., vibration during emergency braking > vibration during gentle braking). The vibration is triggered 0.3 seconds in advance; 2) "Controller / seat belt linkage": if the passenger is wearing a VR controller (or the VR seat integrates a seat belt), the controller / seat belt on the same side will tighten slightly when changing lanes, simulating "early feedback of centrifugal force in a real vehicle".

[0089] - Spatial perception warning: "Sound field / air pressure micro-adjustment": When anticipating an uphill slope, the VR device's built-in micro air pressure module slightly increases the pressure (simulating the "pressure" of climbing); when anticipating high-speed driving, the sound field switches to "gradually enhanced wind noise" to simulate speed changes in advance.

[0090] Remote driver feedback calibration: 1) "Real-time feedback acquisition": The VR device has a built-in heart rate sensor and head motion tracking. If the remote driver's heart rate suddenly increases (>100 beats / min) or the head shakes frequently (indicating discomfort), the warning intensity will be automatically reduced (e.g., reduced vibration, reduced sound volume), and the warning advance time will be shortened (from 0.5 seconds to 0.3 seconds); 2) "Personalized configuration": The remote driver can set the "warning sensitivity" in the VR interface (e.g., elderly mode → 0.6 seconds in advance, low intensity; young mode → 0.3 seconds in advance, high intensity). The system records preferences and automatically adapts.

[0091] The VR multi-sensory warning solution has the following advantages: 1) It is not just "trajectory prediction", but also "scenario-based behavior". Combining future road conditions and driving intentions, the prediction results are labeled with "behavioral tags" (such as "avoidance lane change"). The VR warning is more accurately matched to the scene, rather than simply prompting "you need to change lanes"; 2) At the same time, the warning is transformed into part of the VR scene (trajectory light strip, view linkage, directional vibration). The remote driver perceives it naturally and does not destroy the immersion; 3) It reuses the previous multi-source sensor fusion and low-latency transmission technology, which is highly practical and does not require a lot of additional hardware.

[0092] In one implementation, such as Figure 9 As shown, the remote driving control process can respond to the needs of passengers inside and outside the vehicle, and may include the following steps S401 to S403: Step S401: Obtain the requirement type generated by the vehicle side.

[0093] In this implementation, passengers can generate request types through triggering. For example, passengers can trigger request types by pressing a physical shortcut on the armrest of the seat inside the vehicle. They can also trigger them by inputting voice commands (such as "slow down if you feel carsick" or "turn the heat up a bit").

[0094] Step S402: Send a request prompt to the virtual reality device according to the request type.

[0095] In this embodiment, the method of request prompting can be determined based on the type of request and the current driving scenario on the vehicle; based on the method of request prompting, the request prompt is sent to the virtual reality device.

[0096] Specifically, the system can determine the method of request notification based on the passenger's needs and the current driving scenario in the vehicle, and then send the notification to the virtual reality device accordingly. Specifically, the vehicle can intelligently categorize needs into "urgent needs" (e.g., motion sickness, physical discomfort, hazard avoidance warnings) and "non-urgent needs" (e.g., adjusting temperature, changing music, changing route). Different notifications can be sent to the virtual reality device for different categories. The vehicle can also send notifications based on the current driving scenario. In high-risk scenarios (e.g., the vehicle is making a sharp turn, meeting oncoming traffic, or avoiding obstacles), non-urgent notifications will be temporarily suppressed, with only minor notifications for urgent needs remaining. In stable scenarios (e.g., driving at a constant speed in a straight line, waiting at a traffic light), the complete notification will be pushed to the virtual reality device—including semi-transparent icons at the edge of the field of vision (emergency red flashing, non-emergency blue silent) and corresponding 3D spatial sound effects.

[0097] Step S403: The virtual reality device responds to the demand prompts.

[0098] In this implementation, the virtual reality device can respond to requests based on prompts. For example, urgent requests can be confirmed with a single click using the VR controller, or the system can automatically execute a preset solution; non-urgent requests can be processed in batches or confirmed individually when it is safe to do so. After processing, the results will be synchronized to the passenger's device and communicated to the passenger via voice prompts and VR visual text (or slight seat vibration), forming a complete closed loop of "making a request → being processed → receiving feedback".

[0099] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.

[0100] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0101] Another aspect of this application provides a computer-readable storage medium.

[0102] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the remote driving control method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described remote driving control method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a storage device comprising various electronic devices, such as a magnetic disk, hard disk, optical disk, flash memory, read-only memory, random access memory, etc. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0103] Another aspect of this application provides a vehicle-side solution.

[0104] In one embodiment of the vehicle terminal according to this application, the vehicle terminal may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program, which, when executed by the at least one processor, implements the method described in any of the above embodiments. The vehicle terminal described in this application may include driving equipment, intelligent vehicles, robots, and other devices.

[0105] In some embodiments of this application, the vehicle may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. Optionally, the vehicle may further include an autonomous driving system for guiding the vehicle to drive autonomously or assisting in driving. The processor communicates with the sensor and / or the autonomous driving system to perform the methods described in any of the above embodiments. The processor may be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor may be implemented in software, in hardware, or a combination of both.

[0106] Another aspect of this application provides a virtual reality device.

[0107] In one embodiment of a virtual reality device according to this application, the virtual reality device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described in any of the above embodiments.

[0108] Another aspect of this application provides a cloud server.

[0109] In one embodiment of a cloud server according to this application, the virtual reality device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described in any of the above embodiments.

[0110] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.

Claims

1. A remote driving control method, characterized in that, The method is applied to a vehicle-mounted terminal; the vehicle-mounted terminal is communicatively connected to a virtual reality terminal; the method includes: Based on the vehicle-mounted environmental perception device, obtain the environmental perception results of the environment in which the vehicle is located; Based on the environmental perception results and the map information of the environment where the vehicle is located, obtain the dynamic target segmentation results of the environment where the vehicle is located; The dynamic target segmentation result is encoded to obtain a dynamic target encoding result, so that the virtual reality device can obtain a virtual reality image rendering result based on the dynamic target encoding result and the vehicle's position information, thereby realizing remote driving control of the vehicle.

2. The remote driving control method according to claim 1, characterized in that, The step of obtaining the dynamic target segmentation result of the vehicle's environment based on the environmental perception result and the map information of the intelligent environment includes: Based on the map information, obtain static road element data of the environment in which the vehicle is located; Based on the environmental perception results and the static road element data, and using a dynamic object detection algorithm, the dynamic targets in the environment where the vehicle is located are obtained, and the dynamic targets are segmented to obtain the dynamic target segmentation results.

3. The remote driving control method according to claim 2, characterized in that, The step of segmenting the dynamic target to obtain the dynamic target segmentation result includes: For each environmental sensing device, the dynamic target is segmented from the environmental sensing result of the environmental sensing device, and the position coordinates of each dynamic target are obtained. The segmented image corresponding to the dynamic target and the position coordinates are used as the dynamic target segmentation result.

4. The remote driving control method according to claim 2, characterized in that, The step of encoding the dynamic target segmentation result to obtain the dynamic target encoding result includes: The dynamic target segmentation results corresponding to multiple environmental sensing devices are transformed to the same coordinate system; The dynamic target segmentation results corresponding to the multiple environmental sensing devices after conversion are fused to obtain a dynamic target fusion result; The dynamic target fusion result is encoded to obtain the dynamic target encoding result.

5. The remote driving control method according to claim 1, characterized in that, The step of enabling the virtual reality device to obtain a virtual reality image rendering result based on the dynamic target encoding result and the vehicle-side location information includes: The dynamic target encoding result and the vehicle's location information are transmitted to the virtual reality device, so that the virtual reality device performs the following steps: Three-dimensional reconstruction is performed based on the dynamic target encoding results to obtain the dynamic target three-dimensional reconstruction results; Based on the vehicle's location information, the virtual static road conditions of the environment in which the vehicle is located are obtained, and based on the virtual static road conditions, the three-dimensional reconstruction results of the static road conditions are obtained. Based on the dynamic target 3D reconstruction results and the static road condition 3D reconstruction results, obtain the virtual reality image rendering results; or, The dynamic target encoding result is transmitted to the cloud server, so that the cloud server performs the following steps: Three-dimensional reconstruction is performed based on the dynamic target encoding results to obtain the dynamic target three-dimensional reconstruction results; Based on the location information, the virtual static road conditions of the environment where the vehicle is located are obtained, and based on the virtual static road conditions, the three-dimensional reconstruction results of the static road conditions are obtained. Based on the dynamic target 3D reconstruction results and the static road condition 3D reconstruction results, the virtual reality image rendering results are obtained; The virtual reality image rendering result is transmitted to the virtual reality device.

6. The remote driving control method according to claim 1, characterized in that, The method further includes: Based on the environmental perception results of multiple environmental perception devices on the vehicle, multi-source data fusion is performed to obtain multi-source perception fusion results; Based on the environmental perception results from multiple environmental perception devices on the vehicle, feature extraction is performed to obtain feature extraction results; Based on the multi-source perception fusion result and the feature extraction result, the trajectory of the vehicle is predicted to obtain the trajectory prediction result; The trajectory prediction result is sent to the virtual reality device so that the virtual reality device can display the trajectory prediction result, thereby achieving compensation for the remote driving control of the vehicle.

7. The remote driving control method according to claim 6, characterized in that, Based on the environmental perception results from the multiple environmental perception devices on the vehicle, feature extraction is performed to obtain the feature extraction results, including: Obtain static road element data of the environment in which the vehicle is located; Based on the static road element data and the environmental perception results, feature extraction is performed to obtain the road constraint information of the environment in which the vehicle is located and the motion trend information of the vehicle, which are used as the feature extraction results.

8. The remote driving control method according to claim 1, characterized in that, The method further includes: Based on the environmental perception results and the vehicle-road cooperative data, the future road condition perception results of the vehicle are obtained; Based on the future road condition perception results and the vehicle's operating status, obtain the vehicle's behavior prediction results; The behavior prediction result is sent to the virtual reality device so that the virtual reality device can provide multi-sensory warnings based on the behavior prediction result.

9. The remote driving control method according to claim 8, characterized in that, The step of obtaining the future road condition perception result of the vehicle based on the environmental perception result and the vehicle-road cooperative data includes: Based on the environmental perception results and the static map information of the environment where the vehicle is located, the obstacle trajectory prediction results of the environment where the vehicle is located are obtained. Based on the vehicle-road cooperative data, obtain the road condition prediction results of the environment in which the vehicle is located; The future road condition perception result is obtained based on the obstacle trajectory prediction result and the road condition prediction result.

10. The remote driving control method according to claim 1, characterized in that, The method further includes: Obtain the type of requirement generated by the vehicle; Based on the demand type, a demand prompt is sent to the virtual reality device so that the virtual reality device can respond to the demand prompt.

11. The remote driving control method according to claim 10, characterized in that, Sending a request prompt to the virtual reality device according to the request type includes: The method of requesting information is determined based on the type of request and the current driving scenario on the vehicle. Based on the aforementioned request prompting method, the request prompt is sent to the virtual reality device.

12. A remote driving control method, characterized in that, The method is applied to virtual reality devices; The virtual reality device is communicatively connected to the vehicle; the method includes: Obtain virtual reality image rendering results; wherein the virtual reality image rendering results are obtained based on the dynamic target encoding results generated by the vehicle and the location information of the vehicle, wherein the dynamic target encoding results are obtained in the following manner: based on the environmental perception device of the vehicle, obtain the environmental perception results of the environment in which the vehicle is located; based on the environmental perception results and the map information of the environment in which the vehicle is located, obtain the dynamic target segmentation results of the environment in which the vehicle is located; encode the dynamic target segmentation results. Based on the virtual reality image rendering results, remote driving control is performed on the vehicle.

13. The remote driving control method according to claim 12, characterized in that, The process of obtaining virtual reality image rendering results includes: The generated dynamic target encoding result and the vehicle's location information are obtained from the vehicle terminal. Three-dimensional reconstruction is performed based on the dynamic target encoding results to obtain the dynamic target three-dimensional reconstruction results; Based on the location information, the virtual static road conditions of the environment where the vehicle is located are obtained, and based on the virtual static road conditions, the three-dimensional reconstruction results of the static road conditions are obtained. The virtual reality image rendering result is obtained based on the dynamic target 3D reconstruction result and the static road condition 3D reconstruction result.

14. The remote driving control method according to claim 12, characterized in that, The process of obtaining virtual reality image rendering results includes: Receive virtual reality image rendering results from a cloud server, wherein the virtual reality image rendering results are obtained by the cloud server in the following manner: Obtain the dynamic target encoding result generated by the vehicle and the location information of the vehicle; Three-dimensional reconstruction is performed based on the dynamic target encoding results to obtain the dynamic target three-dimensional reconstruction results; Based on the location information, the virtual static road conditions of the environment where the vehicle is located are obtained, and based on the virtual static road conditions, the three-dimensional reconstruction results of the static road conditions are obtained. The virtual reality image rendering result is obtained based on the dynamic target 3D reconstruction result and the static road condition 3D reconstruction result.

15. The remote driving control method according to claim 12, characterized in that, The method further includes: Obtain the trajectory prediction results generated by the vehicle; The trajectory prediction results are displayed to compensate for the remote driving control of the vehicle.

16. The remote driving control method according to claim 12, characterized in that, The method further includes: Obtain the behavior prediction results from the vehicle end; Based on the predicted behavior, a multi-sensory early warning system is implemented.

17. The remote driving control method according to claim 12, characterized in that, The method further includes: Obtain the requirement prompt generated by the vehicle; wherein the requirement prompt is generated based on the requirement type of the vehicle. Based on the aforementioned requirement prompts, the vehicle-side requirement response is performed.

18. A remote driving control method, characterized in that, The method is applied to a cloud server; The cloud server is connected to both the virtual reality device and the vehicle-mounted communication device; the method includes: Obtain the dynamic target encoding result generated by the vehicle and the location information of the vehicle; Based on the dynamic target encoding result and the vehicle's location information, a virtual reality image rendering result is obtained, and the virtual reality image rendering result is sent to the virtual reality device so that the virtual reality device can remotely control the vehicle based on the virtual reality image rendering result. The dynamic target encoding result is obtained as follows: based on the vehicle's environmental perception device, the environmental perception result of the environment in which the vehicle is located is obtained; based on the environmental perception result and the map information of the environment in which the vehicle is located, the dynamic target segmentation result of the environment in which the vehicle is located is obtained; and the dynamic target segmentation result is encoded.

19. The remote driving control method according to claim 18, characterized in that, The step of obtaining the virtual reality image rendering result based on the dynamic target encoding result includes: Three-dimensional reconstruction is performed based on the dynamic target encoding results to obtain the dynamic target three-dimensional reconstruction results; Obtain the location information of the vehicle; based on the location information, obtain the virtual static road conditions of the environment in which the vehicle is located, and based on the virtual static road conditions, obtain the three-dimensional reconstruction results of the static road conditions; The virtual reality image rendering result is obtained based on the dynamic target 3D reconstruction result and the static road condition 3D reconstruction result.

20. A vehicle end, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the remote driving control method according to any one of claims 1 to 11.

21. A virtual reality device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the remote driving control method according to any one of claims 12 to 17.

22. A cloud server, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program, which, when executed by the at least one processor, implements the remote driving control method according to any one of claims 18 to 19.

23. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the remote driving control method according to any one of claims 1 to 19.

Images