Remote driving method based on remote desktop control technology and related device

Abstract

This application provides a remote driving method and related equipment based on remote desktop control technology, belonging to the field of remote driving technology. The method includes: collecting driving environment data and vehicle status data through vehicle body sensors; rendering a local operation screen based on the driving environment data and vehicle status data to obtain a driving operation video stream; synchronizing incremental pixels of the driving operation video stream to a remote operating terminal using remote desktop control technology, and collecting the synchronization time of the pixel increment process; dynamically allocating transmission redundancy to key frames in the driving operation video stream based on the synchronization time; encoding the key frames according to the transmission redundancy to obtain redundant data packets; and synchronizing the incremental pixels of the driving operation video stream and the redundant data packets together to the remote operating terminal. This application can reduce the data interaction latency of remote driving and improve the safety of remote driving.

Description

Technical Field

[0001] This application relates to the field of remote driving technology, and in particular to a remote driving method and related equipment based on remote desktop control technology. Background Technology

[0002] With the development of autonomous driving technology, remote driving uses onboard controllers to capture, encode, and transmit video to a remote control center. However, existing traditional video streaming solutions generally suffer from drawbacks such as high latency, high bandwidth requirements, and fluctuating video quality. Especially in poor network environments, the stability of the video stream is difficult to guarantee, seriously affecting the remote driver's real-time control of the vehicle and potentially even inducing safety risks. Summary of the Invention

[0003] The main objective of this application is to propose a remote driving method and related equipment based on remote desktop control technology, aiming to reduce the data interaction latency of remote driving and improve the safety of remote driving.

[0004] To achieve the above objectives, one aspect of this application proposes a remote driving method based on remote desktop control technology, comprising the following steps: The vehicle body sensors collect driving environment data and vehicle status data; The driving environment data and the vehicle status data are rendered locally to obtain a driving operation video stream. The incremental pixels of the driving operation video stream are synchronized to the remote operation terminal using remote desktop control technology, and the synchronization time of the pixel increment process is collected. Based on the synchronization time, the key frames in the driving operation video stream are dynamically allocated for transmission redundancy. The keyframes are encoded according to the transmission redundancy to obtain redundant data packets; The incremental pixels of the driving operation video stream are synchronized together with the redundant data packets to the remote operation terminal.

[0005] In some embodiments, synchronizing incremental pixels of the driving operation video stream to a remote operating terminal via remote desktop control technology includes the following steps: Determine the expected load overhead by maximizing the data payload efficiency of the message structure; The incremental pixels of the driving operation video stream are encoded to obtain encoded data, and the encoded data is structured and encapsulated according to the expected load overhead to obtain driving message data. The driving message data is synchronized to the remote operating terminal using remote desktop control technology.

[0006] In some embodiments, the step of dynamically allocating transmission redundancy for key frames in the driving operation video stream based on the synchronization time includes the following steps: Calculate the response delay gain based on the synchronization time. The transmission redundancy of key frames in the driving operation video stream is dynamically allocated based on the response delay gain; wherein the transmission redundancy of the key frames is inversely proportional to the response delay gain.

[0007] In some embodiments, the remote driving method based on remote desktop control technology further includes the following steps: Monitoring the channel quality index values ​​of the command transmission channel; If the channel quality index value is less than the expected quality threshold, driving control commands are transmitted first, followed by cockpit interaction commands.

[0008] In some embodiments, the remote driving method based on remote desktop control technology further includes the following steps: Calculate the total end-to-end latency from the start of data acquisition to synchronization with the remote operating terminal; The delay compensation displacement is determined based on the total end-to-end delay, the current vehicle speed, and the current longitudinal acceleration of the vehicle. The time delay compensation displacement and the vehicle's current heading angle are input into the prediction model to predict the vehicle trajectory, and driving control commands are determined based on the predicted vehicle trajectory.

[0009] In some embodiments, the remote driving method based on remote desktop control technology further includes the following steps: The real vehicle trajectory is obtained based on the aforementioned remote desktop control technology; The prediction error is calculated based on the actual vehicle trajectory transmitted back and the predicted vehicle trajectory at the same time point. The prediction model parameters are adjusted based on the prediction error.

[0010] To achieve the above objectives, another aspect of this application proposes a remote driving system based on remote desktop control technology, comprising: The first module is used to collect driving environment data and vehicle status data through body sensors; The second module is used to perform local operation screen rendering on the driving environment data and the vehicle status data to obtain a driving operation video stream. The third module is used to synchronize the incremental pixels of the driving operation video stream to the remote operation terminal through remote desktop control technology, and to collect the synchronization time of the pixel increment process. The fourth module is used to dynamically allocate transmission redundancy to key frames in the driving operation video stream based on the synchronization time. The fifth module is used to encode the keyframe according to the transmission redundancy to obtain redundant data packets; The sixth module is used to synchronize the incremental pixels of the driving operation video stream with the redundant data packets to the remote operation terminal.

[0011] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0012] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0014] The embodiments of this application include at least the following beneficial effects: This application provides a remote driving method, system, electronic device, storage medium, and program product based on remote desktop control technology. This solution directly renders the driving operation video stream from driving environment data and vehicle status data locally. The incremental pixels of the driving operation video stream are synchronized to the remote operating terminal via remote desktop control technology. This method of directly rendering the screen locally and synchronizing it to the remote operating terminal via remote desktop technology avoids multi-level forwarding and encoding / decoding losses, reducing data interaction latency compared to traditional remote driving data interaction solutions. Furthermore, this solution collects the synchronization time of the incremental pixel process, dynamically allocates transmission redundancy for key frames in the driving operation video stream based on the synchronization time, encodes the key frames according to the transmission redundancy to obtain redundant data packets, and synchronizes the incremental pixels of the driving operation video stream and the redundant data packets together to the remote operating terminal. This prioritizes the remote synchronization of key frames when the synchronization time is high, thereby improving remote driving safety. Attached Figure Description

[0015] Figure 1 This is a flowchart of a remote driving method based on remote desktop control technology provided in an embodiment of this application; Figure 2 This is a flowchart of the remote driving interaction logic based on remote desktop control technology provided in the embodiments of this application; Figure 3This is an overall flowchart of remote operation provided in the embodiments of this application; Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0018] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0019] This application provides a remote driving method and related equipment based on remote desktop control technology. This solution can reduce the data interaction latency of remote driving and improve the safety of remote driving.

[0020] The remote driving method based on remote desktop control technology provided in this application relates to the field of remote driving technology. This remote driving method based on remote desktop control technology can be applied to a system composed of an in-vehicle terminal and a remote driving terminal. The remote driving terminal can be a terminal device. In some embodiments, the remote terminal device can be a smartphone, tablet computer, laptop computer, or desktop computer, but is not limited to these.

[0021] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0022] Figure 1 This is an optional flowchart of a remote driving method based on remote desktop control technology provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0023] S101 collects driving environment data and vehicle status data through body sensors; S102, render the local operation screen based on the driving environment data and vehicle status data to obtain the driving operation video stream; S103 uses remote desktop control technology to synchronize incremental pixels of the driving operation video stream to the remote operation terminal and collects the synchronization time of the pixel increment process. S104, dynamically allocate transmission redundancy to key frames in the driving operation video stream based on synchronization time. S105, Encode the keyframes according to the transmission redundancy to obtain redundant data packets; S106 synchronizes the incremental pixels of the driving operation video stream with redundant data packets to the remote operation terminal.

[0024] Steps S101 to S106, as illustrated in this embodiment, can be specifically applied to an in-vehicle terminal. It directly renders the driving environment data and vehicle status data locally to obtain a driving operation video stream. Incremental pixels of the driving operation video stream are synchronized to the remote operating terminal via remote desktop control technology. This method of directly rendering the screen locally and synchronizing it to the remote operating terminal via remote desktop technology avoids multi-level forwarding and encoding / decoding losses, reducing data interaction latency compared to traditional remote driving data interaction schemes. Furthermore, the in-vehicle terminal collects the synchronization time of the incremental pixel process, dynamically allocates transmission redundancy for key frames in the driving operation video stream based on the synchronization time, encodes the key frames according to the transmission redundancy to obtain redundant data packets, and synchronizes the incremental pixels of the driving operation video stream and the redundant data packets together to the remote operating terminal. This prioritizes the remote synchronization of key frames when the synchronization time is high, thereby improving remote driving safety.

[0025] In step S101 of some embodiments, driving environment data refers to the environmental video stream collected by the vehicle body camera, and vehicle status data refers to vehicle status data such as the vehicle's current position, speed, acceleration, and driving direction.

[0026] In one specific embodiment, a multi-sensor integrated system is deployed on the vehicle side. Each sensor is connected to the vehicle-side domain controller via an in-vehicle Ethernet or CAN bus. To ensure timing consistency, the trigger sampling frequency of each sensor is set to be the same.

[0027] set up The total amount of raw data generated by the sensor per unit time is calculated using the following formula:

[0028] in, For the number of sensors, For the first The resolution of each sensor, For the number of channels, This represents the sampling frame rate. The amount of raw data collected can be used to initialize bandwidth allocation strategies.

[0029] In another specific embodiment, considering that sensor data acquisition under natural driving conditions may introduce noise, this embodiment preprocesses the acquired vehicle underlying signals (such as driving environment data and vehicle status data). A Butterworth low-pass filter is used to eliminate random noise caused by vehicle vibrations, and this filter is represented as follows: ; in, The amplitude response of the filter. Angular frequency, The cutoff angular frequency, This represents the filter order.

[0030] In some embodiments, please refer to Figure 2 The flowchart shown illustrates the remote driving interaction logic based on remote desktop control technology. Data from cameras around the vehicle needs to be converted via a deserializer before being connected to the SOC (H.265) core processing module of the autonomous driving controller. The timing logic for video processing is initialized within the SOC, while the cockpit-side driving software and remote desktop service module are simultaneously initialized within the autonomous driving controller. Subsequently, the video stream can be sent to the cloud cockpit (i.e., the remote control terminal) via the operator network through the remote desktop service, thereby enabling remote driving.

[0031] In step S102 of some embodiments, after collecting data, the domain controller directly renders the driving environment data and vehicle status data locally to obtain a driving operation video stream. Specifically, the rendered driving operation video stream includes a real-time vehicle driving video stream (including driving environment information and vehicle status information) and an operation interface. The operation interface may include some vehicle operation controls, and control commands can be input to the vehicle by triggering the operation controls.

[0032] In step S103 of some embodiments, incremental pixels of the driving operation video stream are synchronized to the remote operation via remote desktop control technology. In this embodiment, extremely low operation feedback latency is achieved by executing video processing logic on the vehicle-side controller and using the remote desktop protocol to complete interface synchronization. Specifically, the vehicle-side domain controller collects data in real time from sensors such as cameras and performs image encoding and graphical rendering directly on the local computing node, generating the vehicle driving video stream and operation interface within the controller. This process restricts video display to the local operating environment of the domain controller, avoiding the multi-level forwarding and encoding / decoding losses that occur during traditional video stream transmission over the network.

[0033] In step S104 of some embodiments, while synchronizing the incremental pixels of the driving operation video stream to the remote operating terminal using remote desktop control technology, the synchronization time of the pixel increment process is collected. The synchronization time can be obtained based on the timestamp analysis of each data during the data interaction process. Based on the synchronization time, the transmission redundancy of key frames in the driving operation video stream is dynamically allocated. Specifically, if the synchronization time is too long, the transmission redundancy of key frames can be increased, thereby improving the reliability of the remote operating terminal in restoring the display of the driving operation video stream, and thus improving remote driving safety. In this embodiment, key frames refer to video frames containing important target events, such as surrounding moving objects and traffic lights.

[0034] In steps S105 to S106 of some embodiments, keyframes are encoded according to transmission redundancy to obtain redundant data packets. The incremental pixels of the driving operation video stream and the redundant data packets are then synchronized to the remote control terminal. In cases where synchronization takes too long (i.e., network conditions are poor), the remote control terminal can also recover the keyframes based on the redundant data packets, and then display the video stream based on these keyframes, thereby improving the robustness of remote control operation.

[0035] According to some embodiments of this application, please refer to Figure 3 The overall remote operation flowchart shown first illustrates the process: the vehicle's domain controller first captures video and then sends it to the remote cockpit via a 5G-TBOX. The remote cockpit's interactive device then displays the environmental video and vehicle status, obtains the driver's input commands, and synchronizes these commands back to the vehicle. Simultaneously, either the vehicle or the remote cockpit can send vehicle status information to the cloud server for synchronized display via HTTP or MQTT proxy.

[0036] In some embodiments, the step S103 of synchronizing incremental pixels of the driving operation video stream to the remote operation terminal via remote desktop control technology may include, but is not limited to, the following steps: S201, determine the expected load overhead by maximizing the data payload efficiency of the message structure; S202, the incremental pixels of the driving operation video stream are encoded to obtain encoded data, and the encoded data is structured and encapsulated according to the expected load overhead to obtain driving message data; S203 uses remote desktop control technology to synchronize driving message data to a remote operating terminal.

[0037] In this embodiment, the passive transmission of traditional video streams can be transformed into active exchange based on state information, thus reducing the burden on data transmission in weak network environments. Specifically, in this embodiment, data transmission can employ lightweight data communication protocols such as MQTT or WebSocket. The system introduces a data payload efficiency evaluation mechanism, which minimizes the header overhead of the protocol link layer and application layer, and performs compact encoding of the payload structure of multi-sensor structured data to maximize the proportion of effective data in the transmission packet, thereby improving data payload efficiency. The formula is expressed as follows: ; in, This represents the expected payload overhead (i.e., the length of the encoded data for incremental pixels in the structured encapsulated driving operation video stream). This represents the header overhead of the protocol link layer and application layer. By reducing... And optimize The encapsulation structure enables rapid exchange of critical control information under network fluctuation conditions. This embodiment employs this high payload-to-weight ratio encapsulation strategy, which significantly reduces redundant load on network transmission, allowing critical control information to still be exchanged and resolved at the millisecond level even under network fluctuation conditions.

[0038] In some embodiments, step S104 may include, but is not limited to, the following steps: S301, calculate the response delay gain based on the synchronization time; S302, dynamically allocate transmission redundancy for key frames in the driving operation video stream based on response delay gain; wherein, the transmission redundancy of key frames is inversely proportional to the response delay gain.

[0039] In this embodiment, a direct interactive connection from the client to the vehicle-side domain controller is used to synchronize the locally rendered interface pixels to the cabin-side display in real time. To quantify the contribution of this architecture to real-time performance, the response latency gain is calculated using the following formula: ; in, Represents the total transmission latency in traditional video streaming mode. For the corresponding software buffering time, This embodiment of the application shows the synchronization time for the remote desktop to only synchronize pixel increments. By reducing pixel synchronization overhead, the operating accuracy of the remote driver is made close to that of on-site driving.

[0040] Please continue to refer to Figure 2 The vehicle-mounted computing chip (SOC) generates a video stream, which is then used by local software to synthesize a UI interface. This UI is subsequently mirrored to the 5G cloud cockpit via a remote desktop service, enabling integrated interaction of video, status, and commands. The aforementioned response latency gain is used to verify the real-time advantages of the "local rendering" and "mirror synchronization" architecture compared to traditional video transmission methods. Based on the response latency gain, the transmission redundancy of key frames is dynamically adjusted to ensure that the lag in the visual feedback received by the driver is below the operational safety threshold, thereby improving the safety of remote operation. Specifically, when a decrease in reliability indicators is detected (i.e., a decrease in response latency gain), the transmission redundancy of key frames is automatically increased, thus maximizing the operational determinism and robustness of remote control without sacrificing overall real-time performance.

[0041] In some embodiments, the remote driving method based on remote desktop control technology in this application may also include, but is not limited to, the following steps: S401, monitoring the channel quality index value of the command transmission channel; S402: When the channel quality index value is less than the expected quality threshold, drive control commands are transmitted first, followed by cockpit interaction commands.

[0042] In this embodiment, the remote driving method is mainly applied to a remote operation terminal. The command transmission channel can refer to a network layer channel or link layer channel used specifically for transmitting control commands in remote desktop control technology. Channel quality indicators can be measured by network bandwidth, link rate fluctuation, and network throughput bottlenecks, etc., and this embodiment does not impose specific limitations. Specifically, the channel quality indicator value can be evaluated by comparing the bandwidth requirements of traditional video streaming modes with those of current lightweight protocol transmission in real time. The greater the difference, the less bandwidth is used in the data transmission process of this embodiment, i.e., the higher the channel quality indicator value.

[0043] This embodiment further mitigates the negative impact of network transmission latency by prioritizing and scheduling control commands. Specifically, driving control commands related to vehicle dynamics safety (such as braking and steering) are defined as high priority, while ordinary cockpit interaction commands (such as playing music and adjusting seats) are defined as low priority. When network bandwidth constraints or link fluctuations are detected, i.e., when the channel quality index value is lower than the expected quality threshold, high-priority commands are granted absolute transmission rights, allowing them to dynamically surpass queued ordinary data packets for priority transmission. Through this mechanism, the end-to-end latency of critical driving signals can be maintained within a safe threshold under complex network conditions, effectively avoiding the risk of control failure due to data congestion.

[0044] According to some embodiments of this application, based on a real-time bandwidth assessment mechanism, network throughput bottlenecks can be dynamically identified, and differentiated transmission priorities can be assigned to pixel update messages and control command messages in the remote desktop protocol at the vehicle end. When the network bandwidth is detected to be lower than a preset threshold, the system prioritizes ensuring the real-time reliable interaction of high-priority control commands, thereby maintaining the continuity and stability of remote driving operations.

[0045] In some embodiments, the remote driving method based on remote desktop control technology in this application may also include, but is not limited to, the following steps: S501, calculates the total end-to-end latency from the start of data acquisition to synchronization with the remote operation terminal; S502, determine the time delay compensation displacement based on the total end-to-end time delay, the current vehicle speed, and the current longitudinal acceleration of the vehicle; S503 inputs the time delay compensation displacement and the vehicle's current heading angle into the prediction model to predict the vehicle trajectory, obtains the predicted vehicle trajectory, and determines driving control commands based on the predicted vehicle trajectory.

[0046] In this embodiment, in order to accurately measure the time delay from "vehicle-side perception" to "cabin-side display", a PTP-based clock synchronization mechanism needs to be established between the cabin and the vehicle.

[0047] set up For the cabin end clock, For the vehicle-end clock, synchronization deviation Represented as:

[0048] in, The time points for sending and receiving synchronization messages at the cabin end. , This represents the response time point at the vehicle end. The synchronization deviation is used to calibrate the reference error of the cabin and vehicle end clocks. This offset is used to correct the timestamp of real-time data packets, improving the accuracy of measurement delay.

[0049] This embodiment establishes a theoretical latency decomposition model for the remote driving link, which serves as the benchmark for subsequent latency optimization, i.e., the end-to-end total latency. Defined as: ; in, The corresponding time is the time it takes for the hardware link from the camera to the vehicle-side computing chip SOC interface, through the deserializer. The computation time for video data processing, timing logic calculation, and generating the local rendering interface in the corresponding SOC core module; Due to network latency, The latency is the time required for remote desktop pixel synchronization. The lower the latency, the higher the security redundancy, and the stronger the system's ability to cope with emergencies.

[0050] This embodiment introduces a motion prediction algorithm, which combines historical data with the current state to calculate possible driving conditions in the future and make control adjustments in advance to reduce the negative impact of network latency.

[0051] First, the vehicle's current dynamic parameters are used to compensate for the command execution effect, including time delay compensation and displacement compensation. The expression is as follows: ; in, This represents the current real-time vehicle speed. For longitudinal acceleration, This represents the total end-to-end latency. This compensation logic significantly improves the operational stability and safety of remote driving under complex network conditions.

[0052] use and the vehicle's current heading angle The predicted driving state is transformed into a series of continuous coordinate points to obtain the predicted vehicle trajectory. In one example, the predicted vehicle trajectory can be obtained through a machine learning model (i.e., a prediction model), which can predict trajectory points according to the following... The computational logic formula is constructed and expressed as follows: ; ; in, This represents the vehicle's initial position coordinates at the moment the control command is sent. The generated motion trajectory point set includes not only position information but also corresponding timestamp indices. The predicted vehicle trajectory is synchronized to the cabin interface in real time, allowing the driver to input driving control commands that take time delays into account based on the predicted trajectory. Furthermore, the deviation between the actual vehicle's transmitted trajectory and the predicted trajectory enables quantitative analysis of the remote control closed-loop accuracy.

[0053] In some embodiments, please continue to refer to Figure 2 Control commands are sent from the "command transceiver" module in the cloud cockpit and transmitted back to the "command transceiver control" module on the vehicle via the 5G network. The predicted motion trajectory point sequence is rendered in real time in the "real-time vehicle status display" module, allowing the driver to intuitively compare the predicted trajectory with the actual vehicle's motion trend.

[0054] In some embodiments, the remote driving method based on remote desktop control technology in this application may also include, but is not limited to, the following steps: S601 acquires the real vehicle's feedback trajectory based on remote desktop control technology; S602, calculate the prediction error based on the real vehicle return trajectory and the predicted vehicle trajectory at the same time point; S603, adjust the parameters of the prediction model based on the prediction error.

[0055] In this embodiment, the predicted vehicle trajectory generated by the above prediction model is periodically compared with the actual vehicle trajectory fed back by the vehicle sensors to calculate the deviation of the prediction model, and then the root mean square error is used. The accuracy of the prediction model is evaluated as follows: ; in, To predict the location of trajectory points of a vehicle's trajectory, These are the actual trajectory points located from the vehicle's sensors, transmitting the actual vehicle's trajectory back to the real vehicle. This represents the number of samples within the sliding window. When the error value exceeds the safety threshold, a safety redundancy mechanism will be triggered, enhancing robustness under network fluctuation conditions by adjusting prediction parameters or increasing the sampling frequency. This embodiment compares latency and prediction accuracy in real-world scenarios. Exceeding the preset security threshold, or In the event of a sudden increase, the cabin-side piloting software will immediately trigger the safety redundancy strategy.

[0056] This application provides a low-latency remote driving method based on remote desktop control technology. This method utilizes a vehicle-side domain controller for localized video rendering and interface generation, establishing a direct pixel-level mapping connection from the cabin client to the vehicle-side controller, effectively avoiding the multi-level forwarding and encoding / decoding losses in traditional video stream transmission. By adopting a lightweight data transmission protocol instead of a high-bandwidth video stream and introducing a key instruction priority scheduling and motion prediction compensation mechanism, this method can dynamically adjust data exchange strategies and pre-compensate latency deviations, ultimately achieving significantly reduced latency, greatly reduced network bandwidth consumption, and improved real-time performance and adaptability of remote control. Furthermore, real-time status feedback and accurate motion trajectory prediction can effectively prevent potential safety risks, thereby improving the operational reliability, robustness, and driving safety of remote driving.

[0057] This application also provides a remote driving system based on remote desktop control technology, including: The first module is used to collect driving environment data and vehicle status data through body sensors; The second module is used to render the local operation screen based on driving environment data and vehicle status data to obtain a driving operation video stream. The third module is used to synchronize incremental pixels of the driving operation video stream to the remote operation terminal through remote desktop control technology, and to collect the synchronization time of the pixel increment process. The fourth module is used to dynamically allocate transmission redundancy for key frames in the driving operation video stream based on the synchronization time. The fifth module is used to encode keyframes according to transmission redundancy to obtain redundant data packets; The sixth module is used to synchronize the incremental pixels of the driving operation video stream with redundant data packets to the remote operation terminal.

[0058] It is understood that the methods described in the above method embodiments are applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0059] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0060] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0061] Please see Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901. The input / output interface 903 is used to implement information input and output; The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0062] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0063] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0064] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0065] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0066] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0067] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0068] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0069] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0070] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0071] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0072] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0073] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0074] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0075] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A remote driving method based on remote desktop control technology, characterized in that, Includes the following steps: The vehicle body sensors collect driving environment data and vehicle status data; The driving environment data and the vehicle status data are rendered locally to obtain a driving operation video stream. The incremental pixels of the driving operation video stream are synchronized to the remote operation terminal using remote desktop control technology, and the synchronization time of the pixel increment process is collected. Based on the synchronization time, the key frames in the driving operation video stream are dynamically allocated for transmission redundancy. The keyframes are encoded according to the transmission redundancy to obtain redundant data packets; The incremental pixels of the driving operation video stream are synchronized together with the redundant data packets to the remote operation terminal.

2. The remote driving method based on remote desktop control technology according to claim 1, characterized in that, The step of synchronizing incremental pixels of the driving operation video stream to the remote operation terminal via remote desktop control technology includes the following steps: Determine the expected load overhead by maximizing the data payload efficiency of the message structure; The incremental pixels of the driving operation video stream are encoded to obtain encoded data, and the encoded data is structured and encapsulated according to the expected load overhead to obtain driving message data. The driving message data is synchronized to the remote operating terminal using remote desktop control technology.

3. The remote driving method based on remote desktop control technology according to claim 1, characterized in that, The step of dynamically allocating transmission redundancy for key frames in the driving operation video stream based on the synchronization time includes the following steps: Calculate the response delay gain based on the synchronization time. The transmission redundancy of key frames in the driving operation video stream is dynamically allocated based on the response delay gain; wherein the transmission redundancy of the key frames is inversely proportional to the response delay gain.

4. The remote driving method based on remote desktop control technology according to claim 1, characterized in that, The remote driving method based on remote desktop control technology also includes the following steps: Monitoring the channel quality index values ​​of the command transmission channel; If the channel quality index value is less than the expected quality threshold, driving control commands are transmitted first, followed by cockpit interaction commands.

5. The remote driving method based on remote desktop control technology according to claim 1, characterized in that, The remote driving method based on remote desktop control technology also includes the following steps: Calculate the total end-to-end latency from the start of data acquisition to synchronization with the remote operating terminal; The delay compensation displacement is determined based on the total end-to-end delay, the current vehicle speed, and the current longitudinal acceleration of the vehicle. The time delay compensation displacement and the vehicle's current heading angle are input into the prediction model to predict the vehicle trajectory, and driving control commands are determined based on the predicted vehicle trajectory.

6. The remote driving method based on remote desktop control technology according to claim 5, characterized in that, The remote driving method based on remote desktop control technology also includes the following steps: The real vehicle trajectory is obtained based on the aforementioned remote desktop control technology; The prediction error is calculated based on the actual vehicle trajectory transmitted back and the predicted vehicle trajectory at the same time point. The prediction model parameters are adjusted based on the prediction error.

7. A remote driving system based on remote desktop control technology, characterized in that, include: The first module is used to collect driving environment data and vehicle status data through body sensors; The second module is used to perform local operation screen rendering on the driving environment data and the vehicle status data to obtain a driving operation video stream. The third module is used to synchronize the incremental pixels of the driving operation video stream to the remote operation terminal through remote desktop control technology, and to collect the synchronization time of the pixel increment process. The fourth module is used to dynamically allocate transmission redundancy to key frames in the driving operation video stream based on the synchronization time. The fifth module is used to encode the keyframe according to the transmission redundancy to obtain redundant data packets; The sixth module is used to synchronize the incremental pixels of the driving operation video stream with the redundant data packets to the remote operation terminal.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images