Multi-source perception fusion based low-latency remote tele-driving method and system

Abstract

The application is suitable for the field of intelligent driving, and discloses a multi-source perception fusion low-delay remote control driving method and system. The multi-source perception fusion low-delay remote control driving method comprises the following steps: fusing perception information containing obstacle warning and bird's-eye panoramic view, and using a synchronous source identifier to ensure the space-time alignment rendering of double-way pictures, and then performing directional safety checking on the control instruction according to the real-time perception state, so that the potential collision risk is effectively eliminated, and the safety of remote control driving is significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent driving, and in particular relates to a low-latency remote control driving method and system with multi-source perception fusion. Background Technology

[0002] Traditional remote-controlled driving methods simply transmit raw video footage captured by the vehicle's cameras and basic two-dimensional instrument data independently, then overlay and display them remotely. The issuance of control commands and the rendering of images lack a unified spatiotemporal reference, and safety verification relies heavily on the remote driver's visual observation or fixed network thresholds. This traditional method, with its limited perception, struggles to control dynamic obstacles around the vehicle and its own position, greatly increasing the risk of collisions and posing serious safety hazards. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a low-latency remote control driving method and system based on multi-source perception fusion, which can solve the problem in related technologies that it is difficult to control the dynamic obstacles around the vehicle and its own position and posture, which can easily lead to vehicle collisions and pose serious safety hazards.

[0004] The first aspect of this invention provides a low-latency remote-controlled driving method based on multi-source perception fusion, comprising: Acquire multi-source perception data of the vehicle and perform fusion processing on the multi-source perception data to obtain fused perception information that includes the vehicle's surrounding environment and the vehicle's status. Acquire the vehicle's main view view, and convert the vehicle's main view view view and the fused perception information into streaming media data; The streaming media data is distributed to the remote control terminal via a low-latency network protocol, wherein different synchronization source identifiers are assigned to the fused perception information and the vehicle's main view screen for synchronous rendering on the remote control terminal, and the remote control terminal responds to the synchronously rendered streaming media data by returning vehicle control commands. The system receives vehicle control commands from the remote control terminal and performs targeted security verification on the vehicle control commands based on the real-time security status contained in the fused perception information, thereby obtaining the verification result. The vehicle control command is executed or corrected based on the verification result.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the step of acquiring multi-source perception data of the vehicle and fusing the multi-source perception data to obtain fused perception information including the vehicle's surrounding environment and the vehicle's state includes: Based on the obstacle point cloud data, real-time pose data and chassis status data of the vehicle, coordinate transformation and region judgment are performed on the point cloud data to obtain obstacle collision warning signs in all directions around the vehicle. Based on the real-time pose data, the chassis status data, and the preset vector map data, trajectory prediction and environmental boundary drawing are performed to obtain a bird's-eye view panoramic image containing the vehicle outline, the predicted trajectory line, and the map boundary line. The fused perception information is obtained by associating and integrating the bird's-eye view panoramic image with the obstacle collision warning sign.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the step of acquiring the vehicle's main view image and converting the vehicle's main view image and the fused perception information into streaming media data includes: Hardware encoding is performed on the raw encoded video stream obtained from the vehicle camera and the image frames generated from the fused perception information to obtain the first encoded video stream corresponding to the vehicle's main view and the second encoded video stream corresponding to the fused perception information. The first encoded video stream and the second encoded video stream are encapsulated into data packets based on a real-time transport protocol to obtain the streaming media data, which includes two independent streaming media data streams to be distributed.

[0007] Optionally, in a third implementation of the first aspect of the present invention, distributing the streaming media data to the remote control terminal via a low-latency network protocol includes: A point-to-point connection is established with the remote control terminal according to the WebRTC protocol, and in the generated session description protocol message, a first synchronization source identifier is assigned to the independent streaming media data corresponding to the vehicle's main view screen in the streaming media data, and a second synchronization source identifier is assigned to the independent streaming media data corresponding to the fused perception information in the streaming media data. Through the point-to-point connection, the independent streaming media data carrying different synchronization source identifiers are pushed to the remote control terminal.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, after distributing the streaming media data to the remote control terminal via a low-latency network protocol, the process includes: Monitor the image loss feedback message in the control channel of the low-latency network protocol; If the image loss feedback message is parsed, it is determined whether the time interval between the last keyframe request command sent exceeds the preset anti-shake time threshold. If the anti-shake time threshold is exceeded, a keyframe request instruction is generated and sent to the data acquisition device of the vehicle's main view screen to force the encoder of the data acquisition device to immediately output a keyframe.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the step of generating a keyframe request instruction and sending the keyframe request instruction to the data acquisition device of the vehicle's main view image if the anti-shake time threshold is exceeded includes: If the anti-shake time threshold is exceeded, the image loss feedback message will be converted into a keyframe request instruction that conforms to the hardware control protocol of the data acquisition device. Send the keyframe request command to the network control interface of the data acquisition device.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, the step of performing targeted safety verification on the vehicle control command based on the real-time safety status contained in the fused perception information to obtain a verification result includes: The vehicle control commands are analyzed to obtain the longitudinal speed control quantity and the lateral steering control quantity; Read the real-time obstacle warning sign corresponding to the vehicle's current driving direction from the fused perception information; If the real-time obstacle warning sign indicates that there is an obstacle in the direction of vehicle travel, and the longitudinal speed control value is positive, then a verification result containing an instruction to correct the forward speed component to zero is generated. If the real-time obstacle warning sign indicates that there is an obstacle in the reverse direction of the vehicle, and the longitudinal speed control value is negative, then a verification result containing an instruction to correct the reverse speed component to zero is generated.

[0011] Optionally, in a seventh implementation of the first aspect of the present invention, after the step of receiving the vehicle control command from the remote control terminal, the method includes: Record the latest timestamp of successful reception of the vehicle control command; Calculate the interval duration of consecutive invalid instructions based on the current system time and the latest timestamp; If the interval exceeds a preset safety timeout threshold, the network connection is determined to be interrupted, and an emergency stop command with zero longitudinal speed and zero lateral steering is generated to replace the vehicle control command to be verified.

[0012] Optionally, in an eighth implementation of the first aspect of the invention, after the step of generating an emergency stop command with zero longitudinal speed and zero lateral steering to replace the vehicle control command to be verified, the method includes: Based on the current driving mode status fed back by the vehicle chassis controller, perform mode status monitoring; If, during remote-controlled driving, it is detected that the current driving mode has been switched from remote-controlled mode to another mode, the start-up state will continue to be timed. When the duration of the state maintenance time exceeds the preset state maintenance time threshold, the remote control mode enable command is resent to the vehicle chassis controller to reset and maintain the vehicle in remote control mode.

[0013] Secondly, embodiments of the present invention provide a low-latency remote-controlled driving system based on multi-source perception fusion, the device comprising: The fusion module is used to acquire multi-source perception data of the vehicle and perform fusion processing on the multi-source perception data to obtain fused perception information that includes the vehicle's surrounding environment and the vehicle's status. The conversion module is used to acquire the vehicle's main view image and convert the vehicle's main view image and the fused perception information into streaming media data. The distribution module is used to distribute the streaming media data to the remote control terminal via a low-latency network protocol. Different synchronization source identifiers are assigned to the fused perception information and the vehicle's main view screen for synchronous rendering on the remote control terminal. The remote control terminal responds to the synchronously rendered streaming media data and returns vehicle control commands. The verification module is used to receive vehicle control commands from the remote control terminal, and perform targeted security verification on the vehicle control commands based on the real-time safety status contained in the fused perception information, and obtain the verification result. The instruction processing module is used to execute or correct the vehicle control instructions based on the verification results.

[0014] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: by integrating perception information including obstacle warning and bird's-eye view panoramic view, and using synchronous source identifier to ensure spatiotemporal alignment rendering of dual-channel images, and then performing directional safety verification of control commands based on real-time perception status, potential collision risks are effectively eliminated, and the safety of remote control driving is significantly improved. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of an embodiment of the low-latency remote-controlled driving method based on multi-source perception fusion in this invention. Figure 2 This is a schematic diagram of Embodiment 2 of the low-latency remote-controlled driving method based on multi-source perception fusion in this invention. Figure 3This is a schematic diagram of Embodiment 3 of the low-latency remote-controlled driving method based on multi-source perception fusion in this invention. Figure 4 This is a schematic diagram of an embodiment of a low-latency remote-controlled driving system based on multi-source perception fusion according to the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are protected by this invention.

[0018] It should be noted that the terms "comprising," "including," and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this invention, are intended to cover non-exclusive inclusion. Relational terms such as "first" and "second" in the claims, specification, and accompanying drawings of this invention are used merely to distinguish one entity / operation / object from another, and do not necessarily require or imply any such immediate relationship or order between these entities / operations / objects.

[0019] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0020] Traditional remote-controlled driving methods simply transmit raw video footage captured by the vehicle's cameras and basic two-dimensional instrument data independently, then overlay and display them remotely. The issuance of control commands and image rendering lack a unified spatiotemporal reference, and safety verification relies heavily on the remote driver's visual observation or fixed network thresholds. This single-dimensional perception method struggles to control dynamic obstacles around the vehicle and its own position, greatly increasing the risk of collisions and posing serious safety hazards.

[0021] In view of this, embodiments of the present invention provide a low-latency remote-controlled driving method and system based on multi-source perception fusion. By fusing perception information including obstacle warning and bird's-eye view panoramic view, and using synchronous source identifiers to ensure spatiotemporal alignment rendering of dual-channel images, and then performing directional safety verification of control commands based on real-time perception status, potential collision risks are effectively eliminated, and the safety of remote-controlled driving is significantly improved.

[0022] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0023] Example 1: Figure 1 This illustration shows a flowchart of a low-latency remote-controlled driving method based on multi-source perception fusion, according to an embodiment of the present invention. This method can be applied to systems, such as servers, service clusters, mobile phones, laptops, and super mobile personal computers.

[0024] Specifically, the aforementioned low-latency remote-controlled driving method based on multi-source perception fusion may include the following steps S101 to S104.

[0025] Step S101: Acquire multi-source perception data of the vehicle and perform fusion processing on the multi-source perception data to obtain fused perception information that includes the vehicle's surrounding environment and the vehicle's status.

[0026] In embodiments of the present invention, raw multi-source perception data are acquired in parallel from multiple sensors mounted on the vehicle.

[0027] The obtained multi-source perception data is preprocessed by time synchronization, coordinate system 1 and data association, and then a fusion processing algorithm is executed. For example, the lidar point cloud and visual detection results are fused to improve the accuracy of obstacle recognition. Combined with vehicle pose and map information, a structured fusion perception information is generated, which includes the passable area around the vehicle, the location and attributes of obstacles, lane line information, and the vehicle's pose, speed, heading angle and other states.

[0028] Step S102: Acquire the vehicle's main view view and convert the vehicle's main view view view and fused perception information into streaming media data.

[0029] In an embodiment of the present invention, the vehicle's first-person perspective view is acquired simultaneously. This first-person perspective view, along with fused perception information, is then converted into streaming media data suitable for network transmission. This step encapsulates visual and structured perception information into a continuous media stream.

[0030] Specifically, the encoded original video stream from the vehicle's main view is acquired synchronously through the application programming interface (API) of the onboard camera module. Simultaneously, the fused perception information is transformed into a continuous sequence of image frames by a graphics rendering engine. This image sequence visually displays the vehicle's outline, surrounding obstacles, predicted trajectory, and boundaries in a bird's-eye view format. A hardware encoder is then invoked to independently encode both the main view video stream and the rendered perception information image stream, resulting in a first encoded video stream and a second encoded video stream. These two encoded video streams are then encapsulated into data packets based on a real-time transmission protocol, thus completing the streaming media conversion process and yielding two independent streaming media data streams to be distributed.

[0031] Step S103: The streaming media data is distributed to the remote control terminal via a low-latency network protocol. Different synchronization source identifiers are assigned to the fused perception information and the vehicle's main view screen for synchronous rendering on the remote control terminal. The remote control terminal responds to the synchronously rendered streaming media data and returns vehicle control commands.

[0032] In an embodiment of the invention, streaming media data is distributed to a remote control terminal via a low-latency network protocol. During this process, different synchronization source identifiers are assigned to the streaming media data corresponding to the fused perception information and the streaming media data corresponding to the vehicle's first-person view. This enables the remote control terminal to synchronously render the two streaming media data streams based on these identifiers, thereby providing the operator with a composite visual interface integrating panoramic perception and a first-person perspective. The remote control terminal will then generate and return vehicle control commands in response to this synchronously rendered visual information.

[0033] Specifically, the WebRTC protocol stack is initiated to exchange signaling with the remote control terminal to establish an end-to-end point-to-point connection. When generating the local SDP description, a synchronization source identifier is assigned to the first encoded video stream, and a different SSRC is assigned to the second encoded video stream. After the connection is established, the two RTP streaming media data streams carrying different SSRCs are pushed to the remote control terminal in real time as two independent media tracks through this P2P connection.

[0034] Upon receiving the media stream, the client at the remote control end separates the two streams according to SSRC and simultaneously decodes and renders them within designated windows of the user interface. The operator then makes a judgment based on this synchronously presented composite visual information and generates operation commands through the interactive interface. The remote control application subsequently encapsulates these operation commands into a vehicle control command message of a specific format and sends it back to the vehicle system via the WebRTC data channel or signaling channel.

[0035] Step S104: Receive vehicle control commands from the remote control terminal, and perform targeted security verification on the vehicle control commands based on the real-time safety status contained in the fused perception information to obtain the verification result.

[0036] In an embodiment of the present invention, upon receiving an instruction, it is not executed directly. Instead, a targeted safety verification is performed on the received vehicle control instruction based on the real-time safety status contained in the previously generated and continuously updated fused perception information, resulting in a verification result. This verification process is used to determine the safety of the instruction in the current environment.

[0037] Specifically, the system receives vehicle control command messages sent from the remote control terminal via a data channel, parses them, and extracts core control parameters, such as longitudinal speed control and lateral steering control. Then, it queries the fused perception information to read real-time safety status directly related to the current vehicle's driving direction or command intent, such as forward / rear / side obstacle warning signs, estimated collision time, and lane departure status. The embedded safety verification logic is then invoked to compare the parsed control commands with the current real-time safety status and apply rule-based judgments. This targeted safety verification process outputs a definitive verification result, which may be "Command is safe, execution allowed" or "Command poses a risk, needs to be corrected to a specific value," along with specific correction parameters.

[0038] Step S105: Execute or correct vehicle control commands based on the verification results.

[0039] In an embodiment of the present invention, based on the verification result, corresponding operations are performed. Specifically, if the verification passes, the original vehicle control command is executed; if the verification reveals a safety hazard, the original command is modified, and then the modified command is executed. This step ensures that the final executed control commands all comply with the current real-time safety constraints.

[0040] Specifically, the final action is determined based on the verification results. If the verification result indicates the command is safe, the original parsed vehicle control command is sent to the vehicle's powertrain controller and steering actuator via the vehicle control area network bus or an in-vehicle network interface such as Ethernet. If the verification result indicates that correction is needed, a new control command that conforms to safety constraints is generated based on the correction parameters attached to the verification result. Then, this corrected control command is sent to the vehicle actuators. This step ensures that all control commands ultimately acting on the vehicle chassis pass real-time safety filtering based on the latest environmental perception.

[0041] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: by integrating perception information including obstacle warning and bird's-eye view panoramic view, and using synchronous source identifier to ensure spatiotemporal alignment rendering of dual-channel images, and then performing directional safety verification of control commands based on real-time perception status, potential collision risks are effectively eliminated, and the safety of remote control driving is significantly improved.

[0042] Example 2: In an optional specific embodiment of the present invention, refer to Figure 2 , Figure 2 This is a schematic diagram of Embodiment 2 of the low-latency remote control driving method for multi-source perception fusion in the present invention. The method involves acquiring multi-source perception data of the vehicle and fusing the multi-source perception data to obtain fused perception information that includes the vehicle's surrounding environment and the vehicle's status. Specifically, this may include steps S201 to S203.

[0043] Step S201: Based on the obstacle point cloud data, real-time pose data and chassis status data of the vehicle, perform coordinate transformation and region judgment on the point cloud data to obtain obstacle collision warning signs in all directions around the vehicle.

[0044] In embodiments of the present invention, raw point cloud sequences are acquired from sensors such as lidar as obstacle point cloud data; real-time vehicle pose data are acquired from modules such as integrated navigation systems or visual odometry; and chassis status data containing information such as vehicle speed, wheel speed, and steering angle are parsed from the vehicle controller local area network bus. These data are temporarily stored in a buffer and undergo preliminary timestamp alignment to ensure data time synchronization.

[0045] Transform the point cloud from the sensor coordinate system to a unified local coordinate system or world coordinate system centered on the vehicle.

[0046] Based on multiple pre-defined collision warning zones enclosed around the vehicle's outline (e.g., front, rear, left, right, left-front, right-front fan-shaped or rectangular areas), the converted point cloud is spatially analyzed, and the number of valid point clouds falling within each warning zone is counted. The number of point clouds in each zone is compared to a pre-defined threshold. When the number of point clouds in any warning zone exceeds its corresponding pre-defined threshold, a collision risk is determined for that direction, and a Boolean or hierarchical obstacle collision warning sign representing that direction is generated. Finally, a set of real-time obstacle collision warning signs covering all directions surrounding the vehicle is output.

[0047] Step S202: Based on real-time pose data, chassis status data, and preset vector map data, perform trajectory prediction and environmental boundary drawing to obtain a bird's-eye view panoramic image containing vehicle outline, predicted trajectory line, and map boundary line.

[0048] In an embodiment of the present invention, a bird's-eye view panoramic image is rendered based on real-time pose data and chassis status data. Specifically, the precise position and orientation of the vehicle in the current map are determined based on the real-time pose.

[0049] The system uses a trajectory prediction algorithm, combined with chassis status data such as current vehicle speed and steering angle, to calculate and draw the predicted trajectory of the vehicle in the near future. Simultaneously, it retrieves and extracts environmental boundary information such as lane lines, curbs, and building boundaries from a pre-set vector map database.

[0050] On a preset-sized image canvas, the vehicle's polygonal outline, the calculated predicted trajectory line, and the environmental boundary line retrieved from the map are overlaid and drawn proportionally from a top-down perspective, with the vehicle's current position as the center.

[0051] Optionally, the trajectory line will be dynamically updated according to the vehicle status during the drawing process, and the map boundary will be rendered in a sliding window manner according to the pose change.

[0052] Output a bird's-eye view panoramic image that includes all the above elements and visually demonstrates the vehicle itself and its relationship with the static environment.

[0053] Step S203: The bird's-eye view panoramic image is associated and integrated with the obstacle collision warning sign to obtain fused perception information.

[0054] In an embodiment of the invention, a bird's-eye view panoramic image is used as the base layer, and generated obstacle collision warning signs corresponding to different spatial directions are accurately superimposed on the corresponding warning area positions on the bird's-eye view panoramic image in the form of graphical elements. For example, if the left front warning sign is activated, a conspicuous warning graphic is drawn in the left front area of ​​the vehicle outline in the bird's-eye view. This association process ensures the spatiotemporal consistency between dynamic obstacle perception results and static environment and vehicle status.

[0055] Once integrated, the enhanced bird's-eye view panoramic image of the dynamic early warning information, along with its underlying structured early warning sign data, will be packaged or encoded together and output as fused perception information in subsequent processes.

[0056] In this embodiment of the invention, by defining specific fusion processing steps, abstract obstacle point cloud data, vehicle pose, and map information are generated into an intuitive bird's-eye view panoramic image that integrates vehicle outline, predicted trajectory, static boundaries, and dynamic warning signs. This is equivalent to providing the operator with a real-time updated, top-down digital twin map, reducing their cognitive load and enabling them to grasp the comprehensive spatial relationship between the vehicle and its environment at a glance, thus making faster and more accurate decisions.

[0057] Example 3: In an optional specific embodiment of the present invention, refer to Figure 3 , Figure 3 This is a schematic diagram of Embodiment 3 of the low-latency remote control driving method for multi-source perception fusion in the present invention. The method involves acquiring the vehicle's main view view and converting the vehicle's main view view and fused perception information into streaming media data, which may specifically include steps S301 to S302.

[0058] Step S301: Based on the encoded raw video stream obtained from the vehicle camera and the image frames generated from the fused perception information, hardware encoding is performed respectively to obtain the first encoded video stream corresponding to the vehicle's main view and the second encoded video stream corresponding to the fused perception information.

[0059] In an embodiment of the present invention, two data preparation threads are started.

[0060] In the first thread, the compressed, encoded raw video stream is directly obtained from the camera's built-in hardware encoder at a specific resolution, frame rate, and encoding format by calling the vehicle's software development kit or video capture interface. This video stream represents the vehicle's main viewpoint.

[0061] In the second thread, fusion perception information is received in real time from the fusion perception module. This information is a structured data set containing elements such as vehicle outlines, obstacle warning signs, predicted trajectories, and map boundaries.

[0062] A lightweight graphics rendering engine is invoked to render the received fusion-aware information into a series of consecutive image frames in real time.

[0063] Optionally, the rendering frame rate is synchronized with the frame rate of the main-view video stream for subsequent synchronized distribution.

[0064] The two raw data streams are hardware-accelerated for encoding to reduce system CPU load and meet low latency requirements. The raw video stream from the first thread is either directly used as input or decoded and then re-encoded before being fed into a hardware video encoder. This encoder processes the data and outputs the first encoded video stream in a standard format, corresponding to the vehicle's main viewpoint. Simultaneously, the sequence of fused perception information image frames generated in real-time rendering in the second thread is also fed into another hardware video encoder for real-time compression encoding, generating the second encoded video stream corresponding to the fused perception information.

[0065] Optionally, different encoding parameters can be dynamically configured for the two video streams based on network bandwidth conditions. For example, a higher bitrate can be allocated to the main viewpoint and a relatively lower bitrate can be allocated to the perceptual information view.

[0066] Step S302: Encapsulate the first encoded video stream and the second encoded video stream into a data packet based on the real-time transport protocol to obtain streaming media data, which includes two independent streaming media data streams to be distributed.

[0067] In this embodiment of the invention, after obtaining two independently encoded video streams, encapsulation based on the Real-Time Transport Protocol (RTP) is performed on each video stream. An RTP header is added to each frame of encoded video data. The RTP header contains key fields such as sequence number, timestamp, and synchronization source identifier to support packet loss detection, out-of-order reassembly, and stream synchronization in network transmission. A specific SSRC is assigned to the first encoded video stream, and a different SSRC is assigned to the second encoded video stream, thereby distinguishing the two streams at the packet level.

[0068] After encapsulation, two independent streaming media data streams, each consisting of a series of RTP data packets, are obtained. These two encapsulated independent streaming media data streams are then used as input for the subsequent distribution module.

[0069] In this embodiment of the invention, hardware encoding significantly reduces video compression latency and CPU usage. Encapsulating the two video streams into independent RTP streams ensures the independence of the video streams at the source.

[0070] Example 4: In an optional embodiment of the present invention, the streaming media data is distributed to the remote control terminal via a low-latency network protocol, which may specifically include steps S401 to S402.

[0071] Step S401: Establish a point-to-point connection with the remote control terminal according to the WebRTC protocol, and in the generated session description protocol message, assign a first synchronization source identifier to the independent streaming media data corresponding to the vehicle's main view screen in the streaming media data, and assign a second synchronization source identifier to the independent streaming media data corresponding to the fused perception information in the streaming media data.

[0072] In this embodiment of the invention, the system registers its identity with a signaling server. Next, it generates initialization signaling exchange parameters necessary for establishing a point-to-point connection. When generating these parameters, based on the two prepared independent streaming media data streams, two corresponding media channels are declared, and globally unique identifiers, namely a first synchronization source identifier and a second synchronization source identifier, are generated for each media channel. The parameters containing the media channel declarations and identifiers are sent to the remote control terminal, and the response parameters are received from the other party. Simultaneously, both parties perform network address discovery and traversal negotiation to determine the optimal end-to-end data transmission path. When the signaling exchange is successfully completed and the transmission path is agreed upon, the point-to-point connection with the remote control terminal is determined to be formally established.

[0073] Step S402: Through point-to-point connection, push independent streaming media data carrying different synchronization source identifiers to the remote control terminal.

[0074] In an embodiment of the present invention, after the point-to-point connection is established, pre-encapsulated independent streaming media data packets corresponding to the vehicle's main viewpoint are sequentially read from the transmission buffer. Before transmission, a first synchronization source identifier is written into a specific protocol header field of each data packet. Simultaneously, another pre-encapsulated independent streaming media data packet corresponding to the fused sensing information is read in parallel and bound to a second synchronization source identifier.

[0075] Using an established, low-latency point-to-point network channel, these two streaming media data streams, each carrying different synchronization source identifiers, are pushed in parallel and in real-time to the network receiving address of the remote control terminal. The entire push process continues throughout the remote control session.

[0076] Optionally, during this process, network conditions will be dynamically monitored, and the sending strategy will be adaptively adjusted according to the flow control mechanism built into the protocol.

[0077] In this embodiment of the invention, a peer-to-peer connection is established via the WebRTC protocol to eliminate relay latency, and the synchronization and correlation problem of multiple video streams is solved by explicitly assigning different synchronization source identifiers to two independent streaming media data streams in the SDP. This enables the remote control terminal to accurately identify and separate the two video streams based on SSRC, and to project them to the correct rendering window for synchronized playback, thereby achieving spatiotemporal alignment rendering of the first-person perspective view and the fused perception view.

[0078] Example 5: In an optional embodiment of the present invention, after distributing streaming media data to the remote control terminal via a low-latency network protocol, steps S501 to S503 may be included.

[0079] Step S501: Listen for image loss feedback messages in the control channel of the low-latency network protocol.

[0080] In an embodiment of the present invention, while continuously distributing control information to the remote control terminal via a low-latency network protocol, an independent control signaling listening thread is initiated. This thread listens to a dedicated channel in the low-latency network protocol used for transmitting control information, such as a real-time transmission control protocol channel.

[0081] This channel captures and parses incoming data packets in real time, aiming to identify specific types of feedback control messages that indicate image quality issues at the remote end. Specifically, by parsing the payload type and format of the data packets, it determines whether they are image loss feedback messages, such as image loss indication messages or complete frame request messages. When such messages are successfully parsed, subsequent intervention procedures are triggered.

[0082] Step S502: If a frame loss feedback message is parsed, determine whether the time interval between the last keyframe request command sent exceeds the preset anti-shake time threshold.

[0083] In an embodiment of the present invention, upon parsing a valid image loss feedback message, no action is taken immediately. Instead, anti-shake logic is first executed to prevent sending too many instructions to the hardware device due to high-frequency packet loss in an extremely unstable network environment.

[0084] The system timestamp of the last successful keyframe request command sent to the data acquisition device of the vehicle's main view view is retrieved from the internal memory. Next, the difference between the current system time and this historical timestamp is calculated to obtain the time interval since the last command was sent. This calculated time interval is compared with a preset anti-shake time threshold used to balance response speed and command frequency. If the time interval is less than or equal to the anti-shake time threshold, it is determined that the anti-shake period is in effect, the feedback message is ignored, no intervention is performed, and channel monitoring continues. If the time interval exceeds the anti-shake time threshold, the intervention conditions are met, and the command generation and transmission phase begins.

[0085] In step S503, if the anti-shake time threshold is exceeded, a key frame request instruction is generated and sent to the data acquisition device of the vehicle's main view screen to force the encoder of the data acquisition device to immediately output a key frame.

[0086] In an embodiment of the present invention, after determining that the intervention conditions are met, the generation and transmission of a keyframe request instruction are performed. A new instruction is generated to request the immediate output of a complete intra-frame coded image. The generation logic of this instruction is independent of the conventional video stream coding control loop.

[0087] Subsequently, the generated keyframe request command is sent via its network communication interface to the data acquisition device responsible for providing the vehicle's main viewpoint. This data acquisition device can be an in-vehicle camera or its encoding unit.

[0088] The purpose of sending this instruction is to directly intervene at the source of encoding, forcing the device's hardware encoder to interrupt the current predictive encoding sequence and immediately generate and output a keyframe containing complete image information, thereby providing the remote control terminal with an independent decoding frame that does not depend on the preceding frame and is required for rapid image recovery.

[0089] In this embodiment of the invention, by monitoring the image loss feedback messages on the network control channel, the system can proactively detect the blindness state of the remote end. Combined with anti-shake time threshold judgment, a keyframe request command is sent to the data acquisition device only when necessary. This optimizes the user experience and operational security under weak network conditions.

[0090] Example 6: In an optional embodiment of the present invention, if the anti-shake time threshold is exceeded, a key frame request instruction is generated and sent to the data acquisition device of the vehicle's main view screen, which may specifically include steps S601 to S602.

[0091] In step S601, if the anti-shake time threshold is exceeded, the image loss feedback message is converted into a key frame request instruction that conforms to the hardware control protocol of the data acquisition device.

[0092] In an embodiment of the present invention, after determining that the conditions for sending a keyframe request instruction are met, the abstract feedback information from the network control channel is transformed into concrete instructions that the data acquisition device can recognize and execute. Specifically, a screen loss feedback message is acquired. The screen loss feedback message is a control signaling message that follows a low-latency network protocol format. Instruction conversion is performed according to a preset hardware control protocol mapping rule for the currently used vehicle's main view screen data acquisition device.

[0093] Specifically, the type and content of the image loss feedback message are analyzed, and mapped to a specific application programming interface or command defined in the hardware control protocol of the data acquisition device, used to request the immediate output of a keyframe. A complete and correctly formatted keyframe request instruction is generated according to the data format, method, request header, and payload structure specified by the hardware control protocol. For example, it might be a pre-generated Hypertext Transfer Protocol request pointing to a specific resource on the device's network control interface.

[0094] Step S602: Send a keyframe request command to the network control interface of the data acquisition device.

[0095] In an embodiment of the present invention, after successfully generating a keyframe request command, the command is sent. A request is initiated to the data acquisition device of the vehicle's main viewpoint via its network communication stack. The system establishes a Transmission Control Protocol (TCP) connection with the target device's network control interface or directly sends a User Datagram Protocol (UDP) data packet, according to hardware protocol requirements. Then, the keyframe request command is completely sent to the data acquisition device through this network connection.

[0096] In this embodiment of the invention, by converting network protocol messages into native hardware control protocol commands of the device and sending them directly to the device's network control interface, the most basic and direct control of the encoder is achieved. This bypasses intermediate software steps that may cause delays and fluctuations, essentially issuing a highest-priority hardware interrupt command to the encoder chip, which can boost the image recovery speed under weak network conditions to its theoretical limit.

[0097] Example 7: In an optional embodiment of the present invention, the vehicle control command is subjected to directional safety verification based on the real-time safety status contained in the fused perception information to obtain the verification result, which may specifically include steps S701 to S704.

[0098] Step S701: Parse the vehicle control commands to obtain the longitudinal speed control quantity and the lateral steering control quantity.

[0099] In this embodiment of the invention, vehicle control command data packets from a remote control terminal are continuously monitored and received via a network interface. Upon successful reception, the data packet is decrypted and its format verified. After successful verification, the payload portion of the data packet is parsed. Based on a predefined control command data exchange format, core control parameters are extracted from the parsed structured data. Specifically, longitudinal speed control quantities representing the vehicle's desired acceleration or speed, and lateral steering control quantities representing the vehicle's desired steering angle or curvature are extracted. These parsed control quantity values ​​are temporarily stored in memory as input for subsequent safety checks.

[0100] Optionally, the parsing process may include scaling, unit conversion, or range clipping of the original values ​​to match the input specifications of the vehicle's underlying controller.

[0101] Step S702: Read the real-time obstacle warning sign corresponding to the vehicle's current driving direction from the fused perception information.

[0102] In embodiments of the present invention, real-time environmental perception information is acquired simultaneously with or after parsing control commands. The shared data area, maintained and updated in real-time by the fusion processing module, is accessed to read the current fused perception information. From the structure of the fused perception information, real-time obstacle warning signs directly associated with the vehicle's current driving direction are located and read. For example, if the vehicle is currently in drive or the recent command trend is forward, warning signs in the forward direction are primarily read; if it is in reverse or the command trend is backward, warning signs in the backward direction are primarily read.

[0103] Step S703: If the real-time obstacle warning sign indicates that there is an obstacle in the direction of vehicle travel and the longitudinal speed control value is positive, then a verification result containing the instruction to correct the forward speed component to zero is generated.

[0104] In an embodiment of the present invention, the parsed control quantity is logically compared with the read real-time safety status. First, the status of the real-time obstacle warning sign is checked. The judgment logic is divided into two main branches based on the conflict between the vehicle's direction of movement and the obstacle's location: 1. Forward Direction Conflict Judgment: Determine if the real-time obstacle warning sign indicates the presence of an obstacle in the vehicle's forward direction. Simultaneously, determine if the parsed longitudinal speed control value is positive. If both conditions are true, a forward direction conflict risk is identified, and the corresponding result generation path is initiated.

[0105] 2. Reverse Direction Conflict Judgment: Determine if the real-time obstacle warning sign indicates the presence of an obstacle in the vehicle's reverse direction. Simultaneously, determine if the parsed longitudinal speed control value is negative. If both conditions are true, a reverse direction conflict risk is identified, and the corresponding result generation path is initiated.

[0106] If neither of the above two conflict checks is valid, then the current instruction is determined to have no collision risk in the current environment and enters the safe passage path.

[0107] Step S704: If the real-time obstacle warning sign indicates that there is an obstacle in the reverse direction of the vehicle and the longitudinal speed control value is negative, then a verification result containing the instruction to correct the reverse speed component to zero is generated.

[0108] In an embodiment of the present invention, a corresponding, structured verification result is generated based on the logical judgment result.

[0109] If a forward direction conflict is detected, a verification result containing a correction instruction is generated. This correction instruction corrects the forward velocity component of the longitudinal speed control variable to zero, while retaining the lateral steering control variable and its negative velocity components. The result indicates a verification conclusion that prohibits forward movement but allows reversing or steering.

[0110] If a reverse direction conflict is detected: a verification result containing a correction instruction is also generated. This correction instruction corrects the reverse direction velocity component in the longitudinal speed control variable to zero, while retaining the positive values ​​of the lateral steering control variable and its velocity. The result indicates a verification conclusion that reverse direction is prohibited, but forward or directional movement is allowed.

[0111] If no conflict is determined, a safety-indicating verification result is generated. This safety-indicating verification result is directly converted into the original longitudinal speed control and lateral steering control values ​​without correction.

[0112] The generated verification result will be passed to the subsequent instruction processing module as the basis for its final execution or correction of vehicle control instructions.

[0113] Optionally, the verification results may include additional information, such as conflict type, risk level, or log records.

[0114] In this embodiment of the invention, the control command is deconstructed into longitudinal and lateral components, and obstacle warnings are correlated with the driving direction. The aim is to provide directional correction, prohibiting only the direction of movement that conflicts with the control command, while retaining other harmless control dimensions. This allows the system to respect the operator's intentions to the greatest extent possible while ensuring collision safety.

[0115] Example 8: In an optional embodiment of the present invention, after the step of receiving the vehicle control command from the remote control terminal, steps S801 to S803 may be included.

[0116] Step S801: Record the latest timestamp of successfully receiving the vehicle control command.

[0117] In an embodiment of the present invention, after successfully receiving and completing the preliminary verification of the vehicle control command from the remote control terminal, regardless of the subsequent security verification result of the command, a parallel time recording operation that is independent of the command content processing flow will be executed.

[0118] Obtain the current high-precision system clock time and update this time as a new, latest successful reception timestamp to a storage unit dedicated to recording network connection status.

[0119] Optionally, this timestamp record may include the instruction's sequence number or other identifier to aid in more granular connection state analysis.

[0120] Step S802: Calculate the interval duration of consecutive periods without valid instructions based on the current system time and the latest timestamp.

[0121] In an embodiment of the present invention, a periodic background monitoring task is initiated, which runs independently of the main control loop. Within each monitoring cycle, the interval duration is calculated.

[0122] First, the latest successful reception timestamp is read from the updated storage unit. Next, the current high-precision system clock time is retrieved again. The difference between the current time and the latest read timestamp is calculated; this difference represents the duration of the continuous interval without a valid command since the last successful reception of a valid vehicle control command. This calculation is dynamic and continuous.

[0123] In step S803, if the interval exceeds the preset safety timeout threshold, it is determined that the network connection is interrupted, and an emergency stop command with zero longitudinal speed and zero lateral steering is generated to replace the vehicle control command to be verified.

[0124] In an embodiment of the present invention, after calculating the interval duration of consecutive periods without valid instructions, it is immediately compared with a preset security timeout threshold. The security timeout threshold is a time value preset based on network latency tolerance, control frequency, and security redundancy requirements.

[0125] If the calculated interval is less than or equal to the preset security timeout threshold, the network connection status is determined to be normal or within an acceptable short-term fluctuation range, no security intervention is triggered, and monitoring continues for the next cycle.

[0126] If the calculated interval exceeds the preset safety timeout threshold, it is determined that a network connection interruption or severe delay has occurred, and the remote control terminal may have lost effective control of the vehicle.

[0127] When a connection is determined to be interrupted, the normal subsequent processing flow based on the newly received instructions will be interrupted, and a preset safety instruction will be generated, namely an emergency stop instruction with both longitudinal speed and lateral steering at zero.

[0128] In this embodiment of the invention, by continuously recording the timestamps of received instructions and calculating the intervals between no instructions, the health status of communication can be quantified in real time. This fundamentally reduces the risk of loss of control due to communication failures.

[0129] Example 9: In an optional embodiment of the present invention, after the step of generating an emergency stop command with zero longitudinal speed and zero lateral steering to replace the vehicle control command to be verified, steps S901 to S903 may be included.

[0130] Step S901: Based on the current driving mode status fed back by the vehicle chassis controller, perform mode status monitoring.

[0131] In embodiments of the present invention, a separate chassis status monitoring task is initiated simultaneously with or after the system generates and executes an emergency stop command. This task continuously reads the current driving mode status, which is fed back in real time by the vehicle chassis controller, at a fixed frequency via the vehicle controller LAN bus, in-vehicle Ethernet, or a dedicated control interface. This current driving mode status information is used to identify the current control source of the vehicle chassis, such as automatic driving mode, remote control mode, manual mode, or safety mode. The current driving mode status is compared in real time with internally maintained variables that record the system's expected control mode.

[0132] Step S902: If, during the remote control driving process, it is detected that the current driving mode has been switched from remote control mode to another mode, then the start state maintenance timer is activated.

[0133] In an embodiment of the present invention, logical judgments are performed during continuous monitoring.

[0134] Check if the current driving mode status read is consistent with the system's expected remote control mode. If the status is consistent, it indicates that chassis control has not been unexpectedly seized, and no intervention is required. Continue monitoring for the next cycle.

[0135] If the status is inconsistent, meaning the current driving mode has been switched from remote control mode to another mode, a recovery procedure is triggered. First, a status maintenance timer is started, its value is cleared to zero, and then it begins to increment. This timer is used to quantify the duration of the control mode being in an abnormal state.

[0136] Step S903: When the duration of the state maintenance time exceeds the preset state maintenance time threshold, the remote control mode enable command is resent to the vehicle chassis controller to reset and maintain the vehicle in remote control mode.

[0137] In an embodiment of the invention, after the state maintenance timer is started, its current value, i.e., the duration of the state maintenance timer, is periodically checked. The duration of the state maintenance timer is compared with a pre-set state maintenance time threshold. If the duration is less than or equal to the threshold, it is determined that it may be a transient disturbance, and the system continues to wait and monitor without taking immediate action. If the duration exceeds the threshold, it is determined that the mode switch is continuous and unexpected, and remote control authority has been lost and has not been automatically restored. At this time, a critical recovery operation is performed, such as resending a remote control mode enable command to the designated command interface of the vehicle chassis controller. This command is a specific control message conforming to the vehicle chassis communication protocol specification, which instructs the chassis controller to relinquish control to the remote control system, thereby resetting and maintaining the vehicle in remote control mode. After the command is sent, the state maintenance timer is optionally reset, and monitoring continues to confirm whether the mode recovery was successful.

[0138] In this embodiment of the invention, by continuously monitoring the driving mode status, the system can detect in real time whether control has been unexpectedly seized. The introduction of a "state maintenance timer" avoids erroneous recovery caused by instantaneous state transitions. When the loss of control is confirmed to exceed a threshold, the system automatically resends the enable command to attempt to "regain" control. This mechanism significantly improves the system's robustness and fault tolerance, ensuring that remote control permissions can be automatically reset and maintained under non-malicious interference, thus guaranteeing the continuity and standardization of remote driving tasks.

[0139] Figure 4 This diagram illustrates the structure of a low-latency remote-controlled driving system based on multi-source perception fusion, according to an embodiment of the present invention. The aforementioned low-latency remote-controlled driving system 800 can be configured on a system. Specifically, the low-latency remote-controlled driving system 800 may include: The fusion module 1001 is used to perform coordinate transformation and region judgment on the point cloud data based on the vehicle's obstacle point cloud data, real-time pose data, and chassis status data to obtain obstacle collision warning signs in all directions around the vehicle; based on the real-time pose data, the chassis status data, and preset vector map data, it performs trajectory prediction and environmental boundary drawing to obtain a bird's-eye view panoramic image including the vehicle outline, predicted trajectory lines, and map boundary lines; and it integrates the bird's-eye view panoramic image with the obstacle collision warning signs to obtain fused perception information. The conversion module 1002 is used to perform hardware encoding on the encoded raw video stream obtained from the vehicle camera and the image frames generated from the fused perception information to obtain a first encoded video stream corresponding to the vehicle's main view and a second encoded video stream corresponding to the fused perception information; and to encapsulate the first encoded video stream and the second encoded video stream into a data packet based on a real-time transmission protocol to obtain the streaming media data, which includes two independent streaming media data streams to be distributed. The distribution module 1003 is used to distribute streaming media data to the remote control terminal through a low-latency network protocol. Different synchronization source identifiers are assigned to the fused perception information and the vehicle's main view screen for synchronous rendering on the remote control terminal. The remote control terminal responds to the synchronously rendered streaming media data and returns vehicle control commands. The verification module 1004 is used to receive vehicle control commands from the remote control terminal and perform targeted security verification on the vehicle control commands based on the real-time safety status contained in the fused perception information to obtain the verification result. The instruction processing module 1005 is used to execute or correct vehicle control instructions based on the verification results.

[0140] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows: by integrating perception information including obstacle warning and bird's-eye view panoramic view, and using synchronous source identifier to ensure the spatiotemporal alignment rendering of dual-channel images, and then performing directional safety verification of control commands based on real-time perception status, potential collision risks are effectively eliminated, and the safety of remote control driving is significantly improved.

[0141] The fusion module 1001 can also be specifically used for: performing coordinate transformation and region judgment on the point cloud data based on the vehicle's obstacle point cloud data, real-time pose data, and chassis status data to obtain obstacle collision warning signs in all directions around the vehicle; performing trajectory prediction and environmental boundary drawing based on real-time pose data, chassis status data, and preset vector map data to obtain a bird's-eye view panoramic image containing the vehicle outline, predicted trajectory lines, and map boundary lines; and associating and integrating the bird's-eye view panoramic image with the obstacle collision warning signs to obtain fused perception information.

[0142] The distribution module 1003 can also be specifically used to: perform hardware encoding on the encoded raw video stream obtained from the vehicle camera and the image frames generated from the fused perception information to obtain a first encoded video stream corresponding to the vehicle's main view and a second encoded video stream corresponding to the fused perception information; encapsulate the first encoded video stream and the second encoded video stream into a data packet based on a real-time transmission protocol to obtain streaming media data, which includes two independent streaming media data streams to be distributed.

[0143] The instruction processing module 1005 can also be specifically used to: establish a point-to-point connection with the remote control terminal according to the WebRTC protocol, and in the generated session description protocol message, assign a first synchronization source identifier to the independent streaming media data corresponding to the vehicle's main view screen in the streaming media data, and assign a second synchronization source identifier to the independent streaming media data corresponding to the fused perception information in the streaming media data; and push the independent streaming media data carrying different synchronization source identifiers to the remote control terminal through the point-to-point connection.

[0144] The instruction processing module 1005 can also be specifically used to: monitor image loss feedback messages in the control channel of the low-latency network protocol; if an image loss feedback message is parsed, determine whether the time interval between the last keyframe request instruction and the last sent exceeds a preset anti-shake time threshold; if it exceeds the anti-shake time threshold, generate a keyframe request instruction and send the keyframe request instruction to the data acquisition device of the vehicle's main view image to force the encoder of the data acquisition device to immediately output a keyframe.

[0145] The instruction processing module 1005 can also be specifically used to: if the anti-shake time threshold is exceeded, convert the image loss feedback message into a key frame request instruction that conforms to the hardware control protocol of the data acquisition device; and send the key frame request instruction to the network control interface of the data acquisition device.

[0146] The verification module 1004 can also be specifically used for: parsing vehicle control commands to obtain longitudinal speed control quantities and lateral steering control quantities; reading real-time obstacle warning signs corresponding to the vehicle's current driving direction from the fused perception information; if the real-time obstacle warning sign indicates that there is an obstacle in the vehicle's forward direction and the longitudinal speed control quantity is positive, then generating a verification result containing a command to correct the forward speed component to zero; if the real-time obstacle warning sign indicates that there is an obstacle in the vehicle's reverse direction and the longitudinal speed control quantity is negative, then generating a verification result containing a command to correct the reverse speed component to zero.

[0147] The verification module 1004 can also be specifically used to: record the latest timestamp of successfully receiving vehicle control commands; calculate the interval duration of consecutive invalid commands based on the current system time and the latest timestamp; if the interval duration exceeds the preset safety timeout threshold, it is determined that the network connection is interrupted, and an emergency stop command with zero longitudinal speed and zero lateral steering is generated to replace the vehicle control command to be verified.

[0148] The verification module 1004 can also be specifically used to: perform mode status monitoring based on the current driving mode status fed back by the vehicle chassis controller; if, during remote driving, the current driving mode status is detected to be switched from remote control mode to other modes, then start the status maintenance timer; when the duration of the status maintenance timer exceeds the preset status maintenance time threshold, resend the remote control mode enable command to the vehicle chassis controller to reset and maintain the vehicle in remote control mode.

[0149] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0150] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for various specific applications, but such implementations should not be considered beyond the scope of this invention.

[0151] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. Furthermore, the mutual couplings or direct couplings or communication connections shown or discussed can be indirect couplings or communication connections through some interfaces, devices, or units, and can be electrical, mechanical, or other forms.

[0152] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0153] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0155] The embodiments described above are merely illustrative of the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A low-latency remote-controlled driving method based on multi-source perception fusion, characterized in that, include: Based on the obstacle point cloud data, real-time pose data and chassis status data of the vehicle, coordinate transformation and area judgment are performed on the point cloud data to obtain obstacle collision warning signs in all directions around the vehicle. Based on the real-time pose data, the chassis status data, and the preset vector map data, trajectory prediction and environmental boundary drawing are performed to obtain a bird's-eye view panoramic image containing the vehicle outline, the predicted trajectory line, and the map boundary line. The bird's-eye view panoramic image is associated and integrated with the obstacle collision warning sign to obtain fused perception information; Hardware encoding is performed on the raw encoded video stream obtained from the vehicle camera and the image frames generated from the fused perception information to obtain the first encoded video stream corresponding to the vehicle's main view and the second encoded video stream corresponding to the fused perception information. The first encoded video stream and the second encoded video stream are encapsulated into data packets based on a real-time transport protocol to obtain streaming media data, which includes two independent streaming media data streams to be distributed. The streaming media data is distributed to the remote control terminal via a low-latency network protocol, wherein different synchronization source identifiers are assigned to the fused perception information and the vehicle's main view screen for synchronous rendering on the remote control terminal, and the remote control terminal responds to the synchronously rendered streaming media data by returning vehicle control commands. The system receives vehicle control commands from the remote control terminal and performs targeted security verification on the vehicle control commands based on the real-time security status contained in the fused perception information, thereby obtaining the verification result. The vehicle control command is executed or corrected based on the verification result.

2. The low-latency remote-controlled driving method based on multi-source perception fusion as described in claim 1, characterized in that, The step of distributing the streaming media data to the remote control terminal via a low-latency network protocol includes: A point-to-point connection is established with the remote control terminal according to the WebRTC protocol, and in the generated session description protocol message, a first synchronization source identifier is assigned to the independent streaming media data corresponding to the vehicle's main view screen in the streaming media data, and a second synchronization source identifier is assigned to the independent streaming media data corresponding to the fused perception information in the streaming media data. Through the point-to-point connection, the independent streaming media data carrying different synchronization source identifiers are pushed to the remote control terminal.

3. The low-latency remote-controlled driving method based on multi-source perception fusion as described in claim 1, characterized in that, After distributing the streaming media data to the remote control terminal via a low-latency network protocol, the process includes: Monitor the image loss feedback message in the control channel of the low-latency network protocol; If the image loss feedback message is parsed, it is determined whether the time interval between the last keyframe request command sent exceeds the preset anti-shake time threshold. If the anti-shake time threshold is exceeded, a keyframe request instruction is generated and sent to the data acquisition device of the vehicle's main view screen to force the encoder of the data acquisition device to immediately output a keyframe.

4. The low-latency remote-controlled driving method based on multi-source perception fusion as described in claim 3, characterized in that, If the anti-shake time threshold is exceeded, a keyframe request instruction is generated and sent to the data acquisition device of the vehicle's main view image, including: If the anti-shake time threshold is exceeded, the image loss feedback message will be converted into a keyframe request instruction that conforms to the hardware control protocol of the data acquisition device. Send the keyframe request command to the network control interface of the data acquisition device.

5. The low-latency remote-controlled driving method based on multi-source perception fusion as described in claim 1, characterized in that, The step of performing targeted safety verification on the vehicle control command based on the real-time safety status contained in the fused perception information, and obtaining the verification result, includes: The vehicle control commands are analyzed to obtain the longitudinal speed control quantity and the lateral steering control quantity; Read the real-time obstacle warning sign corresponding to the vehicle's current driving direction from the fused perception information; If the real-time obstacle warning sign indicates that there is an obstacle in the direction of vehicle travel, and the longitudinal speed control value is positive, then a verification result containing an instruction to correct the forward speed component to zero is generated. If the real-time obstacle warning sign indicates that there is an obstacle in the reverse direction of the vehicle, and the longitudinal speed control value is negative, then a verification result containing an instruction to correct the reverse speed component to zero is generated.

6. The low-latency remote-controlled driving method based on multi-source perception fusion as described in claim 1, characterized in that, After the step of receiving the vehicle control command from the remote control terminal, the following steps are included: Record the latest timestamp of successful reception of the vehicle control command; Calculate the interval duration of consecutive invalid instructions based on the current system time and the latest timestamp; If the interval exceeds a preset safety timeout threshold, the network connection is determined to be interrupted, and an emergency stop command with zero longitudinal speed and zero lateral steering is generated to replace the vehicle control command to be verified.

7. The low-latency remote-controlled driving method based on multi-source perception fusion as described in claim 6, characterized in that, After the step of generating an emergency stop command with zero longitudinal speed and zero lateral steering to replace the vehicle control command to be verified, the following steps are included: Based on the current driving mode status fed back by the vehicle chassis controller, perform mode status monitoring; If, during remote-controlled driving, it is detected that the current driving mode has been switched from remote-controlled mode to another mode, the start-up state will continue to be timed. When the duration of the state maintenance time exceeds the preset state maintenance time threshold, the remote control mode enable command is resent to the vehicle chassis controller to reset and maintain the vehicle in remote control mode.

8. A low-latency remote-controlled driving system based on multi-source perception fusion, characterized in that, include: The fusion module is used to perform coordinate transformation and region judgment on the point cloud data based on the vehicle's obstacle point cloud data, real-time pose data, and chassis status data to obtain obstacle collision warning signs in all directions around the vehicle; based on the real-time pose data, the chassis status data, and preset vector map data, it performs trajectory prediction and environmental boundary drawing to obtain a bird's-eye view panoramic image including the vehicle outline, predicted trajectory lines, and map boundary lines; and it integrates the bird's-eye view panoramic image with the obstacle collision warning signs to obtain fused perception information. The conversion module is used to perform hardware encoding on the encoded raw video stream obtained from the vehicle camera and the image frames generated from the fused perception information to obtain a first encoded video stream corresponding to the vehicle's main view and a second encoded video stream corresponding to the fused perception information; the first encoded video stream and the second encoded video stream are encapsulated into data packets based on a real-time transmission protocol to obtain streaming media data, which includes two independent streaming media data streams to be distributed; The distribution module is used to distribute the streaming media data to the remote control terminal via a low-latency network protocol. Different synchronization source identifiers are assigned to the fused perception information and the vehicle's main view screen for synchronous rendering on the remote control terminal. The remote control terminal responds to the synchronously rendered streaming media data and returns vehicle control commands. The verification module is used to receive vehicle control commands from the remote control terminal, and perform targeted security verification on the vehicle control commands based on the real-time safety status contained in the fused perception information, and obtain the verification result. The instruction processing module is used to execute or correct the vehicle control instructions based on the verification results.

Images