Device and method for generating a virtual environment for use in controlling a host vehicle

Abstract

System for generating a virtual environment for use in a vehicle remote control system, comprising an environmental sensor for detecting an environmental condition near the host vehicle, a sensor configured to generate sensor data representing an environment near the host vehicle, a camera configured to capture image data representing the environment near the host vehicle, and a processor configured to determine a network condition, generating a modulated data containing the image data and the sensor data in response to an environmental condition that is less than an environmental threshold and a network condition that exceeds a network threshold condition, and generating the modulated data without the image data in response to at least one of the environmental conditions that exceeds the threshold and a network condition that does not exceed the network threshold condition.

Description

[0001] The present description relates generally to systems and methods for the remote control of a vehicle, and in particular to a method and a device for controlling data transmission in response to communication system conditions and vehicle operating conditions.

[0002] Remote vehicle operation (RVO) systems offer potential benefits such as increased safety, efficiency, and overall convenience for vehicle operators. Remotely operated vehicles can be used in hazardous environments, such as contaminated areas, and can automate parts of the driving task when less human intervention is required, such as when driving on highways. These systems allow a driver or operator to maneuver a vehicle from a remote location, with vehicle data being transmitted from the vehicle to the user and control data from the user to the vehicle. However, current RVO systems have limitations regarding user interaction and safety. They typically require continuous user input throughout the entire driving process.These limitations highlight the need for improved RVO systems that address the problems of constant communication, driver distraction, and incorrect user input.

[0003] Future advancements should focus on developing systems that maintain safety while providing a more user-friendly and reliable experience. One of the biggest challenges is latency. The time it takes for the driver to recognize a situation, react, and send a signal to the vehicle can be critical at high speeds. By the time the vehicle receives the message, the situation on the road may have completely changed. Furthermore, the driver relies on cameras and sensors to gather information, which, compared to a driver physically present in the car, represents a potential interruption. This can lead to misinterpretations and delayed reactions. Additionally, communicating large amounts of data between the vehicle and the vehicle control station can prove difficult when network bandwidth is variable.Sufficient network bandwidth ensures smooth and fast data transmission between the vehicle control station and the vehicle. This is important for two main reasons. First, it allows for the transmission of critical vehicle sensor and control data, enabling timely vehicle data and corresponding maneuvers. Second, high bandwidth enables the transmission of high-resolution video images from the vehicle cameras, allowing the user to accurately perceive the surroundings and react accordingly. If insufficient network bandwidth is available, the vehicle typically enters a safe mode, such as being switched off or moved to a safe location nearby, like the shoulder of the road.

[0004] German patent DE 10 2015 118 489 A1 describes computer devices, systems, and methods for the remote control of an autonomous passenger vehicle. When an autonomous vehicle encounters an unexpected environment, such as roadworks or an obstacle, that is unsuitable for autonomous operation, the vehicle's sensors can collect data about the vehicle and the unexpected environment, including images, radar and LiDAR data, etc. The collected data can be sent to a remote operator. The remote operator can then manually control the vehicle remotely or issue instructions to the autonomous vehicle, which are to be executed by various vehicle systems. The collected data sent to the remote operator can be optimized to save bandwidth, for example, by sending a limited subset of the collected data.

[0005] US 2019 / 0196464A1 describes a remote control device configured to receive a remote control assistance request from an autonomous vehicle and to obtain remote control data in response to the request. Furthermore, the remote control device is configured to store at least some remote control inputs and / or assistance, which are then transmitted to the autonomous vehicle based on those remote control inputs. Upon receiving a subsequent request, the remote control device is configured to repeat at least parts of the previous remote control input and / or to provide an option to activate assistance associated with that remote control input.The remote control device is further configured to train a model to determine from vehicle data an option to present via a remote control interface and / or a presentation configuration of the remote control interface.

[0006] US 2017 / 0192423A1 describes systems and procedures for remotely assisting an autonomous vehicle. The procedure includes collecting sensor data from the autonomous vehicle, identifying a scenario in which assistance is desired, generating an assistance request based on the sensor data, submitting the assistance request to a remote assistance interface, and receiving and processing a response to the assistance request. The remote assistance interface is used to generate the response to the assistance request.

[0007] German patent DE 10 2022 129 929 A1 describes a vehicle system and a method for remotely controlling a self-driving vehicle. The vehicle system is equipped with at least one sensor for generating high-resolution data that characterizes an environment outside the self-driving vehicle, and with a processor that communicates with the at least one sensor. The processor is programmed to generate low-resolution data based on the high-resolution data and to control at least one vehicle actuator based on a driver command. At least one transmitter / receiver transmits the low-resolution data to and receives the driver command from a remote driving system.

[0008] It can be considered a task to specify improved systems and procedures for the transmission and reception of data between a remotely controlled vehicle and a remote vehicle control station.

[0009] The problem is solved by a device according to claim 1 and a method according to claim 9 for generating a virtual environment for use in controlling a host vehicle.

[0010] In one application, a vehicle control system comprises a device according to the invention for generating a virtual environment for use in controlling a host vehicle.

[0011] The device according to the invention for generating a virtual environment for use in controlling the host vehicle comprises an environmental sensor for detecting an environmental condition near the host vehicle, a sensor configured to generate sensor data representing an environment near the host vehicle, a camera configured to capture an image representing the environment near the host vehicle, a processor configured to determine a network condition, and to generate modulated data including the image and sensor data in response to the environmental condition being less than an environmental limit and the network condition exceeding a network limit condition, and to generate modulated data without the image in response to at least one of the environmental conditions exceeding the environmental limit and the network condition not exceeding the network limit condition.to generate a transmitter-receiver configured to send the modulated data to a remote control station and to receive vehicle control data from the remote control station, and a vehicle controller to control the host vehicle in response to the vehicle control data. The processor is further configured to generate a virtual environment in response to the sensor data, where the resolution of the virtual environment is determined in response to a size of the network condition.

[0012] According to one embodiment, the processor is further configured to generate the virtual environment in response to the sensor data, and the modulated data contains the virtual environment for display on the remote control station.

[0013] According to one embodiment, the environmental condition is a visibility that is determined in response to the image.

[0014] According to one embodiment, the network condition is a bandwidth of a communication network.

[0015] According to one definition, the network condition is a bit error rate of a communication network.

[0016] According to one embodiment, the modulated data contains an object list, and each of a plurality of objects listed in the object list is represented as a bounding rectangle.

[0017] According to one embodiment, the modulated data contains an object list, and each of a plurality of objects listed in the object list is represented as one of a plurality of bounding rectangles, and the size of each of the plurality of bounding rectangles is inversely proportional to a size of the network condition.

[0018] According to one embodiment, the modulated data contains an object list and at least one of a plurality of objects listed in the object list is represented as a bounding rectangle that represents a collection of objects within the environment near the host vehicle.

[0019] A method according to the invention for generating a virtual environment for use in controlling a host vehicle comprises: determining an environmental condition in response to sensor data representative of an environment near the host vehicle; determining a network condition in response to transmitter-receiver power data; acquiring an image of the environment near the host vehicle; generating, by a processor, modulated data containing the image and the sensor data in response to the environmental condition being less than a threshold and the network condition exceeding a network power threshold; generating the modulated data without the image in response to at least one of the environmental conditions exceeding the threshold and the network condition being less than the network power threshold; and transmitting the modulated data to a remote control station.Receiving vehicle control data from the remote control station and a vehicle controller to control the host vehicle in response to the vehicle control data. Generating the modulated data further includes generating a virtual environment in response to the sensor data and determining a resolution of the virtual environment in response to a size of the network condition.

[0020] According to one embodiment, a method in which the generation of the modulated data further comprises the generation of the virtual environment in response to the sensor data, and in which the modulated data comprises the virtual environment for display on the remote control station.

[0021] According to one embodiment, the environmental condition is a visibility that is determined in response to a saturation level of the image.

[0022] According to one embodiment, the modulated data contains an object list, and each of a plurality of objects listed in the object list is represented as a bounding rectangle.

[0023] According to one embodiment, the modulated data contains an object list and at least one of a plurality of objects listed in the object list is represented as a bounding rectangle that represents a collection of objects within the environment near the host vehicle.

[0024] According to one embodiment, the modulated data contains an object list, and each of a plurality of objects listed in the object list is represented as one of a plurality of bounding rectangles, and the size of each of the plurality of bounding rectangles is inversely proportional to the size of the network condition.

[0025] According to one embodiment, the network condition is determined in response to a network condition indicator received from the remote control station.

[0026] According to one embodiment, the modulated data is further generated in response to a request from the remote control station to exclude the image from the modulated data.

[0027] The exemplary embodiments are described below in conjunction with the following drawings, where identical numbers denote identical elements and where: Fig. Figure 1 shows an exemplary vehicle system for virtual environment creation for vehicle teleoperation; Fig. Figure 2 shows an exemplary environment for virtual environment creation for vehicle teleoperation; Fig. Figure 3 shows an example block diagram for virtual environment creation for vehicle teleoperation; Fig. Figure 4 shows an exemplary decision and data flow diagram for virtual environment creation for vehicle teleoperation; and Fig. Figure 5 shows a flowchart illustrating a procedure for generating virtual environments for vehicle teleoperation.

[0028] As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic and / or processor component, individually or in any combination, including but not limited to: application-specific integrated circuit (ASIC), an electronic circuit, a processor (common, dedicated or group) and memory executing one or more software or firmware programs, a combinational logic circuit and / or other suitable components providing the described functionality.

[0029] In Fig. Figure 1 shows an exemplary vehicle system 100 for generating virtual environments for the teleoperation of vehicles in accordance with various embodiments. The exemplary system 100 comprises a vehicle 10 with a plurality of sensor devices 40a-40n, a drive system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one control unit 34, and a communication system 36.

[0030] As in Fig. As shown in Figure 1, the vehicle 10 generally comprises a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is mounted on the chassis 12 and essentially encloses the components of the vehicle 10. The body 14 and the chassis 12 can together form a frame. The wheels 16-18 are each rotatably connected to the chassis 12 near a corner of the body 14.

[0031] In various embodiments, the vehicle 10 is an autonomous vehicle, and the control system 100 is integrated into the autonomous vehicle 10 (hereinafter referred to as autonomous vehicle 10). The autonomous vehicle 10 is, for example, a vehicle that is automatically controlled to transport passengers from one place to another. In the embodiment shown, the vehicle 10 is depicted as a passenger car, but it should be understood that any other vehicle, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), watercraft, aircraft, etc., can also be used. In one exemplary embodiment, the autonomous vehicle 10 is a so-called Level Four or Level Five automation system. A Level Four system means "high automation," i.e.,An automated driving system performs all aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request for intervention. A Level 5 system means "full automation" and refers to the complete execution of all aspects of the dynamic driving task by an automated driving system under all road and environmental conditions that a human driver could handle. As can be seen, the vehicle, in various embodiments, can be a non-autonomous vehicle where the human driver requires remote assistance to control the vehicle for a certain period of time, and is not limited to the examples presented here.

[0032] As shown, the vehicle 10 generally comprises a drive system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one control unit 34, and a communication system 36. The drive system 20 may, in various embodiments, comprise an internal combustion engine, an electric machine such as a traction motor, and / or a fuel cell drive system. The transmission system 22 is configured to transmit the power of the drive system 20 to the vehicle wheels 16-18 in selectable gear ratios. According to various embodiments, the transmission system 22 may comprise a continuously variable automatic transmission, a continuously variable transmission, or another suitable transmission. The braking system 26 is configured to exert a braking torque on the vehicle wheels 16-18.The braking system 26 can, in various embodiments, comprise friction brakes, wire-operated brakes, a regenerative braking system such as an electric motor, and / or other suitable braking systems. The steering system 24 influences the position of the vehicle wheels 16-18. Although a steering wheel is shown for illustrative purposes, the steering system 24 may not include a steering wheel in some embodiments considered within the scope of this description.

[0033] The sensor system 28 comprises one or more sensor devices 40a-40n that detect observable conditions of the external environment and / or the internal environment of the autonomous vehicle 10. The sensor devices 40a-40n may include, but are not limited to, radars, lidar, global positioning systems, optical cameras, thermal cameras, ultrasonic sensors and / or other sensors.

[0034] In various embodiments, the sensor devices 40a-40n are arranged at different locations on the vehicle 10. In the exemplary embodiments described here, one or more of the sensor devices 40-40n are implemented as lidar devices. In this respect, each of the sensor devices 40a-40n can include or incorporate one or more lasers, scanning components, optical arrangements, photodetectors, and other components that are suitably configured to scan the environment near the vehicle 10 horizontally and rotatably at a specific angular frequency or rotational speed. In the exemplary embodiments described here, one or more of the sensor devices 40a-40n are configured as optical cameras to capture images of the environment near the vehicle 10.

[0035] The actuator system 30 comprises one or more actuator devices 42a-42n that control one or more vehicle functions, such as, but not limited to, the drive system 20, the transmission system 22, the steering system 24, and the braking system 26. In various embodiments, the vehicle features may also include interior and / or exterior features of the vehicle, such as doors, a trunk, and cabin features such as air conditioning, music, lighting, etc. (not numbered).

[0036] In exemplary embodiments, which still refer to Fig. 1. The communication system 36 is designed to wirelessly transmit information to and from other units 48, such as, but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems and / or remote control stations (RCS) (described in more detail in relation to Fig. 2) In an exemplary embodiment, the communication system 36 is a wireless communication system configured for communication over a wireless local area network (WLAN) using IEEE 802.11 standards or using cellular data communication. However, additional or alternative communication methods, such as a dedicated short-range communication channel (DSRC), are also considered within the scope of this description. DSRC channels refer to one-way or two-way short- to medium-range wireless communication channels specifically designed for use in motor vehicles, as well as a number of corresponding protocols and standards.

[0037] The data storage device 32 stores data for use in the automatic control of the autonomous vehicle 10. In various embodiments, the data storage device 32 stores defined maps of the navigable environment. In various embodiments, the defined maps can be predefined by and obtained from a remote system. For example, the defined maps can be compiled by the remote system and transmitted to the autonomous vehicle 10 (wirelessly and / or via a wired connection) and stored in the data storage device 32. In some exemplary embodiments, this process can be carried out before the vehicle is operated in its environment. In various embodiments, the data storage device 32 stores calibrations for use in aligning the sensor devices 40a-40n.In various embodiments, one or more of the calibrations are estimated as extrinsic parameters using the methods and systems described herein. As can be seen, the data storage device 32 can be part of the control unit 34, separate from the control unit 34, or part of the control unit 34 and part of a separate system.

[0038] The control unit 34 comprises at least one processor 44 and a computer-readable storage device or medium 46. The processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the control unit 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or generally any device for executing instructions. The computer-readable storage devices or media 46 can include volatile and non-volatile memory, such as read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM). KAM is persistent or non-volatile memory that can be used to store various operating variables while the processor 44 is powered off.The computer-readable storage device(s) 46 can be implemented using any number of known storage devices such as PROMs (programmable read-only memory), EPROMs (electrical PROM), EEPROMs (electrically erasable PROM), flash memory, or other electrical, magnetic, optical, or combined storage devices capable of storing data, some of which may be executable instructions used by the control unit 34 in controlling the autonomous vehicle 10.

[0039] The instructions can comprise one or more separate programs, each containing an ordered list of executable instructions for implementing logical functions. When executed by the processor 44, the instructions receive and process signals from the sensor system 28, perform logic, calculations, procedures, and / or algorithms for the automatic control of the components of the autonomous vehicle 10, and generate control signals for the actuator system 30 to automatically control the components of the autonomous vehicle 10 based on the logic, calculations, procedures, and / or algorithms. Although in Fig. While only one control unit 34 is shown in Figure 1, embodiments of the autonomous vehicle 10 may include any number of control units 34 that communicate via any suitable communication medium or combination of communication media and cooperate to process the sensor signals, perform logic, calculations, procedures and / or algorithms, and generate control signals to automatically control features of the autonomous vehicle 10. In various embodiments, one or more instructions from the control unit 34 are embodied in the control system 100 and, when executed by the processor 44, cause the processor to perform the procedures and systems that dynamically align the sensor devices by updating the calibrations stored in the data storage device 32, as described in more detail below.

[0040] In accordance with various embodiments, the control unit 34 implements an autonomous driving system (ADS). Software and / or hardware components of the control unit 34 (e.g., processor 44 and computer-readable storage device 46) are used to provide an autonomous driving system that, in conjunction with the vehicle 10, is used, for example, to automatically control various actuators 30 on board the vehicle 10, thereby controlling vehicle acceleration, steering, and braking without human intervention.

[0041] In various embodiments, the instructions of the autonomous driving system can be structured according to functions or systems. For example, the autonomous driving system can include a computer vision system, a positioning system, a guidance system, and a vehicle control system. As can be seen, in different embodiments, the instructions can be organized into any number of systems (e.g., combined, further subdivided, etc.), since the description is not limited to the examples presented.

[0042] In various embodiments, the computer vision system synthesizes and processes sensor data and predicts the presence, location, classification, and / or path of objects and features in the vehicle's environment 10. In various embodiments, the computer vision system can incorporate information from multiple sensors, including but not limited to cameras, lidars, radars, and / or any number of other sensor types. In various embodiments, the computer vision system receives information from and / or implements information from the control system 100 described herein.

[0043] The positioning system processes sensor data along with other data to determine the position (e.g., a local position relative to a map, a precise position relative to a lane of a road, vehicle direction, speed, etc.) of the vehicle 10 in relation to its environment. The guidance system processes sensor data along with other data to determine a path for the vehicle 10 to follow. The vehicle control system generates control signals to steer the vehicle 10 according to the determined path.

[0044] In various embodiments, the control unit 34 implements machine learning techniques to support the functionality of the control unit 34, such as feature recognition / classification, obstacle avoidance, route traversal, mapping, sensor integration, ground truth determination, and the like.

[0045] In Fig. Figure 2 shows an exemplary environment 200 for virtual environment creation for vehicle teleoperation in accordance with various embodiments. The exemplary environment 200 includes a nearby vehicle 210, a host vehicle 230, a road surface 240, a roadside unit (RSU) 220, and an RCS 250. Vehicle teleoperation is considered a possible solution for remotely assisting vehicles equipped with advanced driver assistance systems (ADAS) in situations where they encounter difficulties or require human intervention. The RCS 250 can act as an extension of the driver, allowing them to control and interact with the host vehicle 230 as if they were physically present.

[0046] An ADAS-equipped vehicle is one capable of perceiving its surroundings and navigating with little or no user input. An ADAS-equipped vehicle perceives its environment using sensors such as radar, lidar, image sensors, and the like. The ADAS-equipped vehicle system can also utilize information from global positioning systems (GPS), navigation systems, vehicle-to-vehicle (V2V) communication, vehicle-to-infrastructure (V2I) technology, and / or drive-by-wire systems to navigate the vehicle. Vehicle automation has been categorized into numerical levels ranging from zero, meaning no automation with complete human control, to five, meaning complete automation without any human control. Various ADAS systems, such as...Cruise control, adaptive cruise control and parking assistance correspond to lower levels of automation, while true “driverless” or autonomous vehicles correspond to higher levels of automation.

[0047] In some exemplary embodiments, the host vehicle 230 can use various sensors, onboard data storage, and V2V communication 211 to receive data, such as image, sensor, or mapping data, from one or more neighboring vehicles 210. The data received via V2V communication 211 can include precise information about the location, speed, and direction of one or more neighboring vehicles 210. Furthermore, V2V communication 211 can provide details about the braking status, turn signals, and even hazard warnings from neighboring vehicles 210. By incorporating this collaborative data stream, the host vehicle 230 can make more informed decisions, such as anticipating maneuvers by surrounding vehicles, mitigating potential collisions, and optimizing traffic flow for greater safety and efficiency.

[0048] Similarly, the host vehicle 230 can receive data via V2I communication 221 to enhance its situational awareness and improve its decision-making processes. This data can include real-time traffic signal timings, enabling optimized vehicle speeds for smoother stops and reduced brake wear. RSUs 220, such as roadside sensors, can transmit information about impending hazards, such as accidents or broken-down vehicles, allowing for rerouting or speed adjustments to improve traffic flow. V2I communication 221 can even provide details about weather conditions along the route, including slippery roads or poor visibility, enabling the autonomous vehicle to adapt its driving strategy to prioritize safety.V2I communication 221 facilitates the transmission of information about available parking spaces, enabling the vehicle to efficiently find a parking space and thereby minimizing traffic jams caused by circling cars.

[0049] In some exemplary embodiments, the RSUs 220 can serve as a backbone for data exchange during remote vehicle control. The RSUs 220 enable seamless communication between the host vehicle 230 and the RCS 250. In some exemplary embodiments, the RSUs 220 can be connected to the RCS 250 via other wired or wireless networks, such as cellular networks, dedicated wired or wireless networks, and / or the internet. The base RSUs 220 can prioritize reliable data transmission, thus ensuring a stable connection for exchanging critical information. This data can include the timing of traffic signals, allowing vehicles to optimize their speed to ensure smoother traffic flow at intersections, or real-time warnings about hazards such as accidents or obstructed vehicles on the road.Some RSU 220s can be equipped with environmental sensors to collect real-time data on weather conditions or traffic flow. This data can be used to dynamically adjust the timing of traffic signals to optimize road capacity and reduce congestion. RSUs can also be used to relay critical traffic information or safety warnings directly to drivers and vehicles equipped with ADAS. For example, an RSU 220 can send warnings in poor visibility due to fog or heavy rain, advising drivers to reduce their speed and maintain a greater following distance.

[0050] The RCS 250 can include a workstation and / or a user interface from which an operator controls the host vehicle 230, which is located at a distance from the RCS 250. The user interface can include joysticks, steering wheels, pedals, buttons, and levers. The workstation can include one or more screens for displaying live video images from cameras mounted on the vehicle and / or data generated by the host vehicle, such as three-dimensional object maps or similar. This gives the operator a real-time view of the surroundings, which is crucial for safe and effective control. Additional displays can show data such as speed, battery level, or sensor readings. The RCS 250 can communicate with the host vehicle via a communication link, such as a cellular network or the RSU 220.Low latency (minimal delay) and high bandwidth are desirable for smooth and responsive control of the host vehicle 230 and reliable transmission of video, sensor data, and other environmental data from the host vehicle 230 to the RCS 250. The RCS 250 can include additional software functionalities for controlling vehicle behavior beyond basic movement, such as autonomous navigation waypoints for pre-programmed movements or the activation or configuration of ADAS algorithms to be executed by the host vehicle's control unit.

[0051] In some exemplary embodiments, data transmission problems may occur between the host vehicle 230 and the RSU 250. These data transmission problems can be caused by signal interference, such as electromagnetic interference from other transmitters, physical obstacles such as buildings or tunnels, or damage to the transmitter or transmitting antenna. Furthermore, increased latency can reduce the responsiveness of the host vehicle systems, causing the vehicle control to become sluggish or unresponsive, thus hindering precise control of the host vehicle 230. Additionally, increased latency can lead to unstable control, as the operator may be controlling the vehicle based on outdated information.To address this problem, the exemplary host vehicle communication system can reduce the amount of data transmitted between the host vehicle 230 and the RSU 250 in response to a limitation of the available bandwidth and / or in response to errors detected in the digital data. Likewise, the RSU 250 and / or the host vehicle control unit can initiate ADAS control algorithms in response to detected data corruption, a reduction in bandwidth, or a detected increase in latency.

[0052] The exemplary teleoperating system can prioritize efficient information transmission by transmitting a processed representation of the vehicle's surroundings instead of raw sensor data, such as video streams. In some exemplary embodiments, the optimized data stream can transmit a list of objects identified by onboard processing, eliminating the need for redundant transmission of static elements, such as road boundaries, obtained from preloaded maps. To further improve bandwidth efficiency, the system can generate bounding rectangles around objects and dynamically group nearby objects so that a single data point is transmitted instead of multiple points, and highlight missing data when environmental conditions, such as fog, limit sensor effectiveness.This adaptive strategy adjusts the information sent to the real-time network conditions and the operator's preferences, ensuring that critical control data is prioritized while minimizing overall data consumption.

[0053] In some exemplary embodiments, the exemplary teleoperating system can be configured to leverage the inherent ADAS capabilities of the host vehicle for enhanced versatility. By integrating these capabilities, the system can seamlessly switch between remote control and ADAS operation based on predefined parameters or driver selection. This allows the system to adapt to diverse environments, such as off-road expeditions and highway driving. For example, in a terrain scenario with limited visibility, the system can prioritize transmitting data about large obstacles that impede progress, while relying on the vehicle's autonomous capabilities for basic navigation.Conversely, on a highway with clear visibility and predictable traffic patterns, the system can switch to a more manual control mode, providing the truck driver with a detailed object list for precise maneuvering. This integration of remote control and autonomous functions enables a highly adaptable and effective system for various operational situations.

[0054] In Fig. Figure 3 shows an exemplary block diagram for a vehicle system 300 for control by a vehicle telecommunication unit according to various embodiments. The exemplary system 300 comprises a transmitter-receiver 305, a lidar 312, a lidar processor 320, a camera 310, an image processor 315, a sensor processor 330, a memory 340, and a vehicle control unit 350.

[0055] The Lidar 312 is designed to emit a light pulse at a known angle and altitude and measure the pulse's travel time. Based on this travel time, the Lidar 312 can then determine the distance to an object at that known angle and altitude. The Lidar 312 can repeat this process for multiple angles and altitudes to generate a point cloud of depths to objects within the Lidar's field of view (FOV). Typically, the light pulses are emitted at regular angular intervals, such as 0.1 degrees, and at regular altitude intervals. The greater the number of detection points grouped in the point cloud, the longer the Lidar takes to fully scan the field of view. A Lidar point cloud with a high density of 3D points requires longer data acquisition intervals but provides higher-resolution data with rich features that can be used for alignment.

[0056] In some exemplary embodiments, the lidar 312 can be configured to couple the detected depths for each of the angles and heights to the lidar processor 320 as individual points or as a point cloud. The lidar processor 320 can then generate a 3D contour of target vehicles in response to the points or point cloud. Furthermore, the lidar processor 320 can generate a 3D representation of the field of view, including the detection and classification of objects within the field of view.

[0057] One or more cameras 310 can be mounted on the host vehicle, each with a field of view (FOV) of an area adjacent to the host vehicle, such as a front FOV, rear FOV, or side FOV. In some exemplary embodiments, the images captured by the different cameras 310 can be combined to produce a continuous FOV view. The forward-view camera 310 can be mounted inside the vehicle behind the rearview mirror or on the front bumper of the vehicle. The cameras 310 can use captured images and / or videos, which can be used to detect vehicles ahead and behind, obstacles, lane markings, lane edges, road features, other road markings, and road hazards during ADAS operation. The images captured by the camera 310 and the data generated from the images can be used to supplement the map data stored in the memory 340.The images captured by the camera(s) 310 can be combined with the image processor 315 for object recognition. The image processor 315 can execute image processing algorithms in response to the image, such as Canny edge detection algorithms to detect edges within the image, and RCNN segmentation to detect vehicle contours. These detected edges can then be used to detect object outlines. These object outlines can be used to set boundaries around the detected object, as well as to identify and classify the detected objects.

[0058] In some exemplary embodiments, the sensor system processor 330, or a comparable processor or sensor data processing system, can receive the two-dimensional edge-detected image data from the image processor 315 and the three-dimensional point cloud from the lidar processor 320. This data can then be used to calibrate the orientation of the sensors so that, for example, detected objects are determined to be in the same location for each sensor. Furthermore, additional processing can be performed to fuse the sensor data and execute other vehicle algorithms in response to the fused data.

[0059] The sensor system processor 330 can be further configured to receive location data from a GPS (Global Positioning System) receiver 335 and store this location data in the memory 340. The memory 340 can be operated to store map data for use by the sensor system processor 330. The memory 340 can further store map data, which may be high-resolution map data containing detailed representations of roads, including precise road positions, lane positions, curves, elevations, known road hazards, and other road details.

[0060] In some exemplary embodiments, the sensor system processor 330 can be configured to monitor network and environmental conditions and to estimate data transmission capabilities. In response to the network and environmental conditions, the sensor system processor 330 can generate a virtual environment for transmission to a remote control system (RCS) to provide a remote teleoperator with a visual representation of the three-dimensional environment around the host vehicle. In some exemplary embodiments, the sensor system processor 330 can reduce the resolution of the virtual environment to be transmitted in response to a reduced bandwidth of the data transmission network, an increase in the bit error rate, or the like. Likewise, the sensor system processor 330 can reduce the sensor data to be transmitted to the RCS in response to environmental conditions such as fog, rain, snow, etc.In the case of dense fog, the sensor system processor 330 may decide not to transmit the video received from the cameras and may transmit the lidar depth map, radar data, and / or generate a virtual environment in response to the lidar and radar data for transmission to the RCS. Additionally, in response to limited bandwidth or a high bit error rate, the sensor system processor 330 may replace object data within the virtual environment with bounding rectangles that cover the positions of the replaced objects. In some exemplary embodiments, multiple bounding rectangles for closely spaced objects may be combined into a single bounding rectangle to further reduce the amount of data to be transmitted.

[0061] In Fig. Figure 4 shows a function block diagram 400, which illustrates an exemplary decision and data flow diagram for generating a virtual environment for vehicle teleoperation in accordance with various embodiments. The function block diagram can generally be divided into in-vehicle functions 403 and RSU functions 423. In some exemplary embodiments, the in-vehicle functions 403 may include environmental sensing 401, additional data acquisition 405, environmental and network monitoring and decision-making 410, data processing 415, and vehicle control 420. The RSU functions 423 may include map data acquisition 425, V2I data acquisition 430, satellite data acquisition 435, virtual environment creation and verification 440, human-machine interface (HMI) 445, and remote control 450.

[0062] Exemplary in-vehicle functions 403 and RSU functions 423 are used to create a virtual environment based on raw and metadata for remote vehicle operation. Remotely operating a vehicle using a virtual environment facilitates the control of a host vehicle, immersing the remote operator in a realistic, real-time virtual environment that replicates the host vehicle's surroundings. To address challenges such as latency and security, the exemplary system and procedures for remote teleoperator operation of a vehicle create the virtual environment, taking into account network congestion, optimizing the representation of the perceived environment, and performing a validation of the created virtual environment.

[0063] The vehicle's in-vehicle functions 403 and in-vehicle systems are designed to detect an environment 401 around the host vehicle. This environment can be detected using vehicle sensors, such as radar and / or lidar, cameras, and the like. Detection 401 can include the detection of static objects, such as buildings or parked vehicles, and dynamic objects, such as pedestrians or approaching vehicles. Additional information 405 is acquired via V2I, V2V, and / or vehicle-to-everything (V2X) communication networks. This additional information can include data on surrounding objects, such as the location and speed of nearby vehicles, and the locations of permanent static objects like pedestrian crossings, curbs, light poles, etc.

[0064] The Vehicle Perception Function 415 is designed to combine the various sensor data and other received data to create a three-dimensional environment and object map in real time. A three-dimensional point cloud can be generated from a collection of depth measurements from the lidar system. The complementary radar technology can continuously transmit radio waves, enabling the detection of nearby objects (including other vehicles and pedestrians) as well as their relative position and speed. Cameras, acting as a visual perception system, capture important data about lane markings, traffic signals, and visual details for object identification.To transform this raw sensor data into a usable map, advanced algorithms such as simultaneous localization and mapping (SLAM) are used to merge the information from each sensor and refine the three-dimensional environment map in real time.

[0065] The 410's environmental and network monitoring and decision-making function can monitor both the health of the communications network and the environmental conditions near the host vehicle. The health of the communications network can be characterized by monitoring critical metrics such as signal strength and throughput. Signal strength, measured by the Received Signal Strength Indicator (RSSI), is evaluated for both individual devices and access points to identify potential coverage gaps or weak signal zones. Network coverage analysis tools enable the localization of dead zones, while ping sweeps provide a comprehensive overview of connected clients and reveal potential rogue devices. Throughput monitoring tracks data transfer rates and helps isolate bottlenecks and congested areas.Packet loss analysis examines the integrity of data transmissions and uncovers potential sources of interference or congested access points. Environmental conditions, such as fog, rain, snow, etc., can also affect data integrity.

[0066] In response to network and weather conditions, the data processing function 415 can determine which data to transmit to the RCS 423 in response to latency and network conditions, in order to provide the remote operator with sufficient data for successful vehicle operation. For example, in foggy conditions, the data processing function 415 can determine that video streaming would not be useful for a remote operator and that the three-dimensional environment map would be more suitable. Similarly, data from a host vehicle sensor can be excluded from transmission to the RCS 423 if the sensor is in a fault condition. In some exemplary embodiments, the data processing function 415 can reduce the resolution of the three-dimensional environment map in response to reduced network bandwidth or a reduced data transmission rate.Similarly, objects within the three-dimensional environment map can be represented as bounding rectangles around the object's location, thereby reducing the amount of data to be transmitted. In some exemplary embodiments, where communication bandwidth is extremely limited, multiple objects can be grouped into a single cluster, resulting in the three-dimensional environment map containing multiple clusters. Clusters can be determined in response to overlapping objects or objects between which the space is not wide enough for the host vehicle. For example, a row of parked cars can be grouped into a single bounding rectangle. Clustering can be further generated in response to host vehicle information, such as vehicle speed, steering angle, location, and optimized field of view, e.g., for a left turn, where information about the right side may be redundant and can be clustered.

[0067] Once the relevant data to be transmitted has been determined by the data processing function 415, the data is transferred to the virtual environment creation and verification function 440. The virtual environment creation and verification function 440 is executed on the RCS 423 and is configured to generate a virtual environment for display on the HMI 445 to enable remote control. The virtual environment creation and verification function 440 can use map data 425, such as Google Maps data and / or HD map data, with V2I data and satellite data 435 and / or imagery to determine the locations of static objects, such as road boundaries, buildings, roadside structures, etc.

[0068] In some exemplary embodiments, the remote teleoperator 450 can determine which data is transmitted from the data system processor 415 via inputs to the HMI 455. For example, if the video data is choppy, has high latency, or poor resolution, the remote teleoperator 450 can choose to receive a virtual environment created by the data system processor 415. In some exemplary embodiments, the logic can calculate which data to transmit, taking into account the network latency, the speed of the ego vehicle, and the estimated speed of other objects in the environment. The request could then be sent to the data system processor 415 to transmit only the sensor data required by the virtual environment creation and verification function 440 to create the virtual environment for display on the HMI 445.

[0069] The HMI 445 can also include user inputs for remote control of the host vehicle 403 by the remote operator 450. The HMI 445 can include a display, such as a computer screen or a virtual reality device, to show the generating virtual environment, including a received object list, a video, or a combination of both, to the remote operator 450. The exemplary HMI 445 can also display a trajectory of the host vehicle for the current steering, braking, and throttle inputs for a defined period of time. The human-machine interface 4445 can include steering inputs that read control signals from a steering device, such as a steering wheel or a control unit, and send the steering signals and the steering device name to the host vehicle control function 420.Similarly, inputs regarding the accelerator and brake pedal positions at the human-machine interface 445 can be transmitted to the host vehicle control function 420. In some exemplary embodiments, the HMI 445 can also include a switching input for controlling a transmission in the host vehicle 403 and / or an emergency stop for halting the host vehicle if necessary.

[0070] In Fig.Figure 5 is a flowchart illustrating an exemplary implementation of Method 500 for generating a virtual environment for vehicle teleoperation in accordance with various embodiments. Method 500 can be configured by a microprocessor or controller in a host vehicle to generate data for transmission to a remote control system (RCS). Method 500 is initially configured to receive data from various host vehicle sensors, such as lidar, radar, cameras, internal measurement units (IMUs), and the like. Method 500 can also receive data from memory, such as map data, and data from other vehicles or infrastructure, such as via V2V and V2I communication networks.

[0071] In response to the received data, Procedure 500 next determines the environmental conditions near the host vehicle. These environmental conditions can include fog, rain, snow, temperature, and ambient light intensity. Some environmental conditions can be determined directly from the relevant sensor data, such as temperature, humidity, and ambient light from a built-in thermometer, hygrometer, or light sensor, respectively. Furthermore, environmental conditions can be inferred in response to other sensor data, such as camera images, lidar data, or similar sources. For example, in response to dense fog, it can be assumed that a fully saturated image and lidar depth data will not indicate an object near the camera within the camera's field of view.

[0072] In response to the environmental conditions, the procedure 500 can next determine whether any environmental conditions 515 are exceeded that would render some of the sensor data unusable. If none of the environmental conditions are exceeded, the procedure 500 transmits the standard sensor data 525. If one or more of the environmental conditions are exceeded, the procedure then modulates the sensor data 520 and determines which sensor data should be excluded from the data to be transmitted to the RCS. For example, in dense fog, the image data that would be unusable for the RCS can be excluded from the data to be transmitted. After the data to be transmitted has been modulated with regard to the exceeded environmental conditions, the procedure 500 then transmits the modulated sensor data 525 to the RCS.

[0073] In response to the transmission of sensor data and the reception of control data from the RCS, Procedure 500 determines the network conditions between the host vehicle and the RCS. These network conditions can be determined by analyzing Key Network Performance Indicators (KPIs). Bandwidth utilization, measured as a percentage of consumed network capacity, provides an important understanding of resource allocation. Latency, the time it takes for data to travel across the network, is critical for real-time applications. Packet loss, the number of data packets that do not reach their destination, significantly impacts data integrity. Monitoring device connectivity and analyzing traffic patterns helps identify potential bottlenecks and congestion points.

[0074] In response to the prevailing network state, Procedure 500 can next modulate sensor data and other data to be transmitted to the RCS. For example, to reduce bandwidth, video can be omitted from the transmitted data, leaving only lidar depth and radar data. This transmitted data can then be used by the RCS to generate a virtual environment with corrected object positions, which is projected for display to a remote driver based on network latency calculations. Similarly, object lists for static and dynamic objects can be modulated to contain only bounding rectangles for specific objects. To further reduce the amount of data to be transmitted, objects can be grouped together if they are close to each other or located in an area that the host vehicle is unlikely to traverse.This cluster boundary rectangle information can then be transferred to the RCS for use in creating the virtual environment.

[0075] In some exemplary embodiments, ADAS systems within the remotely controlled host vehicle can be activated in response to severely degraded network conditions and / or network performance. For example, during highway operation, lane centering and adaptive cruise control can be activated for a limited time under severely impaired network conditions, such that a sufficient virtual environment cannot be generated for display to the remote teleoperator, or in the event of a loss of connection to vehicle control data from the RCS. In some exemplary embodiments, in the event of a communication loss, a safe shutdown procedure can be initiated, in which the host vehicle's ADAS steers the host vehicle to a safe stopping point.

[0076] Once the required data has been modulated in response to the current network state, the modulated data 540 is transmitted to the RCS. The modulated data can then be used with transmitted sensor data to create a virtual environment 545 by the RCS. The virtual environment can then be displayed to a remote user 550 for use in remotely controlling the host vehicle.

Claims

[1] Device for generating a virtual environment for use in controlling a host vehicle (230), comprising: an environmental sensor for detecting an environmental condition near the host vehicle (230); a sensor that is designed to generate sensor data that is representative of an environment (200) in the vicinity of the host vehicle (230); a camera (310) designed to capture an image representative of the surroundings (200) near the host vehicle (230); a processor trained to determine a network condition, and to generate modulated data including the image and sensor data in response to the environmental condition being less than an environmental limit and the network condition exceeding a network limit condition, and to generate the modulated data without the image in response to at least one of the environmental conditions exceeding the environmental limit and the network condition not exceeding the network limit condition; a transmitter-receiver (305) configured to transmit the modulated data to a remote control station (250) and to receive vehicle control data from the remote control station (250); and a vehicle control unit (350) for controlling the host vehicle (230) in response to the vehicle control data; wherein the processor is further trained to generate a virtual environment in response to the sensor data and wherein a resolution of the virtual environment is determined in response to a size of the network condition. [2] Device according to claim 1, wherein the processor is further configured to generate the virtual environment in response to the sensor data and wherein the modulated data includes the virtual environment for display on the remote control station (250). [3] Device according to claim 1, wherein the environmental condition is a visibility that is determined in response to the image. [4] Device according to claim 1, wherein the network condition is a bandwidth of a communication network. [5] Device according to claim 1, wherein the network condition is a bit error rate of a communication network. [6] Device according to claim 1, wherein the modulated data includes an object list and wherein each of a plurality of objects listed in the object list is represented as a bounding rectangle. [7] Device according to claim 1, wherein the modulated data contains an object list and wherein each of a plurality of objects listed in the object list is represented as one of a plurality of bounding rectangles and wherein a size of each of the plurality of bounding rectangles is inversely proportional to a size of the network condition. [8] Device according to claim 1, wherein the modulated data includes an object list and wherein at least one of a plurality of objects listed in the object list is represented as a bounding rectangle that represents a collection of objects within the environment (200) in the vicinity of the host vehicle (230). [9] Method for generating a virtual environment for use in controlling a host vehicle (230), comprising: Determining an environmental condition in response to sensor data that is representative of an environment (200) near the host vehicle (230); Determining a network condition in response to transmitter-receiver performance data, Capturing an image of the surroundings (200) near the host vehicle (230); Generating, by a processor, modulated data including the image and sensor data in response to the environmental condition being less than a threshold and the network condition exceeding a network performance threshold, and generating the modulated data without the image in response to at least one of the environmental conditions exceeding the threshold and the network condition being less than the network performance threshold; Transmitting the modulated data to a remote control station (250); Receiving vehicle control data from the remote control station (250); and a vehicle control unit (350) for controlling the host vehicle (230) in response to the vehicle control data; wherein the generation of the modulated data further includes the generation of a virtual environment in response to the sensor data and wherein a resolution of the virtual environment is determined in response to a size of the network condition.

Images