Method for adjusting driving dynamics parameters for a remote-controllable vehicle, and remote-control system

Abstract

The invention relates to a method for adjusting driving dynamics parameters (40) for a vehicle (14) which is remote-controllable from a control station (12), wherein a sensor device (22) of the vehicle (14) captures image data (18) describing the surroundings of the vehicle (14), wherein the image data (18) are transmitted as an image data stream (18) from the vehicle (14) to the control station (12) via a data connection (16), the method comprising the following steps, which are carried out by a computing device (30): - determining a present transmission quality of the data connection (16) between the vehicle (14) and the control station (12) and, on the basis of the transmission quality, determining a present quality factor of the image data stream (18) or of individual frames of the image data stream (18); - determining, on the basis of the determined present quality factor, a present perception factor (36) describing the degree to which a present perception of the surroundings of the vehicle (14) by a user operating the control station (12) is reduced in relation to a predetermined standard perception; and - adjusting the driving dynamics parameters (40) taking into consideration the present perception factor (36).

Description

[0001] Method for adapting driving dynamics parameters for a remote-controllable vehicle and remote control system

[0002] The invention relates to a method for adapting driving dynamics parameters for a vehicle that can be remotely controlled from a control center, in particular a remotely controlled motor vehicle. A further aspect of the invention relates to a remote control system comprising a control center and a vehicle that can be remotely controlled from this control center. A further aspect of the invention relates to a method for remotely controlling the vehicle by means of the remote control system using adapted driving dynamics parameters. Further aspects of the invention relate to a computing device designed to carry out method steps according to the invention, as well as to a corresponding computer program and a computer-readable storage medium with such a computer program.

[0003] Methods are known in which a vehicle, in particular a motor vehicle, is controlled by a human remote controller, in particular a teleoperator, positioned externally to the vehicle. The teleoperator is typically located in a stationary remote control center and thus in a building in which one or more display devices, in particular screens, are also arranged. On these screens, the remote controller can identify the vehicle to be remotely controlled and / or the surroundings of this remotely controlled vehicle, or this is displayed on the screens. In other words, the teleoperator is located in a control center that can be operated geographically independently of the location of the vehicle.

[0004] Such procedures differ significantly from those in which a driver is located outside of their own vehicle in the immediate vicinity and must be there in order to perform driving maneuvers with the vehicle they have left, for example using a portable remote control held in one hand. For example, parking maneuvers are carried out here. However, such procedures absolutely require, and are only permitted in this case, that the driver is positioned in close proximity to the vehicle with such a portable remote control and must also be able to see the vehicle in order to control such driving maneuvers. This is not the case with significantly different procedures in which the above-mentioned scenario is carried out with a central remote control located far away from the vehicle.The remote control operator in the remote control center is positioned in such a way that he or she cannot directly see the vehicle or the vehicle's surroundings. With such a method, it is therefore not necessary, and indeed may not even be possible, for the remote control operator to be positioned within a specified close range, for example, a maximum of 5 meters, of the vehicle to be remotely controlled.

[0005] Therefore, with such procedures it is more difficult for a remote control operator in a remote control center to be able to maneuver the vehicle safely during corresponding movements.

[0006] During teleoperated driving or the remote-controlled operation of a motor vehicle, the teleoperator typically controls the motor vehicle at least in part based on an image of the motor vehicle's surroundings, for example based on camera images or on an image data stream that is transmitted or conveyed to him by an environmental sensor system or a sensor device of the motor vehicle, in particular by camera sensors of the motor vehicle. In order to operate the motor vehicle safely, it is of great importance to the teleoperator how good the transmission quality of the data connection is, via which the image data stream or individual frames thereof are transmitted to the control center. The quality of the data transmission can be impaired, for example, due to a reduced transmission rate, due to data loss during transmission, or due to latency in the data transmission.Generally speaking, at a reduced transmission rate, the image quality in the control center is degraded compared to a higher transmission rate. In this case, the teleoperator only has access to image material with a lower resolution, so in extreme cases, they have to remotely control the vehicle virtually "blind."

[0007] In practice, it has been shown that transmission conditions between the vehicle and the control center are subject to significant fluctuations. Depending on the load on existing mobile phone connections, often only a reduced transmission rate is available for the connection between the vehicle and the control center. To ensure that data can still be transmitted between the vehicle and the control center, the image data to be transmitted is often compressed. For example, the resolution of the image data can be reduced (spatial downscaling). This means that the teleoperator only has blurred images available. Furthermore, the number of frames transmitted per unit of time over the data connection can be reduced (temporal downscaling) to reduce the transmission rate.In the latter case, the transmitted images may be sharp, but information from the vehicle's surroundings can only be transmitted to the teleoperator with a time delay, which can lead to critical driving situations, particularly if the teleoperator is unaware of the time delay and drives the vehicle at a comparatively high speed that is not appropriate to the time delay.

[0008] The invention is based on the object of reliably improving the safety of remote-controlled driving maneuvers and making remote control more pleasant for a teleoperator, in particular when the transmission quality between the vehicle and the control center is reduced or does not meet a predetermined minimum standard.

[0009] The problem is solved by the subject matter of the independent patent claims. Advantageous further developments are described in the dependent claims, the description, and the figures.

[0010] The invention is based on the finding that, when poor transmission quality is detected, the vehicle can, to a certain extent, protect itself from being remotely controlled by the teleoperator in a manner that is not appropriate to the transmission quality by adapting driving dynamics parameters on the vehicle side.

[0011] The invention therefore provides a method for adapting driving dynamics parameters for a vehicle that can be remotely controlled from a control center. Image data describing the vehicle's surroundings are first acquired by a sensor device or an environmental sensor system of the vehicle. For example, the sensor device can comprise various sensors that acquire environmental data describing the environment or surroundings of the vehicle. This preferably includes camera sensors that acquire the image data, which is transmitted as an image data stream from the vehicle to the control center via a data connection. In other words, the vehicle can "see" its surroundings using the sensors and transmit this view in the form of the image data stream to the control center, where the image of the surroundings can be displayed, for example, on a screen for the teleoperator.For this purpose, the vehicle can operate a data connection to the control center, for example using a known mobile radio technology. According to the invention, a computing device determines a current transmission quality of the data connection between the vehicle and the control center and, based on the transmission quality, determines a current quality factor of the image data stream or individual frames of the image data stream. Preferably, the computing device determines, based on the determined transmission quality, the extent to which the image data must be compressed in order to be able to be transmitted to the control center via the data connection. In this preferred case, the computing device can determine the quality factor based on the degree of compression without having to further analyze the image data. This quality factor can then be assigned to the image data, i.e., the image data stream or individual frames of the image data stream.

[0012] The quality factor can be determined based on a predetermined standard quality factor, wherein the standard quality factor can be assigned to a predetermined standard transmission quality. Preferably, the computing device can access an assignment table in which a respective quality factor can be permanently assigned to a respective transmission quality. Further assignments can also be permanently prescribed in the assignment table. For example, the assignment table can also contain an assignment of a quality factor to a compression level. In other words, the computing device can determine the transmission quality, compress the image data stream to be transmitted based on the transmission quality, and read the associated quality factor from the assignment table based on the compression level it has selected.

[0013] The quality factor can be lowered compared to the standard quality factor if the current transmission quality is worse than the standard transmission quality. In other words, the quality factor can reflect the current transmission quality relative to the standard transmission quality.

[0014] Based on the determined current quality factor, the computing device then determines or estimates a current perception factor or "situation awareness score," which describes the extent to which a user operating the control center's current perception of the vehicle's surroundings is reduced compared to a predetermined normative perception. The computing device can therefore model the perception factor, taking the quality factor into account and based on the predetermined normative perception. The quality factor is thus used to estimate the user's current perception capabilities with regard to the vehicle's surroundings. A comparatively low quality factor therefore leads to a comparatively low perception factor. In contrast, a high perception factor can be estimated or modeled based on a comparatively high quality factor.

[0015] Finally, the computing device adjusts the driving dynamics parameters taking the current perception factor into account. The adjustment is preferably performed automatically on the vehicle side when a predetermined minimum perception factor is undershot, without any intervention from the teleoperator. The adjustment may, for example, involve reducing the vehicle's maximum possible speed depending on the current perception factor. In other words, one of the driving dynamics parameters may be the vehicle's speed. This effectively protects the vehicle from a driving style by the teleoperator that would be inappropriate for the current transmission quality.On the other hand, the teleoperator can also remotely control the vehicle with confidence even in cases of poor transmission quality, as the adjusted driving dynamics parameters effectively protect them from, for example, excessive speed that is inappropriate for the current transmission quality. The teleoperator can therefore confidently press the accelerator pedal to the floor in the control center, knowing that the vehicle-adjusted driving dynamics parameters will protect the vehicle from entering a critical driving situation that they cannot or only inadequately detect due to the poor transmission quality. This makes remote control of the vehicle safer and more pleasant for the teleoperator.

[0016] The invention also includes further embodiments which provide additional advantages.

[0017] One embodiment provides that input parameters characterizing the image data stream are evaluated to determine, estimate, or model the current quality factor. In other words, when modeling the quality factor, the computing device takes into account further input parameters that characterize the image data stream. These include, in particular, a current data transmission capacity of the data connection ("target bitrate") and / or a current spatial resolution of the image data stream or of the individual frames of the image data stream and / or a current temporal resolution of the image data stream or of the individual frames of the image data stream. Based on the target bitrate, the computing device can estimate the data volume that can currently be transmitted via the data connection. The computing device can then perform the described compression of the image data stream, which in turn determines the quality factor.The spatial resolution of the image data stream ("spatial downscaling factor") can be used to determine the extent to which the resolution of the transmitted image data is reduced compared to predetermined standard values. Analyzing the temporal resolution of the image data stream may reveal that the data connection currently only allows a transmission of 15 frames per second, whereas a predetermined standard value may be 30 frames per second.

[0018] The deviations of the aforementioned input parameters from predetermined standard values, individually or in any combination, influence the quality factor of the data connection and thus also the current perception factor, which is then used to adjust the driving dynamics parameters. By taking these input parameters into account, the accuracy of the adjustment can be further improved.

[0019] According to a further embodiment, to determine the current quality factor, a comparison of the image data stream or the individual frames of the image data stream is carried out with image data streams or their frames transmitted in the past. For example, it is conceivable that the vehicle was already maneuvered remotely in the same environment in the past, and that for this purpose the image data stream captured at that time, or at least individual frames thereof, were stored in a storage unit. The computing device can access this storage unit and compare the current image data stream with the earlier recordings. Based on the deviation, conclusions can be drawn about the quality factor of the currently transmitted image data stream and thus also about the current perception factor. The quality factor assigned to the earlier recordings at that time is preferably stored together with the earlier recordings.If the quality of the previous recordings meets the predetermined standard quality, the current perception factor can be modeled very quickly as a function of the standard state in order to be able to make a statement about the extent to which the current perception factor has deteriorated compared to the standard state.

[0020] As described, the image data of the image data stream can be compressed to enable transmission even with low bandwidth or intensive use of the available bandwidth. The compression can include a reduction in the spatial and / or temporal resolution of the image data stream. Preferably, the computing device can detect the limited capacity of the data connection and adapt the bit rate or transmission rate for transmitting the image data stream to the limited capacity. For this purpose, for example, the bit rate of the encoder can be reduced in the usual way or the temporal or spatial resolution of the image data stream can be reduced. The compression parameters can be summarized and referred to as video coding.In such a compressed image or image data stream, objects, especially the farther they are from the vehicle, may be represented by only a few pixels, making them easily overlooked by the teleoperator. This effect increases with increasing compression ratio.

[0021] A further embodiment therefore provides for the video coding to be taken into account when determining the perception factor. To this end, if the image data stream currently transmitted over the data connection is compressed or contains compressed frames, an image quality metric, in particular a non-referenced image quality metric (i.e., without using the non-compressed image data stream, for example, BRISQUE - Blind / Referenceless Image Spatial Quality Evaluator), is applied to the compressed image data stream or its compressed frames to determine the current quality factor.

[0022] The extent to which the described image data compression influences the perception factor can also depend on the content described in the transmitted image data, i.e., what the vehicle's surroundings actually look like. If the vehicle is in an open field, there are few or no objects that can be underrepresented by compression. However, if the vehicle is at a busy intersection, the negative effects of compression become more significant. This can be taken into account by conducting a context analysis for the vehicle's current driving situation and factoring it in when determining the perception factor. For example, the more other road users are currently detected in the vehicle's surroundings as part of the context analysis, the more heavily the compression can be weighted when determining the perception factor.Particularly preferably, the computing device can determine, on the basis of context recognition, image regions in which compression can be performed without negatively affecting the quality factor, for example because there are no or only a few other road users there. A further embodiment provides that, if the image data stream is compressed or contains compressed frames, a comparison is carried out between the compressed image data stream or its compressed frames with an uncompressed image data stream or its uncompressed frames underlying the compressed image data stream or its compressed frames in order to determine the current quality factor. The comparison can preferably be carried out by the computing device in the vehicle, for example on the basis of a known image quality metric, for example PSNR (Peak Signal to Noise Ratio) or SSIM (structural similarity).The latter metric, also known as the structural similarity index, is a method for estimating the perceived quality of digital images and videos. SSIM is used to measure the similarity between two images. The uncompressed image data stream or its frames can be assigned the described standard quality factor. Depending on the size of the deviations between the uncompressed and compressed image data, a reduced current quality factor for the compressed image data can be modeled based on the standard quality factor, which in turn can be used to model the current perception factor.

[0023] According to a preferred development, the uncompressed image data stream or its uncompressed frames underlying the compressed image data stream or its compressed frames are stored in a vehicle-mounted memory for comparison. The uncompressed image data stream or the uncompressed frames are therefore preferably located in a memory unit of the vehicle, since the uncompressed image data is captured by the vehicle's sensor device. This can therefore be stored directly in the vehicle's memory unit, where it is then available for comparison. Since the comparison is performed on the vehicle side, the corresponding computing capacity must be maintained in the vehicle.Alternatively or additionally, the uncompressed image data can also be written by the computing device to an off-board storage device, such as an internet-based cloud environment, using a data connection different from the one used to the control center. There, they can then be made available for the comparison described.

[0024] A further advantageous development provides that the comparison includes determining the mean square error between individual pixels of the compressed image data stream or its compressed frames, on the one hand, and the uncompressed image data stream or its uncompressed frames, on the other. This allows the comparison to be performed particularly quickly and efficiently.

[0025] A further embodiment provides that, in order to determine the current quality factor, a neural network or network, in particular comprising an object detection algorithm, is used to determine an object detection rate in the compressed image data stream or its frames and in the uncompressed image data stream or its frames, wherein the current quality factor is determined taking into account the deviation between the object detection rates.

[0026] In the context of the present disclosure, an object detection algorithm can be understood as a computer algorithm capable of identifying and locating one or more objects within a provided input data set, for example, an input image, for example by defining corresponding bounding boxes or regions of interest (ROIs), and in particular by assigning a corresponding object class to each of the bounding boxes, wherein the object classes can be selected from a predefined set of object classes. The assignment of an object class to a bounding box can be understood as providing a corresponding confidence value or probability that the object identified within the bounding box belongs to the corresponding object class.For example, for a given bounding box, the algorithm may provide such a confidence value or probability for each of the object classes. Assigning the object class may, for example, involve selecting or providing the object class with the highest confidence value or probability. Alternatively, the algorithm may simply specify the bounding boxes without assigning a corresponding object class.

[0027] Here and in the following, an artificial neural network can be understood as software code that is stored on a computer-readable storage medium and represents one or more networked artificial neurons or can simulate their function. The software code can also contain multiple software code components that can, for example, have different functions. In particular, an artificial neural network can implement a non-linear model or a non-linear algorithm that maps an input to an output, where the input is given by an input feature vector or an input sequence and the output can, for example, contain an output category for a classification task, one or more predicted values, or a predicted sequence. The artificial neural network can therefore implement the described object detection algorithm.

[0028] The deviation between the object detection rates describes how many objects could be detected in the compressed image data stream compared to the uncompressed image data stream using the object detection algorithm and thus provides an indication of the information density available to the teleoperator in the compressed image data stream. A suitable approach to determine the deviation could be to measure the overlap between the bounding boxes of the detected objects on the compressed and uncompressed images or frames of the image data stream.

[0029] Preferably, it can also be considered which object classes overlap. For example, if it turns out that the object recognition rates differ significantly, but objects rated as critical, such as pedestrians, are reliably detected even in the compressed image data stream, the quality factor can be reduced only slightly compared to the standard quality factor despite the differing object recognition rates.

[0030] A further embodiment provides that an achievable maximum driving speed of the vehicle and / or a maximum achievable acceleration of the vehicle and / or a steering sensitivity of the vehicle and / or a braking behavior of the vehicle are adapted as driving dynamics parameters. To implement the adaptation of the driving dynamics parameters in the vehicle, the computing device can transmit corresponding control commands to a control device of the vehicle, which in turn, based on the control commands, controls corresponding actuators in the vehicle in order to reduce or increase the characteristic maps of the aforementioned driving dynamics parameters accordingly. For example, the maximum achievable driving speed of the vehicle can be reduced from 10 kilometers per hour to 5 kilometers per hour.

[0031] To adapt the braking behavior of the vehicle, a predetermined minimum distance is preferably adjusted, from which the vehicle automatically triggers braking in response to an object in its surroundings. In other words, the vehicle can have a brake assistance system that is designed to detect the current distance of the vehicle to objects in its surroundings. For this purpose, data describing the distances can be transmitted from the sensor device to the brake assistance system. Such data can be detected, for example, using radar and / or lidar and / or ultrasonic sensors of the sensor device. The distances can also be determined based on the camera data or image data already recorded. The brake assistance system can store the predetermined minimum distance from which automatic braking of the vehicle is to be triggered.To adjust the braking behavior, this minimum distance can now be adjusted, especially if the quality factor is poor.

[0032] Preferably, as an alternative or in addition to adjusting the vehicle's braking behavior, the vehicle's brake pads are moved closer to the vehicle's brake discs from an initial position. This effectively reduces the brakes' reaction time to the braking command triggered by the brake assistance system.

[0033] A further aspect of the invention relates to a computing device configured to execute method steps of the method according to the invention. For this purpose, the computing device may have multiple computing units.

[0034] In the present disclosure, a computing unit can be understood, for example, as a data processing device with processing circuits. A computing unit can therefore perform computing operations to process data. The computing operations can also include indexed accesses to a data structure, for example, a look-up table (LUT).

[0035] A computing unit can in particular comprise one or more computers, one or more microcontrollers and / or one or more integrated circuits, for example one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more single-chip systems (SoCs). The computing unit can also contain one or more processors, for example one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit can also comprise a physical or virtual cluster of computers or other of the aforementioned units.

[0036] A computing unit may also include one or more hardware and / or software interfaces and / or one or more memory units. A memory unit may be embodied as a volatile data memory, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM), or as a non-volatile data memory, for example, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory or flash EEPROM, a ferroelectric random access memory (FRAM),a magnetoresistive random access memory (MRAM) or a phase-change random access memory (PCRAM).

[0037] According to a further aspect of the invention, a computer program with instructions is provided. When the instructions are executed by a computing device or at least one computing unit of the computing device, the instructions cause the computing device or the at least one computing unit to perform a method according to the invention.

[0038] The instructions can be provided, for example, as program code. The program code can be provided, for example, as binary code or assembly code and / or as source code of a programming language, for example, C, and / or as a program script, for example, Python.

[0039] According to a further aspect of the invention, a computer-readable storage medium is provided which stores a computer program according to the invention.

[0040] The computer program and the computer-readable storage medium are each computer program products containing the instructions. A further aspect of the invention relates to a remote control system comprising a control center and a vehicle, in particular a motor vehicle, remotely controllable from the control center, having a sensor device configured to capture image data describing the surroundings of the vehicle and transmit it as an image data stream to the control center via a data connection. The remote control system also comprises a computing device configured to determine a current transmission quality of the data connection between the vehicle and the control center and, based on the transmission quality, to determine a current quality factor of the image data stream or individual frames of the image data stream.The computing device is further configured to determine, estimate, or model a current perception factor (situation awareness score) based on the determined current quality factor. This situation awareness score describes the extent to which a user operating the control center's current perception of the vehicle's surroundings is reduced compared to a predetermined standard perception. Furthermore, the computing device is configured to adapt the vehicle's driving dynamics parameters taking the current perception factor into account.

[0041] A further aspect of the invention relates to a method for remotely controlling a vehicle by means of a remote control system according to the invention using adapted driving dynamics parameters.

[0042] According to a further development of this method, the vehicle's computing device automatically adjusts the driving dynamics parameters when a predefined perception factor is undershot. This significantly increases the safety of remote-controlled operation of the vehicle, as the teleoperator does not have to pay attention to whether the current perception factor falls below the predefined perception factor. This can reduce human errors during remote-controlled operation.

[0043] Alternatively, the driving dynamics parameters can also be adjusted by the computing device in response to a corresponding input command from the control center user, i.e., the teleoperator, if the predefined perception factor is undershot. In this variant, the teleoperator has the decision-making authority over the adjustment of the driving dynamics parameters, which can enrich the user experience. In other words, the teleoperator at the control center can switch to "blind flight mode" if the perception factor drops. For applications or application situations that may arise with a method according to the invention and that are not explicitly described herein, it can be provided that, in accordance with the method, an error message and / or a request to enter user feedback is output and / or a default setting and / or a predetermined initial state is set.

[0044] Further embodiments of the remote control system according to the invention and / or the method according to the invention for remotely controlling a vehicle using the remote control system follow directly from the various embodiments of the method according to the invention for adapting the driving dynamics parameters, and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various embodiments of the method according to the invention for adapting the driving dynamics parameters can be transferred analogously to corresponding embodiments of the further aspects of the invention. In particular, the computing device according to the invention is designed or programmed to carry out a method according to the invention.

[0045] Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown in the figures can be encompassed by the invention not only in the respectively specified combination, but also in other combinations. In particular, the invention can also encompass embodiments and combinations of features that do not have all the features of an originally formulated claim. Furthermore, the invention can encompass embodiments and combinations of features that go beyond the combinations of features set out in the backreferences to the claims or deviate from them.

[0046] In the figures, the same reference symbols denote functionally identical elements.

[0047] The figures show:

[0048] Fig. 1 is a schematic representation of a remote control system according to the invention; Fig. 2 is a schematic process diagram of a first embodiment of the method according to the invention for adapting driving dynamics parameters;

[0049] Fig. 3 is a schematic representation of method steps for determining a perception factor on the basis of compressed and / or uncompressed image data;

[0050] Fig. 4 is a schematic representation of method steps for determining a perception factor based on a comparison of the object recognition rates in compressed and uncompressed image data; and

[0051] Fig. 5 is a schematic representation of a method for remotely controlling a vehicle by means of a remote control system according to the invention.

[0052] Fig. 1 shows a schematic representation of a remote control system 10, comprising a control center 12 and a vehicle 14 that can be remotely controlled from the control center 12. In the embodiment shown, a data connection 16 exists between the vehicle 14 and the control center 12, wherein image data 18 can be transmitted from the vehicle 14 to the control center 12 via the data connection 16. In the embodiment shown, the data connection 16 is also used to transmit control commands 20 from the control center 12 to the vehicle 14.

[0053] The vehicle 14 can have a sensor device 22, which can include a plurality of sensors, for example camera sensors, by means of which the surroundings of the vehicle 14 can be detected. In other words, the sensor device 22 can be configured to capture the image data 18 describing the surroundings of the vehicle 14. This image data 18 can be transmitted to the control center 12 via the data connection 16 in the form of an image data stream or in the form of individual frames.

[0054] The image data 18 can be received in the control center 12 and displayed as display content on a display device 24, for example, a screen, for a user of the control center 12. The user or teleoperator can thus see an image of the surroundings of the vehicle 14 on the screen, which was generated based on the image data 18 captured and transmitted by the camera sensors. Based on the representation on the display device 24, the teleoperator can make navigation decisions for the vehicle 14, which they can transmit to the vehicle 14 in the form of control commands 20 using the data connection 16. The teleoperator can generate the control commands 20, for example, by operating individual control elements 26 of the control center 12. Control elements 26 can be, for example, a steering device and / or a simulation of an accelerator and / or brake pedal in the control center 12.

[0055] The control commands 20 can be forwarded in the vehicle 14 to a control device 28 of the vehicle 14, wherein the control device 28 can be designed to convert the control commands 20 into the corresponding driving maneuvers by controlling corresponding actuators in the vehicle 14.

[0056] The remote control system 10 also includes a computing device 30, which in the exemplary embodiment shown here is arranged in the vehicle 14. However, the computing device 30 can also be arranged externally of the vehicle and communicatively connected to the remote control system 10 via another data connection.

[0057] The computing device 30 can be configured to determine the current transmission quality of the data connection 16, for example, the currently available bandwidth. The computing device 30 can then determine the extent to which the image data 18 must be compressed in order to be transmitted to the control center 12 via the data connection 16. Depending on the degree of compression, the quality of the transmitted image data 18 suffers, so that the display of the image of the surroundings of the vehicle 14 on the display device 24 becomes blurred. The teleoperator is therefore unable to clearly see how to remotely control the vehicle 14 in the surroundings without, for example, colliding with objects in the surroundings.

[0058] In order to nevertheless enable safe remote-controlled operation of the vehicle 14, the computing device 30 can be designed to determine a current quality factor of the image data 18 or of an image data stream formed therefrom based on the transmission quality of the data connection 16, to estimate a current perception factor or “situation awareness score” based on the quality factor, which describes the extent to which a current perception of the surroundings of the vehicle 14 by a user operating the control center 12 is reduced compared to a predetermined standard perception, and to adapt driving dynamics parameters of the vehicle 14 taking into account the current perception factor, as described in more detail below.

[0059] Fig. 2 shows, with reference to the components designated and described in connection with Fig. 1, a schematic process diagram of a first embodiment of the method according to the invention for adapting driving dynamics parameters.

[0060] To adapt the driving dynamics parameters, the computing device 30 may include a perception calculation module 34 and a driving dynamics parameter adaptation module 38, as schematically illustrated in Fig. 2. The modules 34, 38 may be understood as computing units of the computing device 30, as defined in the context of the present disclosure.

[0061] First, input parameters 32.1, 32.2, 32.3, 32.4, which characterize the image data 18 or the image data stream, are acquired in the perception calculation module 34. The input parameters 32.1, 32.2, 32.3, 32.4 can include, for example, a transmission rate of the data connection 16 (target bitrate), a factor for reducing the spatial resolution (spatial downscaling factor), a factor for reducing the temporal resolution (temporal downscaling factor), and / or image data 18 transmitted in the past (previous frames).

[0062] In the perception calculation module 34, the transmission quality of the data connection 16 can be determined based on the input parameters 32.1, 32.2, 32.3, 32.4.

[0063] Based on the transmission quality, the current quality factor of the image data 18 or the image data stream can then be determined. Finally, in the perception calculation module 34, taking the quality factor into account and based on a standard perception, the perception factor 36 can be determined. This describes the extent to which a user operating the control center 12's current perception of the surroundings of the vehicle 14 is reduced compared to a predetermined standard perception.

[0064] The perception factor 36 can then be transmitted to the driving dynamics parameter adaptation module 38, which outputs the driving dynamics parameters 40.1, 40.2, 40.3 adapted to the perception factor 36. The adapted driving dynamics parameters 40.1, 40.2, 40.3 can be, for example, an adapted maximum achievable driving speed, an adapted maximum achievable acceleration, and / or an adapted braking behavior of the vehicle 14.

[0065] Fig. 3 shows, with reference to the components designated and described in connection with Figs. 1 and 2, a schematic representation of method steps for determining the perception factor 36 on the basis of compressed and / or uncompressed image data 18.

[0066] For this purpose, compressed image data 18.1 and optionally also uncompressed image data 18.2 are provided to the perception calculation module 34. In the perception calculation module 34, an image quality metric 42, for example, BRISQUE, is applied to the image data 18.1, 18.2 in order to determine, for example, differences between the quality of the compressed image data 18.1 and the uncompressed image data 18.2. These differences are then provided to a modeling module 44 of the perception calculation module 34, which ultimately generates the perception factor 36 based on the differences.

[0067] With reference to the components identified and described in connection with the preceding figures, Fig. 4 shows a schematic representation of method steps for determining the perception factor 36 based on a comparison of the object recognition rates in compressed and uncompressed image data 18.1, 18.2. According to the embodiment described here, the compressed image data 18.1 and the uncompressed image data 18.2 are each made available to a neural network with an object detection algorithm 46.1, 46.2. There, an object recognition rate is determined for the respective image data 18.1 and 18.2, with the two rates being compared with one another in a comparison module 48. The difference in the object recognition rates is in turn made available to the modeling module 44, which models the perception factor 36 based thereon.

[0068] Finally, Fig. 5 shows, with reference to the components designated and described in connection with the preceding figures, a schematic representation of a method for remotely controlling a vehicle 14 by means of a remote control system 10 according to the invention.

[0069] As described, the remote control system 10 comprises a control center 12 and a vehicle 14, in particular a motor vehicle 14, which can be remotely controlled from the control center 12, having a sensor device 22 which is designed to capture image data 18 describing an environment of the vehicle 14 and to transmit it as an image data stream 18 to the control center 12 by means of a data connection 16, and a computing device 30.

[0070] In a step S2, the computing device 30 determines a current transmission quality of the data connection 16 between the vehicle 14 and the control center 12 and, based on the transmission quality, determines a current quality factor of the image data stream 18 or individual frames of the image data stream 18. In a step S2, the computing device 30 uses the determined current quality factor to determine a current perception factor 34, which describes the extent to which a current perception of the surroundings of the vehicle 14 by a user operating the control center 12 is reduced compared to a predetermined standard perception. In a step S3, the computing device 30 adapts driving dynamics parameters of the vehicle 14, taking into account the current perception factor 34.Finally, in a step S4, the computing device 30 transmits the adjusted driving dynamics parameters 40 to the control device 28 of the vehicle 14, which accordingly restricts the performance parameters of the vehicle such that vehicle maneuvers can only be performed within the framework of the adjusted driving dynamics parameters 40, preferably independently of an override of the adjusted driving dynamics parameters 40 on the control center side.

[0071] The invention is based on the situation where a teleoperated vehicle 14 travels through an area with limited bandwidth, e.g., due to poor network coverage or because many participants share a station. A computing device 30, e.g., a video module, of the teleoperation system or the remote control system 10 can detect the limited channel capacity and reduce the required bit rate for transmitting the image data stream 18 or video stream. For this purpose, the target bit rate of the encoder can be lowered or the temporal and spatial resolution can be reduced. In both cases, the visual quality of the video stream 18 received in the teleoperation cockpit or control center 12 and displayed to the teleoperator will be significantly reduced. The teleoperator therefore has less accuracy in detecting other road users in the vicinity of the vehicle 14.In particular, distant objects are captured by only a few pixels in the current video image and are therefore particularly susceptible to being overlooked. This applies particularly to an image data stream 18 that has been compressed with a high compression rate. Even with a reduction in temporal resolution, rapidly approaching objects could be detected too late. As a result, the driver may overlook important aspects in the surroundings of the vehicle 14.

[0072] When driving a normal car, the driver is expected to adjust their speed to the visibility (e.g., in heavy snowfall or fog). An analogous approach is proposed for the aforementioned problem, where the teleoperator has reduced situational awareness or perception due to increased video compression.

[0073] The invention proposes a system for teleoperation that models the negative effects of video encoding or video compression on the driver's situational awareness or perception and adapts the driving dynamics of the vehicle 14 accordingly. Such a system for adapting the driving dynamics of the teleoperated vehicle 14 based on video quality may comprise two modules (see Fig. 2).

[0074] The first module (perception calculation module 34) models the teleoperator's reduced situation awareness or reduced perception based on the currently used video coding parameters as a perception factor 36. The reference is the teleoperator's perception based on the video stream 18, which has been compressed according to video compression parameters that enable transmission over an ideal transmission channel. The video compression parameters can include the currently used target bit rate, the spatial downscaling factor, or the temporal downscaling factor.

[0075] The second module (vehicle dynamics parameter adaptation module 38) uses the detection result or perception factor 36 of the first module and adapts the vehicle dynamics of the vehicle 14 accordingly. In this way, greater safety of the entire teleoperation system 10 could be achieved.

[0076] Such an adjustment of the driving dynamics parameters could primarily involve speed reduction, as this is the most important driving dynamics parameter for reducing the probability of an accident at high video compression rates. Furthermore, it could be beneficial to limit the acceleration of the vehicle 14 in such cases or to increase the braking sensitivity of the vehicle 14 in order to be prepared for sudden maneuvers. Limiting the steering angle and steering speed could also be possible. The impact of the compression parameters on the actual video quality and thus on the teleoperator's situational awareness could depend heavily on the video content being transmitted. For example, situational awareness could also be derived using an algorithm that is periodically applied to the previous images (see Fig. 3).This algorithm could be based on measuring the quality of each compressed frame using traditional, reference-free image quality metrics such as BRISQUE. Depending on this, situational awareness could be more accurately modeled. This value could be derived either from the vehicle 14 or the control center 12.

[0077] In addition, the compressed previous images could also be compared with the uncompressed previous images. Since the uncompressed frames are not available on the cockpit side, a frame comparison could preferably be performed in vehicle 14. Such a calculation would therefore only be useful if there are no CPU or GPU limitations in vehicle 14. The comparison between compressed and uncompressed images could be performed using traditional image quality metrics such as PSNR or SSIM. Analogous to the above-mentioned approach using a referenceless metric, situation awareness could be modeled using such a metric.

[0078] Another method could be to use a regular object detection algorithm or a neural network and apply it to the compressed and uncompressed image (see Fig. 4). Then, both results could be compared to assess how many objects detected in the uncompressed image can still be detected after compression. A suitable metric for this task could be measuring the intersection between the bounding boxes of the detections on the compressed image.

[0079] Furthermore, a combination of the described approaches (video coding parameters and previous frames) might be possible to model situation awareness as accurately as possible.

[0080] The invention results in a beneficial increase in safety during teleoperation, which can be attributed to several aspects. A particularly advantageous feature is that the teleoperation system detects the reduced situational awareness and automatically reduces the possibility of accidents by adapting the driving dynamics of the teleoperated vehicle. The teleoperator does not have to worry about this themselves, which prevents human error. Furthermore, the reduction in the maximum speed could also provide an additional sense of security because the teleoperator knows that the car is aware of the current situation with the poor network coverage and that they can still drive safely under the given circumstances. This could reduce the possibility of the teleoperator feeling uneasy or even panicking because the video quality is impaired for a certain period of time.Overall, this could not only increase safety but also the driving experience for the teleoperator.

[0081] In a concrete example, individual frames extracted from a sample teleoperation stream might be compressed once with video compression parameters for sufficient channel capacity and once with compression parameters designed for poor network conditions. In the latter case, a 20-fold lower bitrate might be required, for example.

[0082] Such low-bitrate video transmission may be unavoidable in situations where urgent teleoperations must be performed. One such case could be a self-driving car encountering a problem and stopping in the middle of the road in an area with poor network coverage. The car is blocking the road and must therefore be moved out of the way as quickly as possible. The poor network coverage only allows for strong video compression. The teleoperator must therefore make their driving decisions based on poor video quality. To still enable the teleoperator to safely move the stuck car out of the way, the proposed system detects the poor video quality and initiates a change in driving dynamics to avoid accidents.

[0083] In this example, the proposed system could reduce the current maximum speed the teleoperator is allowed to drive to a specific value, e.g., from 10 km / h to 5 km / h. In this way, the proposed system could enable teleoperation under all network conditions, while providing a certain level of safety for all road users.

[0084] Overall, the examples show how a vehicle can be reliably operated remotely even if a teleoperator's perception is reduced.

Claims

Patent claims 1 . Method for adapting driving dynamics parameters (40) for a vehicle (14) that can be remotely controlled from a control station (12), wherein a sensor device (22) of the vehicle (14) acquires image data (18) that describe an environment of the vehicle (14), wherein the image data (18) are transmitted as an image data stream (18) by means of a data connection (16) from the vehicle (14) to the control station (12), comprising the steps carried out by a computing device (30) - determining a current transmission quality of the data connection (16) between the vehicle (14) and the control center (12) and, based on the transmission quality, determining a current quality factor of the image data stream (18) or individual frames of the image data stream (18), - Determining, on the basis of the determined current quality factor, a current perception factor (36) which describes the extent to which a current perception of the surroundings of the vehicle (14) by a user operating the control station (12) is reduced compared to a predetermined standard perception, - Adjusting the driving dynamics parameters (40) taking into account the current perception factor (36).

2. Method according to claim 1, wherein, in order to determine the current quality factor, input parameters (32) characterizing the image data stream (18) are evaluated, in particular a current data transmission capacity of the data connection (16) and / or a current spatial resolution of the image data stream (18) or of the individual frames of the image data stream (18) and / or a current temporal resolution of the image data stream (18) or of the individual frames of the image data stream (18).

3. Method according to one of the preceding claims, wherein to determine the current quality factor a comparison of the image data stream (18) or the individual Frames of the image data stream (18) with image data streams transmitted in the past or their frames.

4. Method according to one of the preceding claims, wherein, if the image data stream (18) is compressed or contains compressed frames, an image quality metric (42), in particular a non-referenced image quality metric (42), is applied to the compressed image data stream (18) or its compressed frames to determine the current quality factor.

5. Method according to one of the preceding claims, wherein, if the image data stream (18) is compressed or contains compressed frames, a comparison is carried out between the compressed image data stream (18) or its compressed frames with an uncompressed image data stream (18) or its uncompressed frames underlying the compressed image data stream (18) or its compressed frames in order to determine the current quality factor.

6. The method according to claim 5, wherein the uncompressed image data stream (18) or the uncompressed frames underlying the compressed image data stream (18) or the uncompressed frames thereof are kept ready for comparison in a vehicle-side memory.

7. The method according to one of claims 5 or 6, wherein the comparison includes determining the mean square error between individual pixels of the compressed image data stream (18) or its compressed frames on the one hand and the uncompressed image data stream (18) or its uncompressed frames on the other hand.

8. The method according to one of claims 5 to 7, wherein a neural network, in particular comprising an object detection algorithm (46), is used to determine the current quality factor in order to determine an object detection rate in the compressed image data stream (18) or its frames and in the uncompressed image data stream (18) or its frames, wherein the current quality factor is determined taking into account the deviation between the object detection rates.

9. The method according to any one of the preceding claims, wherein an achievable maximum travel speed of the vehicle (14) and / or a maximum achievable acceleration of the vehicle (14) and / or a steering sensitivity of the vehicle (14) and / or a braking behavior of the vehicle (14) are adapted as driving dynamics parameters (40).

10. The method according to claim 9, wherein, in order to adapt the braking behavior of the vehicle (14), a predetermined minimum distance is adjusted, from which the vehicle (14) automatically initiates braking in response to an object located in its surroundings.

11. Method according to one of claims 9 or 10, wherein, in order to adapt the braking behavior of the vehicle (14), brake pads of the vehicle (14) are moved closer to the brake discs of the vehicle (14) from an initial position compared to the initial position.

12. Computing device (30) which is designed to carry out method steps according to one of claims 1 to 11.

13. A computer program comprising instructions which, when the computer program is executed by a computing device (30) according to claim 12, cause the computing device (30) to perform the method steps according to any one of claims 1 to 11.

14. A computer-readable storage medium on which the computer program according to claim 13 is stored.

15. Remote control system (10), comprising a control station (12) and a vehicle (14), in particular a motor vehicle (14), which can be remotely controlled from the control station (12), having a sensor device (22) which is designed to capture image data (18) describing an environment of the vehicle (14) and to transmit it as an image data stream (18) to the control station (12) by means of a data connection (16), and a computing device (30) according to claim 12, which is designed to - to determine a current transmission quality of the data connection (16) between the vehicle (14) and the control center (12) and to determine a current quality factor of the image data stream (18) or individual frames of the image data stream (18) based on the transmission quality, - to determine a current perception factor (36) on the basis of the determined current quality factor, which describes the extent to which a current perception of the surroundings of the vehicle (14) by a user operating the control station (12) is reduced compared to a predetermined standard perception, and - to adapt driving dynamics parameters (40) of the vehicle (14) taking into account the current perception factor (36), in particular according to a method according to one of claims 1 to 11.

16. A method for remotely controlling a vehicle (14) by means of a remote control system (10) according to claim 15 using adapted driving dynamics parameters (40).

17. The method according to claim 16, wherein the driving dynamics parameters (40) are automatically adjusted by the computing device (30) in the vehicle (14) when a predetermined perception factor (36) is undershot.

18. The method according to claim 17, wherein the driving dynamics parameters (40) are adapted by the computing device (30) to a corresponding input command of the user of the control station (12) when a predetermined perception factor (36) is undershot.

Images