Vehicle, device for a vehicle, computer program product, and method for processing information transmitted in a teleoperated driving session
By predicting the communication link delay between the vehicle and the remote control center and adjusting the codec parameters, the problem of data transmission delay in ToD is solved, ensuring the timely processing and transmission of vehicle environment data, and improving remote control operation Security.
Patent Information
- Application Number
- CN202111296709.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-11-03
- Filing Date
- 2021-11-03
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-11-03
AI Technical Summary
In remote control operational driving (ToD), the communication link delay problem caused by changes in traffic conditions of the autonomous driving vehicle cannot meet the requirements of safe driving.
By establishing a remote-controlled operating driving session between the vehicle and the remote control center, the delay of the communication link is predicted and the codec parameters are adjusted according to the predicted delay to keep the total delay of information processing and transmission at a tolerable level , ensure timely transmission and processing of data.
Effectively reduces the fluctuations in communication link delays, ensures that the remote control center can process and transmit video data and other information of the vehicle environment within an acceptable delay range, thereby improving the security and reliability of remote control operations.
Smart Images

Figure CN114449021B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a vehicle, a device for a vehicle, a computer program, and a method for processing information for transmission in a teleoperated driving session, and more particularly, but not exclusively, to a concept for adapting the encoding of such information based on the predicted latency of a communication link between the vehicle and a remote control center. Background Art
[0002] Teleoperated driving (ToD) may become a key technology to solve problems in L4 / L5 (L4: highly automated, L5: fully automated) driven vehicles, such as interpretation problems or deadlocks (situations that cannot be resolved by autonomous or automatic control mechanisms alone). These problems occur when an autonomous vehicle (AV) is unable to interpret and resolve a situation (deadlock) due to unclear traffic conditions (e.g., an accident or a construction site). These vehicles may require external instructions from other parties to resolve the situation, which may be a remote control center far away from the vehicle. Therefore, ToD allows the vehicle to be remotely driven by the remote control center and the operator therein, respectively, in a ToD session. To this end, the vehicle may provide video data of its environment to the remote control center and the operator therein. Due to the processing of such video data, the video data may be displayed to the operator with a delay. Due to changes in the quality of service of the communication between the AV and the remote control center, these delays may reach a level that is unbearable for ToD.
[0003] Document WO 2015 / 034405 A1 describes a concept for determining multimedia codec parameters for a multimedia communication session. To this end, the concept comprises: comparing a channel indicator value with a previous channel quality indicator value of a communication channel for the multimedia communication session; and determining whether to delay the adaptation of the values of a set of multimedia codec parameters based on the comparison, and determining which values of the set of multimedia codec parameters to adapt.
[0004] Document US2007 / 0021879 A1 discloses a system that collaboratively and autonomously plans and controls a team of vehicles with subsystems within an environment. The system can adjust the mission plans of unmanned vehicles to maintain the quality of service of available system data links at an effective level.
[0005] Document US2020 / 0107212 A1 discloses a method for predicting the quality of service of a communication link of at least one communication device. The method provides: sending a quality of service prediction request message from the communication device to a communication service prediction server; predicting the quality of service in the communication service prediction server; and sending a quality of service prediction response message back to the communication device.
[0006] The documents relate to vehicle communication and / or predicting the quality of service in vehicle communication. Nevertheless, none of these documents provide improvements to teleoperated driving.
[0007] Document WO 2019 / 180700 A1 relates to devices, systems and methods for autonomous and remotely controlled vehicles. It provides that a vehicle artificial intelligence (AI) unit is configured to: receive input from a vehicle sensor of a vehicle; process the input locally; and wirelessly transmit the input to a remote remote control driving processor. It further proposes: wirelessly receiving a remotely calculated processing result received from a remote artificial intelligence (AI) unit from the remote remote control driving processor; and implementing a vehicle operation command based on the remotely calculated processing result via the autonomous driving unit of the vehicle or via the remote control driving unit of the vehicle.
[0008] Document EP 3 723 412 A1 relates to a system, a carrier, an apparatus, a method and a computer program for user equipment UE and service provision in a mobile communication system. The method for user equipment UE in a mobile communication system comprises: using a service with a set of quality of service QoS requirements in the mobile communication system; receiving information related to a future process of QoS provision of the service from the mobile communication system; and adapting the service based on the information related to the future process of QoS and based on the set of QoS requirements.
[0009] Document US2020 / 0081431 A1 relates to a remote vehicle control system including a video camera system for generating video data. The document provides that a data processing system is connected to a network to transmit data to and receive data from a remote remote control operation site. It also provides that a virtual control system is configured to: receive video; provide a user with a real-time video stream supported by machine intelligent directional frame synthesis; and transmit control instructions to a remote vehicle via a network. Frame synthesis is supported by a convolutional neural network. Frame synthesis can be used to interpolate frames to increase the effective frames per second. Frame synthesis can be used to extrapolate frames to replace lost or damaged video frames.
[0010] Therefore, there may be a need for an improved concept for processing information for teleoperated driving. This need may be met by the subject matter of the attached independent and dependent claims. Summary of the invention
[0011] Embodiments are based on the discovery that a delay / time period for processing (e.g., for encoding and decoding) information (e.g., video data, lidar data, and / or radar data) for transmission in a teleoperated driving (ToD) session can be adjusted with respect to a predicted delay of a communication link used to transmit the information. This allows, for example, the total delay for transmitting and processing the information to be kept below a maximum tolerable limit on the communication link delay. The delay for processing the information is adjusted via one or more codec parameters used to encode the information.
[0012] An embodiment provides a method for a vehicle and for processing information for transmission in a teleoperated driving session. The method includes: establishing a teleoperated driving session between the vehicle and a remote control center. In addition, the method includes: obtaining a predicted latency of a communication link between the remote control center and the vehicle for the teleoperated driving session. In addition, the method includes: adapting at least one codec parameter for encoding the information based on the predicted latency.
[0013] Establishing a teleoperated driving session includes, for example, transmitting a request to remotely control the vehicle to a remote control center and / or transferring control of the vehicle to a remote control center.
[0014] A skilled person having the benefit of the present disclosure will understand that the quality of service (QoS) and in particular the latency of the communication link may change over time and / or with the location of the vehicle, and that previous latency measurements and / or the movement of the vehicle may be used to determine the predicted latency (i.e., the later / future latency of the communication link).
[0015] Adapting at least one codec parameter comprises, for example, selecting a codec for processing the information from a plurality of (different) codecs, and / or determining one or more codec parameters of a codec setting of the codec. For example, one of the codecs H.26x or Moving Picture Experts Group (MPEG), Lossless Codec Library (LCL) codecs may be selected, a compression technique (e.g., discrete cosine transform (DCT) or motion compensation (MC)) may be determined, and / or other characteristics of the codec for processing the information (e.g., buffering time for compression (e.g., video compression)) may be determined by adapting at least one codec parameter. Adapting at least one codec parameter based on or with respect to the predicted delay enables the processing of the information to be adapted early enough to the changing delay of the communication link, for example, so that if the delay of the communication link changes (suddenly), the total delay for processing and transmitting the information is kept at a tolerable level for ToD. The information may be environmental information about the environment of the vehicle.
[0016] In some embodiments, the information includes video data, radar data, object data, and / or lidar data of the vehicle's environment. This enables the remote control center to display a visual representation of the environment to an operator remotely controlling the vehicle, and keeps the overall latency for processing and transmitting such data at a tolerable level for ToD.
[0017] In some embodiments, the method further comprises: obtaining a delay for encoding the information at the vehicle for one or more codec parameters. In addition, the method may include: receiving a delay for decoding the encoded information at the remote control center for one or more codec parameters from the remote control center. In addition, adapting at least one codec parameter may include: adapting at least one codec parameter based on the delay for encoding and decoding the information.
[0018] The delay for encoding the information may be determined by a buffering time for compression of the information (eg, video compression for video data).
[0019] The vehicle may receive (wirelessly) a time delay for decoding the information from a remote control center via the above communication link or a separate communication link.
[0020] This allows, for example, the bearer to adapt at least one codec parameter with respect to the latency for encoding and decoding, for example to keep the total latency including the latency for decoding and encoding at an appropriate level for ToD.
[0021] In some embodiments, the method further comprises: obtaining a maximum delay for transmission of the information in the teleoperated driving session. Furthermore, adapting at least one codec parameter may comprise adapting at least one codec parameter based on the maximum delay.
[0022] The maximum delay is, for example, a (predefined) maximum tolerable delay for processing the information (e.g., video data) and transmitting the information from the vehicle to a remote control center in a ToD application. Adapting at least one codec parameter based on the maximum delay, for example, comprises: adapting at least one codec parameter such that the delay for processing and transmitting the information is lower than the maximum delay. A person skilled in the art will appreciate that this, for example, makes the delay not too high for ToD.
[0023] In some embodiments, obtaining the maximum delay comprises determining the maximum delay based on information about the vehicle and / or traffic conditions.
[0024] A skilled person having the benefit of this disclosure will appreciate that when the vehicle is controlled remotely, the maximum tolerable delay of the information may depend on movement (e.g., velocity or speed), capabilities (e.g., braking power, weight, etc.), and / or traffic conditions (e.g., indicating traffic density, local road traffic regulations), etc. Therefore, determining the maximum delay based on information about the vehicle (e.g., velocity / speed, or capabilities of the vehicle) and / or traffic conditions allows the delay for processing and transmitting the information to be kept below the maximum tolerable delay for the traffic conditions and the vehicle.
[0025] In some embodiments, the method further comprises: obtaining a predicted data rate of the communication link. Furthermore, adapting at least one codec parameter may comprise: adapting at least one codec parameter based on the predicted data rate of the communication link. This, for example, further allows utilizing a data rate for transmitting the information to the remote control center.
[0026] Another embodiment provides a method for a remote control center. The method includes: establishing a remote control driving session between the remote control center and the vehicle. In addition, the method includes: determining a delay for decoding information for transmission in the remote control driving session. The method further includes: transmitting the delay to the vehicle for adapting at least one codec parameter for encoding the information based on the delay. As mentioned above in conjunction with the method for the vehicle, the delay for decoding the encoded information enables the vehicle to adapt one or more codec parameters with respect to the delay for decoding, and, for example, the delay for processing the information at the vehicle and transmitting the information to the remote control center, so as not to exceed the maximum allowable delay for ToD.
[0027] An embodiment further provides a device comprising one or more interfaces configured to communicate in a communication network, and a data processing circuit configured to control the one or more interfaces, wherein the data processing circuit is further configured to perform one of the methods presented herein.
[0028] Further embodiments relate to a computer program having a program code for performing at least one of the methods presented herein, when the computer program is executed on a computer, a processor or a programmable hardware component.
[0029] The embodiment further provides a vehicle including the device disclosed herein. The vehicle is, for example, a land vehicle. In other applications, the vehicle may also be a ship or an aircraft. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Some other features or aspects will be described, by way of example only and with reference to the accompanying drawings, the following non-limiting embodiments of using an apparatus or a method or a computer program or a computer program product, in which:
[0031] Figure 1 A block diagram of an embodiment of a method for a vehicle and for processing information for transmission in a teleoperated driving session is illustrated;
[0032] Figure 2 A block diagram of an embodiment of an apparatus for performing the above method for a vehicle is illustrated;
[0033] Figure 3 A block diagram of an embodiment of a method for remotely controlling a center is illustrated;
[0034] Figure 4 A block diagram illustrating an embodiment of an apparatus for executing the above method for a remote control center; and
[0035] Figure 5 The application of the proposed apparatus and method is schematically illustrated. DETAILED DESCRIPTION
[0036] Teleoperated driving (ToD) is becoming a key enabler for automated driving. ToD can help resolve deadlock situations that automated vehicles cannot resolve on their own. In an increasing number of countries, ToD may also become a legal requirement for automated driving.
[0037] In order to enable an operator in a remote control center to remotely control a vehicle, the vehicle may transmit information about its environment, such as video or other measurement data, to the remote control center. The processing and transmission of this information may imply a time delay, so that the information reaches the operator with a time delay. The greater the time delay, the more difficult and / or dangerous it may be to remotely control the vehicle.
[0038] Those skilled in the art will appreciate that the time delay for transmitting this information may change, for example (suddenly) increase, due to the movement of the vehicle. In some cases, the time delay may therefore become inappropriate for ToD.
[0039] Therefore, there is a need for improved concepts for communicating in a teleoperated driving session.
[0040] This need may be met by the embodiments described below.
[0041] Figure 1 A block diagram of an embodiment of a method 100 for a vehicle and for processing information for transmission in a teleoperated driving session is illustrated.
[0042] The method 100 comprises establishing 110 a teleoperated driving session between a vehicle and a remote control center. For example, in a so-called "deadlock situation" where the vehicle is unable to automatically / autonomously maneuver in a safe manner, the vehicle may establish or initiate a teleoperated driving session between the vehicle and the remote control center. Establishing 110 the teleoperated driving session comprises, for example, establishing a communication link to the remote control center. Optionally, establishing 110 the teleoperated driving further comprises sending a request for the remote control center to take over control of the vehicle.
[0043] A teleoperated driving session may be understood as part or all of the actions for remotely controlling the vehicle for ToD and for communicating between the vehicle and a remote control center.
[0044] Furthermore, the method 100 comprises obtaining 120 a predicted latency of a communication link between a remote control center and the vehicle for a teleoperated driving session. The predicted latency, for example, relates to an estimated latency of the communication link at a future time and is, for example, determined using previous latency measurements as input to an extrapolation for determining the predicted latency. Since the latency of the communication link or a change in the latency may be related to the movement of the vehicle, the position of the vehicle may optionally be used to determine the predicted latency.
[0045] Furthermore, the method 100 comprises adapting 130 at least one codec parameter for encoding the information based on the predicted delay. Adapting 130 at least one codec parameter based on the predicted delay allows, for example, adjusting the time period / duration for processing the information using the at least one codec parameter with respect to the predicted delay. The at least one codec parameter is, for example, determined such that the total delay including the predicted delay and the delay for encoding the information using the codec parameter is at an appropriate level for ToD. This may enable a remote control center or an operator for remotely controlling the vehicle to safely operate the vehicle based on the information.
[0046] It is noted that adapting 130 at least one codec parameter may optionally include adapting a plurality of codec parameters of a codec, or selecting a coding parameter and / or a codec from a plurality of coding parameters or codecs. For example, adapting 130 at least one codec includes selecting a codec from a plurality of codecs, e.g. from a lookup table, as will be set out in more detail later.
[0047] The information includes, for example, video data, radar data, object data and / or lidar data of the vehicle's environment. Video data includes, for example, environmental data from a camera that may be mounted at the vehicle. Lidar data and radar data may include environmental data from a lidar sensor and a radar sensor, respectively. Object data includes, for example, information about the position, size, vehicle type / traffic participant type and / or shape of another vehicle and / or traffic participant.
[0048] In some embodiments, the information may include video data.Thus, the codec and the at least one codec parameter may be a codec parameter or a codec for encoding and / or decoding video data.
[0049] It is to be noted that although the term "codec" refers to a pair of algorithms for encoding and decoding, those skilled in the art will appreciate that, depending on the context, a codec may also represent one of those algorithms. For example, when it is said that "a codec is used for encoding", it means the algorithm used to encode the corresponding codec. Vice versa, the term "codec" specifically refers to the algorithm from decoding associated with decoding the information.
[0050] Optionally, the method 100 includes obtaining delays for encoding and decoding for one or more codec parameters (eg, for a plurality of codecs and / or codec settings specified by the one or more codec parameters).
[0051] To this end, method 100 may include obtaining a delay for encoding the information at the carrier for one or more codec parameters. For example, the delay for encoding the information is determined based on a buffer time of a corresponding codec and / or a codec setting. Those skilled in the art will appreciate that available computing power may also be considered to obtain the delay for encoding the information at the carrier using a corresponding codec or codec setting.
[0052] In addition, method 100 may include: receiving a delay for decoding the encoded information at the remote control center for one or more codec parameters from the remote control center. The vehicle may, for example, request a delay for decoding the encoded information from the remote control center. To this end, the vehicle may (wirelessly) communicate with a remote control center for which one or more codec parameters (e.g., requested codec and / or codec settings, delay) have been requested. Subsequently, the remote control center may determine the delay for decoding the encoded information for the requested codec and / or codec settings. The vehicle and the remote control center transmit the codec parameters and / or delay, for example, using a cellular network or a local area network. In particular, the vehicle and the remote control center may communicate using a communication link.
[0053] In this way, delays for decoding and encoding may be determined for multiple codec parameters (eg, for multiple codecs and / or codec settings).
[0054] Knowing the delays for encoding and decoding, the vehicle may adapt one or more codec parameters with respect to these delays. For example, adapting 130 at least one codec parameter may include adapting at least one codec parameter based on the delays for encoding the information and decoding the encoded information. In this way, one or more codec parameters may be further adapted with respect to the delays for encoding and decoding, so that the total delay including the delay of the communication link for encoding and for decoding may be maintained at an appropriate level for ToD and with respect to (predicted) changes in the delay of the communication link.
[0055] For the total delay, other delays between capturing the information and displaying the information to the operator in the remote control center may also be considered. For example, the total delay also includes: the delay for processing the information at the vehicle before and / or after encoding the information, and the delay for processing the (encoded) information before and / or after decoding the encoded information. Therefore, the total delay indicates the information delay between capturing the information and displaying the information to the operator. Similarly, at least one codec parameter can be determined so that the total delay / delay of the information with respect to the (predicted) variation of the delay of the communication link is maintained at an appropriate level for ToD.
[0056] In some embodiments, method 100 further includes obtaining a maximum latency for transmission of the information in the teleoperated driving session.
[0057] The maximum delay is, for example, the maximum allowed delay for ToD. The maximum delay may be understood as a maximum value or limit for the total delay in which the vehicle may be safely controlled remotely (i.e., by an operator and a remote control center). In particular, the maximum delay may be a predefined constant or variable, as will be described in more detail later.
[0058] In some embodiments, adapting 130 at least one codec parameter comprises adapting at least one codec parameter based on the maximum delay.In this way, an appropriate codec and / or codec settings may be determined for the ToD session such that the maximum allowed delay cannot be exceeded.
[0059] The maximum allowed delay may depend in particular on the vehicle and / or the traffic situation in the environment of the vehicle. The maximum allowed delay for a vehicle may be higher, the slower the vehicle travels and / or the less dense the traffic is (e.g. the fewer other vehicles are in the environment of the vehicle). Thus, obtaining the maximum delay may comprise determining the maximum delay based on information about the vehicle and / or the traffic situation. Thus, at least one codec parameter is determined relative to information about the vehicle (e.g. speed / rate, braking power) and / or relative to the traffic situation (e.g. traffic density, or distance to other vehicles). Ultimately, this may make ToD safer. For example, this allows the operator to have at least a predefined reaction time when controlling the vehicle.
[0060] Furthermore, the method 100 may include obtaining a predicted data rate of the communication link, and adapting 130 at least one codec parameter may include adapting at least one codec parameter based on the predicted data rate of the communication link. The data rate of the communication link is, for example, a maximum available data rate, and the codec parameters are determined such that the maximum available data rate is utilized (e.g., to provide the highest possible video quality for the video data), but the data rate is not exceeded. In this way, the communication link may not be overloaded.
[0061] The method 100 may be performed by the vehicle. In particular, the method 100 may be performed by a device installed on the vehicle.
[0062] Figure 2 FIG. 2 is a block diagram of an embodiment of an apparatus 210 for executing the above-described method 100 for a vehicle. Figure 2 As indicated by the dotted line in , the device 210 is optionally mounted on the carrier 200.
[0063] The apparatus 210 comprises one or more interfaces 212 configured to communicate in a communication network. Furthermore, the apparatus 210 comprises a data processing circuit 214 configured to control the one or more interfaces 212. The data processing circuit 214 is further configured to perform the method 100 presented herein.
[0064] In an embodiment, one or more interfaces 212 may correspond to any component for obtaining, receiving, transmitting or providing analog or digital signals or information, such as any connector, contact, pin, register, input port, output port, conductor, channel, etc. that allows providing or obtaining signals or information. The interface may be wireless or wired, and it may be configured to communicate with other internal or external components, i.e., transmit or receive signals, information. One or more interfaces 212 may include other components to implement corresponding communications in a (mobile) communication system, such components may include transceiver (transmitter and / or receiver) components, such as one or more low noise amplifiers (LNA), one or more power amplifiers (PA), one or more duplexers, one or more antenna duplexers, one or more filters or filter circuits, one or more converters, one or more mixers, correspondingly adapted radio frequency components, etc. One or more interfaces 212 may be coupled to one or more antennas, which may correspond to any transmitting and / or receiving antennas, such as horn antennas, dipole antennas, patch antennas, sector antennas, etc. In some examples, one or more interfaces 212 may involve components for communicating with a remote control center and / or establishing a teleoperated driving session. Additionally, one or more interfaces 212 are coupled to data processing circuitry 214 .
[0065] The interface 212 is configured, for example, to communicate with a remote control center using a wireless local area network and / or a cellular network.
[0066] In an embodiment, the data processing circuit 214 can be implemented using one or more processing units, one or more processing devices, any means for processing (such as a processor, a computer, or a programmable hardware component operable with corresponding adapted software). In other words, the described functions of the data processing circuit 214 can also be implemented using software, which is then executed on one or more programmable hardware components. Such hardware components may include a general-purpose processor, a digital signal processor (DSP), a microcontroller, and the like. The data processing circuit 214 can be configured to perform any of the methods 100 described herein. In an embodiment, the data processing circuit 214 is configured to: establish a remote-controlled driving session between the vehicle and a remote control center to obtain a predicted delay of a communication link between the remote control center and the vehicle for the remote-controlled driving session, and adapt at least one codec parameter for encoding the information based on the predicted delay.
[0067] As mentioned previously with reference to method 100, the device 210 allows the vehicle to communicate information for teleoperated driving (e.g., video data, object data, lidar data, radar data) with a remote control center, and determines at least one codec parameter so that the total delay including the predicted delay and the delay for encoding the information using the codec parameter is at an appropriate level for ToD. This can enable the remote control center or operator for remotely controlling the vehicle to safely operate the vehicle based on the information.
[0068] Optionally, the device 210 is configured to implement / execute other optional embodiments of the method 100. For example, according to an optional embodiment of the method 100, one or more interfaces 212 and the data processing circuit 214 are further configured to: receive a delay for decoding the encoded information at the remote control center for one or more codec parameters from the remote control center, and adapt at least one codec parameter based on the delay for encoding and decoding the information. Therefore, the remote control center can provide such delays to the vehicle 200 and the device 210, respectively.
[0069] Figure 3 A block diagram of an embodiment of a method 300 for a remote control center is illustrated.
[0070] The method 300 comprises establishing 310 a teleoperated driving session between a remote control center and a vehicle. Furthermore, the method 300 comprises determining 320 a delay for decoding information encoded for transmission in the teleoperated driving session. Furthermore, the method 300 comprises transmitting 330 the delay to the vehicle for adapting at least one codec parameter for encoding the information based on the delay. This allows the vehicle to adapt at least one codec parameter with respect to the delay for decoding the encoded information, the encoded information being transmitted in the teleoperated driving session. The information is, for example, encoded by the vehicle for transmission with a remote control center. The method 300 also allows determining the delay for decoding the encoded information for a plurality of codec parameters and transmitting a plurality of determined delays to the vehicle so that the vehicle can adapt at least one codec parameter based on or with respect to the delay from the remote control center.
[0071] In particular, the method 300 enables the vehicle to adapt at least one codec parameter with respect to the latency from the remote control center so that the total latency including the latency for decoding the encoded information is at, or not higher than, an appropriate level for ToD.
[0072] The method 300 may be executed by a remote control center. In particular, the method 300 may be executed by a device installed at the remote control center.
[0073] Figure 4 FIG. 4 is a block diagram of an embodiment of an apparatus 410 for executing the above method 300 for a remote control center. Figure 4 As indicated by the dotted line in , the device 410 is optionally installed at the remote control center 400.
[0074] The device 410 includes one or more interfaces 412 configured to communicate in a communication network. The device 410 further includes a data processing circuit 414, which is configured to control the one or more interfaces 412 and perform one of the methods proposed herein for a remote control center. In particular, the data processing circuit 414 can be configured to perform the method 300 for a remote control center.
[0075] In an embodiment, one or more interfaces 412 may correspond to any component for obtaining, receiving, transmitting or providing analog or digital signals or information, such as any connector, contact, pin, register, input port, output port, conductor, channel, etc. that allows providing or obtaining signals or information. The interface may be wireless or wired, and it may be configured to communicate with other internal or external components, i.e., transmit or receive signals, information. One or more interfaces 412 may include additional components to implement corresponding communications in a (mobile) communication system, such components may include transceiver (transmitter and / or receiver) components, such as one or more low noise amplifiers (LNA), one or more power amplifiers (PA), one or more duplexers, one or more antenna duplexers, one or more filters or filter circuits, one or more converters, one or more mixers, correspondingly adapted radio frequency components, etc. One or more interfaces 412 may be coupled to one or more antennas, which may correspond to any transmitting and / or receiving antennas, such as horn antennas, dipole antennas, patch antennas, sector antennas, etc. In some examples, one or more interfaces 412 may involve components for communicating with the vehicle and / or establishing a remote control driving session. Furthermore, one or more interfaces 412 are coupled to data processing circuit 414 to enable data processing circuit 414 to perform the method for the remote control center using one or more interfaces 412. Thus, for example, data processing circuit 414 is enabled to receive and transmit information using one or more interfaces 412 for performing method 300.
[0076] In an embodiment, the data processing circuit 414 can be implemented using one or more processing units, one or more processing devices, any means for processing (such as a processor, a computer, or a programmable hardware component operable with corresponding adapted software). In other words, the described functions of the data processing circuit 414 can also be implemented using software, which is then executed on one or more programmable hardware components. Such hardware components may include a general-purpose processor, a digital signal processor (DSP), a microcontroller, and the like. The data processing circuit 414 can be configured to perform any of the methods 300 described herein. In an embodiment, the data processing circuit 414 is configured to: establish a remote-controlled driving session between the vehicle and a remote control center to determine a delay for decoding the encoded information transmitted in the remote-controlled driving session, and transmit the delay to the vehicle for adapting at least one codec parameter for encoding the information based on the delay.
[0077] The data processing circuit 414 uses one or more of the interfaces 412, for example, for communication in a teleoperated driving session and transmits the delay to the vehicle. One or more interfaces 412 may be configured to use a wireless local area network and / or a cellular network for communicating with the vehicle and for transmitting the delay to the vehicle.
[0078] As explained before, the method 300 may also be applied to determine and transmit the delay for decoding the encoded information for multiple codec parameters.Thus, the data processing circuit 414 may be configured to determine and transmit the delay for decoding the encoded information for multiple codec parameters.
[0079] Below, reference Figure 5 Applications of the proposed vehicle, method, apparatus and computer program are described.
[0080] from Figure 5 As can be seen in FIG. 5 , a (autonomous driving) vehicle 510 and a remote control center 520 can establish a teleoperated driving session and transmit information for ToD via a (communication) network 530 including the vehicle 510 and the remote control center 520. The network 530 is, for example, a cellular network.
[0081] The quality of service (QoS) of communications on the network 530 may change over time. Changes in QoS may further involve changes in the latency for information transmitted between the vehicle 510 and the remote control center 520. The latency may involve, for example, the duration or delay between capturing the information at the vehicle and displaying the information at the remote control center. Due to changes in QoS, the latency may reach an inappropriate level for ToD. Figure 5As can be seen from the figure, the delay can include different time delays. Figure 5 As can be seen from the figure, these delays may include: the delay L at the vehicle 510 rv , the video encoding delay L at the carrier 510 cod , the delay L in the (communication) network 530 qos , the time delay L at the remote control center 530 cc , and the video decoding delay L at the remote control center 520 decod .
[0082] L cod and L decod The information may depend on the codec and / or codec settings used to encode and decode the information. The information includes, for example, video data from one or more cameras 516 mounted at the vehicle 510. Thus, the codec is a video codec and the codec settings are codec settings of the video codec, respectively. It should be noted that although only video data is used as information for the following description, the proposed concepts / applications may also be applied to other data, such as lidar data, radar data and / or object data.
[0083] L rv and L cc For example, it relates to the delay / latency caused by processing the video data at the vehicle 510 and the remote control center 520 before and / or after encoding and decoding the video data, respectively.
[0084] L qos For example, it refers to the duration or delay between transmitting the encoded video data from the vehicle 510 and receiving the encoded video data at the remote control center 520 via the communication link between the vehicle 510 and the remote control center 520 in the (communication) network 530. Due to the change in QoS, L qos may change. Such changes may cause the duration or delay between capturing the information at the vehicle 510 and displaying the information at the remote control center 520 to exceed an inappropriate level for ToD. In this disclosure, this delay or duration is referred to as the total delay L total (L total =L qos +L rv +L cc +L cod +L decod ).
[0085] In some embodiments, the vehicle 510 provides information about available codecs and / or codec settings to the remote control center 520, and the remote control center 520 determines and provides the latency L for the available codecs and / or codec settings to the vehicle 510.cc and L decod .
[0086] According to the proposed method and / or apparatus, the vehicle 510 may obtain a prediction of the communication link (i.e., an estimated future L qos ) delay (i.e., the predicted L qos ), and adapting the codec and / or codec settings by adapting at least one codec parameter so that when the QoS changes and thus L qos Change the case to L total Maintain at an appropriate level for ToD.
[0087] For example, adapting the codec includes: selecting a (video) codec for encoding and decoding the video data from a plurality of different (available) (video) codecs. The codec is selected, for example, from a lookup table comprising a plurality of different codecs. A skilled person having the benefit of this disclosure will appreciate that different codecs may result in different L cod and L decod Those skilled in the art will also understand that the delay L cod and L decod The lower the L, the higher the data rate required to transmit the video data may be, and vice versa, because higher compression for lower data rates may take more time and thus result in higher latency for encoding the video data. In other words, higher compression may come at the expense of higher latency. Different codecs may exhibit different compression methods, resulting in different L for different codecs. cod and L decod .
[0088] The basic idea of the proposed concept is to obtain the predicted L qos , and for example by selecting a codec from a plurality of codecs so as to qos To adapt the codec parameters so that qos The total delay L total Maintain at an appropriate level for ToD.
[0089] The predicted L qos For example, based on L qos 510, because L qos It may depend on the relative position of an access node of the network 530 or the distance from the access node.
[0090] In order to total Maintaining a suitable level for ToD, for example, determining the maximum total delay L max, and select a codec from the available codecs so that the predicted L qos The predicted L total Less than or equal to L max :
[0091] L total =L rv +L qos +L decod +L cod +L cc <L max L max For example, L is tolerable when the vehicle 510 is remotely controlled. total The maximum delay may be derived based on or in relation to the speed / velocity of the vehicle 510 , its environment, and the reaction time of the operator 526 at the remote control center 520 .
[0092] The change of QoS may also involve a change of the data rate of the communication link. Therefore, the predicted data rate of the communication link may also be obtained for adapting at least one codec parameter (e.g., by selecting a codec from a plurality of codecs) based (also) on the predicted data rate of the communication link.
[0093] Those skilled in the art will appreciate that the predicted data rate may depend on the communication technology used for the communication link (e.g., 3rd Generation Partnership Project (3GPP) or Dedicated Short Range Communication (DSRC)), bandwidth, signal strength, and / or location of the vehicle 510. Thus, the predicted data rate may be obtained based on location and / or communication technology. Alternatively, the predicted data rate may be obtained based on previous measurements of the predicted data rate. The predicted data rate may, for example, relate to the same point in time or time period as the predicted delay.
[0094] Adapting at least one codec parameter based on the predicted delay and the predicted data rate (e.g., by selecting a codec from a plurality of codecs) such as to allow utilization of (but not exceed) the data rate of the communication link (e.g., to provide a desired or highest possible video quality for the video data) while maintaining L in the event of a change in QoS and data rate total Maintain at an appropriate level for ToD.
[0095] In other words, at least one codec parameter (codec and / or codec setting) may be adapted with respect to the PQoS profile such that the total delay does not exceed the maximum (total) delay and the data rate is not exceeded, according to the following formula:
[0096] DR codec <DR qos ∩L total =L rv +L qos +Ldecod +L cod +L cc <L max ,
[0097] DR codec refers to the data rate of the output of the corresponding codec and / or codec arrangement, and where DR qos Refers to the data rate of a communication link.
[0098] In particular, the vehicle 510 may adapt at least one codec parameter (e.g., by adapting a codec setting and / or selecting a codec from a plurality of codecs) such that the highest quality of the video data is provided at the remote control center 520, but without exceeding the data rate DR qos and the maximum (total) delay L max .
[0099] This will be further explained with reference to the following exemplary scenario:
[0100] For example, different delays L are provided relative to each other. cod and L codec And different data rates DR codec Multiple codecs are available for vehicle 510.
[0101] I. In the first exemplary scenario, the predicted QoS provides an increased data rate DR qos and the reduced delay L qos In this scenario, the following codec can be used: This codec has a higher delay L than other codecs. cod and L decod And data rate DR codec Therefore, while providing the highest possible video quality, the total delay L total It is impossible to exceed the maximum (tolerable) delay L max .
[0102] II. In the second exemplary scenario, the predicted QoS provides an increased data rate DR qos , but also provides an increased delay L qos In this scenario, you can use the low latency L compared to other codecs. cod and L decod But it provides a higher data rate DR codec The codec of the
[0103] III. In the third exemplary scenario, the predicted QoS provides a reduced data rate DR qos and the reduced delay L qosIn this scenario, a codec with a higher latency L than other codecs can be used. cod and L decod Provides low data rate DR at the expense of codec The codec of the
[0104] Therefore, in these scenarios, the data rate DR is utilized qos , and the total delay L total Maintaining a suitable level for ToD. This may enable operator 526 to more safely control vehicle 510 from a remote location than if lower video quality and / or higher overall latency were provided.
[0105] As already mentioned, in an embodiment, the corresponding methods may be implemented as computer programs or codes, which can be executed on corresponding hardware. Therefore, another embodiment is a computer program with program code for carrying out at least one of the above methods when the computer program is executed on a computer, a processor or a programmable hardware component. A further embodiment is a computer-readable storage medium storing instructions, which, when executed by a computer, a processor or a programmable hardware component, causes the computer to implement one of the methods described herein.
[0106] Those skilled in the art will readily recognize that the steps of various methods described above can be performed by a programmed computer, such as can determine or calculate the position of a time slot. In this article, some embodiments are also intended to contain program storage devices, such as digital data storage media, which are machine or computer readable and machine executable or computer executable instruction programs are encoded, wherein the instructions are implemented in some or all of the steps of the method described in this article. The program storage device can be, for example, a digital memory, a magnetic storage medium such as a disk and a tape, a hard drive, or an optically readable digital data storage medium. Embodiments are also intended to contain computers programmed to perform the steps of the method described in this article, or contain (field) programmable logic arrays ((F) PLA) or (field) programmable gate arrays ((F) PGA) programmed to perform the steps of the method described above.
[0107] Reference Symbols List
[0108] 100 Methods for Vehicles
[0109] 110 Establishing a remote control driving session
[0110] 120 Get predicted delay
[0111] 130 Adapting at least one codec parameter based on the predicted delay
[0112] 200 Vehicles
[0113] 210 Installation
[0114] 212 One or more interfaces
[0115] 214 Data Processing Circuit
[0116] 300 Methods for remote control center
[0117] 310 Establishing a remote control driving session
[0118] 320 Determine the delay for decoding the encoded information
[0119] 330 transmitting the delay to the carrier for adapting at least one codec parameter based on the delay
[0120] 400 Remote Control Center
[0121] 410 Device
[0122] 412 One or more interfaces
[0123] 414 Data Processing Circuit
[0124] 510 Vehicles
[0125] 516 One or more cameras
[0126] 520 Remote Control Center
[0127] 526 Operator
[0128] 530 Network
[0129] L total , L rv , L qos , L decod , L cod , L cc , L max Latency
[0130] DR codec ,DR qos Data rate
Claims
1. A method (100) for a vehicle (200, 510) and for processing information for transmission in a teleoperated driving session, the method (100) comprising: establishing (110) a teleoperated driving session between the vehicle (200, 510) and a remote control center (400, 520); Obtaining (120) a predicted latency of a communication link between a remote control center (400, 520) and the vehicle (200, 510) for a teleoperated driving session; obtaining, for one or more codec parameters, a delay for encoding the information at the carrier (200, 510); receiving from the remote control center (400, 520) a delay for decoding the encoded information at the remote control center (400, 520) for one or more codec parameters; as well as At least one codec parameter used to encode the information is adapted (130) based on the predicted delay, wherein adapting (130) at least one codec parameter comprises adapting at least one codec parameter based on the delay used to encode and decode the information.
2. The method (100) according to claim 1, wherein the information comprises video data, radar data, object data and / or lidar data of the environment of the vehicle (200, 510).
3. The method (100) according to claim 1 or 2, The method (100) further comprises: obtaining a maximum delay for transmission of said information in a teleoperated driving session, And wherein adapting (130) at least one codec parameter comprises adapting at least one codec parameter based on the maximum delay.
4. The method (100) of claim 3, wherein obtaining the maximum delay comprises: The maximum delay is determined based on information about the vehicle (200, 510) and / or traffic conditions.
5. The method (100) according to claim 1 or 2, The method (100) further comprises obtaining a predicted data rate of the communication link, and Wherein adapting (130) at least one codec parameter comprises: At least one codec parameter is adapted based on a predicted data rate of the communication link.
6. A method (300) for a remote control centre (400, 520) and for processing information for transmission in a teleoperated driving session, the method (300) comprising: Establishing (310) a teleoperated driving session between a remote control center (400, 520) and a vehicle (200, 510); determining (320) a delay for decoding encoded information for transmission in a teleoperated driving session; as well as The delay is transmitted (330) to the carrier (200, 510) for use in adapting at least one codec parameter used for encoding the information based on the delay.
7. An apparatus (210, 410), comprising: one or more interfaces (212, 412) configured to communicate in a communication network; as well as A data processing circuit (214, 414) configured to control the one or more interfaces (212, 412), wherein the data processing circuit (214, 414) is further configured to perform one of the methods (100, 300) of claims 1 to 6.
8. A computer program product having a program code for performing one of the methods (100, 300) of claims 1 to 6 when the computer program product is executed on a computer, a processor or a programmable hardware component.
9. A carrier (200) comprising the device (210) according to claim 7.
