Method for modifying at least one camera-specific parameter in a vehicle

Abstract

The invention relates to a method (100) for modifying at least one camera-specific parameter in a vehicle (1), comprising: - Determine (101) at least one communication characteristic, wherein the at least one communication characteristic specifies a quality of communication between the vehicle (1) and at least one external data processing device (6), - Modifying (102) the at least one camera-specific parameter in the vehicle (1) depending on the determined at least one communication characteristic, wherein the camera-specific parameter is modified for a camera (2) of the vehicle (1). The invention further relates to a computer program, a device, a storage medium, and a vehicle for this purpose.

Description

[0001] The invention relates to a method for modifying at least one camera-specific parameter in a vehicle. Furthermore, the invention relates to a computer program, a device, a storage medium, and a vehicle for this purpose. State of the art

[0002] Current trends in the automotive industry—autonomous driving, connectivity, electrification, and shared mobility (ACES)—are leading to an increase in the amount of data exchanged between the vehicle and the cloud. Some of these applications require deterministic communication between the vehicle and applications distributed in the cloud. To address this, L4S (Low Latency, Low Loss, and Scalable Throughput), an advanced IP-layer congestion control mechanism for routed networks, is increasingly being deployed in cellular networks to enable low latency without impacting overall throughput. L4S provides early warnings to the data source about connection congestion, allowing it to throttle the generated data and avoid long latency and packet loss due to buffer overflows. This is particularly useful for transmitting data that can be variably compressed, such as video / audio streams.

[0003] Parallel to these developments, so-called zonal architectures are being introduced in on-board, or vehicle-internal, networks. Peripheral devices (sensors, actuators, etc.) are connected to zones, which in turn connect them to an on-board computer where the functionality (e.g., compression algorithm) resides. For data exchange between the zones and the on-board computer, link-layer protocols such as the IEEE 1722 Audio / Video Transport Protocol (AVTP) have been proposed. Thus, the data sources are not IP layer endpoints, enabling them to participate in congestion control.

[0004] Data transmission over IP networks is carried out using transport protocols, such as TCP or UDP. TCP transmissions attempt to scale the data rate to efficiently utilize the available network capacity. Several TCP algorithms exist, such as Reno, Cubic, or Prague, which employ slightly different mechanisms to prevent congestion based on feedback from a TCP receiver.

[0005] However, real-time video transmissions are known to have problems with TCP-based communication protocols, as these also lead to retransmissions and delays. Therefore, it is far more common to use UDP-based protocols as the underlying transport protocols for low-latency, real-time video streaming, such as in teleoperated driving. For this, other protocols are used in addition to UDP to properly packetize a video stream and send it to a receiver, such as the Real-Time Protocol (RTP).

[0006] Furthermore, the packets delivered to the recipient can be monitored and statistics such as delays, jitter, packet loss or congestion markers can be sent back to the sender via another protocol, e.g. the Real-Time Control Protocol (RTCP), which runs on top of the TCP protocol stack.

[0007] Another protocol example is QUIC, which is based on UDP but specifies certain extensions beyond it, such as the use of acknowledgments to inform the sender about the received packets.

[0008] By using this information, the sender can adjust their video / audio bitrate to avoid congestion and not generate more data than the available network connection capacity allows. In video streaming, an algorithm for avoiding congestion relies on information provided by the communication protocols, but it is considered part of the application, and one could develop a proprietary collision avoidance algorithm for their application. However, there are several algorithms standardized by the IETF, such as SCREAM, GCC, and NADA.

[0009] In recent years, a novel congestion mechanism with low latency, low losses, and scalable throughput (L4S) has been developed. L4S is gaining popularity and attracting increasing interest in mobile networks, particularly for interactive media streaming, such as gaming, XR, and VR applications. L4S detects an increase in queuing delays on the network and reports it back to the source, which can then reduce its transmission rate (e.g., by increasing its compression rate). When deployed in mobile networks, its use can also be beneficial for automotive and robotics applications, such as teleoperated driving or functional offloading.

[0010] IEEE 1722 specifies the Audio / Video Transport Protocol (AVTP) for the purpose of transmitting data over Layer 2 switched networks, such as audio / video streams in a concert hall. By encapsulating data in simple Ethernet frames, IEEE 1722 aims for a lean transport protocol that can be easily handled by devices with extremely limited resources. Since audio / video data streams are also latency-sensitive, these networks are equipped with real-time extensions and traffic shapers standardized by the IEEE Time-Sensitive Networking (TSN) Task Group. AVTP also includes native hooks for integration with TSN mechanisms, thus enabling the implementation of mixed-criticality applications where data streams can be delivered across the network with differentiated services.

[0011] With the proliferation of Ethernet in automotive networks, AVTP has also become an attractive transport protocol for in-vehicle data distribution. For zonal E / E architectures, characterized by centralized computing resources, AVTP can be used to transport data from sensors and actuators (including speakers and cameras) connected to different zones via an Ethernet network to the vehicle's computers. While AVTP already supports several data formats relevant to the automotive industry (such as CAN, LIN, etc.), the specification is now being extended to include formats for peripheral buses such as I. 2 to include C, SPI, etc. These extensions make it possible to tunnel these data formats over Ethernet.

[0012] Especially for real-time video streaming (e.g., in teleoperated driving), there are several IETF standards that explain how the video codec's bitrate can be adjusted to avoid / reduce congestion when the communication link's capacity fluctuates. The SCREAM algorithm, for example, is based on RTP and RTCP application protocols and also supports L4S.

[0013] In current standardization and implementation according to the latest state of the art, the overload control mechanism, the video codec and the IP network interface are all considered to be located on the same host computer, such as a PC or a mobile device.

[0014] However, a vehicle has a complex internal network architecture that connects sensors (including cameras), actuators, controllers and on-board computers. Disclosure of the invention

[0015] The invention relates to a method with the features of claim 1, a computer program with the features of claim 8, a device with the features of claim 9, a computer-readable storage medium with the features of claim 10, and a vehicle with the features of claim 11. Further features and details of the invention will become apparent from the respective dependent claims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program, the device, the computer-readable storage medium, and the vehicle according to the invention, and vice versa, so that mutual reference is always possible with regard to the disclosure of the invention.

[0016] The invention relates in particular to a method for modifying at least one camera-specific parameter in a vehicle, comprising: - Determining at least one communication characteristic, wherein the at least one communication characteristic specifies a quality of communication between the vehicle, in particular a data processing device, preferably an on-board computer, of the vehicle, and at least one external data processing device, wherein the at least one communication characteristic can be determined, for example, by the data processing device of the vehicle by analyzing data traffic between the data processing device and the external data processing device, in particular on the basis of a receiver report of the data processing device of the vehicle, wherein the determination is preferably carried out using L4S. - Modifying the at least one camera-specific parameter in the vehicle depending on the determined at least one communication characteristic, wherein the camera-specific parameter is modified for a camera of the vehicle, for example using a data processing device of the vehicle and via an internal network of the vehicle, wherein the modification is carried out, for example, using the Audio Video Transport Protocol.

[0017] The camera-specific parameter is, in particular, an internal parameter of the camera, meaning it can be directly modified by the camera itself. During the modification process, a command to modify the camera-specific parameter can be transmitted from a data processing device, especially the vehicle's on-board computer, to the camera via the vehicle's internal network. The vehicle's internal network can, for example, be an Ethernet-based network. The camera and the vehicle's data processing device can be connected to each other via appropriate interfaces, especially Ethernet interfaces.

[0018] L4S stands for Low Latency, Low Loss, Scalable Throughput and is a technology for improving data transmission in networks, especially mobile networks and other IP-based networks. It aims, for example, to minimize latency (delays) and packet loss while ensuring a scalable data rate.

[0019] L4S enables the coexistence of traditional traffic types and a special low-latency traffic type in a shared network by using a concept called "Dual Queue Coupled AQM" (Active Queue Management). This allows data traffic to be split into two queues: one queue for traditional transmissions that tolerate higher latency, and a separate queue for low-latency applications that are served more quickly.

[0020] L4S can help further improve the quality of latency-critical services and enable optimal utilization of network resources. By implementing L4S, networks can offer faster response times and improved stability under high load without significant packet loss.

[0021] The method according to the present invention can advantageously prevent an overload of one or the vehicle's internal network and thus further improve data transmission between the vehicle and the at least one external data processing device.

[0022] At least one communication characteristic can be, for example, at least one of the following: - A latency in data transmission between the vehicle, in particular a data processing device, preferably an on-board computer, of the vehicle, and the at least one external data processing device, - A number of lost data packets during data transmission between the vehicle, in particular a data processing device, preferably an on-board computer, of the vehicle, and the at least one external data processing device, - A time interval between which data packets are received during data transmission between the vehicle, in particular a data processing device, preferably an on-board computer, of the vehicle, and the at least one external data processing device, - A delay marker, where the delay marker indicates whether and for how long a particular data packet was held in a queue before it was sent.

[0023] The camera-specific parameter can, for example, relate to a recording characteristic of the camera, where the recording characteristic is preferably selected from a camera frame rate, a camera resolution, and / or a camera color mode, i.e., for example, a black and white mode. In this way, modifying the recording characteristic can advantageously reduce the amount of data transmitted by the camera over the vehicle's internal network, thus reducing the load on the vehicle's internal network.

[0024] Furthermore, it is possible that the camera-specific parameter relates to encoding by the camera, whereby the bitrate of the encoding is adjusted during modification. Thus, it is possible for the camera to encode video recordings in a predefined format and for the bitrate of the encoding to be adapted to changing network conditions according to the invention. This enables, in particular, a dynamic modification of the video data volume to the available bandwidth. Through flexible control of the bitrate, overloads in the vehicle's internal network can be avoided and a stable video signal can be ensured.

[0025] It is possible that the method according to the invention is used in a vehicle. The vehicle can be, for example, a motor vehicle and / or a passenger vehicle and / or at least partially automated / autonomous. The vehicle can have vehicle equipment, for example, for providing an autonomous driving function and / or a driver assistance system. The vehicle equipment can be designed to control the vehicle at least partially automatically and / or accelerate and / or brake and / or steer.

[0026] The invention also relates to a computer program, in particular a computer program product, comprising instructions which, when executed by at least one computer, cause it to execute the method according to the invention. Thus, the computer program according to the invention offers the same advantages as those described in detail with reference to a method according to the invention.

[0027] The invention also relates to a data processing device configured to execute the method according to the invention. The device can, for example, comprise at least one computer which executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. A non-volatile data storage device can also be provided in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0028] The invention may also relate to a computer-readable storage medium which contains the computer program according to the invention and / or includes instructions which, when executed by at least one computer, cause it to execute the method according to the invention. The storage medium is, for example, designed as a data storage device such as a hard drive and / or non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.

[0029] The invention also relates to a vehicle comprising: - A device for data processing according to the invention, in particular an on-board computer of the vehicle, - An in-vehicle network, - A camera, wherein the data processing device and the camera are connected to each other via the vehicle's internal network. The vehicle according to the invention thus offers the same advantages as those described in detail with reference to a method according to the invention.

[0030] Furthermore, the method according to the invention can also be implemented as a computer-implemented method. Alternatively or additionally, at least one of the disclosed method steps can be computer-implemented and / or carried out automatically.

[0031] Further advantages, features, and details of the invention will become apparent from the following description, in which exemplary embodiments of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description can each be essential to the invention individually or in any combination. The drawings show: Fig. 1 a schematic visualization of a method, a device, a storage medium and a computer program according to exemplary embodiments of the invention, Fig. 2 a schematic representation of a vehicle and an external data processing device according to exemplary embodiments of the invention.

[0032] In Fig. Figure 1 shows a method 100, a device 10, a storage medium 15 and a computer program 20 according to exemplary embodiments of the invention.

[0033] Fig. Figure 1 shows in particular an embodiment of a method 100 for modifying at least one camera-specific parameter in a vehicle 1. In a first step 101, at least one communication characteristic is determined, wherein the at least one communication characteristic specifies a quality of communication between the vehicle 1 and at least one external data processing device 6. In a second step 102, the at least one camera-specific parameter in the vehicle 1 is modified depending on the determined at least one communication characteristic, wherein the camera-specific parameter for a camera 2 of the vehicle 1 is modified, for example, using a data processing device 4 of the vehicle 1 and via an internal network 3 of the vehicle 1.

[0034] According to exemplary embodiments of the invention, a mechanism for bypassing extended IP congestion control for endpoints of a link layer is provided. The invention specifically considers a case in which a camera 2 of the vehicle 1, which can, for example, generate a live video stream, is connected via Ethernet to a data processing device 4 of the vehicle 1, in particular an on-board computer. The camera 2, for example, only supports the IEEE 1722 standard for video transmission and does not provide any algorithms for avoiding congestion or a protocol stack (such as RTP and RTCP) for communication over a wireless network outside the vehicle 1.Therefore, in certain scenarios, the bitrate adjustments (including those involving L4S mechanisms) cannot be used by default, as feedback regarding an overload of the vehicle's internal network 3 may not reach camera 2 and / or may not be correctly interpreted by camera 2.

[0035] In such a case, camera 2 may continue to transmit pre-configured video streams over the vehicle's internal network 3, even though the video quality is unnecessarily high. This requires additional power and can, in some cases, lead to unfavorable conditions (e.g., overload, bottlenecks, delays, or jitter) for resource utilization in the vehicle's internal network 3.

[0036] Fig. Figure 2 shows an example of video / image transmission from a camera 2 in the vehicle 1 via an in-vehicle network 3 and a data processing device 4 of the vehicle 1, in particular an on-board computer, using appropriate interfaces 5 to a remote receiver, i.e., an external data processing device 6 such as a (cloud) server. In this case, the camera 2, for example, only generates a raw image and, in particular, does not perform any encoding. The camera 2 can have a camera sensor 8, an Ethernet block 7, and an interface 5, in particular an Ethernet interface.

[0037] According to an initial scenario, it can be assumed that there is an in-vehicle camera 2 that can only generate a raw image without an encoder, as is also the case in Fig.Figure 2 illustrates this. An Ethernet block 7 can encapsulate the raw data from the camera sensor 8 using the Audio Video Transport Protocol (AVTP) to stream the video to the data processing device 4 of the vehicle 1, in particular the on-board computer. To avoid sending excessive amounts of raw data to the data processing device 4 of the vehicle 1 (where it must also be compressed), camera-specific parameters on the camera 2 relating to the image capture itself can be modified, as shown in exemplary embodiments. These camera-specific parameters include, for example, a frame rate and a resolution, or a color mode, i.e., whether, for example, only black-and-white images are to be captured.

[0038] A decision to modify these camera-specific parameters is preferably made by the data processing device 4 of the vehicle 1, in particular based on at least one communication characteristic such as receiver reports and local context information and configurations.

[0039] The camera-specific parameters are determined in particular via interfaces such as I 2 Configured in C or SPI. According to the invention, information about parameter changes can be transmitted to camera 2 in vehicle 1 via AVTP. This can be easily done via the I 2 The C and SPI formats of AVTP are used, which are intended for tunneling these protocols over Ethernet. According to the invention, existing AVTP mechanisms can advantageously be used to modify camera-specific parameters in case of overload.

[0040] According to a second scenario, the vehicle's in-vehicle camera 2 is capable of encoding video with a specific codec (e.g., h264), but congestion avoidance algorithms (and the RTP / RTCP protocol stack) are executed by the vehicle's data processing device 4, specifically the on-board computer. Similar to the first scenario, the challenge lies in transmitting the result of a congestion avoidance algorithm to camera 2 via the AVTP protocol and the Ethernet-based in-vehicle network 3. In this case, the vehicle's data processing device 4 can modify another camera-specific parameter, namely a codec bitrate setting. When the command with the new bitrate arrives at camera 2, this command can be used to modify the local bitrate setting of the encoder in camera 2.In this context, camera 2 of vehicle 1 has, in particular, coding functions that can be modified.

[0041] Using the GPC (General Purpose Control) message format defined in the AVTP, these camera-specific parameters, required for encoding the video stream, can be sent to camera 2. Based on these camera-specific parameters, camera 2 can modify the encoder and modify the resulting video stream, for example, according to the overload prevention algorithms executed on the data processing device 4 of vehicle 1.

[0042] Accordingly, according to exemplary embodiments of the invention, existing AVTP mechanisms can be used to integrate the overload control algorithms hosted on the data processing device 4 of the vehicle 1 into the compression mechanisms executed on the camera 2 that generates the video streams.

[0043] The preceding explanation of the embodiments describes the present invention solely by way of examples. Naturally, individual features of the embodiments can be freely combined with one another, provided this is technically feasible, without departing from the scope of the present invention.

Claims

[1] Method (100) for modifying at least one camera-specific parameter in a vehicle (1), comprising: - Determine (101) at least one communication characteristic, wherein the at least one communication characteristic specifies a quality of communication between the vehicle (1) and at least one external data processing device (6), - Modifying (102) the at least one camera-specific parameter in the vehicle (1) depending on the determined at least one communication characteristic, wherein the camera-specific parameter is modified for a camera (2) of the vehicle (1). [2] Method (100) according to claim 1, characterized by that at least one communication characteristic is at least one of the following: - A latency in data transmission between the vehicle (1) and the at least one external data processing device (6), - A number of lost data packets during data transmission between the vehicle (1) and the at least one external data processing device (6), - A time interval between which data packets are received during data transmission between the vehicle (1) and the at least one external data processing device (6), - A delay marker, where the delay marker indicates whether and for how long a particular data packet was held in a queue before it was sent. [3] Method (100) according to any one of the preceding claims, characterized by , that the camera-specific parameter relates to a recording characteristic of the camera (2), wherein the recording characteristic is preferably selected from a frame rate of the camera (2), a resolution of the camera (2) and / or a color mode of the camera (2). [4] Method (100) according to any one of the preceding claims, characterized by, that the camera-specific parameter concerns encoding by the camera (2), whereby, within the modification (102), a bit rate of the encoding is adjusted. [5] Method (100) according to any one of the preceding claims, characterized by , that the modification (102) is performed using the Audio Video Transport Protocol. [6] Method (100) according to any one of the preceding claims, characterized by , that as part of the modification (102) a command to modify the camera-specific parameter is transmitted from a data processing device (4), in particular an on-board computer, of the vehicle (1) via a vehicle-internal network (3) to the camera (2) in order to modify the camera-specific parameter. [7] Method (100) according to claim 6, characterized by, that the vehicle's internal network (3) is an Ethernet-based network and the camera (2) and the data processing device (4) of the vehicle (1) are connected to each other via appropriate interfaces (5). [8] Computer program (20) comprising instructions which, when the computer program (20) is executed by at least one computer (10), cause it to execute the method (100) according to any of the preceding claims. [9] Device (10) for data processing which is configured to carry out the method (100) according to any one of claims 1 to 7. [10] Computer-readable storage medium (15) comprising instructions which, when executed by at least one computer (10), cause it to perform the steps of the method (100) according to any one of claims 1 to 7. [11] Vehicle (1), comprising: - A device (10) for data processing according to claim 9, - An in-vehicle network (3), - A camera (2) wherein the data processing device (10) and the camera (2) are connected to each other via the vehicle's internal network (3).

Images