BUNDLING OF CAMERA AND RADAR RAW DATA CHANNELS

Description

[0001] The invention relates to a device and a method for bundling two data channels via a point-to-point connection, for example based on a CSI-2 protocol, in particular for bundling camera and radar raw data channels. The invention further relates to a use, a program element, and a computer-readable medium.

[0002] Point-to-point connections, especially communication links, are used to transmit data over a line.

[0003] One use case is transmitting raw data from image and radar sensors over such lines. Another use case is that this data is acquired in a vehicle and used, for example, for driver assistance systems. A group of these applications can be so-called ADAS (Advanced Driver Assistance Systems). These systems combine, visualize, and / or validate large amounts of data from various sensors—e.g., radar, lidar, ultrasound, laser, video-based systems such as cameras, etc. The different sensor types often differ, for example, in their data rate and / or the number of wire pairs used. Therefore, in an application that includes both sensor types, a dedicated receiver must be available for each sensor type within a common processing unit, or a separate processing unit must be implemented for each sensor.This leads to a high level of effort, both in terms of processing units and in terms of cables and connectors.

[0004] DE 102015224782 A1 discloses a method for operating a radar device for a motor vehicle, in which operating data from an RF device are transmitted to and evaluated by a microcomputer. The RF device can have an interface in the form of a data bus for transmitting baseband data from an environmental sensing system. The required operating data can be transmitted via the digitally designed data bus, which is conventionally designed exclusively for transmitting baseband data. For this purpose, the interface can be designed, for example, according to the LVDS or MIPI-CSI2 specification.

[0005] EP 3324548 A1 discloses a bus system for vehicle systems through which a large number of processing units can exchange data. A camera and a radar control unit can transmit their data to a conversion unit, where the data can be combined and modulated for transmission. Transmission to an ADAS control unit can take place via a common, high-speed serial bus.

[0006] The object of the present invention is to provide an improved transmission solution. This object is achieved by the subject matter of the independent claims. Advantageous embodiments are the subject matter of the dependent claims, the following description, and the figures.

[0007] A device for bundling two data channels via a point-to-point connection has a first data channel that transmits a multitude of sequences of first data via a first line, wherein a data gap of a first predefined size is arranged between each two sequences of first data.

[0008] In many point-to-point connections, especially serial connections, data is transmitted in blocks. This means that a sequence of data is transmitted, followed by a pause or "blanking phase" during which no data is transmitted, and then another sequence of data. Thus, there is a data gap between each pair of data sequences. This gap can vary in length or always be the same, depending on the bus protocol. Some bus protocols allow for a predefined size of this gap. This predefined size can always be the same; a minimum size may also be defined, depending on the bus protocol. In the following, the first predefined size of the gap refers to the amount of data that is "net" available—that is, after deducting physical and logical protocol overhead—for example, for the potential insertion of further data.

[0009] The device further comprises a second data channel that transmits a multitude of sequences of secondary data over a second line. This second data channel operates at a different data rate. The second data channel may have a different number of lines or, in one embodiment, a different bus protocol.

[0010] The first data channel is designed for transmitting raw camera data, and the second data channel for transmitting raw data from a radar, lidar, ultrasound, and / or laser system. Radar systems, in particular, often have a significantly lower data rate than camera systems in many ADAS applications.

[0011] The device further comprises a third data channel that transmits a sequence of third data over a third line. In one embodiment, the third data channel uses the same bus protocol as the first data channel.

[0012] Furthermore, the device comprises a multiplexer with a multiplexer and a buffer. A first input of the multiplexer is connected to the first line, and the first input of the multiplexer is connected to the multiplexer. Drivers, for example, can be arranged between the first input of the multiplexer and the multiplexer; however, the first input of the multiplexer can also be directly connected to the multiplexer. The multiplexer has a second input connected to the second line. This second input is connected to the multiplexer via the buffer. The buffer is configured to temporarily store the second set of data from the second line. The multiplexer has an output connected to the third line and to an output of the multiplexer.Drivers can be arranged between the output of the multiplexer and the multiplexing device; however, the output of the multiplexer can also be directly connected to the multiplexing device.

[0013] The multiplexer transmits the sequence of initial data from the first input of the multiplexing device to the output of the multiplexing device. This occurs essentially without delay; that is, the initial data is essentially passed directly to the output of the multiplexing device, specifically without any storage elements in this path.

[0014] The multiplexer transmits the sequence of second data from the second input to the output when the data gap begins at the first input and when the buffer containing second data is filled with a second predefined size, where the second predefined size is less than or equal to the first predefined size. The predefined size of the data gap is known. This size can be specified, for example, by the definition of the bus protocol, by settings during bus initialization, or by a determination prior to each data gap. When the buffer containing second data is filled with at least the second predefined size, the data gap is filled with this data. According to the invention, the multiplexer is thus configured to make logical decisions about whether and when to insert second data into the data gap of the first data.As a result, the sequences of third data at the output of the multiplexing device exhibit a bundling of sequences of first data and sequences of second data. In one embodiment, the buffer memory can be filled with dummy data after a predefined number of data gaps, so that even sensors with a low data rate transmit their data to the output of the multiplexing device within a predefined time. In another embodiment, multiple second inputs with corresponding buffer memories can also be provided to acquire a large number of sensor data, for example, from different sensor types. The total amount of data from the second data channels must not, on average, exceed the total number of data gaps in the first data channel.

[0015] The described device utilizes the third data channel with a higher utilization rate than the first. Furthermore, compared to a system that transmits the first and third data channels over separate lines, the device saves on cables, connectors, driver chips, serializers, and similar components. Advantageously, this occurs without any significant data delay on the first data channel, as no storage elements are located on this path, and the data on the first channel is transmitted with a higher priority. Another advantage of the device is that, due to the higher integration and the reduced number of lines, further miniaturization of the systems is possible. This leads, in particular, to a reduction in power consumption and an improvement in the overall system's thermal performance.

[0016] In one embodiment of the device, the point-to-point connection operates on the basis of a CSI-2 protocol.

[0017] CSI-2 is a protocol standardized by the Mobile Industry Processor Interface (MIPI) Alliance. "CSI" stands for "Camera Serial Interface".

[0018] In one embodiment, the first and second data channels have different CSI channel numbers. The CSI-2 protocol optionally allows data packets to be assigned a virtual channel number. In one embodiment, channel 0 can be designated for video data and channel 1 for radar data in the so-called packet header of CSI-2. This makes it possible to distinguish and separate radar and video data in the central control unit and transmit this data via a common physical interface, namely the third data channel. In one embodiment, the third data channel, i.e., for example, the connections from sensors or sensor clusters to the central control unit, is implemented via a single LVDS coaxial cable.

[0019] According to one embodiment, the length of a packet of second data is noted in a header element of the packet of second data, so that the multiplexer can decide, based on the length, whether and what dimensioning or partitioning of the second data is carried out in the buffer memory, or whether the buffer memory can be emptied immediately, i.e., the data contained in the buffer memory is inserted by the multiplexer into the data gap of the first data.

[0020] Other design variants include: pre-known data packet lengths or a special marker at the beginning of each relevant data block that allows differentiation regardless of length (very similar to the described channel principle, i.e., the assignment of data packets to virtual channel numbers).

[0021] In one embodiment, the buffer memory is designed as a FIFO (First-In, First-Out). This allows for simple and quick allocation to the transmission times of the data on the second data channel.

[0022] In one embodiment, the data rate of the first data channel is more than twice as high, and in particular more than four times as high, as the data rate of the second data channel. This dimensioning prevents the buffer from overflowing.

[0023] In one embodiment, the multiplexing device and the buffer memory are implemented as an FPGA, a CPLD (Complex Programmable Logic Device), or an ASIC, in particular as a single FPGA, a single CPLD, or a single ASIC. Other programmable logic circuits are also conceivable for this purpose. This enables a compact and cost-effective implementation of the device, even for small to medium-sized production runs.

[0024] In an FPGA, CPLD, or ASIC, the logic of the device can be implemented as a finite state machine according to one embodiment. This allows for a particularly clear design and relatively simple verification of functionality.

[0025] In other words, the FPGA, CPLD, or ASIC is configurable (programmable or generable) in such a way that the decision as to whether and when the sequence of second data is transferred from the second input to the output of the multiplexing device is made by a finite state machine implemented on the FPGA, CPLD, or ASIC.

[0026] The invention further relates to a method for bundling two data channels based on a CSI-2 protocol, using the device described above. The method comprises the following steps: Transmitting a plurality of sequences of first data over a first line to a first input of a multiplexing device, wherein the first input is connected to a multiplexer, and wherein a data gap of a predefined size is placed between each pair of first data sequences. Transmitting a plurality of sequences of second data over a second line to a second input of the multiplexing device, wherein the second input is connected to the multiplexer via the buffer memory, which temporarily stores the second data from the second line. When the buffer memory is full of second data of the predefined size at the beginning of the data gap, inserting the sequence of second data into the data gap. Transmitting a plurality of sequences of third data, which consist of a bundling of sequences of first data and sequences of second data.

[0027] The device in question is used in particular for bundling camera and radar raw data channels.

[0028] The invention further relates to a program element which, when used to program an FPGA or a CPLD or to generate an ASIC, instructs the FPGA, the CPLD or the ASIC to carry out the described method.

[0029] The invention further relates to a computer-readable medium on which the said program element is stored.

[0030] For further clarification, the invention is described with reference to embodiments illustrated in the figures. These embodiments are to be understood as examples only, and not as limitations.

[0031] This shows: Fig. 1 : the schematic representation of a device for transmitting raw camera and radar data according to the state of the art; Fig. 2 :the schematic representation of an embodiment of a device for bundling and transmitting raw camera and radar data according to the invention; Fig. 3 : the schematic representation of an embodiment for inserting radar raw data into a data stream with camera raw data; Fig. 4 : An overview of a method for bundling two data channels over a point-to-point connection.

[0032] Fig. 1Figure 10 shows a device 10 for transmitting raw camera and radar data according to the prior art. The device 10 includes a sensor device 100 with two data sources for sensor data: a radar transceiver 200 and an image sensor 300. Transmitters 205 and 305 send the raw radar data via data line 210 and the raw camera data via data line 310. The sensor device 100 has a transmitter 280 and 380 for each data channel for high-speed data transmission. These transmitters can be implemented, for example, as LVDS serializers (LVDS: Low Voltage Differential Signaling). The data is transmitted via the two lines 285 and 385 and received and processed by a control unit 500. The control unit 500 has a receiver 290 and 390 for each data channel. These receiving devices can be implemented, for example, as LVDS deserializers.The receiving devices 290 and 390 then forward the data to a control unit 525, with a receiver module 527, for processing the radar raw data and to a control unit 535, with a receiver module 537, for processing the camera raw data.

[0033] Fig. 2 Figure 1 shows a schematic representation of an embodiment of a device 20 for bundling and transmitting raw camera and radar data according to the invention. The data sources 200 and 300 for sensor data are essentially the same as in Figure 20. Fig. 1 .The data sources are transmitted via lines 210 and 310 to the drivers (receiver modules) 420 and 430. The control unit 400 combines the raw camera and radar data into a common line 110. For this purpose, the raw camera data is transmitted via the receiver module 430 to the first input 453 of the multiplexer 450. The raw camera data consists of numerous sequences of initial data 311, which are transmitted via the first line 310 and the receiver module 430. A data gap 312 of a predefined size is positioned between each pair of initial data 311 sequences. For a representation of the data sequences, see [reference to be inserted here]. Fig. 3 . The first data 311 are transmitted to the multiplexer 455, which is part of the multiplexing device 450, and forwarded to the output 451. No storage elements are arranged along the path from the first input 453 to the output 451.

[0034] The multiplexing device 450 also has a second input 452. From there, the raw radar data is transferred to a buffer memory 457. The buffer memory 457 can be implemented as a FIFO (First In First Out). In one embodiment, the size of the buffer memory 457 is chosen to be larger than the data gap 312. Before the start of each data gap 312, the multiplexing device 450 checks the fill level of the buffer memory 457. If the buffer memory 457 has reached or exceeded a fill level corresponding to the size of the data gap 312, then the multiplexing device 450, by means of the multiplexer 455, inserts raw radar data from the buffer memory 457 into the data gap 312. In one embodiment, the data gap 312 cannot be fully utilized; that is, data is inserted when the fill level of the buffer memory 457 is less than the data gap 312.In one embodiment, the buffer memory can be filled with dummy data after a predefined number of data gaps, so that even radar sensors with a low data rate can transmit their data to the output of the multiplexing device within a predefined time. The start of the dummy data can be marked by an escape sequence.

[0035] In one embodiment, the length of the radar raw data can be recorded in a header element of the radar raw data, so that the logic of the control unit 400 can decide whether and which dimensioning or partitioning is carried out, or whether the buffer can be emptied immediately.

[0036] The shared data is then transmitted from output 451 via the shared line 110 to the shared transmitter 180. The sensor 100 transmits the shared data via a single line 185 to the control unit 500. The receiver 190 of the shared data can then split the data channels again (not shown). In the illustrated embodiment, the receiver 190 forwards the shared data to a control unit 515, with a receiver module 517 that processes the shared data.

[0037] In one embodiment, the sensor device 100 and the control unit device 500 are implemented on the same assembly. In this case, the connecting cables 185 and the interface drivers 180 and 190 can be omitted, and the transmitter 440 and the receiver module 517 can communicate directly with each other.

[0038] Fig. 3Figure 1 shows a schematic representation of an embodiment for inserting radar raw data, i.e., second data 211, into a data stream containing camera raw data 311, i.e., first data 311. The horizontal axis represents time t. The data rate of the radar raw data 211 can be, for example, 60 MB / s, while the data rate of the camera raw data 311 is 252 MB / s. It is clearly visible that a data gap 312 of a predefined size is placed between each pair of first data 311 sequences. For example, the first data packet 311.0 of the first data 311 corresponds to a line N of a camera image, and the second data packet 311.1 corresponds to line N+1 of the same camera image. The data gap 312 between them corresponds, for example, to a camera line blanking phase.

[0039] For example, a first data package A of radar raw data 211 is divided into three sub-packets A1, A2, A3.

[0040] The data subpackets A1, A2, A3 are transmitted via line 210 and receiver 420 to buffer memory 457, thus filling it (see Fig. 2 ). At time t1, buffer memory 457 is filled with data of a second predefined size, which is less than or equal to the first predefined size. At the beginning of the data gap 312, i.e., at time t2, the buffered data 212 is inserted into the data gap 312. This creates a common data stream, i.e., third data 111. This is done at Fig. 3based on three radar data packets A (A1, A2, A3), B (B1, B2, B3) and C (C1, C2, C3) (211). These are each divided into buffer fillings A1, A2, A3, etc. (212) in order to be filled into the respective data gaps 312 of the camera raw data stream 311. This results in the combined third data stream 111, in which, for example, data packets 111.1a and 111.2a originate from the two camera raw data packets 311.1 and 311.2 (camera image line N+1 and camera image line N+2), and the interposed data packet 111.1b originates from the buffered radar data packet A1 of buffer data 212. The data rate of the combined third data stream 111 is, for example, also 252 MB / s, and is therefore preferably identical to the data rate of the camera raw data stream 311.

[0041] At time t3, all data of radar data packet A (A1, A2, A3) have been transmitted, and buffer memory 457 is empty. At time t4, the first subpackage B1 of the following radar data packet B is not yet fully stored in the buffer. The resulting data gap in the camera raw data stream 211 is not filled with radar data. However, from time t5 onward, sufficient data for subpackage B1 is present in the buffer. With the end of the current camera raw data packet 311.5 (camera image line N+5), the transmission of radar subpackage B1 from the buffer into the combined data stream 111 as a new data packet 111.5b begins at time t6.

[0042] At time t7, sufficient data for the first subpacket C1 of the radar data packet C is present in buffer memory 457. However, at this time, the camera's line blanking (the data gap between the camera raw data packets of lines N+9, 311.9 and N+10, 311.10) has progressed too far. Therefore, subpacket C1 could no longer be transmitted until the beginning of the following camera image line N+10 (camera raw data packet 311.10). The transmission is thus delayed until the next data gap in the camera raw data stream 311. With the end of the camera raw data packet of camera image line N+10, 311.10, i.e., at time t8, the transmission of radar packet C1 from the buffer memory to the combined data stream 111 begins as a new data packet 111.10b.

[0043] One possible implementation uses a FIFO as buffer memory 457. The input 452 of the FIFO 457 is fed by the second data stream 211. The output of the FIFO provides the buffered data packets 212 of the radar raw data stream 211 to the multiplexer 455.

[0044] The camera raw data stream 311 is present at a first input 453 of the multiplexer 455. The one through the

[0045] The combined data stream 111 is routed from the output 451 of the multiplexer 455 to the transmitter 440 or to the serializer 180 as the sending device for the combined data. The serializer 180 converts the combined data stream 111 into serial signals (e.g., LVDS). The serial data signals can be transmitted to the control unit 515 for evaluation of the raw sensor data.

[0046] Fig. 4 shows an overview of a method 600 for bundling two data channels over a point-to-point connection.

[0047] In step 601, a multitude of sequences of first data 311 are transmitted via a first line 310 to a first input 453 of a multiplexing device 450. The first input 453 is connected to a multiplexer 455, which is part of the multiplexing device 450. A first data gap 312 of a predefined size is arranged between each pair of sequences of first data 311.

[0048] In step 602, a plurality of sequences of second data 211 are transmitted via a second line 210 to a second input 452 of the multiplexing device 450, wherein the second input 452 is connected to the multiplexer 455 via the buffer memory 457, which temporarily stores the second data 211 from the second line 210.

[0049] In step 603, at the beginning of data gap 312, it is checked whether the buffer memory 457 is filled with second data 211 of the predefined second size, where the second predefined size is less than or equal to the first predefined size. If the buffer memory 457 is filled with second data of the predefined size at the beginning of data gap 312, a sequence of the buffered second data 212 is inserted into the data gap, thus forming a sequence of third data 111.

[0050] In step 604, the sequence of third data 111, which has a bundling of sequences of first data 311 and of sequences of second data 211, is transferred to the common data line 110, so that the sequences of third data 111 have a bundling of sequences of first data 311 and of sequences of second data 211.

[0051] It should be further noted that "comprehensive" and "comprising" do not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. It should also be noted that features or steps described with reference to one of the above embodiments may also be used in combination with other features or steps of other embodiments described above. Reference numerals in the claims are not to be considered limitations. List of reference symbols

[0052] 10 Device 20 Device 100 Sensor Device 110 Common Transceiver Data Line 111 Third Data 180 Common Data Transmitter 185, 285, 385 Lines 190 Common Data Receiver 200 Radar Transceiver 205 CSI-2 Transmitter, Radar Data 210 Radar Transceiver Data Line 211 Second Data 212 Buffered Second Data 280 Radar Transceiver Transmitter 290 Radar Transceiver Receiver 300 Image Sensor 305 CSI-2 TransmitterImage data 310 Image sensor data line 311 First data 312 Data gap 380 Image data transceiver transmitter 390 Image data transceiver receiver 400 Common data control unit 420 Radar transceiver receiver 430 Image sensor receiver 440 Transmitter 450 Multiplexer 451 Multiplexer output 452 Second multiplexer input 453 First multiplexer input 455 Multiplexer 457 Buffer 500 Control unit 510 Common transceiver data line 517 Receiver module 520 Radar transceiver data line 515, 525, 535 Control units 527 Radar data receiver 530 Image sensor data line 537 Image data receiver 550 Central control unit 600Procedure 601 .. 604Procedure steps A1, A2, A3Buffer fillings A, B, CRadar data packets,

Claims

1. A device (20) for bundling two data channels via a point-to-point connection, comprising: - a first data channel that transmits a plurality of sequences of first data (311) over a first line (310), wherein a data gap (312) of a first predefined size is arranged between each two sequences of first data (311), - a second data channel that transmits a plurality of sequences of second data (211) over a second line (210), wherein the first data channel transmits camera raw data and the second data channel transmits raw data from a radar, lidar, ultrasound and / or laser system, and wherein the second data channel has a different data rate than the first data channel, - a third data channel that transmits a sequence of third data (111) over a third line (110), and - a multiplexing device (450) with a multiplexer (455) and a buffer storage device (457), wherein a first input (453) of the multiplexing device (450) is connected to the first line (310) and the first input (453) is connected to the multiplexer (455), a second input (452) of the multiplexing device (450) is connected to the second line (310) and the second input (452) is connected to the multiplexer (455) via the buffer memory (457) which is configured to temporarily store the second data (211) from the second line (210), and an output (451) of the multiplexing device (450) is connected to the third line (110), and the output (451) is connected to an output of the multiplexer (455), and wherein the multiplexer (455) is configured to, transmit the sequence of first data (311) from the first input (453) to the output (451), to make logical decisions about whether and when to insert second data into the data gap of the first data and transmit the sequence of second data (211) from the second input (452) to the output (451) when the data gap (312) begins at the first input (453) and when the buffer memory (457) is filled with second data (211) of a second predefined size, wherein the second predefined size is less than or equal to the first predefined size, so that the sequences of third data (111) comprise a bundling of sequences of first data (311) and of sequences of second data (211).

2. The device (20) according to claim 1, wherein the point-to-point connection operates on the basis of a CSI-2 protocol and wherein the first data channel and the second data channel have different CSI-2 channel numbers.

3. The device (20) according to claim 1 or 2, wherein the length of a packet of second data (211) is noted in a header element of the packet of second data (211), so that the multiplexer (455) is configured to decide, based on the length, whether and what dimensioning or partitioning of the second data (211) is carried out in the buffer memory (457), or whether the buffer memory (457) can be emptied immediately.

4. The device (20) according to any one of the preceding claims, wherein the buffer storage (457) is designed as a FIFO.

5. The device (20) according to any one of the preceding claims, wherein the data rate of the first data channel is more than twice as high, in particular more than four times as high, as the data rate of the second data channel.

6. The device (20) according to any one of the preceding claims, wherein the multiplexing device (450) and the buffer memory (457) are implemented as an FPGA, as a CPLD or as an ASIC, in particular as a single FPGA or as a single ASIC.

7. The device (20) according to claim 6, wherein the FPGA, CPLD or ASIC is configured such that the decision as to whether and when the sequence of second data (211) is transferred from the second input (452) to the output (451) of the multiplexing device (450) is made by a finite state machine implemented on the FPGA, CPLD or ASIC.

8. A method for bundling two data channels based on a CSI-2 protocol, using a device (20) configured to perform the following steps: - transmitting a plurality of sequences of camera raw data as first data (311) via a first line (310) to a first input (453) of a multiplexing device (450), wherein the first input (453) is connected to a multiplexer (455), and wherein a data gap (312) having a first predefined size is arranged between each two sequences of first data (311); - transmitting a plurality of sequences of raw data from a radar, lidar, ultrasound and / or laser system as second data (211), wherein the second data (211) have a different data rate than the first data (311), via a second line (210) to a second input (452) of the multiplexing device (450), wherein the second input (452) is connected to the multiplexer (455) via the buffer memory (457) which temporarily stores the second data (211) from the second line (210); - making logical decisions about whether and when to insert second data into the data gap of the first data - if the buffer memory (457) is filled with second data (211) having a second predefined size at the beginning of the data gap (312), wherein the second predefined size is smaller than or equal to the first predefined size, inserting the sequence of second data (211) into the data gap (312); - transmitting a plurality of sequences of third data (111) comprising a bundle of sequences of first data (311) and sequences of second data (211).

9. A use of a device according to any one of claims 1 to 7 or of a method according to claim 8 for bundling camera and radar raw data channels.

10. A program element which, when used to program an FPGA or CPLD or to generate an ASIC, instructs the FPGA, CPLD or ASIC to perform the method according to claim 8.

11. A computer-readable medium on which a program element according to claim 10 is stored.