Environmental sensor with sequential processing line partial exposure

Abstract

The invention relates to a method for operating an environmental sensor (1), wherein exposures of a number h_a of physical lines from pixels of a pixel matrix are started sequentially (S1), and wherein physical exposure results of the h_a physical lines are read out (S2) and taken into account in data processing together with virtual exposure results of a number h-h_a of virtual lines forming a vertical blanking interval, wherein of a total number h processing lines exactly a number N processing lines are exposed simultaneously, of which a number N_a of physical lines are exposed simultaneously, wherein such processing lines that are exposed simultaneously at one time are separated by a number d-1 of unexposed processing lines.where for a minimum number N_a_min of simultaneously exposed physical lines N_a_min = floor(h_a / d) and where for a maximum number N_a_max of simultaneously exposed physical lines N_a_max = ceil(h_a / d), with N_a_min ≤ N_a ≤ N_a_max.,

Description

[0001] The invention relates to a method for operating an environment sensor of a vehicle, as well as a system for environmental detection of a vehicle.

[0002] Modern vehicles, such as passenger cars, typically use a variety of cameras. While interior cameras can, for example, record seat occupancy or passenger condition, exterior cameras are used to monitor the environment, for example, to record the behavior of other road users.

[0003] If two camera- or 2D-sensor-based signal processing applications, such as a digital rearview mirror in a vehicle and a detection system for other road users, require the same or a similar camera / sensor viewing direction but different (minimum) frame rates f1 (first signal processing application) and f2 (second signal processing application) as well as different vertical (minimum) opening angles α1 (first signal processing application) and α2 (second signal processing application), then, in addition to using two (camera- or 2D-) sensors with the parameters f1 and α1 or f2 and α2 respectively, it is fundamentally possible to use a single sensor with the parameters max(f1, f2) and max(α1, α2). The data rate generated by a sensor is proportional to its frame rate f and proportional to its number of processing lines h per frame. The number of processing lines h, in turn, increases with the vertical opening angle α of the sensor.The data rate of a combined sensor is at least as high as the data rate of each of the two individual sensors. If one individual sensor has a significantly higher frame rate and the other individual sensor has a significantly higher vertical field of view (or a significantly higher number of processing lines), then the data rate of the combined sensor is also significantly higher than the data rate of either individual sensor. Correspondingly, the (manufacturing) costs for the data transmission channels from the sensor to the two processing units implementing the first and second signal processing applications, respectively, increase with the data rate. It is important to note that these increased manufacturing costs also affect manufacturing variants of the system that each include only one of the two signal processing applications.This means that the ability to offer one signal processing application on the market increases the manufacturing costs of the other signal processing application. This is particularly disadvantageous when the demand for both signal processing applications differs significantly, with higher volumes typically resulting in greater cost pressure.

[0004] If one signal processing application requires camera images with a wider vertical field of view, while another requires camera images with a relatively higher temporal resolution, it would be logical to use two separate cameras with correspondingly different characteristics. Similarly, it would only be logical to use a single camera if the requirements regarding the field of view and temporal resolution are also similar, or if the requirements of one camera represent a subset of the requirements of the other. However, using a single camera with comparable performance generally results in higher manufacturing costs.

[0005] Two signal processing applications for cameras facing against the vehicle's main direction of travel are, on the one hand, environmental sensing with the detection of other road users, particularly emergency vehicles, for the execution of an automated driving function at Level 2+ / Level 3 or higher, and, on the other hand, a digital interior mirror. Both signal processing applications require a rear-facing camera with a very similar viewing direction and similar positioning within the vehicle. If both signal processing applications are to be available in one vehicle, the question arises whether a single camera is sufficient for both applications, or whether two different cameras are required for the signal processing applications with their differing image characteristics.The different signal processing applications differ, for example, in that the signal processing application for a digital rearview mirror requires a camera with a smaller vertical field of view but 60 fps (frames per second), while the signal processing application for environmental monitoring in automated driving requires a larger vertical field of view but only 10 to 30 fps. A camera with a larger vertical field of view and 60 fps, which would thus meet the requirements of both signal processing applications, may exceed cost expectations or may not be available.For so-called "Serdes" chips ("Serdes" is a portmanteau of "serializer / deserializer") used to transmit a video data stream from a camera to a control unit, and which support the higher data rate required for the wider field of view and 60 fps, essentially the same considerations apply. Since a customer should only be able to order one of the two signal processing applications per vehicle, a concept is desirable that does not lead to increased manufacturing costs when only one of the two signal processing applications is ordered.

[0006] Prior art includes CMOS image sensors with a matrix-shaped pixel sensor unit that can be read out sequentially. DE 10 2009 053 281 A1 relates to an image sensor, in particular a CMOS image sensor for digital cameras, with a plurality of pixels arranged in rows and columns, wherein each pixel comprises: a light-sensitive detector element to generate an electrical charge from incident light during an exposure process, a readout node, a transfer gate to which a transfer control pulse can be applied to enable charge transfer from the detector element to the readout node, and a reset device to reset a charge present in the readout node to a reference value; and a control device for controlling the transfer gate and the reset device of the respective pixel.wherein the control device is designed such that the respective pixel is read out in several readout steps during the continuous charge generation in a single exposure process, such that for each of the several readout steps a respective transfer control pulse is applied to the transfer gate and a respective readout result is then generated, wherein the reset device is activated between the transfer control pulses of the respective exposure process, and wherein only the last transfer control pulse enables a complete charge transfer of the respective charge present in the detector element to the readout node.

[0007] DE 102 45 912 A1 further relates to a method for operating a spatially resolving, optoelectronic sensor device with a sensor surface consisting of a plurality of light-sensitive sensor elements, and with a recording optic that maps a spatial light distribution in an environment detected by it onto the sensor surface, wherein a subset of sensor elements is activated for the duration of an assigned activation interval for the integration of incident light to generate a corresponding partial image of the environment, and with an illumination device that, in temporal relation to the activation intervals, illuminates the environment detected by the recording optic at least temporarily and in certain areas with light of a wavelength range integrable by the sensor elements.wherein, to generate a calculated image of the environment, the sensor area is activated as a sequence of n-tuples of pairs of activation intervals and associated subsets of sensor elements such that within an n-tuple, n subsets of sensor elements are activated during n activation intervals under different lighting conditions of the environment caused by corresponding control of the lighting source, and partial images recorded within an n-tuple are processed together in a data processing device.

[0008] Furthermore, image sensors are known from the prior art that can be read out to obtain images with high resolution and, optionally, low resolution. US 2024 / 0406600 A1 relates in this context to a sensor device comprising: a plurality of pixel arrays, each pixel array comprising a plurality of pixels, each pixel comprising a light sensor element configured to generate and store a charge in response to incident light.Each pixel array comprises a charge storage device configured to receive charge from each of the light sensor elements of the pixel array, a first plurality of switches, wherein each switch of the first plurality of switches is connected between a respective pixel of the pixel array and the charge storage device, a second plurality of switches comprising a high-resolution selector switch and a low-resolution selector switch, wherein each of the high-resolution selector switch and the low-resolution selector switch is connected in parallel to an output of the charge storage device, and a plurality of pixel output lines, wherein each pixel output line is configured to output signals representing pixel values ​​corresponding to one or more pixel arrays coupled to the respective pixel output line.

[0009] US Patent 2023 / 179891 A1 further relates to a vertically stacked image sensor with HDR imaging functionality. The image sensor comprises a first substrate, a pixel array organized into a plurality of pixel subarrays, each pixel comprising a photoelectric element for integrating a photocharge during each of a plurality of subframe exposures, a transfer gate, and a buffered charge-to-voltage converter. A first charge accumulation element of the charge-to-voltage converter is operatively connected to at least one second charge accumulation element via a gain switch. The image sensor includes control circuitry configured to trigger a partial or complete transfer and temporally interleaved at least two rolling shutter control sequences.On the second substrate, separate readout blocks are provided for each pixel subarray, each comprising an A / D conversion unit, a pixel memory logic and a pixel memory unit in a pipeline architecture.

[0010] Further technological background information is revealed in US 2025 / 0 071 435 A1.

[0011] The object of the invention is to enable sensory detection of a vehicle's environment by a single sensor unit, in particular a single camera, which is suitable for various requirements of different vehicle applications and where the hardware requirements for the sensor unit are as low as possible.

[0012] The invention is defined by the features of the independent claims. Advantageous further developments and embodiments are the subject of the dependent claims.

[0013] A first aspect of the invention relates to a method for operating an environmental sensor of a vehicle, wherein exposures of a number h_a of physical lines comprising pixels of a pixel matrix of the environmental sensor are started at least partially sequentially, and wherein physical exposure results of the h_a physical lines are read out and, together with virtual exposure results of a number h-h_a virtual lines forming a vertical blanking interval, are taken into account during data processing, characterized in that, of a total number h processing lines with h≥h_a, comprising the h_a physical and the h-h_a virtual lines, exactly a number N of processing lines are exposed simultaneously at a plurality of different times, of which a number N_a of physical lines are exposed simultaneously, wherein such processing lines that are exposed simultaneously at a time,are separated by a number d-1 of unexposed processing lines, i.e., spaced apart by these, where for a minimum number N_a_min of simultaneously exposed physical lines N_a_min = floor(h_a / d) and where for a maximum number N_a_max of simultaneously exposed physical lines N_a_max = ceil(h_a / d), with N_a_min ≤ N_a ≤ N_a_max.

[0014] Preferably, the N_a physical lines are exposed simultaneously in phase.

[0015] N is the number of processing lines exposed simultaneously; this includes physically exposed physical lines as well as virtually exposed virtual lines. Preferably: h=N*d.

[0016] Furthermore, the following applies: N_a_min = floor(h_a / d) where N_a_min is the minimum number of simultaneously exposed physical lines, where floor() is the rounding function that assigns to each real number the nearest non-larger integer. N_a_max = ceil(h_a / d) where N_a_max is the maximum number of simultaneously exposed physical lines, where ceil() is the rounding function that assigns to each real number the nearest non-smaller integer. If h_a is an integer multiple of d, then N_a_max = N_a_min = h_a / d; otherwise, N_a_max = N_a_min + 1. In general, therefore, N_a_min ≤ N_a ≤ N_a_min + 1.

[0017] Due in part to the history of the cathode ray tube (CRT), where an electron beam builds an image line by line from top to bottom, requiring a certain amount of time after displaying the bottom line to return to the position of the top line before building the next image, it is still common practice in modern digital video recording, processing, transmission, and display systems to insert a pause, known as a vertical blanking interval, after each line of recording, processing, transmission, or display. The duration of the vertical blanking interval can be converted into a number of virtual lines, corresponding to the number of physical lines that could be recorded, processed, transmitted, or displayed in the same amount of time at a constant processing rate. The number of virtual lines in the blanking interval is the ratio of the blanking interval duration to the line period T_h.When considering video systems, it is common practice to add the equivalent number of virtual lines during the blanking interval to the number of actual physical lines to obtain a total of h processing lines per frame. The timing of video signal processing can then be more easily described based on the total number h processing lines per frame, rather than on the physical number of lines h_a and the blanking interval duration. This corresponds to the timing of video signal processing in a hypothetical video system that actually processes h physical lines but does not include processing pauses due to a vertical blanking interval. When the "exposure," "capture," or "transmission" of a processing line is mentioned in the following, it can refer to either a physical or a virtual line, depending on the context.In the case of a physical line, actual exposure, capture, or data transmission takes place. In the case of a virtual line, however, there is only a correspondingly long pause in processing or, alternatively, the processing of meaningless data.

[0018] Each physical row is formed from physical pixels, and each physical pixel comprises one or more sensor elements arranged in a specific way (for example, horizontally and / or vertically adjacent) which are simultaneously illuminated by ambient light. The grouping of sensor elements, or rather the sample values ​​they generate, into a "pixel" due to simultaneous illumination does not necessarily imply a semantic relationship between these sample values. Therefore, the definition of "pixel" used here—according to which, for example, in the case of simultaneous illumination of all sensor elements within a physical row, each row of the pixel matrix comprises only a single pixel—may differ from other common definitions of the term "pixel."

[0019] The environmental sensor preferably provides a 2D matrix of measured values ​​and is particularly preferably a camera. The environmental sensor sequentially exposes physical lines of an image matrix, with the start of the exposures of the physical lines being offset from one another in time, preferably by a predetermined time offset for each immediately following physical line. Exposure also occurs for the virtual lines, but only virtually, for the purpose of providing a virtual exposure result. The environmental sensor has a total number of processing lines h, which comprise h_a physical lines and any additional virtual lines in the vertical blanking interval; h ≥ h_a.

[0020] After each exposure of a physical line is read, that line is reset to be ready for another exposure and subsequent readout. Time-shifted exposure simplifies the time-shifted readout of exposure results, which in turn optimizes the use of an available communication channel for outputting the results of the processing lines after their respective exposures. This is because, unlike a pure global shutter principle, not all processing lines are read out simultaneously, but sequentially, allowing the results of the processing lines to be transmitted one after the other. This offers advantages because, unlike a global shutter, neither an image buffer is required to achieve a constant bandwidth requirement, nor is a transmission channel needed to handle highly fluctuating bandwidth demands.

[0021] The fact that the processing lines, which are exposed simultaneously at different times, are separated from each other by unexposed processing lines means that at least a subset of simultaneously exposed processing lines are not directly adjacent. This allows the results of the exposures of processing lines to be distributed across the image height and subsequently read out directly or indirectly, and the results to be output in a staggered manner.

[0022] The condition that up to N processing lines are exposed simultaneously at a multitude of different times explicitly includes the possibility that this affects different processing lines over time, but the total number of these different times is always at most N.

[0023] In particular, at any given time, at most N processing lines are exposed simultaneously, preferably continuously. N is a natural number greater than 1. A total of h processing lines are used, where the number of line periods of the virtual lines of the vertical blanking interval is counted as processing lines. Preferably, starting simultaneously from each time T_h * k (where T_h is a fixed processing line period duration; k is an integer index denoting a time step), the number N processing lines with the respective processing line indices k mod h, (k - d) mod h, (k - 2 d) mod h, (k - 3 d) mod h, ..., and (k - (N-1) * d) mod h are exposed, where the number h processing lines are preferably designated by the processing line indices 0 to h - 1 according to their spatial position relative to each other, and where preferably h = N * d.

[0024] The term "vertical" refers to a direction perpendicular to a given processing line in the plane of the pixel matrix. A vertical extension of the pixel matrix corresponds to one image height. After the exposure time T_exp has elapsed, i.e., from time T_h * k + T_exp, the data of the respective exposure is acquired by sampling and, in particular, analog-to-digital conversion. The results of the exposures of these N processing lines, including N_a physical lines, are then transmitted via parallel channels and / or time-division multiplexing—for example, via a single channel in the order of the processing line indices listed above.

[0025] According to an advantageous embodiment, exactly a number N of processing lines are exposed at any given time.

[0026] According to a further advantageous embodiment, processing lines with the following processing line indices from an ascending line numbering from 0 to h-1 are exposed simultaneously in phase, where k is an integer and denotes a time step, and mod is the modulo operator: k mod h, (k−1*d) mod h,(k−2*d) mod h,(k−3*d) mod h,...(k−g*d) mod h, with g=N−1.

[0027] According to a further advantageous embodiment, the exposure results read out by the N processing lines are output via a single communication channel in a time-division multiplex, using a FIFO memory from which the exposure results are output sequentially and therefore not simultaneously. FIFO stands for "first in first out".

[0028] When N processing lines are simultaneously acquired using time-division multiplexing over a single channel, the acquired processing lines are transferred in parallel to a shift register, specifically a first-in-first-out (FIFO) memory comprising, and in particular consisting of, N line storage units, and subsequently output sequentially over a period of duration T_h. Here, T_h is a processing line period, which describes a time offset between the start of exposures of successively exposed processing lines and corresponds to the reciprocal of the processing line or horizontal frequency. The duration of the transmission period for N processing lines is independent of N, since the output data rate is proportional to N, and the transmission duration of a single processing line is therefore T_h / N. The output order of the simultaneously exposed processing lines can be arbitrarily defined.

[0029] According to a further advantageous embodiment, the exposure of each of the N processing lines begins with a time offset, cascaded in pixel-wise time-division multiplexing with a processing line-to-processing line time offset of T_h / N / M, where T_h is a predetermined processing line period and M is the number of pixels per processing line. The advantage of this embodiment is that only one scanning unit, instead of N, is required to detect the exposed pixels, although this scanning unit operates at N times the sampling rate.

[0030] According to another advantageous embodiment, the exposure of the N processing lines begins with a respective time offset, cascaded in time with a processing line-to-processing line time offset of T_h / N, where T_h is a predetermined processing line period duration.

[0031] In this variant, the exposure of each N processing line does not begin exactly simultaneously, but is staggered in time with the processing line-to-processing line time offset T_h / N, such that each exposure ends immediately before the transmission of the respective processing line. This change has the minor disadvantage that successive exposure periods of the same processing line are no longer exactly equidistant, but offers the (cost) advantage that no processing line buffers are required to transmit the (now only quasi-)parallel processing lines sequentially. Likewise, only one scanning unit is needed instead of N for capturing the exposed processing lines, although this unit operates at N times the sampling rate.By acquiring N processing lines in parallel or quasi-parallel instead of just one processing line, the data rate and required bandwidth increase by a factor of N, regardless of the parallel and / or time-division multiplex transmission shown, and correspond to the value for conventional non-parallel processing line acquisition with N times the frame rate.

[0032] According to a further advantageous embodiment, only the exposure result of a processing line with a processing line index (k - i*d) mod h is output in a communication channel, wherein the exposure result is captured at the respective time T_h * k + T_exp, where i is a fixed value from the set {0, 1, 2, .., N-1}, mod is the modulo operator, T_h is a predetermined processing line period duration and T_exp is an exposure duration of a respective processing line.

[0033] Here, a video stream is output via a communication channel connected to the environmental sensor. This stream contains only the exposure results of one of the N processing lines exposed in parallel or quasi-parallel. The output processing line, captured at time T_h * k + T_exp, has the index (k - i*d) mod h, where i is a fixed value in {0, 1, 2, ..., N-1}. This video stream has the full vertical resolution of h processing lines and a single frame rate. The term "vertical" refers to a direction perpendicular to the respective processing line path in the plane of the pixel matrix. One vertical extent of the pixel matrix corresponds to one image height.

[0034] According to a further advantageous embodiment, only the results of exposures from processing lines that are located within a predetermined vertical area of ​​the pixel matrix of the environmental sensor are read and output.

[0035] According to a further advantageous embodiment, two video data streams, each comprising a result of the exposures, are output in time-division multiplex from the environment sensor via a communication channel, wherein in a first of the video data streams, a respective result of the exposure is output for all processing lines of the pixel matrix immediately following the end of the exposure, and for a second of the video data streams, a respective result of the exposure is temporarily stored in a processing line memory for a respective duration of a result that is transmitted in the first of the video data streams and subsequently transmitted.

[0036] A further aspect of the invention relates to a system for sensing the environment of a vehicle, comprising an environment sensor for a vehicle, which is configured to initiate exposures of a number h_a of physical lines, comprising pixels of a pixel matrix of the environment sensor, at least partially sequentially, and wherein the system is configured to read out physical exposure results of the h_a physical lines and to consider them together with virtual exposure results of a number h-h_a virtual lines, forming a vertical blanking interval, during data processing, characterized in that the environment sensor is configured to expose exactly a number N of processing lines simultaneously from a total number h processing lines with h≥h_a, comprising the h_a physical and the h-h_a virtual lines, at a plurality of different times, of which a number N_a of physical lines are exposed simultaneously.where such processing lines that are exposed simultaneously at one time are separated by a number d-1 of unexposed processing lines, where for a minimum number N_a_min of simultaneously exposed physical lines N_a_min = floor(h_a / d) and where for a maximum number N_a_max of simultaneously exposed physical lines N_a_max = ceil(h_a / d), with N_a_min ≤ N_a ≤ N_a_max.

[0037] According to a further advantageous embodiment, the system further comprises a first processing unit configured to perform a first data processing procedure for the operation of a vehicle, and comprising a second processing unit configured to perform a second data processing procedure for the operation of the vehicle, and comprising a first decimation unit configured to discard a first part of the video data stream output by the environment sensor for the first processing unit, and comprising a second decimation unit configured to discard a second part, different from the first part, of the video data stream output by the environment sensor for the second processing unit.

[0038] Advantages and preferred further developments of the proposed system result from an analogous and substantive transfer of the above statements made in connection with the proposed procedure.

[0039] Further advantages, features and details will become apparent from the following description, in which - possibly with reference to the drawing - at least one embodiment is described in detail.

[0040] They show: Fig. 1: A conventional rolling shutter method for an environmental sensor according to an embodiment of the invention. Fig. 2: Schematic representation of a method for operating an environmental sensor according to an embodiment of the invention. Fig. 3-9: Explanations of the procedure of Fig. 2 according to further embodiments of the invention. Fig. 10-16: Schematic topologies for processing units connected to the environmental sensor according to exemplary embodiments of the invention.

[0041] The representations in the figures are schematic and not to scale.

[0042] Fig. Figure 1 shows a conventional rolling shutter principle according to the state of the art in a time diagram. Time runs horizontally, while the various processing lines of the image sensor are plotted vertically. In the Fig. In the state-of-the-art rolling shutter principle shown in point 1, each processing line is exposed for a certain period of time, which determines the horizontal length of the bars. Fig. 1. At the end of each exposure of a processing line, data is transmitted, which outputs the result of the exposure, and the exposure is reset. The duration of this data transmission is symbolized by the last segment of each bar. The start of the respective exposures of the processing lines, and thus the data transmission for each exposed processing line, occurs staggered across the individual processing lines, allowing the corresponding data transmission to also be staggered. This ensures optimal utilization of a communication channel with a data stream and thus enables the time-shifted transmission of the data received via the processing lines. The time offset applied between two pairs of processing lines corresponds to the so-called processing line period, the reciprocal of the processing line or horizontal frequency.If the bandwidth of the transmission channel is just sufficient, then (as in . Fig. Figure 1 shows the duration of the transmission of a processing line, precisely the processing line period. In this case, data is transmitted continuously from the end of the exposure of the topmost image processing line until the end of the exposure of the bottommost image processing line. This is followed by the so-called vertical blanking interval, a transmission pause lasting several processing line periods, after which the data of the topmost processing line of the subsequent image is transmitted. Only two consecutive readout sequences are shown as examples; however, these are generated repeatedly throughout the entire operating time of the environmental sensor 1.

[0043] Fig. Figure 2 shows a method for operating an environment sensor (1) of a vehicle, wherein exposures of a number h_a of physical lines comprising pixels of a pixel matrix of the environment sensor 1 are started at least partially sequentially S1, and wherein physical exposure results of the h_a physical lines are read out S2 and taken into account in data processing together with virtual exposure results of a number h-h_a of virtual lines forming a vertical blanking interval, wherein of a total number h processing lines with h≥h_a comprising the h_a physical and the h-h_a virtual lines, exactly a number N processing lines are exposed simultaneously at a multitude of different times, of which a number N_a of physical lines are exposed simultaneously, wherein such processing lines that are exposed simultaneously at a time are separated by a number d-1 of unexposed processing lines.where for a minimum number N_a_min of simultaneously exposed physical lines N_a_min = floor(h_a / d) and where for a maximum number N_a_max of simultaneously exposed physical lines N_a_max = ceil(h_a / d), with N_a_min ≤ N_a ≤ N_a_max.,

[0044] Fig. Figure 3 shows an exemplary application scheme for the processing lines of an environmental sensor 1. Simultaneous, phase-coherent acquisition of N processing lines with transmission via N parallel channels C#0, C#1, ..., C#(N-1) for information transmission is described in Fig. Figure 3 illustrates this for N = 3 and h = 12. The number of processing line periods of the vertical blanking interval, and consequently the duration of the blanking interval, is shown in Fig. 3 zero. Since only three of the twelve processing lines are exposed and read out in phase at any given time, and the selection of the N exposed processing lines varies over time, the letters A, B, and C are used to clarify the different time-staggered selections of N processing lines. Starting at time T_h * k, the N processing lines with the indices k mod h, (k - d) mod h, (k - 2 d) mod h, (k - 3 d) mod h, ... (k - (N-1)*d) mod h are exposed simultaneously and in phase. After the exposure time T_exp has elapsed, i.e., from time T_h * k + T_exp, the data of the respective exposure is acquired by sampling and, in particular, an analog-to-digital conversion, and the results of the exposures of these N processing lines are subsequently transmitted.The reading of the respective result of each exposure of a processing line is carried out for the respective processing line sequences C0 to C11, B0 to B11 and A0 to A11. In this process, N processing lines are exposed simultaneously at any given time.

[0045] Fig. Figure 4 shows another variant of the operation of the environmental sensor 1. When acquiring the N processing lines with transmission in line-by-line time-division multiplexing over a single channel, the acquired processing lines are transferred—optionally in parallel or in pixel-by-line time-division multiplexing—within a time period T_c into a line buffer and then output sequentially from it, line by line, over a period of time T_h. The time available for the transmission of a single processing line is thus T_h / N. (For the time period T_r, which an implementation for acquiring and transferring the processing line data to the buffer, T_c <= T_h must hold.) Fig. 4 (where T_c is assumed to be 0 for an idealized scenario). The output order of the simultaneously exposed processing lines can be arbitrarily defined. For time-division multiplexing of N virtual channels VC#0, VC#1, .. VN#(N-1) over a single real communication channel, for example, a processing line exposed from time T_h * k with index (k - n*d) mod h can be delayed by the time n * T_h / N and thus sent from time T_h * k + T_exp + T_c + n * T_h / N. (Here, let n ∈ {0, 1, 2, .. N - 1}.) This is in Fig. 4 for N = 3 and h = 12. The number of line periods of the vertical blanking interval and, accordingly, the duration of the blanking interval is shown in Fig. 4 is zero. The temporary storage of a processing line in a line buffer is represented by a field labeled "T". In the virtual channels VC#2 to VC#0, a readout of each exposure is transmitted strictly sequentially. This means that, in particular, packet B4 is transmitted only after packet A8, without any simultaneous overlap. Furthermore, packet C0 is only transmitted once packet B4 has been transmitted, packet A9 only after C0 has been transmitted, and so on.

[0046] Fig. Figure 5 shows another variant of the operation of the environmental sensor 1, also illustrated by way of example for N = 3 and h = 12. Fig. In step 5, the exposure of the N processing lines does not begin exactly simultaneously, but is cascaded in time with the processing line-to-processing line time offset T_h / N, such that each exposure ends immediately before the transmission of the respective processing line. This change has the minor disadvantage that successive exposure periods of the same processing line are no longer exactly equidistant, but offers the (cost) advantage that no line buffer is required to transmit the processing lines sequentially, as they are now exposed only in close proximity to each other instead of simultaneously. Likewise, only one scanning unit is needed instead of N for capturing the exposed processing lines (although this unit operates at N times the sampling rate).By simultaneously or in close succession capturing N processing lines instead of just one processing line, the data rate and required bandwidth increase by a factor of N, regardless of the parallel and / or time-division multiplex transmission shown, and correspond to the value for a conventional non-parallel processing line capture with N times the frame rate.

[0047] Fig. Figure 6 shows another variant of the operation of the environmental sensor 1. In this variant, the image sensor can be used in parallel or alternatively to the output accordingly. Fig. 3, Fig. 4 or Fig. 5. Output a video stream that displays only one of the N at a time ( Fig. 3 and Fig. 4) or promptly ( Fig. 5) exposed processing lines, as in Fig. The output is shown for N = 3 and h = 12. The output processing line, captured at time T_h * k + T_exp, has the index (k - i*d) mod h, where i is a fixed value ε {0, 1, 2, .., N-1}. This video stream has the full vertical resolution of h processing lines and a single frame rate.

[0048] Likewise, in parallel or as an alternative to the expenditures, the following can also be done: Fig. 3, Fig. 4, Fig. 5, Fig. 6 a data stream is output in which only one of the N is output at a time ( Fig. 4) or promptly ( Fig. 5) is contained in exposed processing lines that lie within a selected area containing up to d = h / N arbitrary adjacent processing lines of the image. This is in Fig. Figure 7 is shown as an example for N = 3 and h = 12, where, of the total h = 12 processing lines with indices 0 to 11, (for example) d = h / N = 12 / 3 = 4 processing lines with indices 3, 4, 5, and 6 are selected as the image area to be transmitted. The resulting video stream has only one Nth of the full vertical resolution, thus containing only d = h / N processing lines per frame, but has N times the frame rate. Video streams according to Fig. 6 and according to Fig. 7 require compared to a video stream according to Fig. 3, Fig. 4, Fig. 5 or, for a video stream with conventional non-parallel line capture, N-fold frame rate, and full frame height h, each transmission channel with a data rate and bandwidth reduced by a factor of 1 / N is used. These therefore correspond again to the value for conventional non-parallel line capture with a single frame rate. When simultaneously transmitting a video stream with full frame height h and a single frame rate according to... Fig. 6 and a video stream with image height d = h / N and N-fold frame rate according to Fig. Therefore, the total data rate and required total bandwidth is 2 / N of the value for a video stream with image height h, N-fold frame rate, and conventional, non-parallel line-by-line processing. For N > 2, this comparison thus results in a bandwidth saving.

[0049] Fig. Figure 8 shows a time-division multiplex transmission of two such video streams, using the same example parameters as in Fig. 6 and Fig. 7 are used (N = 3 and h = 12; the video stream with reduced image height contains the 12 / 3 = 4 processing lines with indices 3, 4, 5 and 6). After simultaneous exposure and data acquisition, the exposure result of the processing line of one video stream is immediately transmitted, for example as in Fig. Figure 8 shows the video stream for the full image height h, and the exposure result of the processing line of the other video stream is temporarily stored in a processing line buffer for the duration of this transmission and subsequently transmitted.

[0050] In a further embodiment, it is provided that instead of the simultaneous exposure and acquisition of N processing lines, only the (for example, as in Fig. As shown in Figure 8, two processing lines are simultaneously exposed and captured, which in the respective embodiment of the invention are actually output (and transmitted) by the image sensor. Fig. Note 8: For every Nth exposure / capture of any processing line of the video stream with image height d = h / N, the same processing line must be exposed / captured simultaneously for the video stream with full image height h. This means that at this point, instead of two temporal processing line exposures / captures, only one needs to occur.

[0051] Fig. 9. Another variant of the operation of the environmental sensor 1. The concept of timely instead of simultaneous exposure and acquisition of N processing lines (see Fig. 5) is also on the in Fig. The embodiment of the invention shown in Figure 8 is transferable, in which only two processing lines are exposed and captured in parallel. As in Fig. As illustrated in Figure 9, the exposure of the two processing lines does not occur exactly simultaneously, but rather is cascaded in time with the processing line-to-processing line time offset T_h / 2, such that each exposure ends immediately before the transmission of the respective processing line. It is particularly important to note that with each Nth acquisition of any processing line of the video stream with frame height d = h / N, the exposure can be either delayed by the time offset T_h / 2 or (as in Figure 9) Fig. (as shown in Figure 9) the same processing line must be captured in advance for the video stream with full frame height h. Since two separate exposures would overlap if T_exp > T_h / 2, but as a rule T_exp >> T_h / 2 applies and overlapping exposures of the same processing line are physically impossible, the second exposure / capture must be omitted and replaced by a copy of the first capture, preferably temporarily stored in a processing line buffer. This adjustment results in the theoretical, but practically irrelevant disadvantage (due to T_exp >> T_h / 2), that (as in Figure 9) Fig. 9. In the multiplex sequence shown – exposure / capture of the processing line of the video stream with image height h first) successive exposure periods of the same processing line in the video stream with image height d = h / N are no longer exactly equidistant, or (with the reversed multiplex sequence) in the video stream with image height h, all processing lines that are also contained in the other video stream are always exposed too early by the duration T_h / 2. The (cost) advantage of the in Fig. 9 shown variation of Fig. The advantage lies in the fact that only one scanning unit is needed instead of two to capture the two exposed physical lines of the processing lines, although this unit operates at twice the sampling rate.

[0052] In one version, the image area for the video stream can be defined with image height d = h / N (see below). Fig. 7, Fig. 8 and Fig. 9) can be selected statically or dynamically during operation, i.e., shifted stepwise from image to image up or down. If the selected image section d = h / N is to include processing lines starting from processing line y0, then processing lines are exposed, captured, and transferred in the following cyclical sequence: y0, y0+1,y0+2,..y0+d−1, y0, y0+1,y0+2,..y0+d−1 y0, y0+1,y0+2,..y0+d−1, y0, y0+1, y0+2,..y0+d−1,

[0053] To change the beginning of the image section from processing line y0 to processing line y1, i.e., to shift it down by dy := y1 - y0 processing lines (dy > 0) or up by -dy processing lines (dy < 0), a single image (highlighted below by a '←' symbol) is lengthened by dy processing lines or shortened by -dy processing lines: y0, y0+1,y0+2,..y0+d−1, y0, y0+1,y0+2,..y1+d−1,← y1, y1+1,y1+2,..y1+d−1, y1, y1+1,y1+2,..y1+d−1,

[0054] The altered length of a single frame temporarily decreases (dy > 0) or increases (dy < 0) the frame rate, and the phase of the video stream is permanently shifted forward or backward. Preferably, instead of shifting the image area by a larger number of processing lines, several successive shifts by smaller numbers of processing lines are performed. This allows, firstly, the temporary deviation of the frame rate from its target value to be kept within the tolerance range of the video sink (e.g., a screen), and secondly, avoids a jump in the image area that might be perceived as unpleasant by the viewer.

[0055] For dy, a non-positive integer dy_min can be set as a minimum value to ensure a minimum number of processing lines d + dy_min for each image. Preferably, a constant number of processing lines, at most d + dy_min, is displayed.

[0056] In one embodiment, the image section data stream is written to a FIFO buffer, read from it, and sent to the video sink. When reading an image, if dy > 0, the number of additional image lines dy is skipped, and if dy < 0, the number of missing processing lines -dy is inserted. This ensures that the video sink receives a video data stream with a constant frame rate and constant phase.

[0057] In the Fig. In the implementation shown in Figure 10 of a system with an environment sensor 1 for a vehicle, the sensor data output by the environment sensor 1 are received by the two parallel-connected processing units P1 and P2. Unit P1 requires the full image height but only 1 / N of the frame rate of the environment sensor 1, while unit P2 requires only 1 / N of the image height but the full frame rate of the environment sensor 1. The environment sensor 1 delivers the complete data at the full data rate r as shown in Figure 10. Fig. Figures 4 and 5 illustrate this, or he provides selected data at a data rate of 2r / N as in Fig. 8 or Fig. Figure 9 is shown. The downstream decimation units D1 and D2 each reduce the data rate to r / N by discarding images accordingly. Fig. 6 (D1) or by discarding the top and bottom image edges accordingly Fig. 7 (D2). If only the function of processing unit P1 is required, then P2 can be omitted, as in Fig. Shown in 11. The decimation unit D2 is inactive in this case or can also be omitted. Conversely, if only the function of the processing unit P2 is required, then P1 can be omitted, as shown in Fig. 12 is shown. Decimation unit D1 is inactive in this case or can also be omitted. Fig. Figure 13 (above) shows a series connection of the processing units P1 and P2. Here, the environmental sensor 1 transmits the complete sensor data at full data rate r or correspondingly via an optionally present, but then switched off, i.e., forwarding the data unchanged, decimation unit D1. Fig. 8 or Fig. 9 selected data points with a data rate of 2r / N are sent to the processing unit P2. This forwards the sensor data, reduced to the data rate r / N, to the processing unit P1 via an (active) decimation unit D1'. Fig. Figure 13 (below) shows a (possible manufacturing) variant of the same system in which the processing unit P2 and the downstream decimation unit D1' are omitted. Instead, the decimation unit D1, which is downstream of the environmental sensor 1, is active. The in Fig. The arrangement shown in Figure 13 (above and below) is particularly suitable if P1 implements a standard equipment function and P2 an optional equipment function. Since P1 only needs to receive the data rate r / N in any case, its manufacturing costs are not increased by offering the optional equipment P2. Regarding the environmental sensor 1, it is possible to implement it either with a selectable output data rate between r (or 2r / N) and r / N (decimator D1 off for optional equipment, on for standard equipment) or in two variants, each with a fixed output data rate of r (or 2r / N) or r / N (decimator D1 present in the standard equipment, or not present in the optional equipment). In the latter case, an increase in the manufacturing costs of the standard equipment caused by offering the optional equipment can also be avoided for the environmental sensor 1.

[0058] Fig. Figure 14 shows another possible implementation of the invention, in which, unlike in Fig. 12. Processing unit P2 implements a possible basic equipment function and processing unit P1 implements a possible optional equipment function.

[0059] Fig. 15 or Fig. 16 show variations of the realizations in Fig. 13 or Fig. 14, in which the environmental sensor 1 is used accordingly Fig. 8 or Fig. 9 selected data is generated at a data rate of 2r / N and sent to P1 or P2, from where the unchanged data stream can be forwarded to P2 or P1 respectively.

[0060] Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for example, the scope of protection, the possible applications, or the configuration of the invention.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Claims

[1] Method for operating an environment sensor (1) of a vehicle, wherein exposures of a number h_a of physical lines comprising pixels of a pixel matrix of the environment sensor (1) are started at least partially sequentially in time (S1), and wherein physical exposure results of the h_a physical lines are read out (S2) and taken into account in data processing together with virtual exposure results of a number h-h_a of virtual lines forming a vertical blanking interval, characterized by, that of a total number h processing lines with h≥h_a, comprising the h_a physical and the h-h_a virtual lines, exactly a number N processing lines are exposed simultaneously at a multitude of different times, of which a number N_a of physical lines are exposed simultaneously, wherein such processing lines exposed simultaneously at a time are separated by a number d-1 of unexposed processing lines, wherein for a minimum number N_a_min of simultaneously exposed physical lines N_a_min = floor(h_a / d), where floor() is the rounding-down function that assigns to each real number the nearest non-larger integer, and wherein for a maximum number N_a_max of simultaneously exposed physical lines N_a_max = ceil(h_a / d), with N_a_min ≤ N_a ≤ N_a_max, where ceil() is the rounding-up function that assigns to each real number the nearest non-smaller integer. [2] Method according to claim 1, wherein exactly a number N of processing lines are exposed at any given time. [3] Method according to one of the preceding claims, wherein processing lines with the following processing line indices from an ascending line numbering from 0 to h-1 processing lines are exposed simultaneously in phase, where k is an integer and denotes a time step, where mod is the modulo operator: k mod h, (k−1*d) mod h,(k−2*d) mod h,(k−3*d) mod h,... (k−g*d) mod h, with g=N−1. [4] Method according to one of the preceding claims, wherein the exposure results read out by the N processing lines are output via a single communication channel in a time multiplex, wherein a FIFO memory is used from which the exposure results are output sequentially and therefore not simultaneously. [5] Method according to one of claims 1 to 4, wherein each start of the exposure of the N processing lines is cascaded in time with a respective time offset in pixel-wise time multiplexing with a processing line-to-processing line time offset of T_h / N / M, where T_h is a predetermined processing line period duration and M is the number of pixels per processing line. [6] Method according to any one of claims 1 to 4, wherein the exposure of each of the N processing lines begins with a time offset and is cascaded with a processing line-to-processing line time offset of T_h / N, where T_h is a predetermined processing line period duration. [7] Method according to one of the preceding claims, wherein in a communication channel only the exposure result of a processing line with a processing line index (k - i*d) mod h is output, wherein the exposure result is captured at the respective time T_h * k + T_exp, wherein i is a fixed value from the set {0, 1, 2, .., N-1}, mod is the modulo operator, T_h is a predetermined processing line period duration, and T_exp is an exposure duration of a respective processing line. [8] Method according to one of the preceding claims, wherein only exposure results of exposures from processing lines are read and output which are located within a predetermined vertical area of ​​the pixel matrix of the environmental sensor (1). [9] Method according to one of the preceding claims, wherein two video data streams, each comprising an exposure result, are output in time-division multiplex from the environment sensor (1) via a communication channel, wherein in a first of the video data streams, an exposure result is output for each processing row of the pixel matrix immediately following the end of the exposure, and for a second of the video data streams, an exposure result is temporarily stored in a line buffer and subsequently transmitted for a respective duration of an exposure result that is transmitted in the first of the video data streams. [10] System for sensing the environment of a vehicle, comprising an environment sensor (1) for a vehicle, which is configured to initiate exposures of a number h_a of physical lines comprising pixels of a pixel matrix of the environment sensor (1) at least partially sequentially in time, and wherein the system is configured to read out physical exposure results of the h_a physical lines and to take them into account in data processing together with virtual exposure results of a number h-h_a of virtual lines forming a vertical blanking interval, characterized by, that the environmental sensor (1) is configured to expose exactly a number N of processing lines simultaneously from a total number h of processing lines with h≥h_a, comprising the h_a physical and the h-h_a virtual lines, at a multitude of different times, of which a number N_a of physical lines are exposed simultaneously, wherein such processing lines that are exposed simultaneously at a time are separated by a number d-1 of unexposed processing lines, wherein for a minimum number N_a_min of simultaneously exposed physical lines N_a_min = floor(h_a / d), where floor() is the rounding function that assigns to each real number the nearest non-larger integer, and wherein for a maximum number N_a_max of simultaneously exposed physical lines N_a_max = ceil(h_a / d), with N_a_min ≤ N_a ≤ N_a_max, where ceil() is the rounding function,which assigns to each real number the nearest non-smaller integer. [11] System according to claim 10, comprising a first processing unit (P1) configured to perform a first data processing method for the operation of a vehicle, and comprising a second processing unit (P2) configured to perform a second data processing method for the operation of the vehicle, and comprising a first decimation unit (D1) configured to discard a first part of the video data stream output by the environment sensor (1) for the first processing unit (P1), and comprising a second decimation unit (D2) configured to discard a second part, different from the first part, of the video data stream output by the environment sensor (1) for the second processing unit (P2).

Images