Method for creating at least one encoding rule for encoding an image captured by an optical sensor, method for encoding an image captured by an optical sensor, and information processing unit - Patents.com

By creating encoding rules based on the frequency distribution of light signal values, the method optimizes memory usage and detail preservation in image encoding, addressing the inefficiencies of uniform encoding methods.

JP7588646B2Active Publication Date: 2024-11-22ROBERT BOSCH GMBH
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2022537019
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-12-17
Filing Date
2020-11-13
Publication Date
2024-11-22
Estimated Expiration
2040-11-13

AI Technical Summary

Technical Problem

Image processing systems face challenges in storing or processing images with varying detail levels efficiently, often requiring high resolutions that consume excessive memory due to the use of uniform encoding methods.

Method used

A method for creating encoding rules based on the frequency distribution of light signal values in images, assigning codewords to optimize memory efficiency by adapting to the detail requirements of different image regions, using different codeword lengths and distributions to encode images efficiently.

Benefits of technology

This approach allows for memory-efficient encoding of images by varying the resolution depth based on the image content, minimizing memory usage while maintaining high detail sharpness, especially in areas with many objects, and enabling efficient storage and transmission of images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007588646000001
    Figure 0007588646000001
  • Figure 0007588646000002
    Figure 0007588646000002
  • Figure 0007588646000003
    Figure 0007588646000003
Patent Text Reader

Abstract

The present invention relates to a method (600) for creating at least one encoding rule (105) for encoding an image (115) captured by an optical sensor (110), the method comprising at least the steps of: reading (610) the image (115) captured by the optical sensor (110) and creating (620) a frequency distribution (122) of occurrences of optical signal values ​​(123) at various pixels (118) in the image (115). The method (600) further comprises the step of assigning (630) code words (135) to the optical signal values ​​(123) using the frequency distribution (122) to create at least one encoding rule (105) for encoding the image (115) captured by the optical sensor (110).
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The invention takes as its starting point methods and information processing units according to the generic concepts of the independent claims. Computer programs are also the subject of the invention. [Background technology]

[0002] In image processing systems, the problem is often to store or process images that display a wide variety of surroundings with as little memory cost as possible, and still reproduce as many details as possible from this image with sufficient accuracy.For this reason, very high image resolutions are often used, but this very high resolution again requires huge memory requirements for storing or encoding this image, even if individual areas of this image do not contain much detail to be displayed and therefore a more coarse encoding method is sufficient.

[0003] On this basis, the approach presented herein provides a method for generating at least one coding rule for coding an image captured by an optical sensor, a method for coding an image captured by an optical sensor, as well as an information processing unit using at least one of these methods and finally a corresponding computer program according to the independent claims. Advantageous variants and improvements of the device presented in the independent claims are possible thanks to the measures recited in the dependent claims. Summary of the Invention [Means for solving the problem]

[0004] According to the approach proposed herein, a method for creating at least one coding rule for coding an image captured by an optical sensor is provided, comprising at least - reading an image captured by an optical sensor; - creating a frequency distribution of occurrences of optical signal values ​​at different pixels in the image; - assigning code words to the optical signal values ​​using the frequency distribution to create at least one coding rule for coding of an image captured by an optical sensor.

[0005] The optical sensor may be, for example, a camera or the like. A picture element in an image may be, for example, understood as a pixel, in which the brightness or another light parameter acting at the pixel's location is captured and can be stored or transferred in the form of a light signal value. The light signal value may be, for example, brightness, light intensity in a particular spectral region (color), polarization, extended spectral channels, or the like. A code word may be a symbol or a combination of symbols, by means of which the light signal value can be stored, for example, in an ordinal system of code words. A coding rule may be an assignment rule that assigns a light signal value or a light signal value range to the corresponding code word, respectively, and vice versa.

[0006] The finding underlying the approach presented herein is that a very efficient possibility for the coding of an image or at least one fragment of an image is achieved by assigning various light signal values ​​to code words, taking into account the frequency distribution of occurrence of light signal values ​​on this image (or image fragment). This achieves a coding of image signal values ​​provided by pixels or picture elements, which allows for as good a consideration as possible of further objects displayed in the image or image fragment. It is then exploited that these further objects also feature an increase in light intensity in the corresponding pixels in the image or image fragment, and thus the determination of the frequency distribution of brightness or other light parameters in the individual pixels of the image also provides information about how many objects or different or distinguishable objects are to be coded in the image. This allows a very memory-efficient coding of the image by subsequent application of the coding rules created, which also allow a high degree of detail in terms of the resolution of the individual objects in the image and, accordingly, the selection of the appropriate format of the code words to be used. This allows, for example, that images or image fragments in which only a few objects or details need to be distinguished can be coded using coding rules which only result in a small codeword space, and conversely, images or image fragments in which many objects or details have to be distinguished can be coded using coding rules which result in a larger codeword space.The use of the frequency distribution of the light signal values ​​thus opens up the possibility of using coding rules which are respectively adapted to the image to be coded, thereby enabling the most memory-efficient coding possible of the images or image fragments.

[0007] Furthermore, in a preferred embodiment of the approach presented herein, in the assigning step, code words are assigned to the light signal values ​​such that in ranges of light signal values ​​with a high frequency in the image, adjacent code words are assigned light signal values ​​with a small difference between the light signal values ​​to be assigned to the code words. Additionally or alternatively, in ranges of light signal values ​​with a low frequency in the image, adjacent code words may be assigned light signal values ​​with a large difference between the light signal values ​​to be assigned to the code words. Such an embodiment of the approach presented herein provides the advantage that for areas in the image with very similar brightness or light parameters, code words with a very fine distinction between the brightness or light parameters are used, and conversely, areas in the image with very large brightness differences are coded by code words with a less fine distinction between the light signal values. In this regard, the distinguishability of the code words may arise based on the light signal values ​​assigned to the respective code words depending on the coding rules. This allows a very efficient coding of the image content, which allows a high sharpness of detail, especially for areas with many different objects, and thus allows the coding of the image with a varying resolution depth depending on the requirements.

[0008] In a particularly preferred embodiment of the approach presented herein, in the reading step, an image representing one fragment of the overall image captured by the optical sensor is read. In this case, at least one further fragment of the overall image captured by the optical sensor cannot be read and used for creating the frequency distribution. Such an embodiment provides the advantage that different coding rules can be created for individual partial fragments of the overall image, thereby allowing an improvement in the efficiency of the coding or storage of the overall image, for example when different coding rules can be used for different fragments of the overall image depending on the requirements.

[0009] Furthermore, an embodiment of the approach proposed herein is conceivable, in which in a reading step, a further image taken by an optical sensor is read, in a creating step, a further frequency distribution of occurrences of light signal values ​​in various different further pixels in this further image is created, and in an assigning step, this further frequency distribution is used to assign further code words to the light signal values ​​in order to create a further encoding rule. The further image may for example relate to a region of the overall image that is radially symmetrically further outside the optical center of the overall image. If instead the overall image is divided into tiles, the further image may also correspond to a further tile from this overall image. Such an embodiment of the approach presented herein provides the advantage of creating different encoding rules for different images / partial images, and thus making it possible to select one encoding rule from a plurality of encoding rules for subsequent use, respectively, for the most suitable selection in the case of the use in question, whereby a coding of the image as memory-efficient as possible may be achieved.

[0010] In a particularly preferred embodiment of the approach proposed herein, further codewords are used in the allocating step, which have a larger codeword length than the codewords of the encoding rule, so that different requirements regarding the level of detail can be taken into account particularly well when encoding images with many objects.

[0011] Furthermore, according to another embodiment of the approach presented herein, in the creating step a frequency distribution can be created for the optical signal values ​​of a predefined spectral sub-region of optical visible light and / or for the optical signal values ​​of the spectral regions detectable by the sensor (e.g. the ultraviolet region and / or the near infrared region (NIR) and / or the short wave infrared region (SWIR) or the far range infrared region (FIR)) and / or for the optical signal values ​​of the optical parameters detectable by the sensor, and / or in the assigning step a binary sequence can be used as the code word, in particular as the code word having the same predefined bit length. In this regard, in the case of a particular spectral sub-region of optical visible light representing a specific color, for example red, green, blue, etc., it can be exploited that there is a particularly large amount of information about the details of the image, which can then also be used for the determination of the code word or the encoding rule. The use of binary sequences as code words in the encoding rule, in particular the use of code words of the same length, offers the advantage of simplifying the use of this encoding rule for encoding the image, in terms of numerical or circuit technology.

[0012] Furthermore, an embodiment of the approach proposed herein is conceivable, in which in the allocating step, the lowest valued codeword is assigned an optical signal value that is within a tolerance range around the lowest optical signal value in the frequency distribution. Additionally or alternatively, in the allocating step, the highest valued codeword may be assigned an optical signal value that is within a tolerance range around the highest optical signal value in the frequency distribution. Such a tolerance range may, for example, include a range of 10-20 percent around the respective lowest or highest optical signal value from the frequency distribution. Such an embodiment of the approach proposed herein provides the advantage of utilizing the codewords or the codeword space created by the correspondingly assigned optical signal values ​​as optimally as possible.

[0013] A method for coding an image taken by an optical sensor, based on the approach proposed herein, in order to use the dates occurring in the individual pixels, taking advantage of the frequency distribution, as well as possible and in a memory-efficient way, for coding the image, comprising: - reading at least one image and an encoding rule, the encoding rule representing an assignment of a light parameter of a pixel of the image to one of a number of distinguishable code words, according to which at least one first difference between the brightness and / or light parameter assigned to two adjacent code words is different from a second difference between the brightness and / or light parameter assigned to two further adjacent code words; - assigning a respective codeword to at least a respective one of a plurality of pixels of the image in order to encode the image. An image may here be understood as an image sensor signal that represents the overall view of the sensor surroundings captured by the sensor. However, it is also conceivable that the image is given by only a fragment of the overall image, in which case this fragment is coded using an encoding rule. In this respect, the encoding rule may be, for example, an allocation rule that has been determined by applying the frequency distribution shown in the present specification of the brightness or light parameters of the individual pixels in an image.

[0014] In a particularly preferred embodiment, the reading step involves a selection of one of a number of encoding rules, the selection being made in particular using a frequency distribution of occurrences of light signal values ​​at different pixels in the image. Such an embodiment offers the advantage that for an image or image fragment a respectively suitable encoding rule can be selected for a memory-efficient encoding of the image / image fragment, whereby in the allocation step the information which of the encoding rules was used for encoding the image / image fragment is preferably stored in order to allow as fast and unambiguous a decoding of the image / image fragment as possible.

[0015] Furthermore, in a particularly advantageous embodiment of the approach proposed herein, in the reading step at least one further image and a further encoding rule different from the one image are read, the encoding rule being different from the further encoding rule, and in the allocating step a respective one further code word of the further encoding rule is allocated to at least one respective one of the pixels of the further image in order to code the image. Such an embodiment provides the advantage that different encoding rules can be used for different images or fragments or regions of the overall image, whereby depending on the actual situation in the image or fragment or region of the overall image a respectively suitable encoding rule for memory-efficient coding can be used.

[0016] In order to avoid optically broken edges occurring between individual fragments of the overall image as much as possible during the decoding of the overall image, according to a further embodiment of the approach presented herein, in the assigning step, pixels in the edge region of the image can be assigned auxiliary values ​​determined using the code word and the further code word. Alternatively or additionally, according to this embodiment, in the assigning step, further pixels in the edge region of the further image can be assigned further auxiliary values ​​determined using the code word and the further code word. In this regard, in particular, when determining the auxiliary value and / or the further auxiliary value, an interpolation can be performed using at least one code word and the further code word. Such an interpolation can be performed, for example, in such a way that for pixels located further inside the image, a higher weight of the code word for the coding of the image is adopted, which is also higher than the weight of the code word for the coding of the further image that borders the image.

[0017] Alternatively or additionally, according to a further embodiment, in the reading step, an image and a further image that at least partially overlap can be read in, so that a decoding of the image and the further image or of a partial fragment of the overall image can be performed, whereby optical artifacts in the decoded image or the overall image can be kept as low as possible or completely avoided.

[0018] Embodiments of the method may for example be implemented in software or hardware or in a mixed form consisting of software and hardware, for example in an information processing unit.

[0019] The approach presented herein further provides an information processing unit configured to implement, control or execute the steps of one of the variations of the method presented herein in a corresponding mechanism. This implementation variation of the invention in the form of an information processing unit can also quickly and efficiently solve the problem underlying the invention.

[0020] For this purpose, the information processing unit may have at least one calculation unit for processing signals or data, at least one memory unit for storing signals or data, at least one interface to sensors or actuators for reading sensor signals from sensors or for outputting data or control signals to actuators, and / or at least one communication interface for reading or outputting data, which is integrated into a communication protocol. The calculation unit may be, for example, a signal processor, a microcontroller or the like, whereby the memory unit may be a flash memory, an EEPROM or a magnetic memory unit. The communication interface may be configured for wireless and / or wired reading or outputting of data, in this regard, a communication interface which may read or output wired data may read or output these data, for example electrically or optically, from or to a corresponding data transmission line.

[0021] In the present application, an information processing unit may be an electrical device that processes the sensor signals and outputs control and / or data signals accordingly. The information processing unit may have an interface that may be implemented by hardware and / or software. In the case of a hardware implementation, the interface may for example be part of a so-called system ASIC, which contains the very different functions of the information processing unit. However, it is also possible that the interface is a dedicated integrated circuit or at least partially consists of separate components. In the case of a software implementation, the interface may for example be a software module that is present together with other software modules on a microcontroller.

[0022] A computer program product or computer program which can be stored on a machine-readable medium or memory medium, such as a semiconductor memory, hard disk memory or optical memory, and which has a program code used to implement, execute and / or control the steps of a method according to one of the above-mentioned embodiments, is also advantageous, particularly when this program product or program is executed on a computer or device. Exemplary embodiments of the approaches presented herein are illustrated in the drawings and are explained in more detail in the following description. [Brief description of the drawings]

[0023] [Figure 1] FIG. 2 is a schematic block diagram of an information processing unit for creating coding rules for the coding of images captured by an optical sensor; [Diagram 2] 2 is a diagram showing an overall image, for example, output as an image from a camera as an optical sensor based on the display of FIG. 1. FIG. [Figure 3A] The left subfigure shows the overall image and each considered overall image fragment as an image for determining the frequency distribution, and the right subfigure shows a graph of the frequency distribution and the correspondingly assigned code words obtained from this overall image fragment. [Figure 3B] The left subfigure shows the overall image and each considered overall image fragment as an image for determining the frequency distribution, and the right subfigure shows a graph of the frequency distribution and the correspondingly assigned code words obtained from this overall image fragment. [Figure 3C] The left subfigure shows the overall image and each considered overall image fragment as an image for determining the frequency distribution, and the right subfigure shows a graph of the frequency distribution and the correspondingly assigned code words obtained from this overall image fragment. [Figure 3D]The left subfigure shows the overall image and each considered overall image fragment as an image for determining the frequency distribution, and the right subfigure shows a graph of the frequency distribution and the correspondingly assigned code words obtained from this overall image fragment. [Figure 4] FIG. 2 is a schematic block diagram of an information processing unit for encoding an image captured by an optical sensor; [Diagram 5] 1 is a schematic diagram of a fragment or image and a further fragment or image in preparation for encoding of an image captured by an optical sensor; [Figure 6] 2 is a flow chart of one exemplary embodiment of a method for creating at least one encoding rule for encoding an image captured by an optical sensor; [Figure 7] 1 is a flow chart of one exemplary embodiment of a method for encoding an image captured by an optical sensor. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] In the following description of preferred exemplary embodiments of the present invention, the same or similar reference numerals are used for similarly acting elements shown in different figures, without repeating the description of these elements.

[0025] 1 shows a schematic block diagram of an information processing unit 100 for creating an encoding rule 105 for encoding an image 115 captured by an optical sensor 110. The information processing unit 100 firstly comprises an interface 117 for reading an image 115 captured by the optical sensor 110, which is formed for example as a camera. This captured image 115 is represented for example as a two-dimensional array of a number of picture elements 118 (also called pixels), which are arranged for example by rows and columns. In this case, these pixels 118 represent objects 119 within the surroundings captured by the optical sensor 110, which display a certain brightness, possibly also a certain predefined color, and at their respective positions represent the objects 119 in the image 115.

[0026] The information processing unit 100 further comprises a creating unit 120, in which a frequency distribution 122 is created, for which the divisions of brightness or light parameters or light signal values ​​123 occurring in pixels 118, for example at the abscissa of a graph for displaying the frequency distribution 122, are represented by the number 124 of brightness / light signal values ​​123 occurring in pixels 118 of the image 115, for example at the ordinate 124 of the graph for displaying the frequency distribution 122. Thus, by means of the created frequency distribution 122, it can be recognized which light signal values ​​123 and in what number 124 occur in each pixel 118 of the image 115 throughout the entire range. Based on the frequency distribution 120 it can thus be very easily recognized which light signal values ​​123 should be coded with a very fine gradation for a detailed and accurate display or coding of the image, and for which light signal values ​​123 a coarser gradation is sufficient, without suffering too high an information loss, which would make it impossible to recognize the object 119 with the accuracy required for a subsequent automatic evaluation, for example for highly autonomous driving of a vehicle.

[0027] The creating unit 120 may also be understood as in which the data content of all the optical signal values ​​is analyzed, and therefore the creating unit 120 may be designated as an analyzing unit.

[0028] The information processing unit 100 further comprises an allocation unit 130 formed for allocating code words 135 to the optical signal values ​​123. In this regard, the code words 135 can be symbols or combinations of symbols resulting in a code word space in which the optical signal values ​​123 can be encoded. For example, the code words 135 can be binary strings, for example, 4, 6, 10, 12, 16, 20 or 24 bits long. In this regard, it is preferred that code words 135 of the same length are used for encoding the optical signal values ​​123 in the image 115, so that for example all code words 135 have a length of 4 bits. In this case, the allocation rule of the code words 135 to the optical signal values ​​123 is the encoding rule 105. A detailed description of the procedure of the allocation of the code words 135 to the optical signal values ​​123 and vice versa is explained in more detail below.

[0029] Here, the above-described procedure can be repeated at least once more, but preferably several times, so that various scenes of the arrangement of objects 119 within the periphery of the optical sensor 110 can be represented as precisely as possible. In this regard, at least one further image 115' different from the image 115 can be read in from the optical sensor 110. This further image 115' can also be formed by a corresponding arrangement of further pixels 118', from which a further frequency distribution 122' can be determined, in which further code words 135' are assigned to the light signal values ​​123 in accordance with a further encoding rule 105'.

[0030] Furthermore, the above-described process does not have to be limited to scanning performed one after the other in time, but can also be achieved by reading out spatially separated sensor elements, as is the case for example with a Split-Pixel configuration or a multi-channel sensor or even with a stacked detector (as in FOVEON).

[0031] It is also conceivable that the image 115 (or the further image 115') is merely a fragment of an overall image in which the arrangement of the objects 119 within the periphery of the optical sensor 110 is reproduced. In this respect, for example, the image 115 and the further image 115' may be taken simultaneously, but display different sub-ranges of this overall image. This makes it possible to create corresponding coding rules 105 (or 105') for the various sub-ranges of the overall image, so that, depending on the required detail accuracy of the representation or encoding, the information loss caused by this encoding can be as small as possible.

[0032] It is also conceivable that the information processing unit 100 is placed in a laboratory environment and that images 115 or 115', which have been captured prior to this time by the optical sensor 110 while driving in a real environment and which have only been temporarily stored, are read in via the interface 117. This means that the creation of the coding rule 105 does not have to take place in real time immediately after the generation of the images 115.

[0033] From now on, in order to explain the procedure in the assignment of code words 135 to optical signal values ​​123 in more detail, the print makes reference to a number of exemplary images 115 or fragments of images 115 in the following figures, in which respect it is immaterial whether the image 115 used for the frequency distribution 122 to be determined is only a fragment of the entire image or the entire image itself. Therefore, for reasons of brevity, the procedure is explained in more detail on the basis of only one image 115 loaded, which should not be understood as limiting in the following description.

[0034] 2 shows an overall image 200, as output by a camera as the optical sensor 110 based on the representation of FIG. 1, for example as image 115. In this regard, a number of objects 119 can be recognized, for example vehicles, doors in the walls of a tunnel, lamps in the tunnel, road markings in the tunnel, and the entrance to the tunnel. Now, for example, dividing the overall image 200 shown in FIG. 2 into three rows and three columns results in nine fragments of this image, which are subsequently taken into account, respectively, as images 115, for determining separate frequency distributions 122 and used as a basis for determining the encoding rules 105.

[0035] In Figures 3A, 3B, 3C and 3D, the left sub-figures show the entire image 200 and the respective fragments considered as images 115 for determining the frequency distribution 122, while the right sub-figures display a graph of the frequency distribution 122 and the correspondingly assigned code word 135 (here 4 bits in length, resulting in a code word space of 16 ordered code words) obtained from the image 115.

[0036] In this respect, in Fig. 3A, a fragmentary image 115 from the bottom row of the middle column of the overall image 200 is used, which fragmentary image 115 displays a large portion of a wet and therefore reflective roadway in a tunnel in the overall image 200. As a corresponding frequency distribution 122, a very homogeneous range of the light parameter brightness occurs in the middle part of the graph in the right subfigure, which range contains few particularly high or particularly low light parameters of brightness. In other words, the light parameters, for example brightness, in this image 115 are very close together, and therefore, due to the high resolution of details within this range of the frequent light signal values ​​123 in the image 115, adjacent code words 135 are assigned light signal values ​​123 with small differences between the light signal values ​​123 to be assigned to the code words 135. This results in a very steep trajectory of the curve graph 300 in the right subfigure of Fig. 3A when the code words 135 are connected to each other in the ascending direction by connecting lines. In this regard, it can also be seen that the allocation of these code words 135 is such that for the lowest code word 135, one optical signal value 132 is assigned that is around (e.g. within a tolerance range) the lowest optical signal value from the frequency distribution. The lowest code word may be a code word that has no further preceding code words 135 in the ordered sequence. Furthermore, for the highest code word 123, one optical signal value 123 is assigned that is around (e.g. within a tolerance range) the highest optical signal value 123 from the frequency distribution 122. The highest code word may be a code word that has no further subsequent code words 135 in the ordered sequence. Thereby, the allocation of code words 135 can very efficiently code the dynamic range of occurrence of the optical signal values ​​123, for example for very advantageous recovery in a later processing stage.

[0037] In this regard, in Fig. 3B, the fragment image 115 in the top row of the right column of the overall image 200 is used, which fragment image 115 shows a large part of the ceiling plate and lamps above the roadway in the tunnel in the overall image 200. As a corresponding frequency distribution 122, in the middle and right parts of the graph in the right partial view of Fig. 3B, the presence of brightly shining lamps results in a very heterogeneous range of the brightness 123 as a light parameter, which range includes not only the middle part but also a particularly high brightness 123. In other words, the brightness as a light parameter in this image 115 is partially very far from each other, and therefore, due to the high resolution of details in the first middle (and also right) range of the frequent light signal values ​​123 in the image 115, adjacent code words 135 are assigned light signal values ​​123 with a small difference between the light signal values ​​123 to be assigned to the code words 135. Between these two ranges of frequently occurring optical signal values ​​123, there is a further range of optical signal values ​​123 that hardly ever occur in the image 115. No fine distinction is required between these optical signal values ​​123, so generally speaking, in the range of optical signal values ​​123 that are less frequent (relative to the other optical signal values ​​123) in the image 115, adjacent code words 135 are assigned optical signal values ​​123 with a large difference between the optical signal values ​​123 to be assigned to the code words 135. This results in a very steep trajectory of the curve graph 300 in the range of accumulated optical signal values ​​123, if the code words 135 are connected to each other in the ascending direction in the right sub-diagram of Fig. 3B. In this case, in the range between both ranges where the optical signal values ​​123 frequently occur, this curve graph 300 has a clearly flatter slope. It can also be seen again that the assignment of these code words 135 assigns the lowest valued code word 135 an optical signal value 123 that is around (e.g., within a tolerance range) the lowest optical signal value 123 from the frequency distribution 122. Furthermore, the highest valued code word 123 is also assigned an optical signal value 123 that is around (e.g., within a tolerance range) the highest optical signal value 123 from the frequency distribution 122.

[0038] In this regard, in Fig. 3C, the image 115 as a fragment in the middle row of the right column of the overall image 200 is used, which image 115 as a fragment shows a number of vehicles and doors as objects 119 in the overall image 200. In the corresponding frequency distribution 122, in the middle and right parts of the graph in the right sub-view of Fig. 3C, the brightly lit tunnel walls again result in a partially inhomogeneous range of the brightness 123 as the light parameter, which is not as strongly manifested here as in Fig. 3B with the actively emitting lamps. The curve graph 300 is therefore also not as steep as the curve graph of Fig. 3A, but at least steeper than the middle part of the curve graph 300 of Fig. 3B.

[0039] In this regard, in Fig. 3D, a fragment image 115 from the bottom row of the right column of the overall image 200 is used, which fragment image 115 displays a number of reflective road markings as objects 119 in the otherwise dark roadway in the overall image 200. As a corresponding frequency distribution 122, in the middle part of the graph in the right sub-view of Fig. 3D, the brightly reflective road markings again result in a fairly homogeneous range of brightness 123 as the light parameter, which here is located entirely at lower light signal values ​​123 compared to the frequency 124 of the light signal values ​​123 of Fig. 3A. The curve graph 300 is therefore again quite steep like the curve graph of Fig. 3A, but starting and ending at the respective lower light signal values ​​123.

[0040] Now, in order to be able to carry out the most memory-efficient encoding of the image 115, the determined encoding rules 105 can advantageously be used from now on, as will be explained in more detail below on the basis of a corresponding information processing unit 400 for encoding the image 115 captured by the optical sensor 110.

[0041] 4 shows a schematic block diagram of an information processing unit 400 for encoding an image 115 captured by an optical sensor 110. The information processing unit 100 firstly comprises an interface 117 for reading at least one fragment of an image 115 from the optical sensor 110, which is again formed for example as a camera. This read image 115 is for example again represented as a two-dimensional array of a number of picture elements 118 (also called pixels), which are arranged for example by rows and columns. In this case, these pixels 118 represent objects 119 within the surroundings captured by the optical sensor 110, which display a certain brightness, possibly also a certain predefined color, and at their respective positions represent the objects 119 in the image 115.

[0042] Furthermore, the coding rule 105 and possibly at least one further coding rule 105' can be loaded via the interface 117, for example from a memory 405, which is here arranged outside the information processing unit 400 (but can also be arranged in the information processing unit 400). This coding rule 105 or the further coding rule 105' is, for example, an assignment rule as determined in accordance with the above description. The coding rule can, for example, represent an assignment of a brightness or light parameter of a pixel of an image fragment to one of a number of distinguishable code words, according to which at least one first difference between the brightness 123 as a light parameter assigned to two adjacent code words 135 differs from a second difference between the brightness 123 as a light parameter assigned to two further adjacent code words 135. In particular, an coding rule 105 can thereby be loaded which represents a non-uniform assignment of light signal values ​​to the code words 135, as obtained by taking into account the frequency distribution of occurrences of brightness as various light parameters in the image 115. Afterwards, in the allocation unit 130, for example, a respective code word is assigned to at least one of the pixels of the image fragment in order to code the image 115 and obtain therefrom an encoded image 420. This encoded image 140 can then be further processed, for example in an image processing unit not shown in Fig. 4, for example in a driver assistance system of the vehicle, stored or transmitted, for example to a central computing unit or other road user for further processing.

[0043] Furthermore, an analysis of the frequency of occurrence of brightness as a light parameter in the image 115 can be performed, for example already in the interface 117, and depending on the analysis results obtained, one particular encoding rule from among the encoding rules 105 can be selected. Such a possibility offers the advantage of selecting the encoding rule 105 to be used already before the actual encoding or assignment of code words to the pixels 118, which encoding rule 105 is based on a very similar frequency distribution also present in the image 115 to be encoded at the time. This allows a memory-efficient encoding of the image 115 to be achieved very quickly and at low numerical or circuit-technical costs, by downloading or loading the encoding rule 105 that is best suited for encoding the image 115 at the time.

[0044] As already detailed above when determining the encoding rules 105 or 105', the loaded fragments of the image 115 or 115' may be (at least partially different from one another) sub-sections of the overall image 200, which are captured at the same time but which nevertheless represent different regions of the overall image 200. That is to say, also when encoding the image 115, individual fragments of the image / overall image 200 (here referred to as image 115 or further image 115') may be encoded by different encoding rules 105 or 105' (i.e. one encoding rule 105 and at least one further encoding rule 105') corresponding to the representation of the sub-views 3A, 3B, 3C and 3D of FIG. 3. This allows to fully take into account different realities or different required sharpness of details in the different images 115 or 115'.

[0045] Now, if an overall image 200 corresponding to the representation in Fig. 2 is divided and coded into a number of (partial) images 115 or 115', decoding of the image 420 coded in this way section by section may result in image artifacts, for example edges at the joints of the sections. Such image artifacts should be avoided as far as possible, especially if an automatic further image processing for recognizing objects or situations is to be performed so that the scene in front of the optical sensor 100 can be automatically evaluated. In order to avoid such image artifacts or edges, a further optimized form of coding of the image 115 may then be used, as will be explained in more detail below.

[0046] Fig. 5 shows a schematic diagram of a fragment or image 115 and a further fragment or further image 115', which are in adjacent sections of the overall image 200 as shown in Fig. 2. If now a pixel 118 of the image 115 is coded according to the coding rule 105 and a further pixel 118' of the further image 115' is coded according to the further coding rule 105', these image artifacts may arise, for example, in a boundary area 500 where the image 115 borders the further image 115'. To avoid this, for example an auxiliary value 510 can be read for the pixel 118 of the image 115, which auxiliary value 510 is determined using the code word 135 of the coding rule 105 assigned to said pixel 118 and the further code word 135 assigned according to the further coding rule 105' to the further pixel 180' of the further image 115'. The determination of the auxiliary value 510 can, for example, be performed using an interpretation in which the closer the pixel 118 is to the boundary line 500, the higher the weight of the further code word 135'. The determination of the further auxiliary value 520 assigned to the further pixel 118' of the further image 115' can also, for example, be performed using an interpretation in which the closer the further pixel 118' is to the boundary line 500, the higher the weight of the code word 135. In this way, a transition as gradual or continuous as possible of the code words 135, 135' or the auxiliary values ​​510, 520 assigned to the respective pixels 118, 118' can be achieved, whereby corresponding image artifacts can possibly be avoided.

[0047] Alternatively or additionally, the image 115 or the further image 115' may be selected or determined such that these images at least partially overlap (as this is indicated by the dashed representation of the further image 115' in Fig. 5), so that for example for one and the same pixel 118 or 118' one code word 135 on the one hand and one further code word 135' on the other hand are determined. In this case, for example, a corresponding auxiliary value 510 may be determined using the code word 135 and the further code word 135', for example by signaling.

[0048] 6 shows a flow chart of one exemplary embodiment of a method 600 for creating at least one encoding rule for encoding an image captured by an optical sensor, the method 600 having at least a step 610 of reading an image captured by an optical sensor. The method 800 further includes a step 620 of creating a frequency distribution of occurrences of light signal values ​​at different pixels in the image. The method 600 finally includes a step 630 of assigning code words to the light signal values ​​using the frequency distribution to create at least one encoding rule for encoding an image captured by the optical sensor.

[0049] 7 shows a flow chart of one exemplary embodiment of a method 700 for encoding an image captured by an optical sensor. The method 700 comprises a step 710 of reading at least one fragment of an image and an encoding rule, which represents an assignment of one brightness or light parameter of one pixel of the fragment of the image to one of a plurality of distinguishable code words, according to which at least one first difference between the brightness or light parameter assigned to two adjacent code words is different from a second difference between the brightness or light parameter assigned to two further adjacent code words. The method 800 further comprises a step 720 of assigning a respective one code word to at least one respective pixel of the plurality of pixels of the fragment of the image in order to encode the image.

[0050] In summary, it can be seen that the approach presented herein, both by the disclosed method for encoding the images taken by the optical sensor and for creating at least one encoding rule for encoding the images taken by the optical sensor, opens up the possibility of reducing the required data depth for the transmission, processing and storage of the sensor data from the optical sensor, by means of a non-lossless information reduction. In this regard, particular attention may be paid to the various aspects presented in particular below.

[0051] 1.) Information reduction is chosen such that the information loss is variably adjustable in space and time.

[0052] A spatially variable adjustment in this context may mean that the information compression function may be performed according to a "region of interest" on a two-dimensional array (e.g. in the case of an image sensor) or within a three-dimensional volume element (in the case of a 3D sensor) as the image 115 captured by the sensor. Special forms of such selection of images that are particularly useful for encoding include, for example, a.An image or image fragment that is radially symmetrically distributed about its optical center; b. Images or image fragments that span separate predefined tiles, e.g. top left, top middle, top right, ... c. A relatively widely spread image or image fragment with defined coordinates of the information present in the rasterized form, e.g. the target coordinates x, y, in which function applications can be performed within an arbitrarily defined variable range around the target coordinates. For practical reasons, the fragment or image 115 is usually selected symmetrically around the target coordinates.

[0053] In this context, time-variable adjustment means that the criteria for compression are not fixed at system start-up, but a) to react to fast changing conditions where it is desirable to change the compression on a frame-by-frame basis (e.g. entering a tunnel), b) To allow adaptation to slow parameter drifts (e.g. day / night transitions, temperature drift, dirt, or aging drift), It may be that it is sometimes changed.

[0054] 2) The information reduction is chosen so that the impact on subsequent processes is harmless or minimal (i.e., below a certain threshold).

[0055] Advantageously, under given detection conditions, the compression is selected such that the best possible use is made of the available hardware resources (which may also be dynamically allocated) (see item 1.2).

[0056] Here, it may be mentioned by way of example that this is also the case in the human visual system. 3.) Information reduction a) Based on statistics for local regions or for the entire image. In an example, this is luminance information, further parameters being chrominance or even depth information (e.g. derived from stereo disparity or a time of flight (TOF) signal). b) Based on the desired target bit depth of the data.

[0057] 4.) The information reduction in the various tiles by linear overlap functions is modified / adapted in such a way that no discontinuous jumps in important parameters (luminance, contrast, color, etc.) appear at the edges of the selected tiles, which is advantageous for the avoidance of block artifacts.

[0058] 5.) Already before the transmission to the downstream system takes place, and in order to keep the bandwidth requirements of this system small and reduce the energy demands of the entire system, the entire processing chain is performed "in situ", for example on the image sensor or on an information processing unit integrated therein.

[0059] If an exemplary embodiment includes an "and / or" combination between a first feature and a second feature, this should be read as meaning that this exemplary embodiment has both the first feature and the second feature according to one embodiment, and either only the first feature or only the second feature according to a further embodiment.

Claims

1. A method (600) for creating at least one encoding rule (105) for encoding an image (115) captured by an optical sensor (110), comprising: - reading (610) said image (115) taken by said optical sensor (110); - creating (620) a frequency distribution (122) of occurrences of optical signal values ​​(123) at different pixels (118) in said image (115); - assigning (630) code words (135) to optical signal values ​​(123) using said frequency distribution (122) in order to create at least one encoding rule (105) for encoding said image (115) captured by said optical sensor (110), To the lowest valued codeword (135) an optical signal value (123) is assigned that is within a tolerance range around the lowest optical signal value (123) from said frequency distribution (122) and / or to the highest valued codeword (135) an optical signal value (123) is assigned that is within a tolerance range around the highest optical signal value (123) from said frequency distribution (122); the different pixels (118) representing objects (119) within the image (115) captured by the optical sensor (110); The method further comprises a step of performing an automatic evaluation for autonomous driving of a vehicle, in which the object (119) is recognized in an encoded image (420) obtained based on the optical signal values ​​(123) to which the code words (135) are assigned in the assigning step (630), In said reading step (610), a further image (115') captured by said optical sensor (110) is read, in said creating step (620) a further frequency distribution (122') of occurrences of optical signal values ​​(123) in different further pixels (118') in said further image (115') is created, and in said assigning step (630) further code words (135') are assigned to said optical signal values ​​(123) using said further frequency distribution (122') to create a further encoding rule (105'), the encoded image (420) obtained on the basis of the optical signal values ​​(123) to which the code word (135) has been assigned is encoded by assigning to pixels in edge regions of the image (115) auxiliary values ​​determined using the code word (135) and the further code word (135'); The method (600), wherein the auxiliary value is determined by increasing the weight of the further codeword (135') the closer the pixel (118) is to the boundary line of the edge region, and increasing the weight of the codeword (135) the closer the further pixel (118') is to the boundary line.

2. In the assigning step (630), the code words (135) are assigned to optical signal values ​​(123) such that, in a range of optical signal values ​​(123) with a high frequency in the image (115), adjacent code words (135) are assigned optical signal values ​​with a small difference between the optical signal values ​​(123) to be assigned to the code words (135) and / or, in a range of optical signal values ​​(123) with a low frequency in the image (115), adjacent code words (135) are assigned optical signal values ​​with a large difference between the optical signal values ​​(123) to be assigned to the code words (135); 2. The method (600) of claim 1, wherein the codewords (135) are a plurality of codewords (135) that are distinguishable from one another, each of the codewords (135) being adjacent to one another in a codeword space by being ordered.

3. 3. The method (600) of claim 1 or 2, wherein in the reading step (610), an image (115) is read that represents a fragment of the overall image (200) captured by the optical sensor (110).

4. 2. The method (600) of claim 1, wherein in the allocating step (630) a further codeword (135') is used having a codeword length greater than the codeword (135) of the encoding rule (105).

5. 5. A method (600) according to any one of claims 1 to 4, characterized in that in the creating step (620), the frequency distribution (122) is created for optical signal values ​​in a predetermined spectral sub-region of optical visible light and / or optical signal values ​​in a spectral region perceptible by the sensor (110) and / or optical signal values ​​(123) of an optical parameter detectable by the sensor (110).

6. 6. The method (600) of claim 5, wherein in the step of allocating (630), the code words (135) are binary sequences used as code words having the same predetermined bit length.

7. An information processing unit (100, 400) adapted to execute each step of the method (600, 700) according to any one of claims 1 to 6 in a corresponding unit (117, 120, 130; 117, 130).

8. A computer program adapted to carry out the steps of the method (600, 700) according to any one of the claims 1 to 6.

9. A machine-readable memory medium having stored thereon the computer program of claim 8.

Citation Information

Patent Citations

  • Image coding / Decoding method

    JP1995298258A

  • Environment recognition apparatus

    JP2007072987A

  • Encoding processor, method, and computer program

    JP2008113439A

  • Unit, method, and program for processing image

    JP2009044633A

  • Imaging apparatus and control method therefor

    JP2016005103A