Method and system for processing image information using artificial neural networks

Abstract

The invention relates to a method and a system for processing image information with an artificial neural network. For a method for processing image information of a vehicle imaging sensor in an artificial neural network (1), the artificial neural network comprises at least one encoder (2) and at least one decoder (3), wherein the artificial neural network solves a classification task with a plurality of classes and / or solves a regression task providing quantized digital output information according to a plurality of quantization intervals, wherein the artificial neural network (1) outputs a plurality of feature maps at an output interface, wherein a class is assigned to an image region of the image information by the feature map or quantized digital output information relating to the image information is encoded output, in such a way that the individual input contents in the respective matrix region of the feature map jointly generate a code, wherein information compression is achieved by the encoding in such a way that the output quantity of the feature map is smaller than the class quantity or the quantization interval quantity.

Description

Technical Field

[0001] This invention relates to methods and systems for processing image information provided by vehicle imaging sensors using artificial neural networks, and to vehicles having such systems. Background Technology

[0002] The complexity of neural networks and the computational cost in neural networks in relation to input representation Or output representation The computational cost increases proportionally with the increase in dimensionality. In other words, the more image information that needs to be processed, the greater the computational cost. In convolutional networks, so-called convolutional neural networks (CNNs) provide output information in so-called feature maps. The number of feature maps provided significantly affects the computational cost in convolutional networks.

[0003] In classification applications, such as image semantic segmentation, so-called one-hot encoding is currently widely used for the output representation. Here, the output of classification information for each image region (e.g., a single pixel, a group of pixels, or the entire image) uses either a vector or a cipher with a length equal to the number of different categories. Ignoring estimation inaccuracies in neural networks, the cipher at a single location has, for example, a first binary information of 1, or otherwise only a second binary information of 0. Therefore, the number of feature maps and the cipher length or vector length increase linearly with the number of different categories.

[0004] Especially in pixel-based methods such as image semantic segmentation, each pixel of the input image information is assigned a category, and the dimension of the feature map is correspondingly the dimension of the image information to be processed. This requires high computing power and storage capacity, which can cause problems, especially for situations with low computing power and low storage capacity, such as in-vehicle computing units.

[0005] Furthermore, it is known that solutions to regression problems in neural networks can be achieved by transforming them into classification problems and solving those classification problems. If the regression problem is solved by transforming it into a classification problem using so-called one-hot encoding, the number of feature maps equals the number of quantization intervals (based on which the continuous output variables of the regression problem are quantized). Here, the computational and storage costs also increase linearly with the number of quantization intervals that affect the accuracy of the output variables of the regression problem. Summary of the Invention

[0006] Based on this, the present invention aims to propose a method for processing image information in artificial neural networks, which can reduce computation and memory requirements while maintaining high estimation accuracy.

[0007] This objective is achieved by a method having the features described in independent claim 1. Preferred embodiments are the subject of the dependent claims. A system for processing image information in an artificial neural network is the subject of independent claim 12, and a vehicle having such a system is the subject of independent claim 15.

[0008] According to a first point of view, the present invention relates to a method for processing image information from a vehicle imaging sensor in an artificial neural network. The neural network includes at least an encoder and a decoder. The neural network preferably refers to a convolutional neural network (CNN). The artificial neural network is configured to solve classification tasks with multiple categories and / or regression tasks that provide digital output information quantized according to multiple quantization intervals. The artificial neural network outputs multiple feature maps at its output interface. Assigning categories to image regions of image information through the feature maps, or encoding digital output information relating to the quantization of image information, is output by means that the inputs in the corresponding matrix regions of the feature maps collectively generate a code. In other words, a code or vector is generated from multiple values ​​input in corresponding regions (e.g., the same row and the same column) of the feature maps, which reproduces the category or quantization interval (e.g., the tenth quantization interval out of one hundred) in an encoded form. An "image region" can be, for example, a single pixel, a group of multiple pixels, all image information, or the entire image. Information compression is achieved through encoding by ensuring that the number of outputs from the feature maps is less than the number of categories or quantization intervals.

[0009] The technical advantage of the proposed method is that the number of feature maps can be significantly reduced by encoding the output information, thereby significantly reducing the computation and memory requirements of the neural network.

[0010] According to one embodiment, the input content of the feature map constituting the cipher is distorted due to estimation errors, thereby deriving a quasi-binary cipher, i.e., a cipher in which the value at a single cipher position not only has zeros and ones, but also decimal numbers between 0 and 1. This cipher is converted into a binary-encoded cipher through a decision function.

[0011] According to one embodiment, the following decision function is used:

[0012]

[0013] Here, It is the binary number that constitutes the position of the binary cipher, p i is the corresponding value at the position of the input content in the feature map corresponding to the binary cipher (i.e., the value distorted by the estimation error), and t is the decision threshold. Thus, the value at the position of the quasi-binary cipher can be correspondingly converted into the value of the binary cipher.

[0014] According to one embodiment, the binary-encoded password is decoded into classification information or rational numbers. Here, the classification information is an indication of which category a corresponding image region can be assigned to according to a predetermined classification, or the classification information can be constructed into information in the form of rational numbers relating to the image region, such as depth information about the image region. Depending on the application, an "image region" may be, for example, a single pixel, a group of multiple pixels, all image information, or the entire image.

[0015] According to one embodiment, the classification task is an image semantic segmentation task, and / or the regression task is a task for monocular depth estimation. Therefore, the method is preferably used in a vehicle's driver assistance system, which enables at least semi-automatic control of the vehicle based on classified objects and surrounding environmental information with depth information.

[0016] According to one embodiment, the artificial neural network is a multi-task network, that is, the artificial neural network has an encoder in the form of a multi-task encoder that provides computational operations for classification and regression tasks, and / or the artificial neural network has a decoder in the form of a multi-task decoder that provides computational operations for classification and regression tasks. The proposed method can establish a simplified and unified multi-task encoder or multi-task decoder that can solve both classification and regression tasks because it can beneficially transform regression tasks into classification tasks, where a relatively small output representation (i.e., a decimal number of feature maps) can be generated despite a high number of quantization intervals. Simultaneously, the neural network can be trained using a single error function.

[0017] According to one embodiment, an artificial neural network is trained based on an error function that determines the total difference between the values ​​of the positions of the cipher (especially a quasi-binary cipher) estimated by the neural network and the corresponding values ​​of the ideal cipher in the training data. In other words, this provides ground-truth training data containing the ideal cipher that should be derived as the output information of the neural network without estimation error. The estimation result of the neural network to be trained provides a quasi-binary cipher for the image information of this training data. The error function can determine the total difference between the estimated quasi-binary cipher and the corresponding positions of the ideal cipher.

[0018] According to one embodiment, the artificial neural network is trained based on either a binary cross-entropy error function (BCI) or a focal loss error function (FOC). These error functions measure error based on two functional components; therefore, not only can the estimated value of a password position being "1" be included in the error measurement, but the estimated value of a password position being "0" can also be included in the error measurement.

[0019] A beneficial approach is to use a loss-fehlerfunktion function, which relies on weighting each task based on its bit validity. For example, the binary cross-entropy error function can be used according to the following formula:

[0020]

[0021] Wherein, the weighting coefficient w i Choose from the following:

[0022]

[0023] Where E represents the classification error, and N is the number of classes. These are the estimated output values, w i The weighting coefficients and y i It is based on the ideal values ​​of the training data. i constitutes the control variable, where i = 0 represents a binary bit with the smallest significant bit value (least significant bit).

[0024] Weighting is beneficial in regression, but should not be used in classification (w). i =1).

[0025] According to one embodiment, before applying the error function, the estimated password position value is adjusted and adapted using a constraint function. The constraint function is preferably an sigmoid function. This restricts the estimated password value to a value between 0 and 1.

[0026] According to one embodiment, the password is decoded into rational numbers before the error function is applied, and the error function determines the difference between the decoded rational numbers and the ideal values ​​based on the training data. This allows for improved error estimation, particularly in regression problems.

[0027] According to another perspective, the present invention relates to a system for processing image information from a vehicle imaging sensor in an artificial neural network, the artificial neural network comprising at least an encoder and a decoder. The artificial neural network is configured to solve classification tasks with multiple categories and / or regression tasks providing digital output information quantized according to multiple quantization intervals. Furthermore, the artificial neural network is configured to output multiple feature maps at an output interface, wherein the feature maps assign categories to image regions of image information, or encode the quantized digital output information relating to the image information; that is, the input content in the corresponding matrix regions of the feature maps collectively generates a code, wherein information compression is achieved through encoding, i.e., the number of output feature maps is less than the number of categories or the number of quantization intervals.

[0028] According to one system embodiment, the imaging sensor is an onboard LiDAR sensor, radar sensor, or camera. The system can then process image information provided by the vehicle's sensors and use the information provided through a neural network for driver assistance functions.

[0029] According to one system embodiment, the artificial neural network is a multi-task network, that is, the artificial neural network has an encoder in the form of a multi-task encoder that provides computational operations for classification and regression tasks, and / or the artificial neural network has a decoder in the form of a multi-task decoder that provides computational operations for classification and regression tasks. Here, the classification task is an image semantic segmentation task, while the regression task is a task for monocular depth estimation. The advantage of doing so is that system resources are saved by combining the classification and regression tasks in a single multi-task network.

[0030] According to a final point of view, the present invention relates to a vehicle including the system described according to one of the embodiments.

[0031] In the context of this invention, "quasi-binary cryptography" refers to a cryptography that, due to the inaccuracy of neural network estimation, contains not only numerical values ​​of 0 and 1, but also decimal numbers that are neither 0 nor 1. However, quasi-binary cryptography can be converted into binary cryptography using suitable functions such as decision functions.

[0032] In this invention, a "feature map" refers to the data structure that outputs the neural network at the output interface. A feature map is, for example, a multi-dimensional, matrix-like data structure that includes multiple values, particularly multiple numerical values.

[0033] In the context of this invention, the terms “approximately,” “substantially,” or “roughly” refer to an error of + / -10%, preferably + / -5%, from the respective precise values, and / or the error is irrelevant to the function.

[0034] Other structural forms, advantages, and application possibilities of the invention also arise from the following description and drawings of the embodiments. Here, all described and / or illustrated features constitute the basic subject matter of the invention, either independently or in any combination, and are independent of their summary in the claims or their recapitulation. The content of the claims is also part of the related description. Attached Figure Description

[0035] The present invention will now be described in detail with reference to the accompanying drawings of the embodiments. Wherein:

[0036] Figure 1 An exemplary schematic diagram of a neural multitasking network for processing image information is shown;

[0037] Figure 2 This diagram illustrates an exemplary representation of neural network output information in the form of multiple feature maps when using a one-hot encoding method.

[0038] Figure 3 An exemplary schematic diagram is shown illustrating the representation of neural network output information in the form of multiple feature maps using the proposed binary encoding method; and

[0039] Figure 4 An exemplary schematic diagram for neural network training is shown. Detailed Implementation

[0040] Figure 1 The illustration shows a schematic block diagram of an artificial neural network 1, configured to determine or extract information from image information. In the context of this patent application, image information refers to various multidimensional data containing information about the surrounding environment. This data can be, in particular, information from imaging sensors such as cameras, radar sensors, and lidar sensors. The artificial neural network 1 can be, in particular, a convolutional network (so-called a CNN: convolutional neural network).

[0041] The artificial neural network 1 has at least one encoder 2 and at least one decoder 3. Figure 1 The neural network 1 shown is a so-called multi-task network that can solve multiple image processing tasks simultaneously. In the illustrated embodiment, these image processing tasks are tasks t1 and t2. For example, the first task t1 is a classification task, i.e., the neural network outputs classification information, and the second task is a regression task, i.e., the neural network outputs continuous output variables, for example, in the form of rational numbers.

[0042] For example, in classification tasks, image information of a single pixel or an image region containing multiple pixels is assigned to a specific category. Such classification tasks include, for example, image semantic segmentation.

[0043] For example, a regression task could be to provide depth estimation information for image information. For instance, depth estimation information could be provided for each pixel of the image information to obtain depth information about the image.

[0044] Encoder 2 can be a so-called shared encoder, meaning it can handle encoding tasks for multiple tasks. For different tasks, the neural network can have multiple different decoders 3, which are specifically tailored to each corresponding task.

[0045] However, it should be noted that, in contrast or as an additional measure, a common decoder (the so-called shared decoder) can also be pre-defined, which simultaneously performs decoding for multiple tasks. The compression method described in detail below can also be used for neural networks that solve only a single image processing task.

[0046] Figure 2 This illustrates an example and diagram of an artificial neural network 1 that outputs classification information (encoded according to so-called one-hot encoding) as output information. Multiple so-called feature maps are output as output information at the output interface of neural network 1.

[0047] Feature maps are, for example, multidimensional data structures, such as matrices containing multiple values. Here, the values ​​refer to specific regions of image information transmitted as input to the neural network 1. A region can be, for example, a single pixel or a group of multiple pixels.

[0048] Figure 2 The regions with numerical values ​​always refer to the same image regions, such as image information that defines pixels. These numerical values ​​are expressed as vectors. p The code is constructed in the form of...

[0049] In the illustrated embodiment, neural network 1 performs a classification task, such as image semantic segmentation; that is, the task of neural network 1 is estimation, involving passwords or vector values. p Which category is the image region assigned to?

[0050] vector p The cipher in this form is encoded using a one-hot encoding method. Here, each feature map is assigned to a specific category. This assignment is defined using numbers from 0 to 7. The encoding is performed such that the image region associated with the cipher is assigned to the category whose feature map has the highest value. In other words, the maximum value in the cipher, or the vector representing the cipher, must be determined accordingly, and the position in the cipher is determined by the category to which the image region belongs.

[0051] In the illustrated embodiment, the feature map has a maximum value of 0.9 at position 3, thereby assigning category 3 to the image information. It should be noted that specific target object categories can be assigned to corresponding feature maps, such as "pedestrian," "cyclist," "motor vehicle," etc., thus allowing the target object to be specified through classification.

[0052] The problem with one-hot encoding is that the number of feature maps increases linearly with the number of categories. This is detrimental to the required computational and memory resources, especially when implementing neural networks in vehicles where the hardware available for such implementations is very limited.

[0053] Figure 3 This illustrates an example and diagram of an artificial neural network 1, which outputs classification information in binary encoded form. Corresponding input values ​​in the feature maps are used to construct a quasi-binary encoded cipher using values ​​(0.2, 0.6, 0.7). The values ​​contained in the feature maps are distorted due to estimation inaccuracies or noise. They can be converted into a binary encoded cipher by comparing them with a threshold. The threshold operation can be:

[0054]

[0055] in, It is a binary number, and it forms the position in the binary cipher, p. i t is the corresponding value of the input content in the feature map at the position corresponding to the binary password (i.e., the value distorted due to estimation error), and t is the decision threshold.

[0056] For example, the threshold t can be 0.5. This yields the binary password (0, 1, 1).

[0057] The category can then be determined by decoding the binary cipher (0, 1, 1), i.e., category 3.

[0058] By using binary encoding of the output information of neural network 1, the number of output feature maps can be reduced to less than the number of categories. Specifically, binary encoding can reduce the number of feature maps from N to log2(N), where N is the number of categories in the classification task. The compression ratio compared to one-hot encoding is... Clearly, the compression factor increases with the number of categories N, meaning that the larger the number of categories N, the more computational resources the proposed method saves.

[0059] As mentioned above, the proposed method can also be used to solve regression tasks, where image regions of image information are assigned as persistent output variables.

[0060] By outputting the number of feature maps corresponding to the number of quantization intervals as the output information of the neural network, the regression problem can be transformed into a classification problem. For example, if the range of values ​​of the output variable is subdivided into 1024 quantization intervals, then 1024 feature maps will be output as the output information of the neural network without using the proposed method.

[0061] To reduce the number of feature maps, the output information of the neural network is provided in binary encoding form, where the quasi-binary code (before the thresholding operation) or the binary code (after the thresholding operation) represents the quantization interval, from which the output variable in real number form can be derived.

[0062] For example, if real numbers in the range between 0 and 1 are output as output information, the real numbers themselves are not output; instead, the quantization interval of the binary code corresponding to the real numbers is output.

[0063] For example, if a binary cipher [1011010011] is output through a neural network (i.e., after applying a threshold operation), the quantization gap y is first derived from this binary cipher. This can be done using the following formula:

[0064]

[0065] Where n is the length of the binary cipher, x i It is the value at the i-th position of the binary cipher, where i = 0 indicates the binary bit with the least significant bit.

[0066] Therefore, the quantization interval in the binary cipher [1011010011] is 723. Given that there are a total of 1024 quantization intervals within the range of 0 and 1, the actual output variable is rounded to two decimal places, resulting in 0.71.

[0067] Therefore, the disclosed methods for processing image information can also be applied to regression problems.

[0068] The following is based on Figure 4 A method for training a neural network 1 is described, such that the neural network can output encoded output information in a feature map in the form of binary cipher or essentially binary cipher.

[0069] For image information used to train neural network 1, it is important that a learning database (also known as benchmark truth) containing feature maps with an ideal code for that image information is available. For example, for image information reproducing traffic conditions in an area around a vehicle, the nominal classification of the image region or image information pixels is predefined (in... Figure 4These classification information (referred to as target values) are output to the feature map after being binary encoded. For example, corresponding regions in the feature map always refer to the same image region. Thus, for example, a value of "0" in the upper left region of the first feature map may relate to a specific pixel of image information or a defined image region.

[0070] The training method for neural network 1 is to transmit image information with a learning database as input information to the neural network in order to perform, for example, classification estimation.

[0071] from Figure 4 As can be seen, due to estimation errors, there is a difference between the value of the feature map output by neural network 1 and the value of the feature map in the learning database.

[0072] To train the neural network, an error function is used, and the estimation error is minimized based on its results, such as using gradient descent, and the weights of the adapted neural network 1 are adjusted in the process.

[0073] In classification tasks, error functions can be used to handle the assignment of multiple categories; that is, error functions that can simultaneously optimize classification results for multiple categories (also known as "multi-label classification") are used. Examples of such error functions are the binary cross-entropy error function or the focus loss error function. The values ​​of the feature maps output by the neural network can be modified, for example, by a constraint function, particularly a sigmoid function, and then fed into the error function. The constraint function allows the values ​​of the feature maps estimated by the neural network to be between 0 and 1.

[0074] An example of the binary cross-entropy function is:

[0075]

[0076] in

[0077]

[0078] Where E represents the classification error, and N is the number of classes. These are the estimated output values, w i It is the weighting coefficient, y i It is based on the ideal values ​​of the training data. i constitutes the control variable, where i = 0 represents a binary bit with the smallest significant bit value (least significant bit).

[0079] Classification errors based on error functions can be addressed, for example, by adjusting and adapting the weights of the neural network using a backpropagation algorithm.

[0080] Another possibility for determining the error is to decode the binary cipher into real or decimal numbers before applying the error function. That is, instead of comparing the estimated cipher to the ideal binary cipher, the estimated value obtained by decoding in real or decimal form is compared to the ideal value, which is also in real or decimal form. This allows the use of other error functions, such as those that give the absolute error, for example:

[0081] E=|yp| (Formula 4)

[0082] Where E represents the classification error, p is the estimated output value, and y is the ideal value based on the training data.

[0083] The following formula is generally applicable:

[0084]

[0085] in, Given the classification error, E is the error function, and D(y) and D(p) are functions for decoding the binary ideal cipher estimated by the neural network or the training data.

[0086] Decoding the estimated binary ideal cipher from the cipher or training data into real or decimal numbers can be done based on the following formula:

[0087]

[0088] Where N is the number of quantization intervals, x i It is the corresponding value at the position of the estimated password or the binary ideal password of the training data.

[0089] The following example illustrates the advantages of using an error function after decoding a cipher.

[0090] If we assume that the password estimated by the neural network after applying the threshold calculation according to Formula 1 is, for example, (0111111111), which corresponds to the decimal number 511, but the ideal password obtained from the training data is (1000000000), which corresponds to the decimal number 512, then the binary difference in the password does not correspond to the actual difference in the decoded number. This is related to training the neural network for a regression task. Therefore, especially beneficial for regression tasks, it is helpful to embed the decoding into the error function, as shown in Formula 5, that is, to perform error estimation after decoding.

[0091] The invention has been described above with reference to embodiments. It should be noted that various modifications and variations may be made without departing from the scope of protection defined by the patent claims.

[0092] List of reference numerals in the attached diagram:

[0093] 1. Artificial Neural Networks

[0094] 2 Encoders

[0095] 3 Decoders

[0096] t1 First Task

[0097] t2 Second Task

Claims

1. A method for processing image information from a vehicle imaging sensor in an artificial neural network (1), said artificial neural network comprising at least one encoder (2) and at least one decoder (3), wherein, Artificial neural networks solve classification tasks with multiple categories and / or regression tasks that provide digital output information quantized according to multiple quantization intervals. In this case, the artificial neural network (1) outputs multiple feature maps at the output interface. The feature maps assign categories to image regions of image information, or encode the quantized digital output information related to the image information. That is, the input content in the corresponding matrix regions of the feature maps collectively generates a code. Information compression is achieved through encoding, meaning the number of outputs from the feature maps is less than the number of categories or the number of quantization intervals. The input content of the feature map constituting the cipher is distorted due to estimation errors and is converted into a binary-encoded cipher through a decision function. The following decision function is used: in, It is the binary number that constitutes the position of the binary cipher, p i t is the corresponding value of the input content in the feature map at the position corresponding to the binary password, and t is the decision threshold.

2. The method according to claim 1, characterized in that, The binary-encoded cipher is decoded into categorical information or rational numbers.

3. The method according to claim 1 or 2, characterized in that, The classification task is an image semantic segmentation task and / or the regression task is a monocular depth estimation task.

4. The method according to claim 1 or 2, characterized in that, Artificial neural networks (1) are multi-task networks, that is, encoders (2) are multi-task encoders that provide computational operations for classification and regression tasks, and / or decoders (3) are multi-task decoders that provide computational operations for classification and regression tasks.

5. The method according to claim 1 or 2, characterized in that, An artificial neural network is trained based on an error function, which determines the total difference between the values ​​of the positions of the password estimated by the neural network and the corresponding values ​​of the ideal password in the training data.

6. The method according to claim 1 or 2, characterized in that, Artificial neural networks are trained using the binary cross-entropy error function or the focus loss error function.

7. The method according to claim 5, characterized in that, Before applying the error function, the position values ​​of the password estimated by the neural network are adjusted and adapted using a constraint function.

8. The method according to claim 7, characterized in that, The constraint function is a sigmoid function.

9. The method according to claim 1 or 2, characterized in that, Before using the error function, the password is decoded into rational numbers, and the error function determines the difference between the decoded rational numbers and the ideal values ​​based on the training data.

10. A system for processing image information from a vehicle imaging sensor in an artificial neural network (1), said artificial neural network comprising at least one encoder (2) and at least one decoder (3), wherein, An artificial neural network (1) is configured to solve classification tasks with multiple categories and / or regression tasks that provide digital output information quantized according to multiple quantization intervals. The artificial neural network (1) is configured to output multiple feature maps at the output interface, assign categories to image regions of image information through the feature maps, or encode and output quantized digital output information related to image information. That is, the input content in the corresponding matrix regions of the feature maps jointly generates a code, where information compression is achieved through encoding, meaning the number of outputs of the feature maps is less than the number of categories or the number of quantization intervals. The input content of the feature map constituting the cipher is distorted due to estimation errors and is converted into a binary-encoded cipher through a decision function. The following decision function is used: in, It is the binary number that constitutes the position of the binary cipher, p i t is the corresponding value of the input content in the feature map at the position corresponding to the binary password, and t is the decision threshold.

11. The system according to claim 10, characterized in that, Imaging sensors can be vehicle-mounted lidar sensors, radar sensors, or cameras.

12. The system according to claim 10 or 11, characterized in that, The encoder (2) is a multi-task encoder that provides computational operations for classification and regression tasks, where the classification task is an image semantic segmentation task and the regression task is a monocular depth estimation task.

13. A vehicle comprising the system according to any one of claims 10 to 12.

Images