Method for processing image data for the application of a machine learning model

Abstract

The invention relates to a method (100) for processing image data (205) for the application of a machine learning model (200), comprising the steps of: - determining (101) image data (205), wherein the image data (205) result from image capture using a camera (40), - converting (102) the image data (205) into a line-of-sight display, - providing (103) an input for the machine learning model (200) based on the image data (205) in the line-of-sight display.

Description

[0001] Description

[0002] title

[0003] Method for processing image data for the application of a machine learning model

[0004] The present invention relates to a method for processing image data for the application of a machine learning model. Furthermore, the invention relates to a computer program and a device for this purpose.

[0005] State of the art

[0006] It is known from the prior art that computer vision algorithms can be used to enhance images from a camera recording, especially from a vehicle. The adaptation of computer vision algorithms to different camera distortions is usually formulated as a domain adaptation problem. Domain invariance is ensured by a special training procedure in which the same CNN is trained on images with different camera distortions, allowing the resulting weights to be generalized across domains.

[0007] In “X. Peng, YL Murphey, S. Stent, Y. Li, and Z. Zhao, “Spatial focal loss for pedestrian detection in fisheye imagery,” 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 561-569, 2019.” special types of costs are described that reweight samples with different camera distortions to promote the learning of domain-invariant features.

[0008] Disclosure of the invention

[0009] The invention relates to a method having the features of claim 1, a computer program having the features of claim 9, and a device having the features of claim 10. Further features and details of the invention emerge from the respective subclaims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program according to the invention and the device according to the invention, and vice versa, so that with regard to the disclosure of the individual aspects of the invention, reference is always made to each other.

[0010] The invention particularly relates to a method for processing, in particular preprocessing, image data for the application of a machine learning model, comprising the following steps, which are preferably carried out successively and / or repeatedly:

[0011] - Determining image data, wherein the image data result from image capture with a camera, preferably of a vehicle,

[0012] - Transferring, preferably converting, the image data into a visual ray representation,

[0013] - Providing an input for the machine learning model based on the image data in the ray of sight representation, preferably to apply the machine learning model with the input, preferably to evaluate the image data, e.g. to classify and / or segment it.

[0014] The input is based on a line-of-sight representation and thus, in particular, on a camera-independent representation of the image data. This has the advantage that the machine learning model, preferably in the form of an artificial neural network, can be applied independently of the specific camera type. The vehicle can be designed, for example, as a motor vehicle, preferably a passenger car. In particular, the vehicle has a driving function for autonomous driving and / or for assisting a driver of the vehicle. The driving function can be designed to control the vehicle, e.g., to steer and / or accelerate it, based on the image data and / or an output of the applied machine learning model.

[0015] Furthermore, it is conceivable that the image data in the line of sight representation are represented by pixels, wherein the provision of the input can comprise the following step for further preprocessing: - Providing a grid representation in which a grid has a plurality of grid cells, wherein the pixels are assigned to the grid cells.

[0016] The grid density and the exact assignment can be hyperparameters of the method according to the invention. In this context of machine learning, a hyperparameter is, in particular, a parameter whose value is used to control the learning process. The geometric shape of the grid can, if necessary, be predefined and, for example, correspond to a projection surface of the visual rays, preferably the image plane, of the visual ray representation.

[0017] The image points can be points on the grid at which the line of sight rays arrive. In other words, the image points can each represent an intersection point with a line of sight in the line of sight representation. Accordingly, the image points can also be referred to as ray intersection points.

[0018] The term "sight ray" is commonly used in the field for a representation such as that described, for example, in Z. Zhang, Camera Parameters (Intrinsic, Extrinsic), pp. 81-85. Boston, MA: Springer US, 2014, or Ikeuchi, K. (eds) Computer Vision. Springer, Boston, MA. Essentially, this representation describes a point at which the light ray that created a point or pixel in the image data encountered a hypothetical plane (also referred to as a projection surface within the scope of the invention) on the optical path. Therefore, this representation is also referred to as a "sight ray representation." The hypothetical plane is often also referred to as the "image plane." It should be noted that, despite the designation "plane," any virtual surface such as a sphere can also be constructed as an image plane, and the sight ray representation of an image, i.e., the image data, can be calculated relative to this surface.

[0019] The image data can initially be present in an image representation, e.g. a pixel matrix representation. This can be the representation of the image data as it is usually provided by the camera. The image data can then be converted into the line of sight representation. In the next step, the line of sight representation can be converted into a representation that refers to a grid, preferably a two-dimensional (2D) grid. This can be referred to as providing the grid representation. This means that a grid, preferably a 2D grid, is specified on the image plane (or the projection surface onto which the line of sight was projected) and each line of sight is assigned to a grid cell in this grid. The image points can therefore be those points on the grid at which the line of sight arrives on the image plane or projection surface.

[0020] Furthermore, it is optionally provided that the pixels are assigned to the grid cells in different numbers, whereby the provision of the input includes the following step for further preprocessing:

[0021] Performing a normalization based on the grid representation, preferably by normalizing a distance between a cell center point of the respective grid cell and the pixels in the grid representation assigned to the respective grid cell.

[0022] The grid representation can be converted to a convolutional representation and / or normalized to a 2D grid of fixed-length vectors. This can have the advantage of being processable with machine learning algorithms, particularly standard CNN approaches.

[0023] Furthermore, the method according to the invention can achieve the advantage that the image data can be converted into a distortion-free representation, so that the machine learning model can be used as a generic algorithm, for example, for detecting objects in the image data or for semantic segmentation of the image data. This enables the use of the method according to the invention, for example, in machines such as vehicles or robots, to control the machine based on object detection.

[0024] Furthermore, within the scope of the invention, it can be provided that a feature map is calculated through normalization, wherein the feature map for each grid cell comprises a feature vector calculated from the pixels of the respective grid cell. This enables a number of tasks, such as semantic segmentation or object recognition, to be performed based on the feature map using the machine learning model. For example, labels can also be mapped to the grid representation to enable training of the machine learning model.

[0025] It may also be possible for the normalization to be based on applying a neural network to the pixels of the respective grid cell. For example, by applying the neural network in the form of a CNN (convolutional neural network), the feature map can be obtained at the 2D grid level. This allows for direct modeling of camera distortions. Optionally, the neural network can also include at least one convolutional layer.

[0026] Furthermore, it is conceivable for the feature map to comprise a single feature vector, preferably with at least one channel, per grid cell. Applied to all grid cells, this process can result in a feature map with the same dimensions as the 2D grid described above.

[0027] Preferably, within the scope of the invention, it can be provided that the machine learning model is applied with the provided input, wherein a vehicle is preferably controlled based on the application of the machine learning model, wherein the machine learning model is preferably trained using dropout, preferably a special form of dropout. A vehicle function such as a driver assistance function and / or an autonomous driving function can be provided, which evaluates an output of the applied machine learning model in order to control the vehicle based thereon. Furthermore, a dropout in the first layer can be used during the training of the machine learning model in order to make the machine learning model robust against changes in the optical parameters.

[0028] The machine learning model, hereinafter also referred to as the model for short, can have been previously trained using image data from a camera of a specific type. Due to the advantageous conversion to the line of sight representation, the machine learning model trained in this way is nevertheless suitable for use with other camera types. The type and / or the imaging errors such as camera distortions of the camera when applying the machine learning model can differ from a camera that was used to train the machine learning model. Thus, the model can preferably be trained such that it is invariant to differences in the optical properties between cameras. Thus, the method according to the invention can preferably be used to apply models such as CNNs to camera distortions that were not yet taken into account during training.

[0029] Directly modeling camera distortions also offers several advantages. First, explicitly modeling prior knowledge about the distortions can reduce the learning problem. A neural network does not need to implicitly learn the transformations between different distortions because they are explicitly modeled in the CNN structure. This frees up capacity in the model that can be used to learn features relevant to the task at hand. Furthermore, explicitly training invariance to camera distortions enables the use of different machine learning models, for example, in the form of deep learning models trained for completely different camera distortions, especially when the sensors of the two cameras are compatible.For example, the machine learning model could also be used on fisheye cameras, even if the model was not trained for such camera distortions. This also means that applying a trained machine learning model to a dataset with a different camera distortion (e.g., oblique-view images) does not require extensive labeling and retraining.

[0030] The invention also relates to a computer program, in particular a computer program product, comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to the invention. The computer program according to the invention therefore brings with it the same advantages as have been described in detail with reference to a method according to the invention. The invention also relates to a device for data processing which is set up to carry out the method according to the invention. For example, a computer which executes the computer program according to the invention can be provided as the device. The computer can have at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program is stored and from which the computer program can be read out by the processor for execution.

[0031] The invention may also provide a computer-readable storage medium comprising the computer program according to the invention. The storage medium is designed, for example, as a data storage device such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.

[0032] Furthermore, the method according to the invention can also be implemented as a computer-implemented method.

[0033] Further advantages, features, and details of the invention will become apparent from the following description, which describes embodiments of the invention in detail with reference to the drawings. The features mentioned in the claims and in the description may be essential to the invention individually or in any combination. They show:

[0034] Fig. 1 : an example visualization of the different optical parameters of cameras.

[0035] Fig. 2: a definition of a 2D grid on the image plane for converting sight rays into relative representations.

[0036] Fig. 3: An application of a CNN to the relative line-of-sight representation. Fig. 4: An exemplary implementation of the embodiment according to Fig. 3, in which the CNN is a transformer network.

[0037] Fig. 5: Embodiments of the invention visualized in blocks.

[0038] Fig. 6 shows a schematic visualization of a method, a device and a computer program according to embodiments of the invention.

[0039] In the following figures, identical reference numerals are used for the same technical features, even in different embodiments.

[0040] Since cameras typically record visual information using a method in which the individual sensors are arranged in a grid, the output of such cameras is a matrix of pixel values. Examples of such pixels are shown in Fig. 1. It is often useful to first convert this pixel matrix into another representation as follows. Using the camera's own parameters, one can calculate the coordinates at which the light hits the image plane for each pixel (see, for example, Z. Zhang, Camera Parameters (Intrinsic, Extrinsic), pp. 81-85. Boston, MA: Springer US, 2014). Essentially, this representation describes a point at which the light ray that created the pixel encountered a hypothetical plane on the optical path. Therefore, this representation is also called a "line-of-sight representation." Note that this representation can always refer to a plane, usually referred to as the "image plane."However, one can construct any virtual surface and compute the ray-of-sight representation of an image relative to that surface. To train camera-independent models, it may be useful to choose a virtual surface that can be easily generalized between different cameras, such as a sphere around the camera.

[0041] Fig. 1 illustrates the various optical parameters of cameras. The first image (Fig. 1a) uses barrel distortion, which results in more pixels being imaged at the sides of the image than in the center. Conversely, the second image (Fig. 1b) uses a distortion that images more pixels in the center and fewer at the edges.

[0042] Computer vision algorithms that leverage deep learning are extremely successful in a wide variety of applications. They rely on a specific representation of the information captured by a camera: a series of pixels arranged in a matrix structure with specific neighborhood relationships. An image is typically a multi-valued H x W x C matrix, where H and W are the height and width of the image, and C is the number of channels, e.g., 3 for an RGB image. However, this representation is largely camera-dependent, as the camera's optical parameters determine the pixel representation of, for example, an object in the real world (see Fig. 1). Therefore, conventional machine learning models trained on such images can only learn this camera-specific representation and struggle when this representation changes because a different camera is used—even if the captured object is the same (see, e.g., X. Peng, YLMurphey, S. Stent, Y. Li, and Z. Zhao, “Spatial focal loss for pedestrian detection in fisheye imagery,” 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 561-569, 2019.).

[0043] Embodiments of the invention can overcome these disadvantages by the following steps:

[0044] First, the image can be converted into a ray-of-sight representation using known methods. Second, the points can then be converted into relative coordinates along a 2D grid. Third, a fully convolutional network can be applied to this representation for each grid cell. Fourth, a conventional CNN can then be applied to the resulting feature map to solve a specific task (e.g., semantic segmentation). Fifth, a dropout in the first layer can be used during training to make the model robust to changes in the optical parameters. The steps of the method are described in more detail below.

[0045] First, the image representation, preferably a pixel matrix representation, of an image can be converted into a ray-of-sight representation. This can be done using conventional state-of-the-art methods (see, for example, Z. Zhang, Camera Parameters (Intrinsic, Extrinsic), pp. 81–85. Boston, MA: Springer US, 2014. In: Ikeuchi, K. (eds) Computer Vision. Springer, Boston, MA.). Essentially, this representation consists of a set of points pi e P, where pi = (Xi.yi.rgbi) and x^yt are coordinates on the image plane, and rgbi is the pixel's color information (note that the same method can be used for any color space, not just RGB, and can be trivially extended to other modalities such as RGBD).

[0046] In the next step, the sight ray representation can then be converted into a representation that refers to a two-dimensional (2D) grid. This means that a 2D grid is specified on the image plane (or the plane onto which the sight rays were projected), and each sight ray is assigned to a grid cell in this grid. This serves as preprocessing and only needs to be performed once, as shown in Fig. 2. In this context, it is also referred to as a grid representation. The relative sight ray representation therefore consists of points e L, where = (Axi.Ayi.rgbi), where Ax^Ayi are coordinates relative to the nearest grid point. Essentially, these steps allow the representation of the sight rays to be mapped onto a 2D grid. However, this representation may not yet be directly usable by the machine learning model, since each grid cell has a different number of mapped points.“mapped”) visual rays.

[0047] Fig. 2 visualizes the definitions of a 2D grid on the image plane for converting sight rays into relative representations. Both the grid density and the precise mapping method can be hyperparameters of a method according to embodiments of the invention. In a further step, the relative sight ray representation, preferably a grid representation, can be converted into a convolutional representation. In other words, the relative sight ray representation can be normalized to a 2D grid of fixed-length vectors. This can have the advantage that it can be processed using standard CNN approaches - as shown in Fig. 3. For this purpose, an artificial neural network such as a CNN can be applied to one or more points l tapplied to each grid cell to compute some features for that grid cell. This can, for example, involve one or more one-dimensional convolutional layers and global average pooling. Crucially, the resulting feature map must have a single feature vector (with one or more channels) per grid cell. Applied to all grid cells, the result of this process can be a feature map with the same dimensions as the 2D grid described above. The CNN can be defined in many ways, e.g., as a series of convolutions and poolings with variable input size, as in Fig. 3, or as a transformer network, as in “A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N.Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ArXiv, vol. abs / 2010.11929, 2020.” and illustrated in Fig. 4.

[0048] Fig. 3 visualizes an application of a CNN to the relative ray representation. The CNN preferably operates with a variable input size N_i x D, where is the number of view rays associated with grid cell i, and D is the input dimension (5 for 2D RGB). The output of this block can then be generated in a fully convolutional manner and have the form 1 × F, where F is the chosen feature dimension (this is a hyperparameter of the model, just like all other convolutional layers). Figure 4 shows an alternative implementation of the embodiment of Figure 3, in which the CNN is a Transformer network. Finally, a standard CNN can be applied to the feature map output in the previous step to perform a variety of tasks such as semantic segmentation or object detection. The labels can preferably also be mapped to the previously described grid representation to enable training.

[0049] A particular advantage can be achieved by training the resulting model to be robust to changes in the ray of sight representation. This can be made possible primarily by allowing the mapping of P to L to change dynamically. For example, some form of dropout can be incorporated, where some L points are randomly dropped from each grid cell, effectively resulting in less information being available for the grid cells. Since the CNNs applied to the L points are fully convolutional and therefore shared by all grid cells, this results in a distortion-independent information processing step that works on images from cameras with different optical properties.

[0050] In Fig. 5, embodiments of the invention are visualized in blocks: creation of a (relative) sight ray representation (A), application of a CNN to obtain a feature map on the 2D grid level (B), and application of a standard CNN to the resulting features (C).

[0051] Fig. 6 schematically illustrates the method, as well as a device and a computer program according to exemplary embodiments of the invention. The method 100 for processing image data 205 for the application of a machine learning model 200 may comprise, in a first method step 101, determining 101 image data 205, wherein the image data 205 results from image capture with a camera 40. Then, according to a second method step 102, converting the image data 205 into a visual ray representation may be provided. Subsequently, according to a third method step 103, an input for the machine learning model 200 based on the image data 205 in the visual ray representation may be provided. The provision may, for example, comprise a data transmission in which the image data—possibly after further preprocessing—is transferred to the machine learning model as the input.

[0052] Fig. 2 shows that the image data 205 can be represented in the visual ray representation by pixels 215. Further preprocessing can also make it possible to provide the grid representation 210, in which the grid has a plurality of grid cells, with the pixels 215 being assigned to the grid cells. In this case, the pixels 215 can be assigned to the grid cells in varying numbers. Subsequently, normalization can be provided, in which a distance between the respective cell center and the respective pixels 215 in the grid representation can be normalized, e.g., by a neural network 230, resulting in a feature map 220 (see Fig. 3).

[0053] The feature map 220 can advantageously comprise a single feature vector, preferably with at least one channel, per grid cell. Figure 4 shows that a transformer 230 can also be used as the neural network 230 with a prior generation of "embeddings" 235 to generate the feature map 220.

[0054] The above explanation of the embodiments describes the present invention exclusively by way of examples. Of course, individual features of the embodiments can be freely combined with one another, provided they are technically feasible, without departing from the scope of the present invention.

Claims

Claims 1 . A method (100) for processing image data (205) for applying a machine learning model (200), comprising: Determining (101) image data (205), wherein the image data (205) result from image capture with a camera (40), Converting (102) the image data (205) into a ray of sight representation, providing (103) an input for the machine learning model (200) based on the image data (205) in the ray of sight representation.

2. Method (100) according to claim 1, characterized in that the image data (205) in the line of sight representation are represented by pixels (215), wherein the provision (103) of the input comprises the following step for further preprocessing: providing a grid representation (210) in which a grid has a plurality of grid cells, wherein the pixels (215) are assigned to the grid cells.

3. Method (100) according to claim 2, characterized in that the pixels (215) are assigned to the grid cells in different numbers, wherein the provision (103) of the input comprises the following step for further preprocessing: performing a normalization based on the grid representation (210), preferably by normalizing a distance between a cell center point of the respective grid cell and the pixels (215) assigned to the respective grid cell in the grid representation. Method (100) according to claim 3, characterized in that a feature map (220) is calculated by the normalization, wherein the feature map (220) for each grid cell comprises a feature vector which is calculated from the pixels (215) of the respective grid cell. Method (100) according to claim 3 or 4, characterized in that the normalization is based on an application of a neural network (230) to the pixels (215) of the respective grid cell. Method (100) according to claim 5, characterized in that the neural network (230) comprises at least one convolutional layer. Method (100) according to one of the preceding claims, characterized in that the feature map (220) comprises a single feature vector, preferably with at least one channel, per grid cell.Method (100) according to one of the preceding claims, characterized in that the machine learning model (200) is applied with the provided input, wherein a vehicle (1) is controlled based on the application of the machine learning model (200), wherein the machine learning model is preferably trained by means of dropout. Computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to execute the method (100) according to one of the preceding claims. Device (10) for data processing which is configured to execute the method (100) according to one of claims 1 to 8.

Images