Method for image processing by neural network and method for training neural network

Abstract

The present invention relates to a computer-implemented method for processing an image, comprising the steps of: dividing the image into at least two image regions; grouping at least one of the image regions into at least one group, assigning each of the image regions to only one group or to no group; and applying at least one kernel of a layer of a neural network to the at least one group of image regions, each group of image regions being applied with at most one kernel. The present invention also relates to a computer-implemented method for training a neural network, a device for processing an image in a neural network and / or for training a neural network, and a computer-readable medium.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for processing images in a neural network. Background Art

[0002] Neural networks, more precisely convolutional neural networks (CNNs), are known, for example, from "Object Recognition with Gradient-Based Learning" by Yann LeCun et al. In particular, the paper discusses finding a suitable set of features. CNNs have proven to be particularly well-suited for this task.

[0003] Digital cameras are used in a variety of industrial applications. The scene to be recorded is typically illuminated using an illumination unit and imaged onto the camera's image sensor using an objective lens. Any processing steps can then be applied to the resulting raw image, which can then be transmitted via a corresponding interface.

[0004] Digital cameras are particularly used in traffic engineering, for example, to detect license plates, to check product quality during production, and in surveillance technology. Samples, such as cell samples, are often evaluated in medical technology and biology. This study is often performed using a microscope equipped with a digital camera. The camera records the sample and then provides the user with a digital image of the sample. This image can then be used for further research, such as counting or classifying cells.

[0005] Many other applications of digital cameras and images exist, for example, in the fields of astronomy, medicine, physics, biology, chemistry, and art history, to name a few.

[0006] In order to be able to complete the corresponding tasks, sufficient image quality is required. This means that the image content relevant to the task must be clearly recognizable in the image and not be unrecognizable due to image noise or other influences.

[0007] Furthermore, it is important, especially in the field of medical technology, that the image content is not distorted by correction methods or individual manipulations. Exposure times of a few milliseconds to several minutes are common in microscopy. Therefore, the evaluation time of processing methods should be within a similar timeframe.

[0008] The removal of image noise, so-called denoising, is an important method for improving image quality or for maintaining image content when processing image data.

[0009] Denoising can be performed directly in the camera or in external hardware, such as a computer, such as a client computer with a processor (CPU) or a graphics processing unit (GPU). Therefore, it is desirable to use resources and computation time economically, minimizing the burden on the client computer and conserving hardware resources. Furthermore, computations can be performed in the camera on a field-programmable gate array (FPGA) or specialized hardware, such as a frame grabber or microcontroller (MCU).

[0010] In this case, the demand for resources should also be kept to a minimum, since computing power, energy and chip area are always linked to costs.

[0011] Medical or biological samples are often very sensitive and can be damaged or destroyed by excessive light. Consequently, imaging recordings of such samples are often produced using limited light, for example, at low light intensities, as is often the case in microscopy. Under the most accurate assumption that the recording involves a camera that at least largely complies with the EMVA 1288 standard, the resulting recording has a low signal-to-noise ratio. This can significantly impair the image's legibility and usability. Therefore, it is desirable to improve the signal-to-noise ratio by reducing noise in the image.

[0012] If the sample is exposed to a small amount of light during recording, this typically results in an underexposure of the image sensor. The significant forms of image noise at low exposure are primarily photon noise (also photon shot noise) and dark noise (dominated by electrical readout noise, also known as "read noise").

[0013] At low light levels, it is generally possible to increase exposure time. This generally reduces photon noise, improving image legibility and usability. However, increasing exposure time according to EMVA1288 also increases dark noise and the number of so-called hot pixels in the image.

[0014] So-called dark noise occurs when no light falls on a camera's sensor. This noise is caused, on the one hand, by the dark current of the individual photosensitive elements (i.e., pixels), and, on the other hand, by the noise of the readout amplifier. Dark noise occurs, for example, in fluorescence microscopy.

[0015] Hot pixels are pixels that do not react proportionally to the incident light, so, especially with long exposure times, they reproduce image values ​​that are noticeably too bright (see EMVA1288). The number of hot pixels increases primarily with long exposure times and high ISO or gain values. As the temperature increases, the number of hot pixels increases, so one way to avoid hot pixels and image noise is to keep the camera temperature low. Many digital cameras intended for long exposure times have cooling of the image sensor. Hot pixels are primarily caused by manufacturing inaccuracies and lead to a further deterioration of image quality, which in turn impairs the image's recognizability and usability.

[0016] A large number of “denoising” algorithms already exist, wherein the first methods, however, require a series of image recordings, such as in J. Boulanger et al., “Patch-based non-local functional for denoising fluorescence microscopy image sequences”, 2009.

[0017] Another approach initially generated synthetic training data by adding defined noise (e.g., Gaussian noise) to a noise-free dataset, such as in “Image Denoising and Inpainting with Deep Neural Networks” by J. Xie et al., 2012. However, this does not represent a realistic noise model for digital cameras, as defined by the EMVA1288 standard, since Gaussian noise is not realistic.

[0018] Furthermore, there are many deep learning methods that preferably use CNNs with different network architectures, such as autoencoders, for example, V. Jain et al., "Natural image denoising with convolutional networks," 2019, and H.C. Burger, "Image denoising: Can plain neural networks compete with BM3D?", in Conference on Computer Vision and Pattern Recognition, 2012.

[0019] However, in principle, a known disadvantage of deep learning methods is the amount of data required and the resources and time-consuming training. Another common disadvantage is the lack of generalization of the network. This means that the network often works well for the samples it was trained on, but not for other types of samples.

[0020] In addition to deep learning methods, there are other denoising methods that are based on so-called surface learning or classical methods, such as P. Milanfar, "Fast, Trainable, Multiscale Denosing" 2018, S. Srisuk, "K-meansbased image denoising using bilateral filtering and total variation" 2014 or J. Ehmann, "Real-time video denoising on mobile phones" 2018.

[0021] A disadvantage of these methods is that the various methods, components or filters often have to be trained or developed individually at great expense and then have to be combined. Summary of the Invention

[0022] It is therefore an object of the present invention to overcome the drawbacks of the prior art and in particular to provide a correction method for reducing noise in digital images without distorting the content of the images.

[0023] This object is achieved by the subject matter of independent claim 1. Therefore, the method according to the invention for processing an image comprises the steps of: dividing the image into at least two image regions; grouping at least one of the image regions into at least one group, wherein each of the image regions is assigned to only one group or to no group; and applying at least one kernel of a layer of a neural network to at least one group of the image regions, wherein at most one kernel is applied to each group of the image regions.

[0024] Furthermore, a method for training a neural network, a device for image processing by means of a neural network and for training a neural network, and a computer-readable medium are proposed.

[0025] The kernels proposed here can specifically be convolution kernels, where the layer in which the kernel is applied to the image thus becomes a convolution layer in a neural network. In particular, the fact that the kernel is a convolution kernel means that the kernel is flipped, i.e., the kernel matrix is ​​vertically mirrored. Without vertical mirroring of the matrix, the operation is usually called cross-correlation. However, since cross-correlation within the neural network is always assumed here, the operation is often called convolution even without vertical mirroring of the matrix. Therefore, the layer can be called a convolution layer in any case. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Preferred embodiments of the invention are specified in the dependent claims. Further advantages, features and characteristics of the invention are explained by the following description of preferred embodiments with reference to the accompanying drawings, in which:

[0027] Figure 1 A method according to an exemplary embodiment is shown;

[0028] Figure 2 shows a schematic exemplary image divided into image regions of different sizes;

[0029] Figure 3A shows applying a kernel to an image according to the prior art;

[0030] Figure 3B shows applying a kernel to an image using zero padding according to the prior art;

[0031] Figure 4A and Figure 4B shows applying a kernel to an image according to an exemplary embodiment;

[0032] Figure 5 An image for training a neural network according to an exemplary embodiment is shown;

[0033] Figure 6 A method for training a neural network according to an exemplary embodiment is shown;

[0034] Figure 7 An apparatus for processing an image in a neural network according to an exemplary embodiment is shown; and

[0035] Figure 8 A schematic exemplary method for the application of noise reduction is shown. DETAILED DESCRIPTION

[0036] Hereinafter, the same or similar reference numerals are used for the same or similar components or steps.

[0037] Figure 1 A method 100 for processing an image according to an exemplary embodiment is shown. In this case, the image is first divided into at least two image regions in step 110. The image regions can be of the same size, for example, 8x8 pixels, 16x16 pixels, or 32x32 pixels. Alternatively, the image regions can be of different sizes.

[0038] For example, an image can be divided into image regions by applying a decomposition algorithm or a segmentation algorithm. In this case, the algorithm used can operate in a pixel-oriented, edge-oriented, region-oriented, model-based, or texture-based manner.

[0039] Such algorithms are, for example, watershed transformation, thresholding methods, random walks, quadtrees, octrees, Felzenszwalb-Huttenlocher algorithm, Sobel operator, Laplacian operator, real-time line method, parallel or sequential edge extraction, optimal edge search, active shape models and snakes.

[0040] However, other methods for dividing an image into image regions can be envisaged, and the method explained above serves only as an example. In another embodiment, the image regions may overlap, i.e., there are regions belonging to more than one image region. This is described in more detail below.

[0041] As an example, Figure 2 The partitioning of an image into image regions resulting from the application of a quadtree is shown (H. Samet, "The Quadtree and Related Hierarchical Data Structures," 1984).

[0042] Subsequently, in step 120, the image regions are grouped. In doing so, a group is assigned to at least one image region. Image regions that are not assigned to a group may also be retained. This grouping is also known as classification, categorization, or clustering. No image region may be assigned to more than one group, so that each image region is assigned to only one group or no group.

[0043] For example, image regions are grouped based on one or more of their attributes, such that image regions with the same or similar attributes are assigned to the same group. The similarity metric can be varied to obtain more or fewer groups. Such standard classification techniques are known to those skilled in the art and will not be further explained here.

[0044] One or more attribute criteria of the image regions can be used to group the image regions. Examples of such attribute criteria are the intensity of the image region, the orientation of the intensity gradient within the image region, the color or color value of the image region, the structure within the image region, the edge orientation, and in the case of image regions of different sizes, the size of the image region can also be used for grouping. Other characteristics are also contemplated.

[0045] The methods used for classification can be numerical or non-numerical, statistical or distribution-free, monitored or non-monitored, permanent size or learned, and parametric or non-parametric.

[0046] Exemplary so-called clustering or classification methods are, for example, k-means, support vector machines (SVM) or Gaussian mixture models, the latter also being multivariate. However, other classification methods are known to those skilled in the art, and any suitable algorithm for clustering image regions may be used.

[0047] In a following step 130 , at least one kernel in a layer of the neural network is applied to at least one group of image regions, wherein each group of image regions is applied to at most one kernel.

[0048] Figure 3A The application of kernels to an image to be processed according to the prior art is shown. In this case, each kernel is applied to the entire image in a region.

[0049] By way of explanation, assume as an example an image 310 of size 8x8 pixels. Assume by way of example the following values ​​for the pixels:

[0050]

[0051]

[0052] The kernel has a size of 3x3. After each calculation step, the kernel is shifted to the right by one column. When the end of a row is reached, the kernel is applied to the next row. The size of the shift corresponds to the stride parameter in each case and can also differ from 1. With a stride parameter of 1, two kernels 320 of size 3x3 produce two result matrices 330 of size 6x6 pixels.

[0053] Here, each value of image 310 represents the value of an image point (i.e., pixel) of the image. Depending on the bit depth of the image, the value includes different value ranges. For example, if image 310 is in 8-bit grayscale, the pixel value ranges from 0 to 255.

[0054] Each field of the result matrix 330 (each field corresponds to a pixel of the image) is calculated as the sum of the products of the values ​​of the image 310 and the corresponding values ​​of the kernel 320, for example in a so-called cross-correlation.

[0055] exist Figure 3A In the example shown, each kernel starts at the beginning above the region in the upper left corner of the image 310. This results in the following values ​​of Table 1 for the result matrix:

[0056]

[0057] Table 1

[0058] The sum of the products is then input into the result matrix 330, which in the above case is 75 or 77.

[0059] Then, the kernel is shifted right by one column according to the stride parameter having a value of 1, which results in the following Table 2:

[0060]

[0061] Table 2

[0062] Therefore, the values ​​61 and 42 will be entered into the next field of the result matrix 330 .

[0063] Figure 3A The example shown in shows an image without so-called zero padding. Zero padding means that the edges of the image are surrounded by zero values. When a kernel with k columns and rows is applied, the size of the image is reduced by k-1.

[0064] Without zero padding and with a stride parameter of 1, the dimension of the resulting matrix is ​​reduced in the usual way as follows:

[0065] Results Matrix x =Input image x -(kernel x-1 ),or

[0066] Results Matrix y =Input image y -(kernel y-1 ).

[0067] where result matrix x / y, input image x / y, and kernel x / y represent the x or y dimensions of the result matrix, input image, and kernel, respectively.

[0068] If the number of columns and rows is increased by k-1 in each case before applying the kernel, the resulting matrix again has the same size as the input image. However, even in the case of zero padding, the method shown does not change, so that a person skilled in the art can also easily apply the method to images with zero padding. This is in Figure 3B Due to the additional values ​​at the edges of the image, the image increases to 10x10 pixels and the resulting matrix increases to 8x8 values, which in turn corresponds to the size of the input image.

[0069] Figure 4A and Figure 4B The cross-correlation of the method according to the invention is shown, wherein only one kernel is applied in each case to each of the divided image regions assigned to a group. Figure 4A , at the transition point in the figure marked with IV, from Figures 4A to 4B Form a connection.

[0070] For example, an image 410 having the same value as before is assumed as an input image.

[0071] 11 5 8 3 6 2 5 1 6 15 4 3 6 1 15 9 9 10 4 2 7 3 11 10 3 5 5 4 7 2 4 5 1 2 6 7 8 1 7 6 10 1 10 8 9 4 2 4 11 2 1 11 2 4 1 3 3 1 2 3 1 7 8 9

[0072] exist Figure 4A and Figure 4B In the example, the image regions have the same size, i.e., an original image 410 having 8×8 pixels is divided into four regions, each of which comprises 4×4 pixels. The division of the image 410 into image regions can be seen in image 415. For this example, the four regions are as follows.

[0073]

[0074] For example, Figure 4A and Figure 4B The four regions in are divided into two groups, where the upper left region and the lower right region are assigned to group 1, and the remaining regions are assigned to group 2.

[0075] The individual values ​​of image 410, in turn, represent the values ​​of the image points (i.e., pixels) of the image. Depending on the bit depth of the image, the values ​​include different ranges of values. If image 410 is, for example, 8-bit grayscale, the pixel values ​​range from 0 to 255. Furthermore, depending on the color model of the entire image, image 410 may represent only one channel of the entire image. Thus, for example, if the entire image is in the RGB color model, image 410 may represent only one of the red, green, or blue channels. In a similar manner, and as will be readily appreciated by those skilled in the art, the present method is also applicable to every other color space, such as YCMK, YCbCr, YUV, HSV. Depending on the color space, the corresponding value of the image point represents the intensity, brightness, or saturation of the color channel.

[0076] Since the 3x3 kernel 420 can be placed exactly 4 times in the 4x4 image region if a stride parameter of 1 is to be used within the corresponding image region, four values ​​for each image region produce the result matrix. The stride parameter can also specify different values.

[0077] By way of example, the following values ​​are used for the kernel:

[0078] Kernel 1:

[0079] 1 2 2 1 1 2 1 1 2

[0080] Kernel 2:

[0081] 1 1 1 1 1 1 2 2 2

[0082] In this case, each image region is individually correlated with a corresponding kernel, and the entire result matrix is ​​subsequently reconstructed from the individual results.

[0083] for Figure 4A and Figure 4BIn the example shown in FIG, the corresponding value results of the four image regions represent partial results of the entire result matrix 430. Figure 3A and Figure 3B Compared to the conventional method described in , the regions do not overlap, thus producing a result matrix of 4×4 pixels 430. In the proposed method, for image regions of the same size and a stride parameter of 1, the reduction in size relative to the input image can be calculated relatively easily as follows:

[0084] Results Matrix x =Input image x -(kernel x-1 )×Area x ,or

[0085] Results Matrix y =Input image y -(kernel y-1 )×Area y .

[0086] Where result matrix x / y, input image x / y, and kernel x / y represent the x or y dimension of the result matrix, input image, and kernel, respectively. x / y Indicates the size of the image area in the x or y direction.

[0087] The following applies: Total number of image areas = area x ×Area y .

[0088] For image regions of different sizes, the computation is slightly more complex, but the resizing of the input image can be achieved either by zero-padding of the regions or by object overlap (i.e., padding of adjacent regions).

[0089] In each case, the calculation of the partial results from the image regions and the associated kernels can be performed serially or simultaneously (ie in parallel).It is irrelevant whether the image regions are of equal or different sizes.

[0090] In order to obtain a result matrix of the same size as the input image, one can use a zero-padding method in the proposed method or increase the individual image regions accordingly by filling them with adjacent pixels.

[0091] When image areas overlap, the average value of those pixels for which more than one value exists is formed from all available values ​​for the corresponding pixels when the entire result matrix is ​​compiled.

[0092] The advantages become apparent with correspondingly larger image sizes. For example, with an input image of 128x128 pixels, assuming appropriate padding, 16 filters in a conventional method generate 16 result matrices of 128x128 pixels. Therefore, the total amount of data is 128x128x16.

[0093] However, by using the proposed method, regardless of the number of kernels, only one matrix of size 128x128 is always produced, again assuming corresponding padding, which means a data reduction by a factor of 16. In this way, with the same number of kernels, hardware resources can be saved, and this is especially true even if the data size of the result matrix is ​​the same as that of the input image.

[0094] Furthermore, the proposed method results in a reduction in computational workload since each kernel is applied to each pixel in conventional methods.

[0095] Furthermore, the proposed method can also have a favorable effect on image quality, where image regions of the same group are processed in the same way, but image regions of different groups are processed in another way. Thus, the structure of the input image can be better preserved.

[0096] This has advantages not only for denoising, but also for other tasks such as deblurring or inpainting. Furthermore, classification tasks can be accelerated by, for example, not processing homogeneous image regions, since typically only edges are often of interest.

[0097] In the following, we will explain the application of the proposed method to fluorescence microscopy noise reduction as another application. In this case, the main focus is on reducing hot pixels and dark noise, since according to the EMVA 1288 standard, they are directly dependent on the exposure time.

[0098] It is assumed here that the images to be corrected are recorded by a digital camera mounted on a fluorescence microscope.

[0099] In this case, a clustering algorithm is trained based on training examples so that it can classify image regions created according to the selected attributes. Thus, the created image regions can then be processed in another way by the network according to their label (i.e., which group they belong to).

[0100] The input image here is, for example, a grayscale image or a color image whose noise is to be reduced. The individual image regions are grouped according to a clustering algorithm. Optionally, a list of image coordinates for the individual image regions is provided. The input image is now processed by applying the proposed kernel 130 to the image regions. This results in smaller structures and edges.

[0101] Furthermore, before, during or after applying 130 the kernel to the image region, the input image is processed, for example, by a pooling layer, preferably by a median or minimum pooling layer, thereby reducing the number of hot pixels. The proposed application of the kernel can also be applied to the resulting image of the processing. In order to maintain the original image size, the input image or the resulting image can be processed by a corresponding upsampling layer. For example, a detailed description of the layers can be found in the Keras library documentation.

[0102] Subsequently, the actual application of the kernel and the resulting image from the pooling layer are combined in a merging layer. This can be performed, for example, by adding layers or averaging layers. Finally, other kernel applications among the proposed kernel applications can be applied.

[0103] The network can be trained using a corresponding dataset, which can consist of image pairs, each consisting of a noisy image and an associated noise-free image.

[0104] The network architecture can be trained in a so-called end-to-end approach.

[0105] During inference of the network, it is conceivable that image regions are assigned to specific cluster groups that are not available in the training data. As a result, the filters or kernels of the proposal layer assigned to the corresponding cluster group are not trained.

[0106] Figure 6 A method 200 for training a neural network is shown. Here, the network is trained on at least one image, a so-called test pattern. The test pattern contains desired feature properties, such as different edge orientations, colors or intensities. Figure 5 An exemplary test pattern is shown in .

[0107] In this case, an image divided into at least two image regions is first read in step 210. This division can be established in advance or can be performed in a layer of a neural network in which the kernel is also trained subsequently, in another layer of the neural network, or in another neural network.

[0108] This is followed in step 220 by grouping at least one of the image regions into at least one group, wherein each image region is assigned to only one group or to no group.

[0109] In this case, the grouping can also be established in advance or can be performed in a layer of the neural network in which the kernel is also trained subsequently, in another layer of the neural network, or in another neural network.

[0110] Subsequently, in step 230 , at least one kernel 420 is applied to at least one group of image regions in a layer of the neural network or in one of the neural networks, wherein at most one kernel is applied to each group of image regions.

[0111] The training is explained in more detail below with reference to the example denoising.

[0112] By using test patterns, the clustering algorithm can learn the desired cluster groups accordingly. Furthermore, it ensures that the kernel is trained for all cluster groups. To enable the network to learn denoising methods, for example, the test patterns are superimposed with the corresponding noise. Training with test patterns also prevents the network's weights from overfitting to certain types of examples, such as those in the training data.

[0113] The network thus learns to denoise, for example, for different edge orientations rather than for certain sample types. For example, if the network is trained on edge orientations, significantly less training data is required for this purpose than if the network is trained directly on microscopic samples.

[0114] For training, using the example of denoising, a noisy input image that has been divided into image regions that have also been clustered is fed to the neural network. To determine the success of the training, a corresponding noise-free image is provided. The neural network is then trained using the known backpropagation method. However, since the method according to the present invention applies the kernel only to (i.e., filters to) specific image regions in the image according to the clustering group in each case, this method has the advantage that training can be performed with only one test pattern. It is also possible to use several image pairs consisting of noisy and noise-free images, but in contrast to conventional training methods, a single image pair is sufficient.

[0115] The input for training is a noisy test pattern with grouped image regions. The so-called ground truth (i.e., the ideal resulting image) is the noise-free test pattern. Each filter, or kernel, is adapted using a backpropagation method so that, on average, it performs the best denoising for all image regions to which it is applied.

[0116] Because the image regions of a cluster group all have the same properties, such as the same edge orientation as described above, this filter learns to denoise the image regions that are correspondingly labeled in this property. Of course, this also applies to other filters.

[0117] It is also possible to extend an already existing neural network by the convolutional layer according to the present invention, or to replace an already existing layer in a neural network by the convolutional layer according to the present invention.

[0118] In this case, it is also possible to train only the layers according to the invention.However, in general, the network is usually trained as a whole, not just the layers in the network.

[0119] If the partitioning of the image and the grouping of image regions occur outside the network, corresponding clustering methods need to be pre-trained if they are used.

[0120] Different settings, such as camera gain and exposure time, produce images with varying degrees of noise. Therefore, it is conceivable that, after processing using a noise reduction method, an image with a high degree of noise, for example, may not achieve the same image quality as an image with a lower degree of noise. Therefore, if information about the camera gain and exposure time settings is available, different processing of the input image can be performed accordingly.

[0121] Furthermore, any input dimension is possible after the network has been trained. However, in the case of methods according to the prior art, the input dimension of the classic convolutional layer must be defined at the beginning of training.

[0122] Before applying 130 the at least one kernel to the group of image regions as just discussed, further processing steps 125 may optionally be performed. These further processing steps 125 may comprise conventional convolutions, i.e. applying a kernel according to the prior art, or different processing as already explained above, such as pooling, upsampling, etc.

[0123] However, step 125 may also include applying 130 at least one kernel according to the exemplary embodiment described above. In this case, dividing the image into image regions may be the same as step 130 or different.

[0124] After applying 130 the at least one kernel to the set of image regions as described above, further processing steps 135 may optionally be performed. These further processing steps 135 may include conventional convolutions known to those skilled in the art, but step 135 may also include applying 130 the at least one kernel according to the exemplary embodiments described above. In this case, dividing the image into image regions may be the same as or different from step 130.

[0125] The steps of partitioning 110, grouping 120, and possibly further processing steps 125 and / or 135 discussed above can in this case be performed in a layer of the neural network to which the at least one kernel is applied 130. However, it is also conceivable that some or all of these steps are performed in another layer of the same neural network or in another neural network. Furthermore, some or all of these steps can also be performed outside of the neural network. Furthermore, it is also possible to perform further steps 125, 135 at least partially in parallel with the application 130 of the at least one kernel, so that further processing steps 125, 135 can be performed before, during, or after the application 130 of the at least one kernel.

[0126] Similarly, the same applies to the training method 200 , wherein the steps of reading 210 and grouping 220 may be performed in a layer of the neural network to which the at least one kernel 230 is applied, in another layer of the neural network, in another neural network, or outside the neural network.

[0127] Other examples of processing steps that can be performed within the same layer are, for example, applying an activation function, such as ReLU, Softmax, or adding a bias value, or batch normalization. Outside the layer, in another layer or another network, any other processing by means of a layer, such as pooling, upsampling, or full connection (i.e., dense layer) can then follow.

[0128] If the partitioning and grouping steps in steps 125 and / or 135 differ from step 130, these differences may be that another algorithm is used for partitioning and / or grouping and / or other parameters are used for partitioning and / or grouping.

[0129] However, steps 125 and / or 135 may also include other processing steps of the image, for example, processing steps including, for example, preparatory measures such as cropping, color reduction, contrast changes, etc. Furthermore, it is also conceivable that, while the processing is running, other processing steps are performed in parallel with the kernel application 130. The results of the separate implementations may then be combined or otherwise processed separately.

[0130] Furthermore, the processed image can be output in a subsequent step 140. The output of the image does not differ from the output of images known to those skilled in the art.

[0131] Figure 7 A device 500 for processing images in a neural network is shown. This device comprises at least one memory 510, wherein at least one image is stored in the at least one memory. In addition, the device 500 comprises at least one processor 520, wherein the at least one processor is configured to perform the method for processing images in a neural network or the method for training a layer of a neural network as described above.

[0132] Furthermore, a computer-readable medium comprising instructions is described which, when executed by a processor, cause the processor to perform the steps of the method for processing images in a neural network or the method for training a layer of a neural network as described above.

[0133] Figure 8 The entire noise reduction method is illustrated schematically and by way of example. Here, an input image 810 is initially subdivided into image regions 820. Subsequently, the image regions are divided into cluster groups 1 to 100 using a previously trained clustering algorithm. This results in a labeled input image 830, here represented by groups 1 to 8. The noise-reduced result image 850 is then created in a neural network 840 using the method of the present invention. In this example, the division and grouping steps occur outside the convolutional layers of the neural network.

[0134] It is conceivable that these steps are previously performed in another layer of the same neural network, by another neural network or outside the neural network.

[0135] List of reference numerals

[0136] 100 methods for processing images

[0137] 110 Step: Divide

[0138] 120 Step: Grouping

[0139] 125 Other Steps

[0140] Step 130: Apply at least one kernel

[0141] 135 Other Steps

[0142] 140 Step: Output

[0143] 200 Methods for training neural networks

[0144] 210 Step: Read the image

[0145] 220 Step: Grouping

[0146] Step 230: Apply at least one kernel

[0147] 310 Input Image (Existing Technology)

[0148] 320 cores (existing technology)

[0149] 330 Results Matrix (Prior Art)

[0150] 410 Input Image

[0151] 415 The input image is divided into grouped image regions

[0152] 420 cores

[0153] 430 Results Matrix

[0154] 500 devices

[0155] 510 Memory

[0156] 520 processor

[0157] 810 Input Image

[0158] 820 image area

[0159] 830 The input image is divided into grouped image regions

[0160] 840 Neural Networks

[0161] 850 result images

Claims

1. A computer-implemented method (100) for processing an image (410) using a neural network, comprising: dividing (110) the image (410) into a plurality of image regions; grouping (120) at least two of the plurality of image regions into at least two groups based on one or more attributes of the plurality of image regions, wherein each of the plurality of image regions is assigned to only one group or to no group, and image regions having the same or similar attributes are assigned to the same group; computing partial results for the at least two groups of image regions by applying (130) at least two kernels (420) in a layer of the neural network to the at least two groups of image regions, wherein each group of image regions is applied with at most one kernel, and A result matrix of a layer of the neural network is generated based on the partial results of the at least two groups of image regions, wherein an amount of data of the result matrix is ​​less than an amount of data generated by applying each of the at least two kernels to each of the plurality of image regions.

2. The method (100) according to claim 1, wherein: The at least two kernels are convolution kernels.

3. The method (100) according to claim 1 or 2, in, performing other processing steps (125, 135) on the image before, during, or after applying (130) the at least two kernels; and wherein one or more of the further processing steps (125, 135) and one or more of the steps of dividing (110) and grouping (120) are performed in the layer of the neural network, in another layer of the neural network, in another neural network or outside the neural network.

4. The method (100) according to claim 1 or 2, in, performing further processing steps (125, 135) on the image before, during or after applying (130) the at least two kernels; or wherein one or more of the further processing steps (125, 135) and one or more of the steps of dividing (110) and grouping (120) are performed in the layer of the neural network, in another layer of the neural network, in another neural network or outside the neural network.

5. The method (100) of claim 1, further comprising: After the convolution (130), the processed image is output (140).

6. The method (100) according to claim 1, in, The dividing (110) of the image into a plurality of image regions divides the image into image regions of equal size.

7. The method (100) according to claim 1, in, The dividing (110) of the image into a plurality of image regions divides the image into image regions of different sizes; and / or Wherein, said dividing (110) said image into a plurality of image regions is performed by decomposition.

8. The method (100) according to claim 1, in, The grouping (120) groups image regions having the same attributes in each case into one group.

9. The method (100) according to claim 8, wherein: The same attribute of the image regions corresponds to at least one of the following criteria: orientation of intensity gradients; the color of the image area; the intensity of the image region; structures within the image region; edge orientation; and / or Image areas of the same size in case the image areas are of different sizes.

10. The method (100) according to claim 1, in, The grouping (120) groups the image regions by clustering or classification algorithms.

11. A computer-implemented method (200) for training a neural network, comprising: reading (210) an image divided into a plurality of image regions; grouping (220) at least two of the plurality of image regions into at least two groups based on one or more attributes of the plurality of image regions, wherein each of the plurality of image regions is assigned to only one group or to no group, and image regions having the same or similar attributes are assigned to the same group; and computing partial results for the at least two groups of image regions by applying (230) at least two kernels (420) in a layer of the neural network to the at least two groups of image regions, wherein each group of image regions is applied with at most one kernel, and A result matrix of a layer of the neural network is generated based on the partial results of the at least two groups of image regions, wherein an amount of data of the result matrix is ​​less than an amount of data generated by applying each of the at least two kernels to each of the plurality of image regions.

12. A device (500) for processing an image in a neural network and / or for training a neural network, comprising: at least one memory (510), wherein at least one image is stored in at least one of said memories; At least one processor (520), wherein at least one of the processors is configured to perform the method according to any one of claims 1 to 11.

13. A computer-readable medium comprising instructions which, when executed by a processor, cause the processor to perform the steps of the method according to any one of claims 1 to 11.

Images