Method for creating a reconstructed image, computer program product, and motor vehicle

Abstract

A method for creating a reconstructed image based on an input image using a generative algorithm is described, wherein the input image is incomplete, the generative algorithm uses a model that has at least three layers, a feedback connection between two adjacent layers is provided, and a hyperparameter is provided that defines a strength of a feedback connection of the feedback loop, and the hyperparameter is adjusted.

Description

[0001] A method for creating a reconstructed image, a computer program product, and a motor vehicle are described.

[0002] Methods for creating a reconstructed image, computer program products and motor vehicles of the type mentioned at the beginning are known in the prior art.

[0003] Modern vehicles increasingly employ camera and image processing systems in a wide range of applications, from safety measures to entertainment functions. A specific problem in processing camera images is the occurrence of image noise, as well as the presence of blind spots or obscured areas caused by the camera position, lens contamination, and / or structural limitations of the vehicle.

[0004] Noise problems are mostly caused by the hardware, as the necessary miniaturization of corresponding camera systems, combined with sometimes dark environments, pushes the hardware to its physical limits, since the signal-noise ratio of the sensors used is unfavorable with the sometimes very low amount of light available and small sensor volumes.

[0005] These problems result in important visual information being missing or distorted, leading to limitations in image quality and completeness, especially in 360-degree camera systems and indoor cameras.

[0006] Boutin et al., PLOS Computational Biology, 2021, reveals a hierarchical image processing model based on the principles of predictive coding and organized into two layers. Feedback connections are provided between these layers, the strength of which is determined by a parameter k. FBis defined. This model serves to investigate the effects of top-down feedback on contour integration and image reconstruction.

[0007] A system for image reconstruction is known from US patent 2024 / 135604 A1, comprising an input for receiving image data, a processor, and a memory. The memory stores instructions that instruct the processor to reconstruct an image from the image data using a self-monitoring deep learning model.

[0008] A disadvantage of the known methods is that they are very computationally intensive and inefficient. They are therefore unsuitable for energy-saving use in a motor vehicle. Furthermore, the methods cannot be adjusted during operation.

[0009] The task therefore is to further develop methods for creating a reconstructed image, computer program products and motor vehicles of the aforementioned type in such a way that fast and energy-saving image reconstruction methods can be used in motor vehicles.

[0010] The problem is solved by a method for creating a reconstructed image according to claim 1, a computer program product according to dependent claim 8, and a motor vehicle according to dependent claim 9. Further embodiments and developments are the subject of the dependent claims.

[0011] This document describes a method for creating a reconstructed image based on an input image using a generative algorithm. The input image is incomplete, and the generative algorithm uses a model with at least three layers. A feedback connection is provided between two adjacent layers, and a hyperparameter is defined that determines the strength of the feedback connection within the feedback loop. The hyperparameter is adjusted as needed. In this context, "incomplete" means an image whose content is not clearly recognizable due to image noise, obscured image areas, or other image defects.

[0012] A generative image optimization algorithm with at least three layers and a feedback loop allows for the improvement of input images by capturing deeper structures and features in the input image data. Each of the three layers extracts different features from the image. The result is the ability to learn comprehensive abstractions of the visual data in deeper layers, while the earlier layers specialize in learning local relationships. The integration of a feedback loop within the model enables adaptive adjustment of the learning processes by dynamically adjusting the bias intensity in each layer based on the feedback signal from the deeper layers.

[0013] The layers themselves can have different architectures, for example Convolutional Neural Networks (CNNs) or other deep neural networks.

[0014] The feedback loop allows the algorithm to use the output of one layer to improve the input for an earlier layer. This helps increase the accuracy and quality of image optimization by correcting errors and refining feature extraction. With less feedback from deeper layers, correspondingly less is generated, resulting in more precise and context-aware image reconstructions.

[0015] The hyperparameter mentioned, which defines the strength of a feedback connection in the feedback loop, determines, in practical application of the method, how much new image content is generated. Adjusting this hyperparameter allows the desired result to be achieved for each specific use case.

[0016] Furthermore, it is planned that the image quality of the input image will be determined, whereby the hyperparameter will be adjusted based on the determined image quality.

[0017] Such automatic adjustment of the hyperparameter can occur depending on the need for image reconstruction due to noise.

[0018] The algorithmic determination of the amount of image noise in an image can be achieved using various methods. A common method is calculating the signal-to-noise ratio (SNR), which determines the ratio of signal strength to noise strength. A higher SNR indicates less noise in the image. Another method is histogram analysis, which analyzes the histogram of an image. An image with a lot of noise typically shows a wider histogram because the pixel values ​​vary more. A Fourier transform can also be used to analyze the frequency spectrum of an image. Noise often appears as high-frequency components in the spectrum. By analyzing these components, the noise component can be determined. Another possibility is calculating the variance of the pixel values ​​in an image. Higher variance indicates more noise.Finally, there are various other mathematical models that describe noise in images, such as Gaussian noise. These models can be used to determine or estimate the noise component.

[0019] These methods can be used individually or in combination to obtain an accurate estimate of the noise content in an image.

[0020] In a first, more advanced version, it is planned that the hyperparameter will be adjusted by a user.

[0021] In this way, the user, for example a driver or passenger of the motor vehicle, can adjust the strength of the reconstruction by setting the hyperparameter accordingly until a satisfactory image impression is achieved.

[0022] In a further refinement, it is envisaged that an LCA algorithm with an inhibitory feedback connection is applied in each of the layers.

[0023] A locally competitive algorithm (LCA) is an algorithm that aims to simulate local competition mechanisms in neural networks. The functionality of such an algorithm can be described as follows: The algorithm begins with the initialization of neurons and the establishment of a network structure. Each neuron has a specific activation threshold and is connected to other neurons. The connections between neurons can be either excitatory or inhibitory, meaning they can either promote or suppress the activation of other neurons.

[0024] First, the input data is fed into the network, and the neurons calculate their activation values ​​based on the received signals. Each neuron sums the incoming signals and compares the sum to its activation threshold. If the sum exceeds the threshold, the neuron is activated and sends signals to its connected neurons.

[0025] During this process, local competition occurs between neurons. Neurons that are highly activated suppress the activation of neighboring neurons through inhibitory signals. This results in only the most highly activated neurons maintaining their activation, while the other neurons are suppressed. This mechanism allows the network to respond to the most relevant input patterns and ignore irrelevant information.

[0026] The algorithm continues this process by continuously updating the activation values ​​of the neurons and adjusting the inhibitory signals. This results in a stable activation pattern in the network that reflects the key features of the input data.

[0027] An inhibitory feedback connection is a mechanism in neural networks that serves to suppress the activation of certain neurons. In a locally competitive algorithm (LCA), this connection plays an important role in improving the efficiency and accuracy of the network.

[0028] When a neuron is highly activated, it sends inhibitory signals to neighboring neurons. These signals suppress the activation of neighboring neurons, reducing the likelihood of them also becoming highly activated. This creates a local competition in which only the most relevant neurons maintain their activation, while others are suppressed.

[0029] This mechanism helps the network respond to the most important input patterns and ignore irrelevant information. The inhibitory feedback loop thus ensures that the network operates more efficiently and that resources are used optimally.

[0030] In a further refinement, the algorithm is provided to have a plurality of neurons, each neuron of the algorithm having its own activation threshold, the activation threshold being defined by training the algorithm, whereby the activation threshold is determined during the training of the algorithm by increasing it for frequently activated neurons and decreasing it for less active neurons.

[0031] This approach allows frequently activated neurons to be relieved of their workload, while less active neurons are more actively involved in the process. This leads to more effective removal of image noise and, at the same time, increases the efficiency and speed of the algorithm and thus the reconstruction of image data, allowing it to operate with less computing power.

[0032] In a further, more advanced version, it is envisaged that a feedback connection exists between all neighboring layers.

[0033] This enables the algorithm to learn comprehensive abstractions of the visual data in deeper layers, while the earlier layers specialize in learning local relationships. Integrating a feedback link between all neighboring layers of the model allows for adaptive adjustment of the learning processes by dynamically adjusting the bias intensity in each layer based on the feedback signal from the deeper layers. With less feedback from the deeper layers, correspondingly less is generated, resulting in more precise and context-aware image reconstruction.

[0034] In a further refinement, the model is intended to be a sparse coding model.

[0035] Sparse coding is a technique that aims to efficiently represent data as a linear combination of a small number of basis vectors. The model is based on a supercomplete basis, where the number of basis vectors is greater than the dimension of the data. For a given signal, a coefficient vector is calculated that is as sparse as possible, meaning that only a few basis vectors are used to reconstruct the signal.

[0036] The functionality of sparse coding consists of two main steps: determining the sparse coefficients and optimizing the basis vectors. First, a sparse representation is found for a given signal. In the second step, the basis vectors are updated, assuming the coefficients are fixed, to improve the reconstruction. This iterative process ensures that both the basis and the coefficients are continuously improved.

[0037] One advantage of sparse coding is the efficiency of data representation. Because only a few basis vectors are used to describe a signal, the resulting representation is compact and stores only the most important information. This leads to lower memory requirements and reduced computational overhead. Furthermore, the sparse representation can be more robust against noise, as it focuses on the relevant features of the data and ignores unimportant or distracting information.

[0038] Additionally, sparse coding offers better interpretability compared to other methods because the basis vectors do not need to be orthogonal and can recognize more flexible patterns in the data. Particularly with high-dimensional and structured data such as images or audio, sparse coding can extract useful latent features that reflect the data structure, thus leading to better model results.

[0039] In a further, more advanced embodiment, it is envisaged that the model is trained with a constant hyperparameter.

[0040] This reduces the training effort and surprisingly leads to good results in practice.

[0041] A first independent subject matter relates to a computer program product comprising a computer-readable storage medium on which instructions are embedded which, when executed by at least one computing unit, cause that at least one computing unit to be equipped to execute the procedure of the aforementioned type.

[0042] The process can be executed on one or more computing units, so that certain process steps are executed on one computing unit and other process steps on at least one other computing unit, whereby calculated data can be transmitted between the computing units if necessary.

[0043] Another independent item relates to a motor vehicle with a computer program product of the type described above, wherein the computing unit is neuromorphic.

[0044] A corresponding computing unit features a neuromorphic processor. A neuromorphic processor is a special type of processor designed to mimic the workings of the human brain. Unlike conventional processors based on the von Neumann architecture, a neuromorphic processor uses an architecture based on neural networks and synaptic connections. This architecture allows the processor to process information in a way that closely resembles the operation of biological neurons.

[0045] One advantage of neuromorphic processors is their high energy efficiency. Because the neurons operate in an event-driven manner, they only consume energy when a signal transmission actually takes place.

[0046] Furthermore, neuromorphic processors are particularly suitable for the use of LCA algorithms.

[0047] Further advantages, features, and details will become apparent from the following description, in which – possibly with reference to the drawing – at least one embodiment is described in detail. Identical, similar, and / or functionally equivalent parts are marked with the same reference numerals.

[0048] They show schematically: Fig. 1. the process of reconstructing an image using a generative algorithm; Fig. 2. Training the generative algorithm Fig. 1, as well as Fig. 3. A method for gesture recognition using the generative algorithm.

[0049] Fig. Figure 1 shows a motor vehicle 0 with a computing unit 2.

[0050] In processing unit 2, an algorithm 4 is executed, which has a generative model 4.1 to reconstruct an input image 6 and generate an output image 6'. The output image 6' is partially restored and otherwise denoise-free, so that a complete and clear image impression is obtained.

[0051] Generative model 4.1 uses a deep hierarchical sparse-coding model with a feedback mechanism, which in this case has three layers 4.1a to 4.1c. Feedback is provided between layers 4.1a to 4.1c, adjustable via a hyperparameter β. By adjusting the hyperparameter β, the intensity of the feedback to the earlier layers 4.1a and 4.1b can be varied, thus changing the degree of generative improvement or enhancement of the input image 6.

[0052] The hyperparameter β can be set via an image analyzer 8, which analyzes the input image 6 with regard to noise and, if necessary, completeness.

[0053] On the other hand, this can also be done via user settings 10. A corresponding user interface is provided in the vehicle 0 for this purpose. Such an interface can, for example, be integrated into a central infotainment system of the vehicle 0 and allow the passengers or the driver to operate the functions of algorithm 4. By varying the hyperparameter β, the intensity of the image reconstruction can be modified according to needs and preferences. Such a setting is particularly useful in situations where different visual information is required or when specific image areas require special attention.Furthermore, the user interface can offer the option to switch the process on or off, giving the user the flexibility to activate or deactivate the image processing functions as needed, for example, depending on driving conditions or personal preferences. An intuitive menu can be provided for this purpose, displaying the available options and the current status of the algorithm and its settings.

[0054] Algorithm 4 can be mathematically described as follows: ul=ul+1τ(Dlx−ul−γlGl−βl(γl−Dl+1Tγl+1))γl=Tλ^l(ul) Tα(x)={x,if x≥α0,otherwise , where β Hyperparameter Weight G Similarity matrix γ output L Loss function λ Activation threshold u internal state τ LCA time constant T threshold function x Input

[0055] Matrices are shown in bold, vectors are marked with a vector symbol x̂, and scalar values ​​are described in normal font. Variables indexed with I refer to the older layer of the model.

[0056] By using feedback connections between all neighboring layers, a greater specialization of the different layers can be achieved.

[0057] Fig. Figure 2 shows the training process for algorithm 4.

[0058] Training data 20 is used to train model 4.1. For this purpose, a sparse representation of an image is determined from the training data, and the weights D and activation thresholds λ are adjusted. The procedure is applied with a fixed hyperparameter β.

[0059] The goal of model training is to minimize a loss function L: L(Dl,γl)=12‖γl−1−DlTγl‖22+λ^l‖γl‖0+βl∗12‖γl−Dl+1Tγl+1‖22

[0060] To achieve this, suitable weights D and activation thresholds λ must be found.

[0061] During training, the activation thresholds λ are dynamically adjusted, as these are essential for training further deep layers 4.1b and 4.1c. The activation thresholds λ are adjusted based on the individual activations of the neurons in algorithm 4. Thus, activation thresholds λ are increased for frequently activated neurons and decreased for less frequently activated neurons. This leads to a more even distribution of activation rates among the individual neurons, since an increase in the activation threshold λ results in the neuron being activated less frequently. Consequently, model 4.1 utilizes all neurons with more or less equal intensity, contributing to the model's greater efficiency. Such an adjustment of the activation thresholds λ can be implemented as follows: λ^li=λ^li+{ε,if too many activations of the neuron−ε,if too few activations of the neuron0,otherwise , where ε Increment.

[0062] During subsequent normalization, the weights D are transformed so that they have a mean of 0 and a variance of 1.

[0063] Fig. Figure 3 shows a flowchart of a possible application of the procedure.

[0064] Due to the generative image enhancement, the method is particularly suitable for non-critical applications, e.g. gesture recognition in motor vehicles, although in some situations, e.g. during night driving or in a tunnel or parking garage, very poor lighting conditions prevail, so that images taken with an interior camera of the motor vehicle 2 are very noisy.

[0065] The procedure involves first capturing image data with the interior camera and forwarding it to the computing unit 2, where algorithm 4 is applied to the image data from the interior camera.

[0066] By automatically determining the noise in the image analyzer 8, an automatic adjustment can be provided, whereby, depending on the specific design, a user may have the option to deactivate the automatic adjustment and make a manual adjustment or to deactivate the corresponding algorithm.

[0067] With the improved output images (6'), gesture recognition then takes place. If the gesture is recognized, a corresponding function is activated, e.g., reducing the volume of an audio output in the vehicle interior.

[0068] Then the procedure regarding the gesture is terminated and can begin again.

[0069] Although the invention has been further illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art without departing from the scope of protection of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned as examples are truly only examples and are not to be understood in any way as limiting, for example, the scope of protection, the possible applications, or the configuration of the invention.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims and their legal equivalents, such as a further explanation in the description. Reference symbol list 0 motor vehicle 2 Computing unit 4 Algorithm 4.1 Model 4.1a - 4.1c Layer 6 Entrance image 6' Output image 8 Image analyzer 10 User settings 20 training data β Hyperparameter Weight ε Increment G Similarity matrix γ output L Loss function λ Activation threshold u internal state τ LCA time constant T threshold function x Input

Claims

[1] Method for creating a reconstructed image (6') based on an input image (6) using a generative algorithm (4), wherein the input image (6) is incomplete, the generative algorithm using a model (4.1) having at least three layers (4.1a, 4.1b, 4.1c), a feedback connection between two adjacent layers (4.1a, 4.1b, 4.1c) is provided, a hyperparameter (β) is provided which defines a strength of a feedback connection of the feedback loop, the hyperparameter (β) being adjusted, characterized by , that an image quality of the input image (6) is determined, whereby the hyperparameter (β) is adjusted on the basis of the determined image quality. [2] Method according to claim 1, characterized by , that the hyperparameter (β) is adjusted by a user. [3] Method according to any of the preceding claims, characterized by, that in each of the layers (4.1a, 4.1b, 4.1c) an LCA algorithm with an inhibitory feedback connection is applied. [4] Method according to any of the preceding claims, characterized by , that the algorithm (4) has a plurality of neurons, wherein each neuron of the algorithm (4) has its own activation threshold (λ), wherein the activation threshold (λ) is defined by training the algorithm (4), wherein the activation threshold (λ) is determined during the training of the algorithm (4) by increasing it for frequently activated neurons and decreasing it for less active neurons. [5] Method according to any of the preceding claims, characterized by , that there is a feedback connection between all adjacent layers (4.1a, 4.1b, 4.1c). [6] Method according to any of the preceding claims, characterized by , that the model is a sparse coding model (4.1). [7] Method according to any of the preceding claims, characterized by , that the model (4.1) is trained with a constant hyperparameter (β). [8] Computer program product comprising a computer-readable storage medium on which instructions are embedded which, when executed by at least one computing unit (2), cause the at least one computing unit (2) to be configured to execute the method according to any of the preceding claims. [9] Motor vehicle with a computer program product according to claim 8, wherein the computing unit (2) is neuromorphic.

Images