Method, computer program and end-to-end object detector module for detecting at least one object in an environment with adverse weather conditions, machine-readable storage medium and system

Abstract

A method for detecting at least one object in an environment with adverse weather conditions is specified, the method comprising: receiving an image (10) from an image sensor path (V1); encoding the image (10) with an encoder (20, 21-26) and obtaining at least one latent embedding of the image (10) comprising encoded features of the image (V2); forwarding the image (10) together with the latent embedding through at least one generative machine learning model (30, 31-36), wherein the generative machine learning model (30, 31-36) is configured to denoise effects caused by the adverse weather conditions in the image (10), and obtaining at least one denoised image (11) (V3); processing the denoised image (11) with an object detector (40) and obtaining detected objects (V4).

Description

[0001] The present disclosure relates generally to the field of advanced driver assistance systems (ADAS) and autonomous driving (AD), in particular to a method and a system for detecting traffic objects in weather conditions that are unfavorable with respect to image-based object detection, e.g., in haze or fog.

[0002] Approximately 1.3 million people die annually as a result of traffic accidents, and 20 to 50 million suffer non-fatal injuries. Of these accidents, 22% are caused by adverse weather conditions such as fog, rain, etc. Typically, an ADAS system is equipped with an object detection model to detect objects in the scene. However, state-of-the-art object detection models fail to detect objects under adverse weather conditions. Incorrect detection can lead to a fatal traffic accident. Detection in adverse weather conditions represents one of the most significant challenges for AD systems. Many efforts have been made to mitigate these problems.

[0003] US 2022 414 838 A1 discloses a method and system for image dehazing based on CycleGAN. CN 115 689 932 A discloses a deep neural network-based image defogging method. CN 110 544 213 A discloses an image defogging method based on global and local feature fusion. US 2022 210 350 A1 is directed to an image dehazing method and an image dehazing device using the method. AU 2020 100 274 ​​AD discloses a multi-scale feature fusion network based on GANs for dehazing. CN 111 915 530 A discloses an image dehazing method for haze reduction using an end-to-end adaptive neural network.

[0004] Numerous attempts have been made to mitigate the effects of adverse weather conditions such as rain, haze, etc. However, as of the current state of the art, there is no concrete end-to-end system that performs both minimization and detection of adverse weather conditions such as fog, haze, etc. in a scene within the same framework.

[0005] One object of the invention was to integrate the mitigation of adverse weather conditions, e.g., denoising, haze reduction, rain removal (deraining), etc., with object detection in a single framework.

[0006] The subject matter of the independent claims solves this problem through a system comprising three subsystems: an encoder, a denoising network, and an object detector. The encoder encodes the image features into a latent space, which is used by the denoising network in combination with the input image to learn the noise distribution due to adverse weather conditions. The denoising network uses a set of generative subnetworks to learn the noise distribution and thus generate the enhanced image. Finally, the enhanced image is passed to an object detector to detect the objects present in the scene.

[0007] Specific embodiments of the invention are disclosed in the specification of the claims, definitions and in the accompanying figures.

[0008] In a first aspect, a method is provided for detecting at least one object in an environment with adverse weather conditions.

[0009] The method includes receiving an image from an image sensor path. The image sensor path includes hardware and / or software components. The image sensor path is, for example, the image sensor path of a camera module, which includes a camera sensor, optical components, and / or imager software for data preprocessing. The image sensor can be a camera sensor, a radar sensor, or a lidar sensor. The image is obtained under adverse weather conditions, where objects are not as clearly visible as under favorable conditions.

[0010] The method further comprises encoding the image with an encoder and obtaining at least one latent embedding of the image comprising encoded features of the image. According to one aspect, the encoder comprises an artificial neural network. The encoder has learned an efficient encoding of the image data, meaning that the encoder automatically finds the representations required for feature detection.

[0011] The method further comprises forward-feeding the image together with the latent embedding through at least one generative machine learning model. Parallel processing of the image with the latent embedding eliminates the need for systems with discriminator or decoder networks, as used in the prior art. The generative machine learning model is configured to denoise effects in the image caused by adverse weather conditions. The at least one generative machine learning model is used for all denoising tasks, e.g., defogging, haze reduction, etc. Denoising may include, for example, restoring image brightness, removing reflections, transforming intensity values ​​of the image data, and other image operations. At least one denoised image is obtained.Generally, the generative machine learning model learns the patterns and structure of the input training data and then generates new data with similar features. More specifically, the generative machine learning model used generates denoised images with similar properties to the noisy input images.

[0012] According to one aspect, the generative machine learning model uses a diffusion model. Diffusion models, such as those described in Chapter 2.1 of ar-Xiv:2209.04747v3, are a class of probabilistic generative models that learn to reverse a process that gradually degrades the training data structure.

[0013] The method further comprises processing the denoised image with an object detector and obtaining detected objects. According to one aspect, the object detector comprises a machine learning model, e.g., an artificial neural network, trained to detect, classify, localise, track, or segment objects in image data. This subsequent processing of the obtained denoised image with an object detector realizes an end-to-end adverse weather object detection system that can be connected to any object detection model and trained end-to-end with images taken under adverse weather conditions such as fog, haze, or rain.

[0014] The method can be computer-implemented.

[0015] In a second aspect, a computer program is provided for detecting at least one object in an environment with adverse weather conditions. The computer program comprises instructions that, when loaded or executed by the computer, cause the computer to perform the operations according to the method of any preceding claim.

[0016] The computer program code for executing the instructions may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, procedural programming languages ​​such as C, and hardware programming languages, e.g. for programming FPGA (field programmable gate array) chips.

[0017] The program code can be executed entirely on the computer as a standalone software package, on a standalone hardware unit, or on a remotely located computer, e.g., a cloud server. According to one aspect, the computer program is transmitted via a data signal using software-over-the-air technology.

[0018] In a third aspect, a machine-readable storage medium having instructions stored thereon is provided. The instructions are executable by a machine to cause the machine to perform the operations according to the disclosed method.

[0019] The machine-readable storage medium, e.g., a computer-readable storage medium, may be a computer diskette, a hard disk, a RAM (random access memory) device, a ROM (read-only memory) device, an EPROM or Flash (erasable programmable read-only memory) device, an optical or magnetic storage device. In one aspect, the medium may comprise any non-transitory medium capable of containing, storing, transmitting, distributing, or transporting the program for use by or in connection with the instruction execution engine.

[0020] In a fourth aspect, an end-to-end object detector module is provided for detecting at least one object in an environment with adverse weather conditions. The end-to-end object detector module comprises an encoder and at least one generative machine learning model configured to denoise effects in an image received from an image sensor path, the effects being caused by the adverse weather conditions. The end-to-end object detector module further comprises an object detector, the object detector receiving a denoised image from the generative machine learning model and being configured to detect objects. According to one aspect, the encoder, the generative machine learning model, and / or the object detector are as disclosed in the present specification.

[0021] According to another aspect, the end-to-end object detector module comprises the disclosed SLD-OD or MLD-OD.

[0022] In a fifth aspect, a system for a civil road vehicle, a public transport system, or a public transport infrastructure element is provided. The system comprises at least one image sensor, an end-to-end object detector module, and a transmission module. The end-to-end object detector module comprises an encoder, at least one generative machine learning model configured to denoise effects in the image caused by adverse weather conditions, and an object detector. The system is configured to detect objects according to the disclosed method. The transmission module is configured to transmit a control signal, in response to the detected objects, to a driving control unit of the road vehicle or the transport system and / or to a control unit of the public transport infrastructure element that is in signal transmission communication with the driving control unit.The civilian road vehicle can be a passenger car. The public transport system can be a shuttle system for passenger transport. The road vehicle and / or the transport system can operate autonomously.

[0023] The public transport infrastructure element may, for example, be a traffic light installed along a public road, e.g., at an urban intersection. According to one aspect, the public transport infrastructure element comprises a vehicle-to-all communication module that, in response to the detected objects, transmits at least one control signal to a vehicle-to-all communication module of the road vehicle or the transport system. The road vehicle or the transport system comprises a vehicle-to-all communication module that receives at least the control signal from the transport infrastructure element. The vehicle-to-all communication module of the road vehicle or the transport system may be in signal transmission communication with the driving control unit.

[0024] The control unit can be any control unit, e.g. an electronic control unit (ECU, area or zone ECU, e.g. ADAS / AD area / zone ECU) that controls a driving function, e.g. longitudinal and / or lateral control, drive, etc. With the help of the transmission module, therefore, the civil road vehicle, the public transport system

[0025] The end-to-end object detector module may include central processing units (CPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or tensor processors and other matrix arithmetic processors. The end-to-end object detector module may also include specific software components to implement the disclosed method.

[0026] According to one aspect, the generative machine learning model is configured to denoise effects resulting from rain, fog, haze, snow, clouds, or sunshine.

[0027] According to another aspect, an individual generative machine learning model is used for each denoising effect. The individual generative machine learning models can be subnetworks of the generative machine learning model. Each subnetwork attempts to learn, or has learned, slightly different noise distributions and thus produces an improved image related to the specific denoising task. The outputs of the individual generative machine learning models are aggregated, and the resulting aggregated image is processed by the object detector. This configuration can be referred to as single-level denoiser-based object detection (SLO-OD). According to one aspect, the aggregated outputs are normalized.

[0028] Alternatively, the individual generative machine learning models are processed sequentially, with the output of one individual generative machine learning model being sent as input to the next individual generative machine learning model, and the output of the last individual generative machine learning model being processed by the object detector. Progressive processing increases the number of features available to the denoising subnetworks during learning.

[0029] According to another aspect, the encoder comprises a feedforward network and the latent embedding obtained from the last layer of the encoder is fed forward through the generative machine learning model.

[0030] According to another aspect, an individual generative machine learning model is employed for each denoising effect. The encoder comprises a feedforward network with multiple layers. The individual generative machine learning models are processed sequentially, with an output from one of the individual generative machine learning models being sent as input to the next of the individual generative machine learning models, and the output of the last of the individual generative machine learning models being processed by the object detector. Some of the individual generative machine learning models are fed with latent embeddings from different layers of the encoder, or in other words, with intermediate features from the encoder.The use of latent embeddings from different layers or stages of the encoder provides multiple scales of features to individual generative machine learning models, also called denoising subnetworks. Each feature has a different impact on different denoising operations. This configuration can be referred to as multi-level denoiser-based object detection (MLD-OD).

[0031] According to another aspect, an individual generative machine learning model is employed for each denoising effect. The encoder comprises a feedforward network with multiple layers, where the number of layers corresponds to the number of individual generative machine learning models. The individual generative machine learning models are processed sequentially, with an output of one of the individual generative machine learning models being sent as input to the next of the individual generative machine learning models, and the output of the last of the individual generative machine learning models being processed by the object detector. Each of the individual generative machine learning models is fed with latent embeddings from different layers of the encoder.

[0032] The invention described above will be explained by reference to specific embodiments illustrated in the accompanying figures, in which: Fig. 1 is a diagram of an example object detection based on a single-stage denoiser; Fig. 2 is a diagram of an exemplary object detection based on a multi-stage denoiser; Fig. 3 is a flowchart of an embodiment of the disclosed method; Fig. 4 shows an embodiment of the disclosed system.

[0033] Fig. 1 shows an exemplary embodiment of the end-to-end object detector module 60 in a first configuration of the invention, a so-called single-stage denoising-based object detection (SLD-OD). SLD-OD comprises an encoder 20, a denoising network 30, and an object detector 40. In this configuration, the encoder 20 embeds an image 10 in a latent space. In this way, coded features of the input image 10 are obtained. The encoder 20 could be a special-purpose image processing encoder. The image 10 is an input image for the encoder 20. The image 10 can be an image received from a camera sensor, RADAR, or LIDAR. The final output of the encoder 20, along with the original image 10, is forwarded to each denoising subnetwork 31-36 of the denoising network 30.Each denoising subnetwork 31-36 attempts to learn slightly different noise distributions and thereby learns to enhance the image 10. The outputs of the denoising subnetworks 31-36 are aggregated and normalized to obtain a clear, enhanced image 11. The enhanced image 11 is an output image of the denoising network 30 and an input image for the object detector 40. The resulting enhanced image 11 is then passed to the object detector 40, which performs detection on the enhanced image 11, i.e., the denoised image. An image 50 with detected objects is obtained in which detected objects can be highlighted and, for example, provided with bounding boxes.

[0034] The encoder 20, the denoising network 30 and / or the object detector 40 may be separate artificial neural networks, and in particular deep neural networks. In the example of Fig. 1 and Fig. 2, the denoising network 30 is a generative machine learning model. In one aspect, the denoising network 30 is based on a U-net architecture that allows for dimension preservation. The object detector 40 may comprise a machine learning model, e.g., an artificial neural network, trained to perceive, recognize, classify, segment, localize, and / or track objects.

[0035] For example, the denoising network 30 is based on a generative model that removes fog, haze, rain, etc. from the given image 10. The denoising network 30 can, for example, be based on a denoising diffusion model. Diffusion models, such as those described in Chapter 2.1 of arXiv:2209.04747v3, are a class of probabilistic generative models that learn to reverse a process that gradually degrades the training data structure. The training procedure thus comprises two phases: the forward diffusion process and the backward denoising process. The first phase consists of several steps in which weak noise is added to each input image, with the strength of the noise varying at each step. The training data is gradually destroyed until it becomes pure Gaussian noise. The second phase consists of reversing the forward diffusion process.The same iterative procedure is applied, but in reverse: the noise is removed sequentially, restoring the original image. Therefore, at the time of inference, images are generated by gradually reconstructing them from white random noise. The noise subtracted at each time step is estimated using the denoising network 30.

[0036] In the example of Fig. 1, the denoising network 30 comprises six denoising subnetworks 31-36. In other embodiments, the denoising network 30 comprises fewer or more denoising subnetworks. In the example of Fig. 1, denoising subnetwork 31 could be used to denoise rain effects from image 10. Denoising subnetwork 32 could be used to denoise fog effects from image 10. Denoising subnetwork 33 could be used to denoise haze effects from image 10. Denoising subnetwork 34 could be used to denoise snow effects from image 10. Denoising subnetwork 35 could be used to denoise cloud effects from image 10. Denoising subnetwork 36 could be used to denoise any other adverse weather conditions from image 10. In other embodiments, denoising network 30 comprises a single generative network for all denoising tasks, e.g., rain removal, defogging, etc.

[0037] Fig. 2 shows an exemplary embodiment of the end-to-end object detector module 60 in a second configuration of the invention, a so-called multi-stage denoising-based object detection (MLD-OD). The MLD-OD comprises an encoder 20, the denoising network 30, and the object detector 40. In SLD-OD, the denoising network 30 is fed the latent embedding received from the last layer of the encoder 20, which can limit the range of features for the denoising subnetworks 31-36 to learn. In contrast to SLD-OD, in MLD-OD, noisy images are gradually improved by sending the output of one denoising subnetwork as input to the next denoising subnetwork. Furthermore, the individual denoising subnetworks 31-36 are also guided by the features obtained from the various encoder layers 21-26 of the encoder 20.Using embeddings from different stages of encoder 20 provides denoising subnetworks 31-36 with multiple feature scales, with each image having a different impact on various denoising operations. The enhanced image 11 is passed to the object detector network 40 for object detection.

[0038] Fig. 3 shows an embodiment of the method for detecting at least one object in an environment with adverse weather conditions. In a first step V1 of the method, an image 10 is received from an image sensor path. In a second step V2 of the method, the image 10 is coded with an encoder 20, 21-26, and at least one latent embedding of the image 10 with coded features of the image is obtained. In a third step V3 of the method, the image 10, together with the latent embedding, is passed through at least one generative machine learning model 30, 31-36. The generative machine learning model 30, 31-36 is configured to denoise effects in the image 10 caused by the adverse weather conditions. At least one denoised image 11 is obtained. In a fourth step V4 of the method, the denoised image 11 is processed with an object detector 40 and detected objects are obtained.

[0039] In Fig. 4, the system 70 is used in a transport system 90, e.g., a shuttle, a van, or a bus. The system 70 comprises an image sensor 71 and the end-to-end object recognition module 60, which is one of the Fig. 1 or Fig. 2. The transmission module 80 transmits a control signal to a drive control unit of the transport system 90 in response to the detected objects. Reference symbol 10 images 11 improved or denoised image 20 coders 21 Encoder layer 22 Encoder layer 23 Encoder layer 24 Encoder layer 25 Encoder layer 26 Encoder layer 30 Denoising network 31 Denoising subnetwork 32 Denoising subnetwork 33 Denoising subnetwork 34 Denoising subnetwork 35 Denoising subnetwork 36 Denoising subnetwork 40 object detector 50 image with detected objects 60 end-to-end object detector module 70 systems 71 image sensor 80 transmission module 90 Transport system SLD-OD object detection based on single-stage denoiser MLD-OD object detection based on multi-stage denoising V1-V4 process steps QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2022 414 838 A1

[0003] CN 115 689 932 A

[0003] CN 110 544 213 A

[0003] US 2022 210 350 A1

[0003] AU 2020 100 274 ​​

[0003] CN 111 915 530 A

[0003] Cited non-patent literature

[0000] Chapter 2.1 of ar-Xiv:2209.04747v3

[0012]

Claims

[1] A method for detecting at least one object in an environment with adverse weather conditions, the method comprising: • Receiving an image (10) from an image sensor path (V1); • coding the image (10) with an encoder (20, 21-26) and obtaining at least one latent embedding of the image (10) comprising coded features of the image (V2); • Passing the image (10) together with the latent embedding through at least one generative machine learning model (30, 31-36), wherein the generative machine learning model (30, 31-36) is configured to denoise effects in the image (10) caused by the adverse weather conditions, and obtaining at least one denoised image (11) (V3); • Processing the denoised image (11) with an object detector (40) and obtaining detected objects (V4). [2] The method of claim 1, wherein the generative machine learning model (30, 31-36) is configured to denoise effects resulting from rain, fog, haze, snow, clouds, or sunshine. [3] A method according to claim 1 or claim 2, wherein an individual generative machine learning model (31-36) is used for each denoising effect, and wherein outputs of the individual generative machine learning models (31-36) are aggregated and the resulting aggregated image is processed by the object detector (40), or wherein the individual generative machine learning models (31-36) are processed sequentially, in which case an output of one of the individual generative machine learning models (31-36) is sent as input to the next one of the individual generative machine learning models (31-36) and the output of the last one of the individual generative machine learning models (36) is processed by the object detector (40). [4] A method according to any one of the preceding claims, wherein the encoder (20, 21-26) comprises a feedforward network and the latent embedding obtained from the last layer of the encoder (20) is fed forward by the generative machine learning model (30, 31-36). [5] Method according to claim 1 or 2, wherein • an individual generative machine learning model (31-36) is used for each denoising effect; • the encoder (20) comprises a forward conducting network with several layers (21-26); • the individual generative machine learning models (31-36) are processed sequentially, in which case an output of one of the individual generative machine learning models (31-36) is sent as input to the next of the individual generative machine learning models (31-36) and the output of the last of the individual generative machine learning models (31-36) is processed by the object detector (40); • some of the individual generative machine learning models (31-36) are fed with latent embeddings from different layers (21-26) of the encoder (20). [6] Method according to claim 1 or 2, wherein • an individual generative machine learning model (31-36) is used for each denoising effect; • the encoder (20) comprises a feedforward network with multiple layers (21-26), wherein the number of layers (21-26) corresponds to the number of individual generative machine learning models (31-36); • the individual generative machine learning models (31-36) are processed sequentially, in which case an output of one of the individual generative machine learning models (31-36) is sent as input to the next of the individual generative machine learning models (31-36) and the output of the last of the individual generative machine learning models (31-36) is processed by the object detector (40); • each of the individual generative machine learning models (31-36) is fed with latent embeddings from different layers (21-26) of the encoder (20). [7] A computer program for detecting at least one object in an environment with adverse weather conditions, the computer program comprising instructions which, when loaded or executed by the computer, cause the computer to perform the operations according to the method of any one of the preceding claims. [8] A machine-readable storage medium having instructions stored thereon, the instructions being executable by a machine to cause the machine to perform the operations according to the method of any one of claims 1 to 6. [9] An end-to-end object detector module (60) for detecting at least one object in an environment with adverse weather conditions, the end-to-end object detector module comprising an encoder (20, 21-26), at least one generative machine learning model (30, 31-36) configured to denoise effects in an image (10) received from an image sensor path, the effects being caused by the adverse weather conditions, and an object detector (40), the object detector (40) receiving a denoised image (11) from the generative machine learning model (30, 31-36) and being configured to detect objects. [10] A system (70) for a civil road vehicle, a public transport system (90), or a public transport infrastructure element, wherein the system (70) comprises at least one image sensor (71), an end-to-end object detector module (60), and a transmission module (80), wherein the end-to-end object detector module (60) comprises an encoder (20, 21-26), at least one generative machine learning model (30, 31-36) configured to denoise effects in an image (10) received by the image sensor (71), the effects being caused by the adverse weather conditions, and an object detector (40), wherein the system (70) is configured to detect objects according to the method of any one of claims 1 to 6, and wherein the transmission module (80) is configured tothat, in response to the detected objects, it transmits a control signal to a driving control unit of the road vehicle or the traffic system (90) and / or to a control unit of the public transport infrastructure element, which is in signal transmission connection with the driving control unit.

Images