Device and computer-implemented method for processing digital images

Abstract

A device and a computer-implemented method for processing digital images, the method comprising: providing a sequence of digital images (102), the sequence comprising a first digital image and a second digital image, the first digital image being assigned to a first time point in the sequence, the second digital image being assigned to a second time point in the sequence, the digital images comprising pixels, encoding the pixels of the first digital image with a first encoder into a first feature image, encoding the pixels of the second digital image with the first encoder into a second feature image, and encoding at least the first feature image and the second feature image with a second encoder into a third feature image.

Description

background

[0001] The invention relates to a device and a computer-implemented method for processing digital images.

[0002] Models for processing digital images in real-time applications repeatedly output results from the processing of individual digital images. These models are trained and evaluated in recurring time steps, with only one digital image in each time step. Disclosure of the invention

[0003] A computer-implemented method for processing digital images comprises providing a sequence of digital images, wherein the sequence includes a first digital image and a second digital image, the first digital image being assigned to a first time point in the sequence, the second digital image being assigned to a second time point in the sequence, the digital images containing pixels, encoding the pixels of the first digital image with a first encoder into a first feature map, encoding the pixels of the second digital image with the first encoder into a second feature map, and encoding the first feature map and the second feature map with a second encoder into a third feature map. The third feature map collects the feature maps from images at different time points.This means that the third feature mapping represents time-dependent features from repeatedly processed feature mappings. The second encoder is a iterative model for determining the third feature mapping from the first and second digital images.

[0004] The third feature mapping can be used for various purposes.

[0005] The method includes, for example, predicting the speed of an object, classifying an object represented by the pixels of the second digital image with a first head depending on the third feature mapping, semantic segmentation of the pixels of the second digital image with a first head depending on the third feature mapping, or detecting an object represented by the pixels of the second digital image with a first head depending on the third feature mapping.

[0006] The procedure includes, for example, predicting the speed of the object or classifying the object represented by the pixels of the second digital image with a second head depending on the second feature mapping, or semantic segmentation of the pixels of the second digital image with a second head depending on the second feature mapping, or detecting an object represented by the pixels of the second digital image with a second head depending on the second feature mapping, and performing multitask training of at least two of the first coder and the second coder and of the first head and the second head.

[0007] The second encoder can be trained as the repeating model based on digital images from the sequence.

[0008] For example, the sequence includes a third digital image, wherein the third digital image is assigned to a third point in time in the sequence, encoding the pixels of the third digital image with the first encoder into a fourth feature image, encoding at least the second feature image and the fourth feature image with the second encoder into a fifth feature image, mapping the third feature image with the first encoder into a result associated with the second digital image, and mapping the fifth feature image with the first encoder into a result associated with the third digital image, wherein the procedure involves providing a first reference for the result associated with the second digital image and a second reference for the result associated with the third digital image.The training of at least two of the first encoder and the second encoder and the first head and the second head includes dependence on a loss, wherein the loss includes a first loss expression that depends on the first reference and the result associated with the second digital image, wherein the loss includes a second loss expression that depends on the second reference and the result associated with the third digital image.

[0009] The procedure can involve providing an initial weight for the first loss expression and a second weight for the second loss expression, with the third time point occurring after the second time point and the second weight being greater than the first. This means that the digital image associated with the last time point is weighted more heavily than the other digital image during training. This improves the repeating model's ability to predict the velocity, classify, or semantically segment the last digital image, or to detect objects within the last digital image, while also taking the preceding digital images into account.

[0010] The process can include receiving the first digital image at the first time point, receiving the second digital image at the second time point, and encoding the pixels of the first digital image into the first feature mapping between the first and second time points. This is useful for real-time processing.

[0011] The procedure can involve storing the first feature mapping in a buffer, particularly before the second time, and reading the first feature mapping from the buffer. Consequently, the buffer is an effective way of providing the ability of the iterating model to retain the earlier feature mapping for later processing.

[0012] The procedure can involve storing the second feature mapping instead of the first in the buffer memory due to encoding at least the first and second feature mappings into the third feature mapping. This resets the buffer memory when two feature mappings are used to determine the third feature mapping.

[0013] The method can include reading a feature mapping from the buffer, encoding the read feature mapping, the first feature mapping, and the second feature mapping into the third feature mapping, and storing the second feature mapping in addition to the first feature mapping in the buffer due to the encoding of at least the read feature mapping, the first feature mapping, and the second feature mapping into the third feature mapping, in particular storing the first feature mapping or the second feature mapping instead of the read feature mapping in the buffer. This uses more than two feature mappings to determine the third feature mapping.

[0014] The process can include receiving the first digital image at the first time point, receiving the second digital image at the second time point, and encoding the pixels of the first digital image into the first feature mapping between the first time point and the second time point.

[0015] The procedure can include storing the first feature mapping in a buffer memory, particularly before the second time, and reading the first feature mapping from the buffer memory.

[0016] The digital images can be processed for various real-world applications using the repeating model.

[0017] The method may include receiving digital images from a sensor, in particular a camera, lidar sensor, radar sensor, ultrasonic sensor, motion sensor, processing the digital images to determine the result for at least one of the received digital images, and operating a technical system, in particular a robot, vehicle, household appliance, imaging device, computer-controlled machine, depending on the result.

[0018] A device for processing digital images comprises at least one processor, at least one data storage device and at least one interface for receiving the digital images, wherein the at least one data storage device stores instructions which, when executed by the at least one processor, cause the device to perform the method.

[0019] A computer program may be provided, wherein the computer program includes computer-readable instructions which, when executed by a computer, cause the computer to perform the procedure.

[0020] Further exemplary embodiments are derived from the following description and the drawing; they show: Fig. 1. A schematic representation of a device for processing digital images; Fig. 2. schematically design a model for processing digital images; and Fig. 3. A flowchart that includes the steps of a procedure for processing digital images.

[0021] Fig. Figure 1 schematically represents a device 100 for processing digital images. An exemplary sequence of digital images 102 is shown in Fig. Figure 1 shows the digital images 102, which are assigned to different points in time within the sequence. The digital images 102 comprise pixels.

[0022] The device 100 comprises at least one processor 104, at least one data storage device 106 and at least one interface 108 for receiving the digital images 102.

[0023] The at least one data storage device 106 stores instructions which, when executed by the at least one processor 104, cause the device 100 to perform a method for processing the digital images 102.

[0024] Interface 108 is configured, for example, to receive digital images from a sensor 110. The sensor 110 can be a camera, a lidar sensor, a radar sensor, an ultrasonic sensor, or a motion sensor.

[0025] The device 100 can be configured to process the digital images 102 in order to determine a result for at least one of the received digital images 102.

[0026] The device 100 can be configured to operate a technical system 112, depending on the desired outcome. The technical system 112 is, for example, a robot, a vehicle, a household appliance, an imaging device, or a computer-controlled machine.

[0027] Operating the technical system 112 may involve moving at least one section of the technical system 112, depending on the result.

[0028] Fig. Figure 2 schematically represents a model 200 for processing the digital images 102. The device 100 comprises the model 200.

[0029] The model 200 is configured to predict the velocity of an object depicted in a digital image 102, or to classify or semantic segment a digital image 102, or to detect an object in a digital image 102.

[0030] The model 200 includes a first encoder 202 and a second encoder 204.

[0031] The first encoder 202 is configured to encode the pixels of a single digital image 102 into a feature map 208. The first encoder 202 is configured to sequentially encode the pixels of the digital image 102 into respective feature maps 208.

[0032] The second encoder 206 is configured to code at least two of the feature images 208, which the first encoder 202 determines successively, into a common feature image 210 for two digital images 102.

[0033] The model 200 comprises several heads 212. A first head 212 is configured to map the common feature mapping 210 into a first result 214 of the prediction of the object's velocity, or of classification, or successive segmentation, or of detection for the digital image 102. A second head 212 is configured to map the feature mapping 204 into a second result 214 of the prediction of the object's velocity, or of classification, or successive segmentation, or of detection for the digital image 102.

[0034] The first encoder 202 can comprise a neural network. The first encoder 202 can be the encoder 202 of an object detector backbone. The second head 214 can be the object detection head for the object detector backbone. The object detector backbone can include the first encoder 202. The object detector backbone can be trained to detect objects in digital images. This means that the feature mapping 204 is determined by the encoder of the trained object detector backbone, i.e., the first encoder 202. In addition to forwarding the features 204 to the second encoder 206, the feature mapping 204 can be forwarded to the object detection head, i.e., to the second head 214.

[0035] The second encoder 206 can comprise a neural network such as a ConvLSTM, BIFPN, deformable attention, or Conv3D. The heads 212 can each comprise a neural network.

[0036] The heads 212 can each comprise a very simple neural network such as a simple forward-feedback neural network, FFN, e.g., a single-layer FFN.

[0037] The model 200 can be used to extend the object detector backbone, i.e. to provide the first result 214 through the first head 212 in addition to the second result 214, which is output by the second head 212, i.e. the object detection head.

[0038] The object detector backbone, e.g., the first encoder 202, can be a convolutional neural network-based object detector backbone. Model 200 is not limited to having a convolutional neural network-based object detector backbone as its only embodiment. Model 200 can be used to extend an object detector backbone that includes a conversion encoder as the encoder.

[0039] This means that the first encoder 202 can be a conversion encoder that outputs the feature mapping 204.

[0040] The model 200 can include a buffer storage tank of 216.

[0041] The model 200 can be configured to store at least one feature mapping 208 in the buffer memory 216 and to read the stored, at least one feature mapping 208 from the buffer memory 208.

[0042] The model 200 can be configured to encode at least one feature image 208 from the buffer memory 216 and one feature image 208 from the first encoder 202 into the common feature image 210.

[0043] For example, buffer memory 216 is present in an embedded system that includes model 200. Buffer memory 216 is optional in the training of model 200. The training can use different storage media for the feature mappings 208.

[0044] Fig. Section 3 represents the steps of the procedure.

[0045] The procedure includes step 302.

[0046] Step 302 comprises providing the sequence of digital images 102. The procedure can be based on a given first encoder 202, second encoder 206, and given heads 212. The procedure can include providing the first encoder 202, the second encoder 206, and / or at least one of the heads.

[0047] The sequence of digital images 102 can be received. Receiving the sequence can involve receiving the digital images 102 at different times. Each digital image 102 can be assigned to the time at which the respective digital image was captured or received.

[0048] The digital images 102 can be received by the sensor.

[0049] The digital images 102 can be images taken with the camera.

[0050] The digital images 102 could be Lidar images.

[0051] The digital images 102 could be radar images.

[0052] The digital images 102 could be ultrasound images.

[0053] The digital images 102 can be motion images.

[0054] The procedure is executed in iterations. In each iteration, one input digital image 102 is processed. The input digital images 102 are processed by the procedure in an order corresponding to the time point to which they are assigned in the sequence of digital images 102. For example, an input digital image 102 assigned to an earlier time point is processed before an input digital image 102 assigned to a later time point.

[0055] Step 302 includes initializing the buffer memory 216 with a feature mapping 208.

[0056] The procedure is not limited to initializing the buffer memory 216 with a single feature mapping 208. The buffer memory 216 can be initialized with more than one feature mapping 208. The content of the feature mapping 208 or mappings 208 used to initialize the buffer memory 216 can be, for example, zeros.

[0057] The procedure includes step 304.

[0058] Step 304 involves encoding the pixels of an input digital image 102 of the sequence into a feature mapping 208. The pixels of the input digital image 102 are encoded, for example, with the first encoder 202.

[0059] The procedure includes step 306.

[0060] Step 306 involves reading the feature mapping 208 from the buffer memory 216.

[0061] The procedure is not limited to encoding two feature images 208 into the common feature image 210. More than one feature image 208 can be read from the buffer memory 216 and encoded into the common feature image with the feature image 208 that is provided depending on the digital image 104, in particular by the first encoder 202.

[0062] The procedure includes step 308.

[0063] Step 308 involves coding the feature mappings 208 into the common feature mapping 210. The feature mappings 208 are coded into the common feature mapping 210, for example, using the second coder 206.

[0064] Step 308 includes storing the feature mapping 208 in the buffer memory 216.

[0065] The feature mapping 208 is stored in buffer memory 216, for example, instead of the feature mapping 208 that was read from buffer memory 216.

[0066] The procedure is not limited to storing a single feature mapping 208 in buffer memory 216. If multiple feature mappings 208 are stored in buffer memory 216, the feature mapping 208 determined in the current iteration is stored in buffer memory 216 instead of a feature mapping 208 stored in a previous iteration. For example, the feature mapping 208 in buffer memory 216 from the earliest iteration that buffer memory 216 can store is replaced by the feature mapping 208 of the current iteration.

[0067] The procedure includes step 310.

[0068] Step 310 involves determining the first result 214 depending on the common feature mapping 210. The common feature mapping 210 is mapped, for example, to the first head 212 in the first result 214.

[0069] The first result 214 is, for example, a result of predicting the speed of an object depicted in the input digital image 102.

[0070] The first result 214 is, for example, a result of classifying an object that is represented by the pixels of the input digital image 102.

[0071] The first result 214 is, for example, a result of the semantic segmentation of the pixels of the input digital image 102.

[0072] The first result 214 is, for example, a result of detecting an object that is represented by the pixels of the input digital image 102.

[0073] Step 310 involves determining the second result 214 depending on the feature mapping 204. The feature mapping 204 is, for example, mapped to the second result 214 using the second head 212.

[0074] Steps 302 to 310 can be repeated to determine multiple results 214 such as first results 214 and / or second results 214 for digital images received consecutively in step 302.

[0075] Steps 304 to 310 can be repeated to determine the multiple results 214 for digital images provided as a stack in step 302.

[0076] To train the first component 202 and / or the second component 206 and / or the head 212, the procedure can include a step 312. During training, for example, the weightings of the first component 202 and / or the second component 206 and / or the head 212 are learned.

[0077] The weightings learned in training teach temporal information.

[0078] For the transfer of learning / for fine-tuning, the weights of the first encoder 202, i.e. the encoder of the object detector backbone, and / or the weights of the second head 212 can be frozen, while only the weights of the second encoder 206 and / or the first head 212 are trained.

[0079] In this case, the first component 202 will not learn any temporal information.

[0080] To train from the ground up, the first component 202, the second component 206 and the head 212 are repeatedly trained together.

[0081] Step 312 involves training the encoder 206 and / or the head 212 depending on a loss.

[0082] The encoder 206 is trained based on the results 214 determined for several digital images 102 processed by the model 200, as described for the input digital image 102. For training purposes, each input digital image 102 is assigned a reference that is associated with that specific input digital image 102.

[0083] The digital images 102 are assigned to different points in time within the sequence.

[0084] When the second encoder 206 is trained to predict the speed, the first result 214 is, for example, a speed of the object depicted in the digital image 102.

[0085] When the second encoder 206 is trained to classify, the reference for the first result 214 is, for example, a classification of the digital image 102.

[0086] When the second encoder 206 is trained for semantic segmentation, the reference for the first result 214 is, for example, a segmentation map for the pixels of the digital image 102.

[0087] In one example, the second encoder 206 is trained using multitasking. This means that the second encoder 206 is trained together with several heads 212. In another example, the first head 212 is used for segmentation, specifically to determine a road surface, while the second head 212 is used for detecting objects of a first type, such as a traffic light. In yet another example, an additional head is provided with an additional reference to provide an additional result and is trained to detect objects of a second type, such as a vehicle or a pedestrian.

[0088] When the second encoder 206 is trained to detect the object, the reference for the first result 214 is, for example, the parameters that define a bounding box that identifies the pixels of the digital image 102 that contain the object.

[0089] The loss includes one loss expression per result 214. The loss expression depends on the reference for the respective result 214 and the respective result 214 itself.

[0090] The loss can include weights for the loss expressions, where the weight for a loss expression defines the influence of the loss expression on the loss.

[0091] The loss is, for example, the sum of the loss expressions. Each loss expression comprises, for example, the absolute value of the distance between the respective reference of the respective result 214 and the respective result 214. The weight for each loss expression is, for example, a value between zero and one, which is multiplied by the respective loss expression.

[0092] The training can include determining the loss expressions for the input digital images 102, which are assigned to successive time points in the sequence. The weight for the loss expression for the input digital image assigned to the last of the successive time points can be greater than the weights for the other loss expressions.

[0093] After that, step 302 can be performed to provide more digital images for training.

[0094] To operate the technical system 112, the procedure may include a step 314.

[0095] Steps 302 to 310 and step 314 can be executed sequentially in step 302 to operate a technical system 112 in real time based on the acquisition of a digital image 102 that depicts at least a section of the technical system 112 or the environment of the technical system.

[0096] Step 314 may include the operation of the technical system 112, in particular the robot, the vehicle, the household appliance, the imaging device, the computer-controlled machine, depending on the first result 214.

[0097] Then step 302 can be performed.

Images