Device and computer-implemented method for processing digital images

Abstract

The invention relates to a device and a computer-implemented method for processing digital images, wherein the method comprises: providing a sequence of digital images (102), wherein the sequence comprises a first digital image and a second digital image, wherein the first digital image is assigned to a first point in time in the sequence, wherein the second digital image is assigned to a second point in time in the sequence, wherein the digital images comprise pixels, encoding the pixels of the first digital image into a first feature map using a first encoder, encoding the pixels of the second digital image into a second feature map using the first encoder, and encoding at least the first feature map and the second feature map into a third feature map using a second encoder.

Description

[0001] R. 413330

[0002] - 1 -

[0003] Description

[0004] title

[0005] Device and computer-implemented method for processing digital images

[0006] background

[0007] The invention relates to a device and a computer-implemented method for processing digital images.

[0008] Models for processing digital images in real-time applications repeatedly output results from the processing of individual digital images. These models are trained and evaluated in recurring journals, with only one digital image in each journal.

[0009] Disclosure of the invention

[0010] A computer-implemented method for processing digital images comprises providing a sequence of digital images, wherein the sequence includes a first digital image and a second digital image, the first digital image being assigned to a first time point in the sequence, the second digital image being assigned to a second time point in the sequence, the digital images containing pixels, encoding the pixels of the first digital image into a first feature map using a first encoder, encoding the pixels of the second digital image into a second feature map using the first encoder, and encoding the first and second feature maps together using a second encoder into a third feature map. The third feature map collects the feature maps from images at different time points. This means that the third feature map contains time-dependent features from repeatedly processed feature maps.The second encoder is R. 413330.

[0011] - 2 - a repeating model for determining the third feature mapping from the first digital image and the second digital image.

[0012] The third feature mapping can be used for various purposes.

[0013] The method includes, for example, predicting the speed of an object, classifying an object represented by the pixels of the second digital image with a first head depending on the third feature mapping, semantic segmentation of the pixels of the second digital image with a first head depending on the third feature mapping, or detecting an object represented by the pixels of the second digital image with a first head depending on the third feature mapping.

[0014] The procedure includes, for example, predicting the speed of the object or classifying the object represented by the pixels of the second digital image with a second head depending on the second feature mapping, or semantic segmentation of the pixels of the second digital image with a second head depending on the second feature mapping, or detecting an object represented by the pixels of the second digital image with a second head depending on the second feature mapping, and performing multitask training of at least two of the first coder and the second coder and of the first head and the second head.

[0015] The second encoder can be trained as the repeating model based on digital images from the sequence.

[0016] For example, the sequence includes a third digital image, where the third digital image is assigned to a third point in time in the sequence, encoding the pixels of the third digital image with the first encoder into a fourth feature mapping, encoding at least the second feature mapping and the fourth feature mapping with the second encoder into a fifth feature mapping, R. 413330

[0017] - 3 -

[0018] Mapping the third feature mapping with the first head into a result associated with the second digital image, and mapping the fifth feature mapping with the first head into a result associated with the third digital image, wherein the method includes providing a first reference for the result associated with the second digital image and a second reference for the result associated with the third digital image, training at least two of the first encoder and the second encoder and the first head and the second head depending on a loss, wherein the loss includes a first loss expression that depends on the first reference and the result associated with the second digital image, wherein the loss includes a second loss expression that depends on the second reference and the result associated with the third digital image.

[0019] The procedure can involve providing a first weight for the first loss expression and a second weight for the second loss expression, with the third time point occurring after the second time point and the second weight being greater than the first. This means that the digital image associated with the last time point is weighted more heavily than the other digital image during training. This improves the repeating model's ability to predict the velocity, classify, or semantically segment the last digital image, or to detect objects within the last digital image, while also taking the preceding digital images into account.

[0020] The process can include receiving the first digital image at the first time point, receiving the second digital image at the second time point, and encoding the pixels of the first digital image into the first feature mapping between the first and second time points. This is useful for real-time processing.

[0021] The procedure may include storing the first feature mapping in a buffer memory, particularly before the second time, and reading the first feature mapping from the buffer memory. Consequently, the buffer memory is an effective way of providing the capability of the R. 413330

[0022] - 4 - repeating model, to retain the earlier feature mapping for later processing.

[0023] The procedure can involve storing the second feature mapping instead of the first in the buffer memory due to encoding at least the first and second feature mappings into the third feature mapping. This resets the buffer memory when two feature mappings are used to determine the third feature mapping.

[0024] The method can include reading a feature mapping from the buffer, encoding the read feature mapping, the first feature mapping, and the second feature mapping into the third feature mapping, and storing the second feature mapping in addition to the first feature mapping in the buffer due to the encoding of at least the read feature mapping, the first feature mapping, and the second feature mapping into the third feature mapping, in particular storing the first feature mapping or the second feature mapping instead of the read feature mapping in the buffer. This uses more than two feature mappings to determine the third feature mapping.

[0025] The process can include receiving the first digital image at the first time point, receiving the second digital image at the second time point, and encoding the pixels of the first digital image into the first feature mapping between the first time point and the second time point.

[0026] The procedure can include storing the first feature mapping in a buffer memory, particularly before the second time, and reading the first feature mapping from the buffer memory.

[0027] The digital images can be processed for various real-world applications using the repeating model. R. 413330

[0028] - 5 -

[0029] The method may include receiving digital images from a sensor, in particular a camera, lidar sensor, radar sensor, ultrasonic sensor, motion sensor, processing the digital images to determine the result for at least one of the received digital images, and operating a technical system, in particular a robot, vehicle, household appliance, imaging device, computer-controlled machine, depending on the result.

[0030] A device for processing digital images comprises at least one processor, at least one data storage device and at least one interface for receiving the digital images, wherein the at least one data storage device stores instructions which, when executed by the at least one processor, cause the device to perform the method.

[0031] A computer program may be provided, wherein the computer program includes computer-readable instructions which, when executed by a computer, cause the computer to perform the procedure.

[0032] Further exemplary embodiments are derived from the following description and the drawing; they show:

[0033] Fig. 1 schematically shows a device for processing digital images;

[0034] Fig. 2 schematically shows a model for processing digital images; and Fig. 3 shows a flowchart comprising the steps of a process for processing digital images.

[0035] Fig. 1 schematically depicts a device 100 for processing digital images. An exemplary sequence of digital images 102 is shown in Fig. 1. The digital images 102 are assigned to different points in time within the sequence. The digital images 102 comprise pixels. R. 413330

[0036] - 6 -

[0037] The device 100 comprises at least one processor 104, at least one data storage device 106 and at least one interface 108 for receiving the digital images 102.

[0038] The at least one data storage device 106 stores instructions which, when executed by the at least one processor 104, cause the device 100 to perform a method for processing the digital images 102.

[0039] Interface 108 is configured, for example, to receive digital images from a sensor 110. The sensor 110 can be a camera, a lidar sensor, a radar sensor, an ultrasonic sensor, or a motion sensor.

[0040] The device 100 can be configured to process the digital images 102 in order to determine a result for at least one of the received digital images 102.

[0041] The device 100 can be configured to operate a technical system 112, depending on the desired outcome. The technical system 112 is, for example, a robot, a vehicle, a household appliance, an imaging device, or a computer-controlled machine.

[0042] Operating the technical system 112 may involve moving at least one section of the technical system 112, depending on the result.

[0043] Fig. 2 schematically depicts a model 200 for processing the digital images 102. The device 100 comprises the model 200.

[0044] Model 200 is configured to predict the velocity of an object depicted in a digital image 102, or to classify or semantically segment a digital image 102, or to detect an object in a digital image 102. R. 413330

[0045] - 7 -

[0046] The model 200 includes a first encoder 202 and a second encoder 204.

[0047] The first encoder 202 is configured to encode the pixels of a single digital image 102 into a feature map 208. The first encoder 202 is configured to sequentially encode the pixels of the digital image 102 into respective feature maps 208.

[0048] The second encoder 206 is configured to code at least two of the feature images 208, which the first encoder 202 determines successively, into a common feature image 210 for two digital images 102.

[0049] The model 200 comprises several heads 212. A first head 212 is configured to map the common feature mapping 210 into a first result 214 of the prediction of the object's velocity, or of classification, or successive segmentation, or of detection for the digital image 102. A second head 212 is configured to map the feature mapping 204 into a second result 214 of the prediction of the object's velocity, or of classification, or successive segmentation, or of detection for the digital image 102.

[0050] The first encoder 202 can comprise a neural network. The first encoder 202 can be the encoder 202 of an object detector backbone. The second head 214 can be the object detection head for the object detector backbone. The object detector backbone can comprise the first encoder 202. The object detector backbone can be trained to detect objects in digital images. This means that the feature mapping 204 is determined by the encoder of the trained object detector backbone, i.e., the first encoder 202. In addition to forwarding the features 204 to the second encoder 206, the feature mapping 204 can be forwarded to the object detection head, i.e., the second head 214. R. 413330

[0051] - 8 -

[0052] The second encoder 206 can comprise a neural network such as a ConvLSTM, BIFPN, deformable attention, or Conv3D. The heads 212 can each comprise a neural network.

[0053] The heads 212 can each comprise a very simple neural network such as a simple forward-feedback neural network, FFN, e.g., a single-layer FFN.

[0054] The model 200 can be used to extend the object detector backbone, i.e. to provide the first result 214 through the first head 212 in addition to the second result 214, which is output by the second head 212, i.e. the object detection head.

[0055] The object detector backbone, e.g., the first encoder 202, can be a convolutional neural network-based object detector backbone. Model 200 is not limited to having a convolutional neural network-based object detector backbone as its only embodiment. Model 200 can be used to extend an object detector backbone that includes a conversion encoder as the encoder.

[0056] This means that the first encoder 202 can be a conversion encoder that outputs the feature mapping 204.

[0057] The model 200 can include a buffer storage tank of 216.

[0058] The model 200 can be configured to store at least one feature mapping 208 in the buffer memory 216 and to read the stored, at least one feature mapping 208 from the buffer memory 208.

[0059] Model 200 can be configured to encode at least one feature mapping 208 from buffer memory 216 and one feature mapping 208 from the first encoder 202 into the common feature mapping 210. R. 413330

[0060] - 9 -

[0061] For example, buffer memory 216 is present in an embedded system that includes model 200. Buffer memory 216 is optional in the training of model 200. The training can use different storage media for the feature mappings 208.

[0062] Fig. 3 illustrates steps of the process.

[0063] The procedure includes step 302.

[0064] Step 302 comprises providing the sequence of digital images 102. The procedure can be based on a given first encoder 202, second encoder 206, and given heads 212. The procedure can include providing the first encoder 202, the second encoder 206, and / or at least one of the heads.

[0065] The sequence of digital images 102 can be received. Receiving the sequence can involve receiving the digital images 102 at different times. Each digital image 102 can be assigned to the time at which the respective digital image was captured or received.

[0066] The digital images 102 can be received by the sensor.

[0067] The digital images 102 can be images taken with the camera.

[0068] The digital images 102 could be Lidar images.

[0069] The digital images 102 could be radar images.

[0070] The digital images 102 could be ultrasound images.

[0071] The digital images 102 can be motion images. R. 413330

[0072] - 10 -

[0073] The procedure is executed in iterations. In each iteration, one input digital image 102 is processed. The input digital images 102 are processed by the procedure in an order corresponding to the time point to which they are assigned in the sequence of digital images 102. For example, an input digital image 102 assigned to an earlier time point is processed before an input digital image 102 assigned to a later time point.

[0074] Step 302 includes initializing the buffer memory 216 with a feature mapping 208.

[0075] The procedure is not limited to initializing the buffer memory 216 with a single feature mapping 208. The buffer memory 216 can be initialized with more than one feature mapping 208. The content of the feature mapping 208 or mappings 208 used to initialize the buffer memory 216 can be, for example, zeros.

[0076] The procedure includes step 304.

[0077] Step 304 involves encoding the pixels of an input digital image 102 of the sequence into a feature mapping 208. The pixels of the input digital image 102 are encoded, for example, with the first encoder 202.

[0078] The procedure includes step 306.

[0079] Step 306 involves reading the feature mapping 208 from the buffer memory 216.

[0080] The procedure is not limited to encoding two feature images 208 into the common feature image 210. More than one feature image 208 can be read from the buffer memory 216 and encoded into the common feature image with the feature image 208 that is provided depending on the digital image 104, in particular by the first encoder 202. R. 413330

[0081] - 11 -

[0082] The procedure includes step 308.

[0083] Step 308 involves coding the feature mappings 208 into the common feature mapping 210. The feature mappings 208 are coded into the common feature mapping 210, for example, using the second coder 206.

[0084] Step 308 includes storing the feature mapping 208 in the buffer memory 216.

[0085] The feature mapping 208 is stored in buffer memory 216, for example, instead of the feature mapping 208 that was read from buffer memory 216.

[0086] The procedure is not limited to storing a single feature mapping 208 in buffer memory 216. If multiple feature mappings 208 are stored in buffer memory 216, the feature mapping 208 determined in the current iteration is stored in buffer memory 216 instead of a feature mapping 208 stored in a previous iteration. For example, the feature mapping 208 in buffer memory 216 from the earliest iteration that buffer memory 216 can store is replaced by the feature mapping 208 of the current iteration.

[0087] The procedure includes step 310.

[0088] Step 310 involves determining the first result 214 depending on the common feature mapping 210. The common feature mapping 210 is mapped, for example, to the first head 212 in the first result 214.

[0089] The first result, 214, is, for example, a result of predicting the speed of an object depicted in the input digital image 102. R. 413330

[0090] - 12 -

[0091] The first result 214 is, for example, a result of classifying an object that is represented by the pixels of the input digital image 102.

[0092] The first result 214 is, for example, a result of the semantic segmentation of the pixels of the input digital image 102.

[0093] The first result 214 is, for example, a result of detecting an object that is represented by the pixels of the input digital image 102.

[0094] Step 310 involves determining the second result 214 depending on the feature mapping 204. The feature mapping 204 is, for example, mapped to the second result 214 using the second head 212.

[0095] Steps 302 to 310 can be repeated to determine multiple results 214 such as first results 214 and / or second results 214 for digital images received consecutively in step 302.

[0096] Steps 304 to 310 can be repeated to determine the multiple results 214 for digital images provided as a stack in step 302.

[0097] To train the first component 202 and / or the second component 206 and / or the head 212, the procedure can include a step 312. During training, for example, the weightings of the first component 202 and / or the second component 206 and / or the head 212 are learned.

[0098] The weightings learned in training teach temporal information.

[0099] For the purpose of transferring the learning / for fine-tuning, the weights of the first encoder 202, i.e., the encoder of the object detector backbone, and / or the weights of the second encoder 212 can be frozen, R. 413330

[0100] - 13 - while only the weights of the second coder 206 and / or the first head 212 are trained.

[0101] In this case, the first component 202 will not learn any temporal information.

[0102] To train from the ground up, the first component 202, the second component 206 and the head 212 are repeatedly trained together.

[0103] Step 312 involves training the encoder 206 and / or the head 212 depending on a loss.

[0104] The encoder 206 is trained on the basis of the results 214 determined for several digital images 102 processed by the model 200 as described for the input digital image 102.

[0105] For training purposes, each entered digital image 102 is provided with a respective reference that is assigned to the respective entered digital image 102.

[0106] The digital images 102 are assigned to different points in time within the sequence.

[0107] When the second encoder 206 is trained to predict the speed, the first result 214 is, for example, a speed of the object depicted in the digital image 102.

[0108] When the second encoder 206 is trained to classify, the reference for the first result 214 is, for example, a classification of the digital image 102.

[0109] When the second encoder 206 is trained for semantic segmentation, the reference for the first result 214 is, for example, a segmentation map for the pixels of the digital image 102.

[0110] According to an example, the second coder 206 is trained in multitask learning. This means that the second coder 206 is trained with multiple heads 212 R. 413330

[0111] - 14 - trained together. According to one example, the first head 212 is used for segmentation, in particular to determine a driving surface, the second head 212 is used for the detection of objects of a first type, such as the detection of a traffic signal. According to another example, an additional head is provided with an additional reference to provide an additional result and is trained for the detection of objects of a second type, such as the detection of a vehicle or a pedestrian.

[0112] When the second encoder 206 is trained to detect the object, the reference for the first result 214 is, for example, the parameters that define a bounding box that identifies the pixels of the digital image 102 that contain the object.

[0113] The loss includes one loss expression per result 214. The loss expression depends on the reference for the respective result 214 and the respective result 214 itself.

[0114] The loss can include weights for the loss expressions, where the weight for a loss expression defines the influence of the loss expression on the loss.

[0115] The loss is, for example, the sum of the loss expressions. Each loss expression comprises, for example, the absolute value of the distance between the respective reference of the respective result 214 and the respective result 214. The weight for each loss expression is, for example, a value between zero and one, which is multiplied by the respective loss expression.

[0116] The training may include determining the loss expressions for the input digital images 102, which are assigned to successive time points in the sequence. The weight for the loss expression for the input digital image assigned to the last of the successive time points may be greater than the weights for the other loss expressions. R. 413330

[0117] - 15 -

[0118] After that, step 302 can be performed to provide more digital images for training.

[0119] To operate the technical system 112, the procedure may include a step 314.

[0120] Steps 302 to 310 and step 314 can be executed sequentially in step 302 to operate a technical system 112 in real time based on the acquisition of a digital image 102 that depicts at least a section of the technical system 112 or the environment of the technical system.

[0121] Step 314 may include the operation of the technical system 112, in particular the robot, the vehicle, the household appliance, the imaging device, the computer-controlled machine, depending on the first result 214.

[0122] Then step 302 can be performed.

Claims

R. 413330 - 16 - Claims 1. Computer-implemented method for processing digital images, characterized by providing (302) a sequence of digital images (102), wherein the sequence comprises a first digital image and a second digital image, wherein the first digital image is assigned to a first time point in the sequence, wherein the second digital image is assigned to a second time point in the sequence, wherein the digital images contain pixels, encoding (304) the pixels of the first digital image with a first encoder (202) into a first feature mapping (204), encoding (304) the pixels of the second digital image with the first encoder (202) into a second feature mapping (204), and encoding (308) at least the first feature mapping (204) and the second feature mapping (204) with a second encoder (206) into a third feature mapping (210).

2. Method according to claim 1, characterized by predicting the speed of an object or classifying (310) an object represented by the pixels of the second digital image with a first head (212) depending on the third feature image (210) or semantic segmentation (310) of the pixels of the second digital image with a first head (212) depending on the third feature image (210) or detecting (310) an object represented by the pixels of the second digital image with a first head (212) depending on the third feature image (210).

3. Method according to claim 2, characterized by predicting a speed of the object or classifying (310) the object depicted by the pixels of the second digital image with a second head (212) depending on the second feature image (210) or semantic segmentation (310) of the pixels of the second digital image with a second head (212) depending on the second R. 413330 - 1 7 - Feature mapping (210) or detecting (310) an object mapped by the pixels of the second digital image with a second head (212) depending on the second feature mapping (210) and performing multitask training of at least two of the first coder (202) and the second coder (206) and of the first head (212) and of the second head (212).

4. The method of claim 3, characterized in that the sequence comprises a third digital image, wherein the third digital image is assigned to a third point in time in the sequence, and by encoding the pixels of the third digital image with the first encoder (202) into a fourth feature image (204), encoding at least the second feature image (204) and the fourth feature image (204) with the second encoder (206) into a fifth feature image (210), mapping the third feature image (210) with the first head (212) into a result (214) that is assigned to the second digital image, and mapping the fifth feature image (210) with the first head (212) into a result (214) that is assigned to the third digital image, wherein the method provides a first reference for the result (214) that is assigned to the second digital image, and a second reference for the result (214).that is associated with the third digital image, and the multitask training (312) of at least two of the first encoder (202) and the second encoder (206) and the first head (212) and the second head (212) includes a loss, wherein the loss includes a first loss expression that depends on the first reference and the result (214) associated with the second digital image, wherein the loss includes a second loss expression that depends on the second reference and the result (214) associated with the third digital image, 5. Method according to claim 4, characterized by providing (312) a first weighting for the first loss expression and a second weighting for the second loss expression, wherein the third time point is after the second time point and wherein the second weighting is greater than the first weighting. R. 413330 - 18 - 6. Method according to claim 5, characterized by storing (308) the second feature image (204) instead of the first feature image (204) in the buffer memory (216) due to encoding at least the first feature image (204) and the second feature image (204) into the third feature image (210).

7. Method according to claim 5, characterized by reading (306) a feature image (204) from the buffer memory (216), encoding (308) the read feature image (204) and the first feature image (204) and the second feature image (204) into the third feature image (210) and storing (308) the second feature image (204) in addition to the first feature image (204) in the buffer memory (216) due to the encoding of at least the read feature image (204), the first feature image (204) and the second feature image (204) into the third feature image (206), in particular storing the first feature image (204) or the second feature image (204) instead of the read feature image (204) in the buffer memory (216).

8. Method according to one of the preceding claims, characterized by receiving (302) the first digital image at the first time, receiving the second digital image at the second time and encoding the pixels of the first digital image into the first feature image (204) between the first time and the second time.

9. Method according to one of the preceding claims, characterized by storing (308) the first feature image (204) in a buffer memory (216), in particular before the second time, and reading (308) the first feature image (204) from the buffer memory (216).

10. Method according to one of the preceding claims, characterized by receiving (302) the digital images from a sensor (110), in particular a camera, a lidar sensor, a radar sensor, an ultrasonic sensor, a motion sensor, and processing (304, 306, 308, 310) the digital images to obtain the result (210) for at least one of the R. 413330 - 19 - to determine the received digital images, and the operation (314) of a technical system (112), in particular a robot, a vehicle, a household appliance, an imaging device, a computer-controlled machine, depending on the result (210).

11. Device (100) for processing digital images (102), characterized by at least one processor (104), at least one data storage device (106) and at least one interface (108) for receiving the digital images, wherein the at least one data storage device (106) stores instructions which, when executed by the at least one processor (104), cause the device (100) to execute the method according to one of claims 1 to 10.

12. Computer program characterized by computer-readable instructions which, when executed by a computer, cause the computer to execute the method according to any one of claims 1 to 10.

Images