Identifying objects in images under equiformity or invariance with respect to object size

Abstract

Recognizing objects in an image under equivariance or invariance relative to object size. A method (100) for recognizing at least one object (2a-2c) in at least one input image (3) has the steps of: processing the original image (1) of the object (2a-2c) into at least one original feature map (5, 5a-5c) by a first convolutional neural network (4a) (110); processing the input image (3) into at least one input feature map (6, 6a-6c) by a second convolutional neural network (4b) (120); and combining the at least one original feature map (5, 5a-5c) with at least one • Compare the input feature maps (6, 6a-6c) (130); • Evaluate (140) from the result (130a) of the comparison (130) whether and, if necessary, at what location (2a'-2c') the object (2a-2c) is contained in the input image (3), wherein the convolutional neural networks (4a, 4b) each contain multiple convolutional layers (7a, 7b), and wherein at least one of the convolutional layers (7a, 7b) is at least partially composed of at least two filters (8a-8c) that can be converted to each other by scaling operations.

Description

Technical Field

[0001] The present invention relates to the identification of objects in images, which in particular can be used to track the movement of these objects for driver assistance systems or at least partially automated driving of vehicles. Background Technology

[0002] Driver assistance systems, such as electronic stability programs, continuously observe the current driving conditions in a sensing manner and make a decision about whether to intervene in the vehicle's dynamics, for example, by braking a single wheel. Systems designed to drive the vehicle at least partially automatically continuously intervene in the dynamics and, for this purpose, pre-plan multiple trajectories at intervals of several seconds. One of these trajectories is then selected and traversed based on boundary conditions and optimality criteria.

[0003] In mixed traffic with human traffic participants, especially these human traffic participants and other moving objects, short-term planning changes may be necessary. A method for adapting the trajectory of a vehicle to the behavior of moving external objects is known from DE 10 2018 210 280 A1. Summary of the Invention

[0004] Within the scope of this invention, a method for identifying at least one object in at least one input image has been developed.

[0005] In this method, the original image of the object (Vorlage-Bild) is processed into at least one original feature map (“feature map”) using a first convolutional neural network (CNN). The input image is then processed into at least one input feature map using a second convolutional neural network (CNN). Therefore, instead of investigating the open-ended question of which objects are contained in the input image, the method specifically seeks to determine whether a fully defined object is present in the input image.

[0006] In this context, the term "image" typically includes the assignment of the value of at least one measurement parameter to each position within a two- or three-dimensional spatial coordinate grid. Thus, for example, a camera measures the intensity of light or thermal radiation for each pixel of its image sensor. An ultrasonic sensor similarly measures the spatial distribution of the intensity and flight time of reflected ultrasonic waves. A radar sensor or lidar sensor measures the direction and distance of the position of a reflected radar or laser beam. Therefore, the measurement data for all the aforementioned measurement modes can be considered as images, although this enumeration is not exhaustive.

[0007] Compare at least one original feature map with at least one input feature map based on any metric. Evaluate from the comparison results whether, and if necessary, at what location, the object is included in the input image.

[0008] This type of recognition is particularly advantageous for tracking the motion of objects, also known as "tracking." For this purpose, objects are searched for in a time series of images, and results regarding whether and, if necessary, at what location the object is present in the input image are combined to track the object's motion. In this case, the specific object to be tracked can be fixed from the beginning, but can also be selected from the input image, for example, during runtime. For instance, the motion of multiple objects can also be tracked based on the same time series of images, such as those showing the kinematics of a traffic event.

[0009] Convolutional neural networks (CNNs) each contain multiple convolutional layers. At least one of the convolutional layers consists at least in part of at least two filters, which can be interconverted through scaling operations. Particularly preferred are three to five interconvertible filters. The more interconvertible filters there are, the better the resolution that can be used to additionally evaluate the size of objects in the input image, but the greater the storage requirements for the network.

[0010] Scaling operations that convert filters to each other can involve one, more, or all coordinate directions of the input image.

[0011] It has been recognized that using filters that can be converted to each other by scaling enables the decoupling of the search for semantically consistent objects with those in the original image from the search for objects of the same size as those in the original image when searching for objects.

[0012] In the absence of filters that allow for mutual conversion, the consistency found between objects in the original image and the input image is an indivisible dichotomy (Zweiklang) between appearance consistency (e.g., semantic consistency of shape) and size consistency. Therefore, if, for example, the input image contains multiple objects of very different sizes, and only a single object among these objects is similar in size to an object in the original image, then this size consistency alone already carries such weight in the comparison of the original and input feature maps that objects of similar size in the input image are evaluated as consistent with the searched object in the original image. For this reason, this is not necessarily an obstacle of semantically referring to completely different objects (e.g., referring to a truck (LKW) rather than a passenger car (PKW)).

[0013] The use of interchangeable filters allows for the free choice of whether the feature maps output by the convolutional layers and the convolutional neural network should not change when the size of the objects in the image changes (in which case the image processing is size invariant), or whether these feature maps should change in a predetermined way when the size of the objects in the image changes (in which case the image processing is size equivariant).

[0014] Therefore, it might be meaningful, for example, that driver assistance systems, or systems for at least partially automated driving, similarly detect all the presence of road signs in the vehicle's environment ahead, since road signs indicate road direction and thus, for example, enable long-term planning of the trajectory to be followed by the vehicle. Those road signs closest to the vehicle appear much larger (up to 10 times larger) in the input image provided by the vehicle's sensor system than those furthest from the vehicle that are still just visible. To detect all road signs, it makes sense to process the image in a size-invariant manner.

[0015] For short-term trajectory planning, not all other traffic participants (e.g., pedestrians or other vehicles) visible in the input image are relevant; only those traffic participants close enough to the vehicle to interact with it during the current planning range are relevant. For an exemplary 30-second planning range, it's unnecessary to meticulously trace the movement of another vehicle over a distance of 500 m that the vehicle might catch within 3 minutes with a 10 km / h speed difference. That is, if the distance to the object is known, the limited computing power available onboard to the vehicle, for example, for accurately "tracking" the object, can be focused on those objects currently relevant to the vehicle's traffic.

[0016] This allows for the implementation of interconvertible filters. In a particularly advantageous configuration, at least one convolutional neural network outputs multiple feature maps, with one of the interconvertible filters primarily involved in the creation of each feature map. For example, multiple layers of the convolutional neural network can each include the same layout (Konstellation) of interconvertible filters, such that information in these layers can be processed individually according to different scales. Thus, for example, during transitions between layers, the output of a particular filter from the layout can preferably be passed back to the corresponding filter in the next layer. Lateral connections between scales are not excluded in this case. However, the interconvertible filters, for example, in the last layer, can then each provide a feature map, where the same filter in the layout is essentially always involved in the processing of the feature map in each layer. That is, in general, feature maps of the image are formed for different scales. The processing of the input image is size-equal.

[0017] These feature maps can be advantageously considered individually for comparison with one or more feature maps from other images, respectively. The results of these comparisons can then be aggregated to provide information about the size of objects in the input image.

[0018] In this case, in particular, the size of the object in the input image can be interpolated between the scales involved in the filters in the layout. If, for example, the first filter involves a scale of 1, the second filter involves a scale of 2, and the third filter involves a scale of 3, then the local consistency of 1 determined from the first feature map between the original image and the input image and the local consistency of 0.2 determined from the second feature map are aggregated into an estimate: there exists an object in the input image with a size of approximately 1.2.

[0019] Therefore, a fairly good estimate of the size of objects in the input image can be determined using only a small number of filters.

[0020] Then, from the determined size of the object in the input image and the absolute size of the object that is at least approximately known beforehand, the distance between the sensor used to record the input image and the object can be evaluated. This distance can then be used, for example, to distinguish between objects that are currently strongly related to vehicle traffic and objects that are less related or not related at all.

[0021] For processing where the input image size remains constant, these feature maps can be aggregated using a function symmetric about the arrangement of feature maps provided by interconvertible filters. The results provided by this function can then be further processed within their respective convolutional neural networks.

[0022] Processing input images with constant or variable sizes is advantageous not only in the context of vehicle systems. It is also advantageous, for example, in the optical quality inspection of mass-produced products, if the identification of defects or damage functions as independently as possible of the distance between the camera and the corresponding product at the time of image recording. In the case of access control systems that authorize access based on facial recognition, identification in the near zone of a door to be opened should be as independent as possible of how close a person is to the camera and how large the person appears in the camera image.

[0023] Regardless of whether the processing of the input image is performed in a size-equal or size-invariant manner, at least one original feature map can be compared with at least one input feature map, for example, via a spatially resolved correlation function. The location within the input image where the correlation function is maximized (and / or exceeds a pre-given threshold) can then be evaluated as the location where the object is contained in the input image.

[0024] In a particularly advantageous configuration, convolutional neural networks have a consistent architecture, and their behavior is characterized by the same parameters. This implies that interconvertible filters, through scaling, exist not only in one network or another, but in both networks. Thus, for example, the same convolutional neural network can be used alternately to process the original image and the input image, saving storage space for parameters used throughout the network. If, for example, the motion of the same object needs to be tracked over a longer period of time in a sequence of always new input images, then the first convolutional neural network must only be used once to process the input image into an original feature map. While continuously tracking the object in the sequence of input images, a second convolutional neural network is then used to continuously create new input feature maps, but the original feature maps can be reused repeatedly.

[0025] If the same object is found in the time series of the input image, results regarding whether and, if necessary, the object is present in the input image can be aggregated for tracking the object's motion. This information can be used, in particular, in driver assistance systems or systems for at least partially automating vehicle driving to plan the vehicle's future behavior. Thus, in a particularly advantageous configuration, the input image is detected using at least one sensor carried by the vehicle. The determined motion and / or determined motion intention of the object are considered by the vehicle's driver assistance system and / or by systems for at least partially automating vehicle driving, for planning the trajectory to be traversed by the vehicle, and / or for determining interventions in the vehicle's driving dynamics.

[0026] For this purpose, information about whether and, if necessary, at what location an object is contained in the input image can be provided, in particular, in the form of a “bounding box” in two-dimensional or three-dimensional space, which is assumed to be occupied by an object.

[0027] Motion intention, in particular, can include, for example, predictions about whether an object will maintain its current state of motion or to what extent that state of motion might change within a pre-given time frame. In the case of purely passive, uncontrolled objects (e.g., a lost piece of cargo), motion intention can be predicted essentially based on the laws of physics. Vehicles and pedestrians, however, are controlled by the intentions of their respective traffic participants. In a particularly advantageous configuration, the intention used for this conjecture, and thus the prediction of the object's motion intention, is determined from the temporal changes in the determined position of the object in the input image and from the temporal changes in the determined size and / or distance of the object.

[0028] As previously explained, the distance between the object and the vehicle is particularly important for determining the extent to which the object is traffic-related to the vehicle's behavior in the near future. Therefore, in another advantageous configuration, the determined distance between the sensor and the object is used to determine whether the temporal changes in the object's position are evaluated within a sequence of input images and / or the extent to which the object is related to the vehicle's current traffic conditions. Typically, the computational power available onboard a vehicle is insufficient to truly track every object visible in the vehicle's environment. This situation is roughly comparable to driving lessons learned by human learners, who must also learn to allocate all available attention to the most important objects.

[0029] Interconvertible filters can, in particular, be linear combinations of fundamental functions in a function space with free coefficients. These coefficients can be trained, at least in part, along with other parameters characterizing the behavior of the convolutional neural network. In this case, in addition to the coefficients of the linear combination, the parameters characterizing the behavior of the interconvertible filters can, in particular, include other parameters characterizing at least one geometric transformation of the filter. For example, these parameters can characterize the rotation of the filter.

[0030] This method can be implemented entirely or partially by a computer. Therefore, the invention also relates to a computer program having machine-readable instructions that, when executed on one or more computers, cause the computers to perform the method. In this sense, control devices for vehicles and embedded systems for technical devices, also capable of executing machine-readable instructions, can be considered computers.

[0031] Similarly, the present invention also relates to machine-readable data carriers and / or downloadable products having computer programs. Downloadable products are digital products that can be transmitted via a data network, i.e., downloaded by a user of the data network, and can be sold, for example, in an online store for immediate download.

[0032] In addition, computers may be equipped with computer programs, machine-readable data carriers, or downloadable products. Attached Figure Description

[0033] Other measures to improve the invention are shown in more detail below, together with the description of preferred embodiments of the invention according to the figures.

[0034] Figure 1 An embodiment of a method 100 for identifying at least one object 2a-2c in at least one input image 3 is shown;

[0035] Figure 2 shows the composition of the basic function 9 of the filters 8a-8c that can be converted to each other by scaling, and the effect of these filters 8a-8c in the convolutional neural network 4a.

[0036] Figure 3 shows the effect of the interconvertible filters 8a-8c on the recognition of object 2a in neural networks 4a and 4b. Detailed Implementation

[0037] Figure 1 This is a schematic flowchart of an embodiment of a method 100 for identifying at least one object 2a-2c in at least one input image 3.

[0038] In step 110, the original image 1 of the objects 2a-2c to be identified is processed into at least one original feature map 5, 5a-5c by a first convolutional neural network (CNN) 4a. In step 120, the input image 3 is processed into at least one input feature map 6, 6a-6c by a second convolutional neural network (CNN) 4b. CNNs 4a and 4b each contain multiple convolutional layers 7a and 7b. At least one of the convolutional layers 7a and 7b is at least partially composed of at least two filters 8a-8c, which can be transformed into each other through scaling operations.

[0039] In this case, in particular, multiple feature maps 5a-5c or 6a-6c can be generated according to block 111 or 121, wherein one of the filters 8a-8c is mainly involved in the completion of the feature map.

[0040] Alternatively, according to block 112 or 122, these feature maps can be combined using a function symmetric to the arrangement of feature maps provided by the interconvertible filters 8a-8c. According to block 113 or 123, the results can be further processed in their respective convolutional neural networks 4a, 4b.

[0041] In step 130, at least one original feature map 5, 5a-5c is compared with at least one input feature map 6, 6a-6c. Then in step 140, the result 130a of the comparison 130 is evaluated to determine whether, and if necessary, at which locations 2a'-2c', objects 2a-2c are contained in the input image 3.

[0042] Comparison 130 can be performed individually for multiple feature maps 5a-5c; 6a-6c, for example, according to block 131. Then, according to block 132, the results of these comparisons can be aggregated into information about the size 2a*-2c* of objects 2a-2c in the input image 3. According to block 133, the distance 2a**-2c** between the sensor used to record the input image 3 and the objects 2a-2c can be evaluated from the size 2a*-2c* and the previously known absolute size 2a#-2c# of the objects 2a-2c.

[0043] Typically, according to block 134, at least one original feature map 5, 5a-5c can be compared with at least one input feature map 6, 6a-6c via a spatially resolved correlation function. Then, for example, according to block 141, the location where the correlation function reaches its maximum value and / or exceeds a pre-given threshold can be evaluated as the location 2a'-2c' where the object 2a-2c is contained in the input image 3.

[0044] Specifically, objects 2a-2c can be searched in the time series of input image 3. In step 150, the previously described search can therefore be repeated in other input images 3. In step 160, results regarding whether and, if necessary, at which locations 2a'-2c' are objects 2a-2c contained in input image 3 can then be aggregated for tracking the motion 2a''-2c'' of objects 2a-2c.

[0045] For example, according to block 161, the motion intention for object 2a-2c can be determined from the time change of the determined position 2a'-2c' of object 2a-2c in the input image (3) and from the time change of the determined size 2a*-2c* and / or the distance 2a**-2c** of object 2a-2c. Figure 2a The prediction of ***-2c***.

[0046] In particular, for example, at least one sensor carried by the vehicle can be used to detect the input image. Then, according to block 162, the determined motion 2a''-2c'' of objects 2a-2c and / or the determined motion intention can be considered by the vehicle's driver assistance system and / or by a system for at least partially automating the driving of the vehicle, for planning the trajectory to be traveled by the vehicle, and / or for specifying interventions on the vehicle's driving dynamics. Figure 2a ***-2c***.

[0047] According to block 163, it can be determined, based on the determined distance 2a**-2c** between the sensor and the object 2a-2c, whether the temporal change process of the position 2a'-2c' of the object 2a-2c is evaluated in the sequence of input images 3, and / or the extent to which the object 2a-2c is related to the current traffic conditions of the vehicle. As previously explained, resources for further processing can therefore be focused on the most important objects.

[0048] Figure 2a An example is shown: how can parameters 8* be derived from the basic function 9 using three scales? , and Three filters, 8a-8c, are generated, and these filters can be converted into each other through scaling operations.

[0049] Figure 2b This example illustrates how these filters 8a-8c function when applied in a convolutional neural network 4a to the original image 1 of object 2a and a scaled-down version 1' of that original image 1. In this example, the original image 1 produces a large response to the first feature map 5a, where filter 8a is applied at its maximum scale. Decisively involved in the completion of the first feature map. The scaled-down version 1' produces a qualitatively similar, but quantitatively significantly smaller, response, and this is not a response to feature map 5a, but rather to feature map 5c, where filter 8c operates at the smallest scale. It decisively participates in the completion of the feature map. Therefore, the processing of the original image 1 by the convolutional neural network 4a is equivariant, that is, the result varies in a predictable manner with the size of the object 2a in the original image 1.

[0050] Figure 2c A detailed explanation: How a convolutional neural network 4a with two convolutional layers 7a and 7b obtains the original image 1 in... Figure 2b The results shown utilize filters 8a-8c in the convolutional layers. Information processed at one of the three scales in the first convolutional layer 7a is essentially passed to the same scale in the second convolutional layer 7b. However, lateral connections to the next smaller scale are also present, as long as they exist.

[0051] Figure 3 illustrates how image processing using convolutional neural networks 4a and 4b affects the recognition of object 2a in input image 3, where filters 8a-8c that can be converted between each other by scaling are used in the convolutional neural networks.

[0052] For comparison, Figure 3aThe following describes the processing according to existing technology. Here, the original image 1 is processed into feature map 5 using a first convolutional neural network 4a, the original image showing only the passenger car as object 2a. In addition to the passenger car 2a, the input image 3 also shows a strongly magnified copy 2b of the passenger car and a cargo vehicle 2c. The input image 3 is processed into feature map 6 using a second convolutional neural network 4b. Convolutional neural networks 4a and 4b lack filters 8a-8c that can be converted between each other by scaling.

[0053] Using cross-correlation comparison features, Figures 5 and 6 show strong consistency only in the upper left corner of the input image 3, where the truck 2c is imaged at approximately the same size as the passenger vehicle 2a in the original image 1. Therefore, the passenger vehicle 2a is identified at the incorrect location 2a'.

[0054] Figure 3b The processing is illustrated using convolutional neural networks 4a and 4b, which contain interconvertible filters 8a-8c that can be scaled. In this example, the feature maps provided by the interconvertible filters 8a-8c are combined with a symmetric function, and then only the result of this combination is further processed. Therefore, the processing of the original image 1 and the input image 3 is size-invariant.

[0055] This has the following effect: not only is the passenger car 2a in the upper right of the input image 2a identified as passenger car 2a, but also its much larger copy 2b in the lower half of the input image 3. Thus, two positions 2a' of passenger car 2a are identified, where strongly different sizes are not considered.

[0056] Unlike this, Figure 3c This illustrates that when the feature maps generated by the interconvertible filters 8a-8c are not combined, but rather resemble... Figure 2c How it works when propagated through convolutional neural networks 4a and 4b. The first convolutional neural network 4a then provides multiple original feature maps 5a-5c, and the second convolutional neural network 4b provides multiple input feature maps 6a-6c. Individual comparisons of these feature maps 5a-5c and 6a-6c result in the following: only the passenger car 2a in the upper right of the input image 3 matches the original image 1 not only in shape but also in size. Correspondingly, only one position 2a' of the passenger car 2a is output.

Claims

1. A method (100) for identifying at least one object (2a-2c) in at least one input image (3), the method comprising the steps of: • The original image (1) of the object (2a-2c) is processed into at least one original feature map (5, 5a-5c) (110) by the first convolutional neural network CNN (4a). • The input image (3) is processed into at least one input feature map (6, 6a-6c) (120) by a second convolutional neural network CNN (4b). • Compare the at least one original feature map (5, 5a-5c) with the at least one input feature map (6, 6a-6c) (130); • Evaluate (140) from the results (130a) of comparison (130) whether, and if necessary, at what location (2a'-2c') the object (2a-2c) is contained in the input image (3). The convolutional neural networks (4a, 4b) each contain multiple convolutional layers (7a, 7b), and at least one of the convolutional layers (7a, 7b) is at least partially composed of at least two filters (8a-8c), which can be transformed into each other through scaling operations. At least one of the convolutional neural networks (4a, 4b) outputs multiple feature maps (5a-5c; 6a-6c) (111, 121), wherein one of the interconvertible filters (8a-8c) is primarily involved in the completion of the feature maps.

2. The method (100) according to claim 1, wherein the comparison (130) (131) is performed individually for the plurality of feature maps (5a-5c; 6a-6c), and wherein the results of these comparisons are aggregated into information (132) about the size (2a* -2c*) of the objects (2a-2c) in the input image (3).

3. The method (100) according to claim 2, wherein the distance (2a**-2c**) between the sensor used to record the input image (3) and the object (2a-2c) is evaluated from the size (2a*-2c*) of the object (2a-2c) in the input image (3) and the prior known absolute size (2a#-2c#) of the object (2a-2c) (133).

4. The method (100) according to claim 1, wherein the feature maps are aggregated (112, 122) using a function symmetric about the arrangement of feature maps provided by the interconvertible filters (8a-8c), and the results are then further processed in the corresponding convolutional neural network (113, 123).

5. The method (100) according to any one of claims 1 to 4, wherein the at least one original feature map (5, 5a-5c) is compared with the at least one input feature map (6, 6a-6c) via a spatially resolved correlation function (134), and the position where the correlation function is maximized is evaluated as the position (2a'-2c') where the object (2a-2c) is contained in the input image (3) (141).

6. The method (100) according to any one of claims 1 to 4, wherein the convolutional neural network (4a, 4b) has a consistent architecture and its behavior is characterized by the same parameters.

7. The method (100) according to any one of claims 1 to 4, wherein the object (2a-2c) is searched in the time series of the input image (3) (150), and wherein results (160) are aggregated regarding whether and, if necessary, at which position (2a'-2c') the object (2a-2c) is contained in the input image (3) to track the motion (2a''-2c'') of the object (2a-2c).

8. The method (100) according to claim 7, wherein a prediction (161) for the motion intention (2a***-2c***) of the object (2a-2c) is determined from the temporal change of the determined position (2a'-2c') of the object (2a-2c) in the input image (3) and from the temporal change of the determined size (2a*-2c*) and / or distance (2a**-2c**) of the object (2a-2c).

9. The method (100) of claim 7, wherein the input image is detected using at least one sensor carried by the vehicle, and wherein the determined motion (2a''-2c'') and / or the determined motion intention (2a***-2c***) of the object (2a-2c) are considered by the vehicle's driver assistance system and / or by means of at least partially automating driving the vehicle, planning the trajectory to be traveled by the vehicle and / or determining interventions on the driving dynamics of the vehicle (162).

10. The method (100) of claim 9, wherein a determination (163) is made based on the determined distance (2a**-2c**) between the sensor and the object (2a-2c): whether to evaluate the temporal change of the position (2a'-2c') of the object (2a-2c) in the sequence of input images (3), and / or the extent to which the object (2a-2c) is related to the current traffic conditions of the vehicle.

11. The method (100) according to any one of claims 1 to 4, wherein the interconvertible filters (8a-8c) are linear combinations of basic functions (9) of a function space with free coefficients.

12. The method (100) according to claim 11, wherein the basic function (9) is a Hermitian polynomial.

13. The method (100) of claim 11, wherein, in addition to the coefficients of the linear combination, the parameter (8*) characterizing the behavior of the interconvertible filters (8a-8c) also includes other parameters characterizing at least one geometric transformation of the filters (8a-8c).

14. A computer program product having a computer program, the computer program containing machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100) according to any one of claims 1 to 13.

15. A machine-readable data carrier having a computer program containing machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100) according to any one of claims 1 to 13.

16. A computer equipped with a computer program and / or a machine-readable data carrier having the computer program, the computer program containing machine-readable instructions that, when executed on one or more computers, cause the one or more computers to perform the method (100) according to any one of claims 1 to 13.

Images