Method for detecting objects by means of two neural networks

Abstract

The invention relates to a method and a device for detecting objects, wherein objects are detected in visual data detected by means of a sensor, wherein two neural networks are used for object detection, and the object detection comprises the following method steps: running a first neural network to recognize geometric shapes in the visual data, wherein the detected visual data is used as input quantity for the first neural network; running a second neural network to recognize objects in the visual data, wherein not only the detected visual data but also the geometric shapes recognized by means of the first neural network are used as input quantity for the second neural network.

Description

Technical Field

[0001] The present invention relates to a method and a device for detecting objects, wherein objects are detected in visual data acquired by means of a sensor, wherein two neural networks are used for object detection, and the object detection comprises the following method steps:

[0002] - running a first neural network to identify geometric shapes in said visual data,

[0003] wherein the detected visual data is used as an input for the first neural network;

[0004] - running a second neural network to identify objects in said visual data,

[0005] o In this case, both the detected visual data and the geometric shapes identified with the aid of the first neural network are used as input variables for the second neural network. Background Art

[0006] It is well known in the prior art that, for example, in autonomous driving applications, machine learning using neural networks (also known as deep neural networks, DNNs) can automatically identify objects. While machine learning is unclear about what the DNN learns, specifically how it identifies the object, its recognition rate is significantly superior to that of hand-coded algorithms.

[0007] However, the danger when recognizing objects using DNNs lies in targeted attacks (for example, with the help of adversarial pertubations). In such attacks, the actual object is maliciously manipulated. That is, there is no need to intervene in the database or the learning process of the DNN. For example, graphic information is deliberately added to the actual object (for example, a traffic sign) (for example, by means of a specifically formed label). This information is also detected and evaluated when the object is detected. The additional information is formed so that it affects the evaluation. The recognition of the object is affected by the transmitted additional information. It is therefore possible that, due to the added information, a maliciously manipulated object is recognized as another object. Summary of the Invention

[0008] In contrast, the method according to the present invention advantageously enables accurate object identification even in the event of malicious manipulation of the object. This is achieved according to the present invention by the features specified in the independent claim. Further embodiments of the invention are the subject matter of the dependent claims.

[0009] According to the present invention, a method for detecting objects is provided, wherein objects are detected in visual data acquired by means of a sensor, wherein two neural networks are used for object detection, and the object detection comprises the following method steps:

[0010] - running a first neural network to identify geometric shapes in said visual data,

[0011] wherein: using the detected visual data as input for a first neural network; and running a second neural network to identify objects in the visual data.

[0012] In this case, both the detected visual data and the geometric shapes identified with the aid of the first neural network are used as input variables for the second neural network.

[0013] In this sense, the core of the present invention is to expand the current DNN-based recognition with an additional DNN, wherein the additional DNN is trained only for the recognition of basic geometric shapes (such as horizontal, vertical and inclined lines, circles, N-gons). This geometric information clearly also influences the decisions of the original DNN. In other words, it is proposed to expand the recognition of objects with the help of DNN through targeted geometric information. This makes the algorithm more stable against attacks caused by, for example, malicious manipulation of objects. As a result, a possible DNN recognition rate that is higher than the previous DNN recognition rate is advantageously achieved through geometric evaluation.

[0014] That is, a method for detecting objects is provided. For this purpose, visual data of the surroundings of a motor vehicle are detected, for example, by means of sensors. Objects are detected in the detected data by means of a data processing unit. Two trained neural networks are used to evaluate the visual data. The first neural network is trained to recognize geometric shapes. The visual data is used as an input layer. The second neural network is trained to recognize objects, using the recognized geometric shapes as an input layer in addition to the visual data. Object recognition should be understood to mean not only the identification of objects but also their classification. The sensor for detecting the visual data should be understood in particular to be a video camera of the motor vehicle.

[0015] In one advantageous embodiment, the method is used to operate a driver assistance system in a motor vehicle. A driver assistance system is understood to be any system that performs driving functions and / or assists the driver in guiding the vehicle. For example, a corresponding driver assistance system enables automatic object recognition to enable automated driving. Of course, it can also assist with partially automated driving functions. Alternatively, it can assist with other driver assistance functions, such as the recognition and display of traffic signs.

[0016] In an advantageous embodiment of the method, visual data of the vehicle's surroundings are detected by means of sensors. This means that the vehicle includes a camera, which records at least a portion of the vehicle's surroundings. The resulting data is then evaluated by the method according to the present invention for object recognition.

[0017] In one possible embodiment of the method, the first neural network is trained in one method step for recognizing simple geometric shapes.

[0018] The DNN is first trained to recognize simple geometric shapes. For example, the DNN is trained to recognize the typical shapes of traffic signs. In the example of a priority lane sign, this would be, for example, four diagonal lines and a square as the overall shape. As a basis for DNN learning, a large amount of data from various photographed traffic signs needs to be combined with individual information about the geometric shapes. The recognition of the geometric shapes can then be expanded to include spatial shapes and, if necessary, motion models. A further DNN is then trained not only to process the image information of the visual data but also to learn the geometric shape information of the aforementioned geometric shape DNN as a further input variable. This allows object recognition to be optimized. Methods for learning DNNs and DNN software can call upon established and available methods, such as the TensorFlow Framework, to optimize resource utilization. Of course, learning methods and approaches designed specifically for the application can also be developed and used to achieve the best possible results.

[0019] In a preferred embodiment of the method, the first neural network is trained in one method step for recognizing spatial geometric shapes.

[0020] This means that, in a preceding method step, a first neural network is trained not only for recognizing simple geometric shapes, such as straight lines, but also for recognizing complex geometric shapes of a motor vehicle, such as the body shape. The trained geometry DNN is then used to recognize and classify geometric shapes in the visual data. The information thus determined is transmitted as additional information to a further DNN for optimized object recognition in the visual data.

[0021] In an alternative development of the method, the first neural network is trained in one method step for recognizing movement patterns.

[0022] This means that, in the aforementioned method steps, the first neural network is trained not only to recognize simple geometric models, but also complex motion models, such as the movement patterns of pedestrians, people on bicycles, or animals. A motion model is understood here to mean the dynamic changes of complex geometric shapes. A person, for example, is a complex geometric shape, and their movements are dynamic changes of geometric shapes. The trained motion DNN is then used to recognize and classify motion in visual data. The information thus determined is transmitted as additional information to a further DNN for optimized object recognition in visual data.

[0023] In an advantageous embodiment of the method, the method is designed to reduce the influence of interfering content in the acquired visual data during the detection of objects.

[0024] This is understood to mean the use of a method for detecting objects. Advantageously, the method steps are designed so as to reduce or completely avoid the influence of visual malicious manipulation. This is understood to mean increasing or improving the accurate recognition of maliciously manipulated objects. A measure for this is, for example, the RSR (Ratio of Successful Recognition). Examples of malicious manipulations, also called attacks, include: adversarial interference, general interference, random noise, etc. In other words, the method achieves the effect of reducing the attack or suppressing the attack attempt in the visual data as best as possible.

[0025] In one possible embodiment of the method, the method is designed for identifying traffic signs. This means that the method is used to identify traffic signs. The method can also achieve optimized classification in the case of maliciously manipulated traffic signs.

[0026] The solution proposed here also proposes a device, which is designed to implement, control or realize the steps of a variant of the method described here in a corresponding instrument. Through this embodiment variant of the present invention in the form of a device, the task of the present invention can also be solved quickly and efficiently.

[0027] Here, the term "device" is understood to mean an electronic device that processes sensor signals and outputs control and / or data signals based on them. The device may have an interface, which may be designed as hardware and / or software. In a hardware-based design, the interface may, for example, be part of a so-called ASIC system that contains the various functions of the device. However, it is also possible for the interface to be a separate, integrated circuit or to be composed at least partially of discrete components. In a software-based design, the interface may be a software module that exists alongside other software modules, for example on a microcontroller.

[0028] Therefore, a device is provided that is configured to implement the described method. This device can be understood as a correspondingly designed data processing unit. Advantageously, this device is understood as a data processing unit having two DNNs. In particular, this device is understood as a data processing unit having a geometry DNN and an object DNN.

[0029] Advantageously, a computer program product or a computer program having a program code can be stored on a machine-readable carrier or storage medium, such as a semiconductor memory, a hardware memory or an optical memory, and is used to execute, implement and / or control a method according to one of the aforementioned embodiments, in particular when the program product or the program is executed on a computer or a device. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] It should be noted that the individual features cited in the description can be combined with one another in any technically meaningful manner and represent further embodiments of the invention. Further features and applications of the invention are apparent from the description of the exemplary embodiments with reference to the drawings.

[0031] In the attached figure:

[0032] Figure 1 illustrates the method steps of an exemplary embodiment; and

[0033] Figure 2 A diagram showing an exemplary application. DETAILED DESCRIPTION

[0034] exist Figure 1 The method steps of one embodiment of the present invention are shown in FIG. In a first step S1, visual data of the surroundings of a motor vehicle are detected with the aid of a camera. In a subsequent step S2, the visual data are preprocessed. In the next two steps, objects are detected in the visual data. The visual data are first provided as input (input layer) to a first neural network. In step S3, geometric shapes are detected in the visual data with the aid of the first neural network and are classified. The information obtained in this way is then provided as input to a second neural network. In addition, the detected visual data are also provided as further input to the second neural network. In step S4, objects are detected in the visual data with the aid of the second neural network and are classified. Then, in step S5, a specific action is performed based on the detected object. For example, an autonomous vehicle stops at a recognized stop sign.

[0035] exist Figure 2An exemplary application is shown in FIG. A motor vehicle 1 is shown here. The motor vehicle is designed as an autonomous shuttle bus. In addition to other sensors, the motor vehicle 1 also includes a camera 2 as a sensor for detecting visual data of the surroundings 6. The detection range of the camera is shown with the aid of a dotted line. The visual data detected by the sensor 2 are processed in a data processing unit 3. The data processing unit 3 includes a first neural network 4 and a second neural network 5. Traffic signs 7 and stop signs are located in the detection range of the sensor 2. The traffic sign 7 has an element 8 that acts as an interfering or erroneous content in the detected visual data, a so-called adversarial perturbation. This element 8 is formed in such a way that it strongly influences the evaluation based on the neural network. This can, for example, cause a stop sign to not be recognized as the stop sign or to be recognized incorrectly as another traffic sign.

Claims

1. A method for detecting an object, wherein: An object (7) is detected in visual data acquired by means of a sensor (2), wherein two neural networks (4, 5) are used for object detection and the object detection comprises the following method steps: - running a first neural network (4) to identify geometric shapes in said visual data, wherein the detected visual data is used as input for the first neural network (4); - running a second neural network (5) to identify objects (7) in said visual data, In this case, both the detected visual data and the geometric shapes identified by means of the first neural network (4) are used as input variables for the second neural network, The first neural network is trained only for the recognition of basic geometric shapes. The method is designed to reduce the influence of interference content (8) in detected visual data when detecting an object in the event that the object is maliciously manipulated, wherein the interference content is an adversarial perturbation.

2. The method according to claim 1, wherein Visual data of the surroundings (6) of the motor vehicle (1) are detected by means of the sensor (2).

3. The method according to claim 1 or 2, wherein: The first neural network (4) is trained in a method step for recognizing simple geometric shapes.

4. The method according to claim 1 or 2, wherein: The first neural network (4) is trained in a method step for recognizing spatial geometric shapes.

5. The method according to claim 1 or 2, wherein: The first neural network (4) is trained in a method step for recognizing movement patterns.

6. The method according to claim 1 or 2, designed for recognizing traffic signs (7).

7. A device (2, 3, 4, 5) configured to carry out the method according to any one of claims 1 to 6.

8. A computer program product comprising a program code which, when executed by a computer, causes the computer to perform the method according to any one of claims 1 to 6. 9 . A machine-readable storage medium having a program code stored thereon, wherein when the program code is executed by a computer, the program code causes the computer to execute the method according to claim 1 .

Images