Method and apparatus for reliably identifying objects in video images

Abstract

Computer-implemented method for reliably identifying an object in a sequence of input images (x) received by means of an imaging sensor (30), wherein the positions of light sources in the respective input images (x) are determined from the input images (x) by means of a first machine learning system (NN1), in particular an artificial neural network, and wherein the object (i) in the sequence of input images (x) is identified from the resulting sequence of positions (P1, P2) of the light sources by means of a second machine learning system (NN2), in particular an artificial neural network.

Description

Technical Field

[0001] The present invention relates to: a method for reliably identifying objects within an image detected by an imaging sensor; a system configured to implement the method; a method for training the system; a training apparatus; a computer program; and a machine-readable storage medium. Background Technology

[0002] A method for detecting an object in a provided input signal is known from DE 10 2017 223 264.1, which has not yet been disclosed, wherein the object is detected based on the provided input signal and wherein an actuator is manipulated based on the detection of the object. Summary of the Invention

[0003] Advantages of the present invention

[0004] This invention examines a vision sensor that provides input images for, for example, automated driving functions, even in poorly lit scenes, where the poorly lit scenes have active light sources visible in the images.

[0005] In this situation, it's possible that the input images are improperly adjusted. This can result in only visible light sources in the image, such as vehicle headlights, streetlights, billboards, etc., being visible.

[0006] Standardization can adjust an image, but this may result in the surrounding environment being visible, but the location of the light source not being clearly determined.

[0007] If a machine learning system is taught to identify objects in an input image, it cannot be ruled out that light sources may be associated with vehicles and / or other objects. This can lead to each light source automatically causing the detected object to be identified. This results in problems such as reflections from wet driveways, reflective concrete surfaces, tunnel walls, etc., being identified as ghost objects.

[0008] It is practically impossible to set up real training data containing records of all possible light sources and reflections in the training of a machine learning system.

[0009] In contrast, the method of the present invention has the advantage that the light intensity is matched locally, so that no superposition occurs.

[0010] Furthermore, the method can be trained using simulations of moving and stationary light sources, in which reflections and occlusions for all relevant conditions can be synthetically generated.

[0011] Other aspects and advantageous extensions of the invention are also the subject of this invention.

[0012] In a first aspect, the present invention relates to a computer-implemented method for reliably identifying objects in a sequence of input images received by means of an imaging sensor, wherein the positions of light sources in the respective input images are determined by means of a first machine learning system, in particular an artificial neural network, based on the input images, and wherein the objects in the input image sequence are identified, in particular by means of a second machine learning system, such as an artificial neural network, based on the obtained sequence of light source positions.

[0013] This processing sequence has the advantage that the complex task of distinguishing between light sources and reflections and reliably identifying light sources that are at least partially occluded can be transferred to a machine learning system that can be trained at a reasonable cost. Then, based on the identified light sources, objects can be identified particularly easily. This task can also be transferred particularly well to a machine learning system that can be easily trained.

[0014] In one extended scheme, it can be specified that, in particular with the aid of a second machine learning system, attributes of the objects identified, especially those identified in the input image sequence, are also identified, in addition to the identified objects themselves. This can be implemented particularly easily using a second machine learning system. Attributes could be, for example, the (3D) position of the object captured by the sensor, the (3D) velocity of the object, or the sensor's own velocity, which, together with the object's (3D) velocity, characterizes the relative velocity between the sensor and the object.

[0015] Alternatively or additionally, it can be specified that, especially with the aid of a first machine learning system, the standardization of the corresponding input image is also determined. Standardization can specifically mean that any superimposition that may exist in the input image no longer exists in the standardization. Using this standardization, another source of information can be provided particularly efficiently, and this other source of information can also be used to identify objects. It is particularly advantageous that the first machine learning system not only performs standardization, that is, removing superimposition caused by the light source, but also performs the identification of the position of the light source, because the necessary information content for the corresponding actions in these two actions is closely related. Therefore, this machine learning system can be trained particularly efficiently.

[0016] To further utilize this information source, an extended scheme can specify that, with the aid of a third machine learning system, objects in the corresponding input image and their locations within the image are identified based on the standardization of the input image. Here, the design of the machine learning system is particularly simple because it is very easy to train.

[0017] In another aspect, it can be stipulated that the merging of objects identified in an alternative manner is achieved by means of a fourth machine learning system, that is, the fourth machine learning system identifies safe objects in the input image based on the objects identified by means of the third machine learning system, based on the objects identified from the position sequence of the light source, and especially based on the identified attributes.

[0018] In other respects, the present invention relates to a system configured to implement one of the previously mentioned methods, the system comprising a first machine learning system and a second machine learning system, and optionally also a third machine learning system and / or a fourth machine learning system.

[0019] Building upon this, in another aspect, a method is specified for training a computer implementation of one of these systems, wherein a scene is generated, comprising objects and light sources in pre-given locations within space, and a sequence of synthetic positions of the light sources is generated based on the scene, as would be captured by a camera from a pre-given camera position, wherein a second machine learning system is trained to deduce the positions of the objects based on the positions of the light sources, as would be captured by a camera from that pre-given camera position. This has the advantage that the positions of the light sources can be generated realistically in the simplest way, and thus training data can be generated particularly easily for a variety of conceivable scenes, making the trained system function particularly reliably.

[0020] In one extended scheme, it can be specified that: a first machine learning system is trained using a data record comprising pairs of augmented images and their corresponding pre-given locations, wherein these augmented images are obtained by artificially adding light sources to real images at pre-given locations, and wherein the first machine learning system is trained to determine these pre-given locations based on these augmented images. This is particularly advantageous because: although a large number of images with real light sources are difficult to generate using a suitable number of reflections, etc., light sources can be artificially added to real images, i.e., images captured by sensors, in a relatively simple way, for example, by means of ray tracing.

[0021] In one extended embodiment, it can be specified that the first machine learning system is also trained using a data record that includes pairs of real images and their corresponding augmented images, wherein the first machine learning system is trained to determine the corresponding real image based on these augmented images as standardized data. In this way, the first machine learning system can be trained particularly simply to calculate the light source from the image.

[0022] In other respects, the present invention relates to: a computer program configured to perform the above-described method; and a machine-readable storage medium having the computer program stored thereon. Attached Figure Description

[0023] The embodiments of the present invention will then be further described with reference to the accompanying drawings. In the drawings:

[0024] Figure 1 The structure of an embodiment of the present invention is illustrated schematically;

[0025] Figure 2 An embodiment for controlling at least a partially autonomous robot is illustrated schematically;

[0026] Figure 3 An embodiment for controlling a production system is illustrated schematically;

[0027] Figure 4 An embodiment for a control access system is illustrated schematically;

[0028] Figure 5 An embodiment for controlling and monitoring systems is illustrated schematically;

[0029] Figure 6 An exemplary scenario of detecting objects using sensors is shown;

[0030] Figure 7 An exemplary simplified diagram of a scene detected by sensors is shown;

[0031] Figure 8 An exemplary information flow in the system mentioned above is shown;

[0032] Figure 9 The possible construction of the training device is shown. Detailed Implementation

[0033] Figure 1 An actuator 10 is shown interacting with a control system 40 in its surrounding environment 20. The surrounding environment 20 is detected at preferably periodic intervals by a video sensor 30, which may also be provided by multiple sensors, such as a stereo camera. Sensor signals S from sensor 30, or individual sensor signals S in the case of multiple sensors, are transmitted to the control system 40. Therefore, the control system 40 receives a sequence of sensor signals S. Based on this, the control system 40 determines a control signal A, which is transmitted to the actuator 10.

[0034] The control system 40 receives a sequence of sensor signals S from the sensor 30 in an optional receiving unit 50. This receiving unit converts the sequence of sensor signals S into a sequence of input images x (alternatively, each sensor signal S can be directly used as the input image x). The input image x can be, for example, a fragment of the sensor signals S or further processing of the sensor signals S. The input image x includes individual frames of a video recording. In other words, the input image x is determined based on the sensor signals S. The sequence of input images x is then transmitted to the object identification system 60.

[0035] The object identification system 60 is preferably parameterized by parameters φ, which are stored in and provided by the parameter memory P.

[0036] The object identification system 60 determines an output parameter y based on the input image x. The output parameter y is sent to an optional modification unit 80, which determines a control signal A based on this parameter. This control signal is then sent to the actuator 10 to correspondingly control the actuator 10. The output parameter y includes information about the object detected by the video sensor 30.

[0037] Actuator 10 receives control signal A, is correspondingly controlled, and performs a corresponding action. In this case, actuator 10 may include (not necessarily structurally integrated) control logic that determines a second control signal to be used to control actuator 10 based on control signal A.

[0038] In other embodiments, the control system 40 includes a sensor 30. In still other embodiments, alternatively or additionally, the control system 40 also includes an actuator 10.

[0039] In other preferred embodiments, the control system 40 includes one or more processors 45 and at least one machine-readable storage medium 46 on which commands are stored, which, when executed on the processors 45, cause the control system 40 to perform the method according to the invention.

[0040] In alternative implementations, a display unit 10a is provided in addition to or in addition to the actuator 10.

[0041] Figure 2 The diagram illustrates how the control system 40 can be used to control at least a partially autonomous robot, here at least a partially autonomous motor vehicle 100.

[0042] Sensor 30 may be, for example, a video sensor preferably arranged in the motor vehicle 100.

[0043] The object identification system 60 is configured to reliably identify objects based on the input image x.

[0044] The actuator 10 preferably arranged in the motor vehicle 100 may be, for example, the braking device, drive device, or steering device of the motor vehicle 100. The control signal A can then be determined to cause the actuator or these actuators 10 to be operated so that, especially when certain types of objects are involved, such as pedestrians, the motor vehicle 100, for example, avoids a collision with an object reliably identified by the object identification system 60.

[0045] Alternatively, the at least partially autonomous robot could be other mobile robots (not shown), such as those that move by flying, floating, diving, or walking. The mobile robot could also be, for example, a at least partially autonomous lawnmower or a at least partially autonomous cleaning robot. In these cases, control signal A could also be determined to cause the mobile robot's drive and / or steering mechanisms to be manipulated to prevent the at least partially autonomous robot from colliding with objects identified by object identification system 60, for example.

[0046] Alternatively or additionally, the display unit 10a can be controlled using the control signal A, and for example, the determined safety area can be displayed. For example, in the case where the vehicle 100 has a non-automatic steering system, it is also possible that the display unit 10a is controlled using the control signal A to output an optical or audible warning signal if it is determined that the vehicle 100 is about to collide with one of the reliably identified objects.

[0047] Figure 3 The following embodiment is illustrated, in which the control system 40 is used to operate the production machine 11 of the production system 200 by manipulating the actuator 10 that controls the production machine 11. The production machine 11 may be, for example, a machine for stamping, sawing, drilling and / or cutting.

[0048] Sensor 30 may thus be, for example, an optical sensor that detects characteristics of finished products 12a and 12b. It is possible that these finished products 12a and 12b are movable. It is possible that the actuator 10 controlling the production machine 11 is manipulated based on the detected association between finished products 12a and 12b, so that the production machine 11 correspondingly performs the subsequent processing steps for the correct finished products 12a and 12b. It is also possible that by identifying the correct characteristics of the same finished product 12a and 12b (i.e., in the absence of incorrect associations), the production machine 11 correspondingly uses the same production steps to match the processing of subsequent finished products.

[0049] Figure 4The following embodiment is illustrated, in which a control system 40 is used to manipulate an access system 300. The access system 300 may include physical access control devices, such as door 401. A video sensor 30 is used to detect personnel. The detected images can be interpreted using an object identification system 60. If multiple personnel are detected simultaneously, their identities can be reliably determined, for example, by associating them with each other, such as by analyzing their movements. An actuator 10 may be a lock that activates or deactivates the access control devices based on a control signal A, for example, opening or closing door 401. For this purpose, control signal A can be selected based on the interpretation of the object identification system 60, for example, based on the determined identity of the personnel. Logical access control devices may also be used instead of physical access control devices.

[0050] Figure 5 The following embodiment is shown, in which the control system 40 is used to control the monitoring system 400. Figure 5 The embodiment shown in the diagram differs from this embodiment in that, instead of actuator 10, a display unit 10a is provided, which is controlled by control system 40. For example, object identification system 60 can reliably determine the identity of objects captured by video sensor 30, so as to infer, for example, which objects are suspicious, and control signal A can then be selected to make the object highlighted in color by display unit 10a.

[0051] Figure 6 An exemplary real-world scenario is illustrated, in which a motor vehicle 100, also referred to as the vehicle itself (Ego-Fahrzeug), equipped with sensor 30, travels at its own speed (Ego-Geschwindigkeit) v e Moving forward, and object i, in the example shown, is another vehicle traveling at speed v. i go ahead.

[0052] Figure 7 A schematic view of object i is shown, as it can be captured by sensor 30. Object i has two headlamps, a first headlamp at a first position P1 and a second headlamp at a second position P2, which may cause dissolves in the image. In the captured image, object i is at position P. i .

[0053] Figure 8 An exemplary illustration shows the information flow in an object identification system 60. At successive time points t k t k+1The input image x is fed into the object identification system. The first machine learning system NN1 uses this information to determine the location of the light source in the input image at time point t. k Positions P1, P2(t) k ) and the light source in the input image at a later second time point t k+1 Positions P1, P2(t) k+1 The locations P1 and P2 of the light source can be described, for example, by a semantically segmented image with the categories "with light source" and "without light source". The two determined locations are fed to a second machine learning system NN2, which uses them to determine the object i and its location P. i The object's velocity v i (or a list of objects and their corresponding properties) and their velocity v e .

[0054] The first machine learning system NN1 also determines the normalized x-norm of the input image x and feeds this normalized value to the third machine learning system NN3, which in turn determines the object i and its position P. i (or a list of objects and their corresponding locations).

[0055] The object i and its position P determined by the second machine learning system NN2 i The object's velocity v i (or a list of objects and their corresponding properties) and their velocity v e And the object i and its position P determined by the third machine learning system NN3. i (or a corresponding list) is fed to a fourth machine learning system, NN4. Based on this, the fourth machine learning system determines an output signal y, which includes information about the safe object i and the safe location P of that safe object. i (or a list of objects and their respective security locations).

[0056] Each of these four machine learning systems can be provided, for example, through an artificial neural network. The method illustrated here can be implemented as a computer program and stored in a machine-readable storage medium 46.

[0057] Figure 9 A possible configuration of a training device 140 for training an object identification system 60 is shown. This configuration utilizes parameters φ, which are provided by a parameter memory P.

[0058] The training device 140 can train each of the four machine learning systems NN1, ..., NN4. The corresponding machine learning system to be trained is represented using the attached label NN.

[0059] The training device 40 includes a provider 71 that provides an input parameter e and a nominal output parameter as. The input parameter e is fed to the machine learning system NN to be trained, which determines the output parameter a accordingly. The output parameter a and the nominal output parameter as are fed to a comparator 74, which determines new parameters φ' based on the consistency between the corresponding output parameter a and the nominal output parameter as. These new parameters are then transferred to a parameter memory P and replace the parameter φ there.

[0060] In the first step, a first machine learning system NN1 is trained. This system should determine the location of the light source and standardize the input image. In this case, at a pre-given location P... 1,s P 2,s Artificial light sources are added to the real image xr, which is provided, for example, by a database. This results in an enhanced image xa. This enhanced image is the input parameter e, and the output parameter a is the normalized image and the determined positions P1, P2 of these light sources. The nominal output parameter is the real image xr and the pre-given positions P1, P2. 1,s P 2,s Consistency was determined individually for these images and these location distributions. The first machine learning system, NN1, was trained to output the locations of these light sources and to normalize the images, meaning the light sources were removed again.

[0061] In the second step, a second machine learning system, NN2, is trained. In this case, the positions of objects (such as vehicles with headlights) with light sources are generated, where the light sources have a time series of reflections from a random scene sz. The input parameter e is the position sequence of the light source at two successive time points, and the nominal output parameter as is the position and velocity of the object. The second machine learning system NN2 is trained to accurately reconstruct the positions and velocities of these objects.

[0062] The order of the first and second steps can be chosen arbitrarily.

[0063] In the third step, the third machine learning system, NN3, is trained. This is achieved using a data record (X, Z) containing the input image sequence X = ((x... 1,0 , x 1,1 , ..., x 1,t ), ..., (x n,0 , xn,1 ,..., x n,t And for each sequence (x) k,0 , x k,1 , ..., x k,t For each of these, the following list is included (z k,0 , z k,1 ,..., z k,t This list contains objects and their attributes: position, velocity, and self-velocity. Identical objects in different input images are correlated with each other. Furthermore, a segmentation is provided for each input image x, specifically containing the active light sources contained within it. Using a first machine learning system NN1, the normalized xnorm is determined for the input image x, and this normalization is used as the input parameter e. The normalized output parameter as represents the objects and their positions. A third machine learning system NN3 should be trained to accurately identify objects and reliably reconstruct their positions.

[0064] Regarding the order of step 3, the only thing to note is that step 3 should be performed after step 1 is completed so that the first machine learning system NN1 has been trained.

[0065] In the fourth step, the fourth machine learning system NN4 is trained. In this case, the input image x of the data record (X, Z) described by the entire object identification system 60 is fed to the object identification system. Here, the output parameter a is the output parameter y of the object identification system 60, which, as the nominal output parameter as, maintains the corresponding object and attribute in the data record. Only the parameter φ is matched here; these parameters characterize the fourth machine learning system NN4.

[0066] Regarding the training order, this step should be performed last.

[0067] The method implemented by the training system 140 can be stored on the machine-readable storage medium 146 as a computer program and implemented by the processor 145.

[0068] The term "computer" includes any device used to process computational rules that can be given in advance. These computational rules can exist in the form of software, hardware, or a combination of both.

Claims

1. Computer-implemented method for reliably identifying an object in a sequence of input images received by means of an imaging sensor (30), wherein by means of a first machine learning system (NN1) a position (P1, P2) of a light source in a respective input image is determined from the input image, wherein by means of a second machine learning system (NN2) an object in the sequence of input images is identified from the resulting sequence of positions (P1, P2) of the light source, wherein by means of the first machine learning system (NN1) also a normalization of the respective input image is determined, wherein by means of a third machine learning system (NN3) an object in the respective input image is identified from the normalization of the respective input image, and wherein by means of a fourth machine learning system (NN4) a security object in the input image is identified from the object identified by means of the third machine learning system (NN3) and from the object identified from the sequence of positions of the light source.

2. Method according to claim 1, wherein by means of the second machine learning system (NN2) also an attribute of the identified object in the sequence of input images is identified.

3. Method according to claim 1, wherein by means of the fourth machine learning system (NN4) also the security object in the input image is identified from the identified attribute.

4. Method according to any one of claims 1 to 3, wherein the first machine learning system (NN1) is an artificial neural network.

5. System (60) set up to carry out the method according to any one of claims 1 to 4, comprising: a first machine learning system (NN1) set up to determine from the input image a position (P1, P2) of a light source in a respective input image and to determine a normalization of the respective input image, a second machine learning system (NN2) set up to identify an object in the sequence of input images from the resulting sequence of positions (P1, P2) of the light source, a third machine learning system (NN3) set up to identify an object in the respective input image from the normalization of the respective input image, and a fourth machine learning system (NN4) set up to identify a security object in the input image from the object identified by means of the third machine learning system (NN3) and from the object identified from the sequence of positions of the light source.

6. Computer-implemented method for training the system (60) according to claim 5, wherein a scene is generated, which comprises an object and a light source at a predefinable position within a space, and from which a sequence of synthetic positions of the light source is generated as if taken by a camera from a predefinable camera position, wherein the second machine learning system (NN2) is trained to derive the position of the object from the sequence of synthetic positions of the light source.

7. The method according to claim 6, wherein the first machine learning system (NN1) is trained with data records comprising pairs of an augmented image and the predefinable position it belongs to, wherein the augmented image is obtained by artificially adding a light source at a predefinable position to a real image, and wherein the first machine learning system (NN1) is trained to determine the predefinable position from the augmented image.

8. The method according to claim 7, wherein the first machine learning system (NN1) is further trained with data records comprising also pairs of a real image and the augmented image it belongs to, and wherein the first machine learning system (NN1) is trained to determine the real image it belongs to from the augmented image as normalized data.

9. The method according to any one of claims 6 to 8, wherein the first machine learning system (NN1) is trained first before the third machine learning system (NN3) and then the fourth machine learning system (NN4) as the last one.

10. Training apparatus (140) which is set up to carry out the method according to any one of claims 6 to 9.

11. Computer program product comprising a computer program which is set up to carry out the method according to any one of claims 1 to 4 or 6 to 9 when running on a processor.

12. Machine-readable storage medium (46, 146) on which a computer program is stored which is set up to carry out the method according to any one of claims 1 to 4 or 6 to 9 when running on a processor.

Images