Method for detecting free space in a traffic scene

Abstract

The invention relates to a method for detecting free space in a traffic scene. Furthermore, the invention relates to a computer program, a device, and a storage medium for this purpose.

Description

The invention relates to a method for detecting free space in a traffic scene. The invention further relates to a computer program, a device and a storage medium for this purpose.Prior ArtIn the field of autonomous driving, detection of free space may be an important function that can improve scene understanding by obtaining a boundary between the area in front of a vehicle free of obstacles and the area including non-drivable obstacles. This problem can be addressed in computer vision in different ways with frame-by-frame detection. This is particularly challenging when outliers or missed objects occur in the frame and there is a need for temporal accumulation and aggregation to correct possible false detections that come from a deep learning based approach.Disclosure of the InventionAccording to aspects of the invention there are provided a method having the features of claim 1, a computer program having the features of claim 8, a data processing apparatus having the features of claim 9 and a computer readable storage medium having the features of claim 10. Further features and details of the invention are disclosed in the respective dependent claims, in the description and in the drawings. Features and details described in the context of the method according to the invention also correspond to the computer program according to the invention, the data processing device according to the invention and the computer readable storage medium according to the invention, and vice versa in each case.According to one aspect of the invention, a method for detecting a free space in a traffic scene is provided, wherein the steps can be carried out sequentially and / or repeatedly, comprising:providing image data of the traffic scene, wherein the image data comprise optical flow values and semantic contours for at least one object in the traffic scene, wherein the optical flow values represent an optical flow in the image data and the semantic contours represent a delimitation of the free space by the at least one object in the image data, wherein the image data are provided as image frames which comprise a current frame and at least one frame which precedes the current frame at a respective time,providing relative movement values, wherein the relative movement values are determined based on the optical flow values (of the provided image data) which correspond in particular to the semantic contours (of the provided image data),propagating the semantic contours of the at least one frame of the image data (in particular, preceding the current frame) into the current frame of the image data based on the provided relative movement values.In other words, the traffic scene may be a road environment for vehicles. The method may be carried out by one of the vehicles, also referred to as an "ego vehicle.". The free space may indicate a space free for the ego vehicle to drive on or navigate. The optical flow values may represent a change, i.e., a movement or displacement, of pixels between successive frames in the image data. The semantic contours may in simple words describe a line or a curve or a polygon connecting a lower end of any unpassable object in a direction of travel of the vehicle, and thereby represent a boundary of the clearance.The object may be, for example, another car or curb of the road in the context of the present invention. The semantic contours may be obtained by different object detection methods known in the art. Examples of the object recognition methods are as follows:feature-based methods that extract salient features from the image, such as edges, corners or texture features, and use these features to identify objects,machine learning based methods including classification algorithms such as support vector machines (SVM), random forests or deep learning models such as convolutional neural networks (CNN),segmentation-based methods that segment objects by separating the pixels or regions in the image belonging to the respective object from other background regions.There may be a semantic contour for each frame of the image data. The image data may be a video and may therefore be provided as a plurality of frames. The optical flux values may represent the change in the form of movement or displacement of the pixels. The propagation of the semantic contours can provide the advantage that the semantic contours can be dynamically adapted to the free space using the relative movement values, and thereby describe the free space more precisely.It is contemplated that the optical flow values are represented as (at least) 2D vectors (i.e., at least two-dimensional vectors) for each pixel in the image data, where an X and a Y component of each vector indicate the horizontal and vertical movement of the respective pixel or object represented by the respective pixel, and a size of each vector indicates a size of the respective movement. By representing the optical flow values as the 2D vectors, the change, in particular the movement or displacement, of pixels can advantageously be described more extensively.It may be advantageous if each of the semantic contours comprises a set of nodes, wherein the nodes correspond to pixels in the image data that represent the delimitation of the free space by the at least one object. Further, optical flow values corresponding to a respective node of the semantic contours may be calculated based on those pixels to which the respective node corresponds.Providing the relative motion values may include the following steps:analyzing the set of nodes of the semantic contours to detect whether at least one unavailable optical flow value corresponding to at least one of the set of nodes is present,performing, if the at least one unavailable optical flow value is present, a radial scan of surrounding pixels of the pixels in the image data to which the at least one respective node with the detected unavailable optical flow value corresponds, thereby determining at least one available optical flow value corresponding to a neighborhood of the at least one respective node, in particular by determining whether the object whose boundary of the free space is represented by the at least one respective node corresponds to, in particular is identical to, a neighboring object represented by the pixels in the neighborhood where the optical flow value is available,assigning the determined available optical flow value to the respective node.The unavailable optical flow value may occur due to pixel errors or disturbances in the image data, for example due to sunlight. The radial scan may also be referred to as a radial search and may be based on the assumption that pixels belonging to the same object or type of object have a similar displacement. The radial scan may be advantageous to provide a more detailed semantic contour with fewer errors due to more nodes with optical flow values available for shaping the semantic contour.In one possible embodiment, the propagating comprises the following step:calculating coordinates of the nodes of the semantic contour of the current frame of the image data based on the nodes of the semantic contour of the at least one preceding frame of the image data and the respective relative movement values of the nodes of the semantic contour of the at least one preceding frame of the image data.The calculation can be performed by two additions for each node of the respective semantic contour, in particular an addition for a horizontal movement and an addition for a vertical movement.It can be provided that the relative movement values represent an accumulation of optical flow values over a plurality of frames of the image data in order to take account of static and dynamic objects. The relative motion values may thus advantageously provide information about a motion of the at least one object even in the traffic scene by considering the optical flow values over multiple frames.It is possible that the relative motion values represent an accumulation of optical flow values over at least two frames. With more frames, the accumulation of optical flow values may advantageously result in more precise relative motion values.In another example, the image data results from detection of an external device, in particular a camera, to provide a real-time implementation of the method. By means of the real-time implementation, navigation of an autonomous vehicle through the traffic scene can be advantageously improved.In another aspect of the invention there may be provided a computer program, in particular a computer program product comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the method according to the invention. Thus, the computer program according to the invention may have the same advantages as those described in detail with reference to a method according to the invention.In another aspect of the invention, there may be provided a data processing apparatus adapted to carry out the method according to the invention. As the means, for example, a computer executing the computer program according to the invention may be provided. The computer may include at least one processor that may be used to execute the computer program. In addition, a non-volatile data memory may be provided in which the computer program may be stored and from which the computer program may be read by the processor for execution.According to another aspect of the invention there may be provided a computer readable storage medium comprising the computer program according to the invention and / or instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to the invention. The storage medium may be configured as a data storage device such as a hard disk and / or a non-volatile memory and / or a memory card and / or a solid state drive. The storage medium can be integrated into the computer, for example.Furthermore, the method according to the invention can be implemented as a computer-implemented method.Other advantages, features and details of the invention will become apparent from the following description, in which embodiments of the invention are described in detail with reference to the drawings. In this connection, the features mentioned in the claims and in the description can each be essential for the invention individually or in any combination. The following are shown: FIG. 1 shows a method, a computer program, a computer-readable storage medium and a data processing device according to embodiments of the invention, FIG. 2 shows a schematic representation of an optical flow propagation, FIG. 3 shows a schematic representation of a radial scan, FIG. 4 shows a schematic representation of a profile span of the last 5 propagated free space outputs in the current frame using the prior art ego motion, FIG. 5 shows a schematic representation of a profile span of the last 5 propagated free space outputs in the current frame using the optical flow, FIG. 6 is a schematic representation of a challenging scenario using the prior art ego motion, FIG. 7 is a schematic representation of a challenging scenario using the optical flow.FIG. 1 shows a method 100, a computer program 20, a computer-readable storage medium 15 and a data processing device 10 according to embodiments of the invention.The method according to the embodiment of FIG. 1 comprises a first step 101 in which image data of a traffic scene are provided, wherein the image data comprise optical flow values 3 and semantic contours 1 of at least one object 2 in the traffic scene, wherein the optical flow values 3 represent an optical flow in the image data and the semantic contours 1 represent a delimitation of the free space by the at least one object 2. In a second step 102, relative movement values are provided, wherein the relative movement values are determined on the basis of the optical flow values 3, which correspond in particular to the semantic contours 1. In a third step 103, the semantic contours 1 of at least one frame of the image data are propagated into a current frame of the image data on the basis of the provided relative movement values.More particularly, the present invention proposes an approach to time accumulation and aggregation of multiple free space instances in a set of successive frames of a video sequence. One aspect of the invention may use the optical flow measurements to predict the position of the previous free space instance at the current time. The optical flow may be defined as the displacement of a pixel in an image between two successive frames in a video. The free space is represented, in particular, as a polyline consisting of a set of nodes having a 2D position associated with them, the semantic contour and the optical flow values for both vertical and horizontal displacement.The free space may be initially generated in the image domain of a monocamera. The current approach to free space propagation implies in particular the use of the ego motion, the motion of the video camera between two successive frames. Several problems can arise with this approach. The ego motion may be in 3D coordinates as the camera moves in the real world. This means in particular that a 2D-to-3D transformation is required for all nodes of the free space. Since depth information is not available for this, the intersection with the road can be performed. Rotation and translation may then need to be applied at each node of the free space to be propagated from a previous frame to the current one. These steps may be computationally expensive. Moreover, in particular, the ego motion represents only the motion of the camera in the world, assuming a static world, so that it may totally disregard the dynamic objects present in the scene and not propagated using the ego motion. The accumulation based on the optical flow can overcome the problems with ego motion propagation by eliminating the static world assumption and including the dynamic objects in the tracking process.Finally, there are two major areas where the proposed approach can provide improvements:Calculations: Optical flow propagation can replace the road intersection and the two transforms with two simple additions to the values of the optical flow in the image domain: one for vertical displacement and one for horizontal displacement.Static World Assumption: Dynamic objects can also be propagated because the optical flow follows the motion of the scene, rather than the motion of the camera.Since the optical flow values for each frame are obtained from other hardware or software, they can be applied to the free space nodes generated from the frame-by-frame generation result. There are two steps that can be performed by this approach. First, a correct value of the optical flow may be associated with each free space node, and then the propagation operation may take place with the addition of the optical flow values to the previous image coordinates. 1. flow allocation may become a challenge in a flow field that is not 100% dense, as the free space nodes may be in image coordinates where the optical flow is not available for various reasons. For this, semantic information can be used to find the most appropriate value based on the assumption that object pixels belonging to the same object or type of object have similar displacements, which can thus be done is performing a radial scan or a radial search in a neighborhood region of the unavailable flow cell until an allowed value is found and assigned to the current node. 2. the flow propagation may describe that after the optical flow values are assigned to each free space node generated in one frame, addition may be performed to obtain the coordinates of the nodes in the next frame. This step can perform two additions for each node in the free space polygon. The image coordinates of the past free space polygon can be calculated by adding the flow values. 3. a flow update may be similar to the flow assignment because it is responsible for assigning new values of the optical flow corresponding to the flow ordered in the current frame. The same problem of flow unavailability may occur, which is solved in the same way as above at point 1.FIG. 2 shows a schematic representation of an optical flow propagation. Several objects 2 are visible and there is a semantic contour 1 representing a boundary of the free space. Further, an optical flux value 3 is shown. The optical flux value 3 is indicated by an arrow schematically representing a direction and amount of movement of a pixel. The coordinates of the pixel in the previous frame are updated by adding the optical flow value 3, and in the current frame, the position of the pixel will be at the top of the arrow.FIG. 3 shows a schematic representation of the radial scan, in particular the radial search. An allowed optical flow value for semantic contour 1 can be searched in an adjacent region of an unavailable node of semantic contour 1 until it is found, given compliance with the semantic class of the original point and assuming that similar objects have similar motions. The search is carried out radially, which is marked in FIG. 3 in the form of circles 4.FIG. 4 shows a span of the last 5 propagated free space outputs in the current frame using the ego motion. Five semantic contours 1 and an object 2 are shown.FIG. 5 shows a span of the last 5 propagated free space outputs in the current frame using the optical flow. Five semantic contours 1 and an object 2 are also shown. A comparison between FIGS. 4 and 5 can show that the propagation with the optical flow describes the free space by means of the semantic contour in an improved manner.In FIGS. 6 and 7, a respective schematic representation of a challenging scenario using the prior art ego motion (FIG. 6 ) and the optical flow (FIG. 7 ) is shown. The advantageous effect of using the optical flow can be derived from the semantic contour 1 partially enclosing the object 2 of FIGS. 6 and 7.After obtaining the five propagated frames, a consolidation strategy may be applied to obtain only a free space representing the current frame representing the final output of the semantic path. Initially, since the dynamic nodes were not propagated with the ego motion, they were not part of the consolidation. After propagation based on the optical flow, they may also be consolidated and the results may be noted in particular in some challenging scenarios (direct sunlight).The above explanation of the embodiments describes the present invention in the context of examples. Of course, individual features of the embodiments may be freely combined with each other provided that this is technically implementable without departing from the scope of the present invention.

Claims

Method (100) for detecting a free space in a traffic scene, comprising: - providing (101) image data of the traffic scene, wherein the image data comprise optical flow values (3) and semantic contours (1) for at least one object (2) in the traffic scene, wherein the optical flow values (3) represent an optical flow in the image data and the semantic contours (1) represent a delimitation of the free space by the at least one object (2) in the image data, wherein the image data are provided as image frames comprising a current frame and at least one frame preceding the current frame at a respective time, - providing (102) relative movement values, wherein the relative movement values are determined based on the optical flow values (3), - propagating (103) the semantic contours (1) of the at least one frame, preceding the current frame, into the current frame based on the provided relative motion values.The method (100) according to claim 1, characterized in that the optical flow values (3) are represented as at least 2D vectors for each pixel in the image data, wherein an X and a Y component of each vector indicate the horizontal and vertical movement of the respective pixel, or of the object (2) represented by the respective pixel, and a size of each vector indicates a size of the respective movement.Method (100) according to any of the preceding claims, characterized in that each of the semantic contours (1) comprises a set of nodes, wherein the nodes correspond to pixels in the image data representing the delimitation of the free space by the at least one object (2), and the optical flow values (3) corresponding to a respective node of the semantic contours (1) are calculated on the basis of those pixels to which the respective node corresponds, and the provision (102) of the relative movement values comprises the following steps: - analyzing the set of nodes of the semantic contours (1) in order to detect whether at least one unavailable optical flow value (3) corresponding to at least one of the set of nodes is present, - performing, if the at least one unavailable optical flow value is present, a radial scan of surrounding pixels of the pixels in the image data, to which the at least one respective node with the detected unavailable optical flow value (3) corresponds, whereby at least one available optical flow value (3) is determined which corresponds to a neighborhood of the at least one respective node, in particular by determining whether the object (2) whose delimitation of the free space is represented by the at least one respective node corresponds to, in particular is identical to, an adjacent object (2) which is represented by the pixels in the neighborhood where the optical flow value (3) is available, - assigning the determined available optical flow value (3) to the respective node.Method (100) according to any of the preceding claims, characterized in that each of the semantic contours (1) comprises a set of nodes, wherein the nodes correspond to pixels in the image data representing the delimitation of the free space by the at least one object (2), and the optical flow values (3) corresponding to a respective node of the semantic contours (1) are calculated on the basis of those pixels to which the respective node corresponds, and the propagating (103) comprises the step of: - calculating coordinates of the nodes of the semantic contour (1) of the current frame of the image data on the basis of the nodes of the semantic contour (1) of the at least one preceding frame of the image data and the respective relative movement values of the nodes of the semantic contour (1) of the at least one preceding frame of the image data.Method (100) according to one of the preceding claims, characterized in that the relative movement values represent an accumulation of optical flow values (3) over a plurality of frames of the image data in order to take account of static and dynamic objects (2).Method (100) according to claim 5, characterized in that the relative motion values represent an accumulation of optical flow values (3) over at least two frames.Method (100) according to one of the preceding claims, characterized in that the image data result from a detection of an external device, in particular a camera, in order to provide a real-time implementation of the method (100).A computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out the method (100) of any preceding claim.Data processing device (10) comprising means for carrying out the method (100) according to any one of claims 1 to 7.A computer readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer (10) to carry out the steps of the method (100) of any one of claims 1 to 7.

Images