A method for detecting a stopped state of a vehicle and a ground vehicle

Abstract

According to a method for detecting a stopped state of a vehicle, a plurality of frames of a scene are captured by a camera of the vehicle. A perspective transformation of at least a portion of each of the plurality of frames is performed to obtain a transformed portion of each of the frames. Pixel shifts in corresponding transformed portions of a group of at least two subsequent frames of the plurality of frames are determined. A moving state of the vehicle is determined based on an evaluation of corresponding pixel shifts of each of a plurality of groups of at least two subsequent frames of the plurality of frames.

Description

Technical Field

[0001] Embodiments relate to a method for detecting a moving state, in particular a stopped state, of a vehicle, such as an autonomous vehicle. Background Art

[0002] Visual odometry and driver assistance systems have become increasingly popular in recent years and naturally attract more and more attention from researchers in the field of intelligent robotics and autonomous driving. A frequently occurring task is to detect whether the vehicle is currently stopped. In standard production vehicles, this can be done by using vehicle movement sensors such as wheel ticks.

[0003] Especially in early test vehicles, access to the vehicle's movement sensors may be limited. Sensor data that can be easily obtained from the vehicle or by simple additional equipment are GPS data, IMU data and data from a camera installed in the vehicle. However, the data received from the IMU and GPS do not provide reliable detection of the vehicle's stopped state. The detection of the vehicle's stopped state by GPS fails, for example, not only in tunnels or urban canyons but also often in standard situations. For example, the GPS position may move even if the vehicle is stopped. The detection of the vehicle's stopped state by using an IMU is very sensitive to noise. Small movements of the vehicle, such as those caused by wipers, moving passengers or moving traffic, can indicate movement of the vehicle.

[0004] Therefore, the detection of the stopped state of the vehicle must focus on evaluating the camera data and then performing visual processing. Known visual processing methods that can be applied to the optical detection of the stopped state of the vehicle are, for example, feature point tracking or optical flow analysis in SLAM (Simultaneous Localization And Mapping). However, most prior art optical methods, such as feature point tracking or optical flow analysis, are sensitive to movement in parts of the image of the captured scene. For example, rain splashing on the windshield, moving traffic, movement of wipers, or other movements in the captured image may falsely indicate the movement of the vehicle. In summary, the known prior art optical methods do not allow for robust and reliable detection of the stopped state of the vehicle. Summary of the invention

[0005] The problem to be solved by the present invention is to provide a method for detecting the moving state of a vehicle, in particular the stopped state of a vehicle, by using a robust optical method that is resistant to interference influences that may falsely indicate the movement of the vehicle even if the vehicle is actually stopped.

[0006] The solution to this problem is described in the independent claim. The dependent claims relate to further developments of the invention.

[0007] One embodiment relates to a method for detecting a moving state of a vehicle, in particular a stationary state of a vehicle, which method allows a reliable detection of the moving state of the vehicle even if optical data are affected by interfering factors.

[0008] In a method for detecting the movement state of a vehicle, a plurality of frames of a scene are captured by a camera of the vehicle. The frames may be consecutive frames. A perspective transformation of at least a portion of each of the plurality of consecutive frames may be performed to obtain a transformed portion of each of the frames. Pixel shifts in corresponding transformed portions of a group of at least two, preferably subsequent, frames of the plurality of frames may be determined. The corresponding pixel shifts of each of the plurality of groups of at least two, preferably subsequent frames of the plurality of frames may be evaluated. The movement state of the vehicle may be determined based on the evaluation of the corresponding pixel shifts of each of the plurality of groups of at least two, preferably subsequent frames of the plurality of frames. Preferably, subsequent frames are used, but frames may also be skipped, for example, if they are distorted by foreign objects or noise, or if they cannot be used for other reasons.

[0009] Such an embodiment for detecting the moving / stopping state of a vehicle requires only at least one optical sensor, ie a camera. In particular, the method works with a single camera, which may be a monocular camera.

[0010] The evaluation of the camera's data does not depend on details of the captured images, such as where local motion may have occurred, but rather the method can evaluate global motion of the captured images / frames.

[0011] The frames / images may be captured from a camera from a front view, and the frames / images may be consecutive frames / images. The method may obtain several portions from each front view image. A perspective transformation is performed on each portion of the captured image / frame to better emphasize the movement of the vehicle. For example, each portion of the captured frame / image is transformed from a front view to a side view or a bird's eye view transformed picture portion, in which the movement of the vehicle can be better detected than in the original front view image.

[0012] Several techniques such as Fourier analysis / transformation, structural similarity analysis or joint information entropy analysis of each part of the frame / image can be used to obtain the offset between the two frames. The offset between the two frames can be analyzed for each pair of captured frames to obtain a judgment about the movement state of the vehicle, for example, to obtain a judgment whether the vehicle is stopping.

[0013] This embodiment can analyze the overall changes of the image and can tolerate movement in the parts of the captured image. Therefore, it can even handle extreme scenes, such as scenes with heavy rain or dense traffic. The method is robust to almost all kinds of difficult to analyze scenes, such as moving wipers, scenes with heavy traffic and bad weather conditions. In addition, the method can be applied to any vehicle, regardless of the type of vehicle. It does not require specific equipment except for the camera. In addition, the method is independent of the type of camera used.

[0014] Additional features and advantages are set forth in the detailed description which follows.It is to be understood that both the foregoing general description and the following detailed description are merely exemplary, and are intended to provide an overview or framework for understanding the nature and character of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention will be described below with reference to the accompanying drawings by way of illustration of examples of embodiments without limiting the general inventive concept.

[0016] Figure 1 A flow chart illustrating method steps of a method for detecting a moving state of a vehicle is shown;

[0017] Figure 2 illustrates a frame / image captured by a camera in a front view, wherein the portion of the frame is to be evaluated by the method for detecting the movement state of a vehicle; and

[0018] Figure 3 Exemplary results of Fourier analysis and inverse Fourier transform are illustrated to determine pixel shifts between captured frames.

[0019] Figure 4 An embodiment with two thresholds is shown.

[0020] exist Figure 1 4 shows a first embodiment according to the present invention including steps S1 to S5 for detecting the moving state / stopping state of the vehicle.

[0021] In a first step S1, a plurality of frames of a scene are captured by a camera of a vehicle, and the plurality of frames may be consecutive frames. According to one embodiment, the camera used may be configured as a monocular camera. The plurality of frames may be captured by the camera in a front view form.

[0022] In step S2, a perspective transformation of at least a portion of each of the plurality of frames is performed to obtain a transformed portion of each of the frames. According to one embodiment, the portion of each of the plurality of frames is transformed from a captured front view to a side view or a bird's eye view by perspective transformation. The perspective transformation is used so that the transformed picture portion allows better detection of the movement of the vehicle compared to the original captured image.

[0023] In step S3, pixel offsets in corresponding transformed portions of a set of at least two subsequent frames of the plurality of frames are determined. According to one embodiment, the pixel offsets are determined by calculating a phase correlation. The phase correlation may be calculated by performing a Fourier analysis transform on the corresponding transformed portions of the set of at least two subsequent frames of the plurality of frames, calculating a cross-power spectrum by taking the complex conjugate of one of the Fourier transformed frames, multiplying the Fourier transforms element-wise, normalizing the product element-wise, and performing an inverse Fourier transform of the result.

[0024] According to another embodiment, the pixel offset is determined by performing a structural similarity analysis or a joint information entropy analysis on the corresponding transformed portions of the set of at least two subsequent frames of the plurality of frames.

[0025] In step S4, the corresponding pixel shifts of each of the plurality of groups of at least two subsequent frames in the plurality of frames are evaluated. According to one embodiment, the corresponding pixel shifts are evaluated by calculating at least one mean deviation or standard deviation of the pixel shifts of each of the plurality of groups of at least two subsequent frames in the plurality of frames.

[0026] According to one embodiment, a corresponding threshold value of an average deviation and / or a standard deviation of the pixel offset of each of the plurality of groups of at least two subsequent frames in the plurality of frames is defined. If the average deviation and / or the standard deviation of the pixel offset of each of the plurality of groups of at least two subsequent frames in the plurality of frames exceeds the corresponding threshold value of the average deviation and / or the standard deviation of the pixel offset of each of the plurality of groups of at least two subsequent frames in the plurality of frames, it may be further evaluated.

[0027] In step S5, the movement state of the vehicle is determined based on an evaluation of corresponding pixel shifts of each of a plurality of groups of at least two subsequent frames in the plurality of frames.

[0028] According to one embodiment, when the average deviation and / or standard deviation of the pixel offsets in each of the multiple groups of at least two subsequent frames in the multiple frames are evaluated to be below the corresponding threshold values ​​of the average deviation and / or standard deviation of the pixel offsets in each of the multiple groups of at least two subsequent frames in the multiple frames, the moving state of the vehicle is determined so that, in particular, the stopped state of the vehicle is identified.

[0029] Each method step S1 to S5 may be performed by a control unit of the vehicle, which may be embodied as a processor or a controller.

[0030] Pass below Figure 2 An example of a captured front view image / frame shown in is used to explain the method steps described above. Figure 2 The frame of the scene shown in is captured by a camera of a vehicle. The camera captures the illustrated front view through the front window of the vehicle.

[0031] A perspective transformation is applied to the left part P1 and the right part P2 of the captured image. The definition of a perspective transformation is to establish collineation on another plane by projecting points of one plane from two different projection centers onto the other plane. For example, in Figure 2 In the right part P2 of the captured image, the trapezoid shown in dotted lines is converted into a rectangle R2. A transformation matrix is ​​calculated from four corresponding points, and then the image in the entire trapezoid P2 is transformed.

[0032] Finally, the portion P1 on the left side of the front view is transformed into a picture just like looking out of the left side window of the vehicle. The portion P2 on the right side of the front view of the captured image is transformed into a picture just like looking out of the right side window of the vehicle. The purpose of this transformation is to provide better data for subsequent image analysis, such as subsequent Fourier analysis, because panned images are easier to process for Fourier analysis.

[0033] After the transform described above is performed on each of the frames, Fourier analysis can be performed on the left and right transformed portions of the corresponding frames between the two frames. After the Fourier analysis has been performed, the cross power spectrum is calculated by taking the complex conjugate of one of the Fourier transformed frames, multiplying the Fourier transforms element-by-element, normalizing the product element-by-element, and applying an inverse Fourier transform.

[0034] As a result of the Fourier analysis and the inverse Fourier transformation, a pulse signal 14 is obtained which can extend from the zero plane 12, such as Figure 3 This is illustrated in Figure 10 of . The coordinates of this pulse signal determine the pixel offsets in the transformed parts of two subsequent frames.

[0035] The corresponding pixel offset of each group / pair of two subsequent frames in the plurality of frames is determined. In order to evaluate the corresponding pixel offset of each group of multiple groups of subsequent frames in the plurality of frames, a mean deviation and / or a standard deviation may be calculated. For each of the mean deviation and / or the standard deviation, a corresponding threshold value may be set. If the determined pixel offset is below the corresponding threshold value of the mean deviation and / or the standard deviation, a stopped state of the vehicle may be detected. On the other hand, if the determined pixel offset is above the corresponding threshold value of the mean deviation and / or the standard deviation, a moving state of the vehicle is identified.

[0036] When considering land vehicles, optical analysis can be performed from front view images. In the case of panned images, such as those captured by drones carrying downward directed cameras, there may not be any need to perform a perspective transform.

[0037] about Figure 2 , the method is explained by performing a perspective transformation of the left and right parts of the image. It must be noted that the method is not limited to these left and right parts of the image. Other parts of the captured image can be used for perspective transformation, such as a part of the image directly in front of the vehicle can be used for perspective transformation, for example to perform a bird's eye view transformation.

[0038] The method can be applied to detect the moving state of a vehicle, such as an autonomous vehicle. However, the method is not limited to being used to detect the moving state of a vehicle. The method can be applied to other applications with limited motion sensors, where it is necessary to detect the stationary state of an object by a camera. For example, the method can be used to control a virtual reality scene through a smart phone, or to control the flight of an aircraft, such as being used to control the hovering state of a drone.

[0039] Figure 4 An embodiment is shown where two thresholds are set for the mean and standard deviation, and if it is less than the corresponding threshold setting, the vehicle is stopping, otherwise the vehicle is moving. As shown in the figure, this can give very accurate results.

Claims

1. A method for detecting a stopped state of a vehicle, comprising: A plurality of frames of a scene are captured by a camera of the vehicle, performing a perspective transform of at least a portion of each of the plurality of frames to obtain a transformed portion of the frame in side view, determining corresponding pixel offsets in respective transformed portions of pairs of frames in the plurality of frames, evaluating the corresponding pixel offsets of pairs of frames in the plurality of frames, determining a stopped state of the vehicle based on an evaluation of the corresponding pixel shifts of pairs of frames in the plurality of frames, Wherein determining the corresponding pixel offsets comprises performing phase correlation on respective transformed portions of pairs of frames in the plurality of frames.

2. The method according to claim 1, The camera is embodied as a monocular camera.

3. The method according to claim 1, The plurality of frames are captured in a front view of the camera.

4. The method according to claim 1, The frames are consecutive frames.

5. The method according to claim 1, wherein: Performing the phase correlation comprises: Fourier analysis is performed based on corresponding pairs of frames in the plurality of frames, a cross power spectrum is calculated by taking the complex conjugate of one of the Fourier transformed frames, the Fourier transforms are multiplied element-wise, the product is normalized element-wise, and an inverse Fourier transform is performed on the result.

6. The method according to claim 1, comprising: The corresponding pixel offsets are evaluated by calculating at least one of a mean deviation or a standard deviation of the pixel offsets for each pair of frames in the plurality of frames.

7. The method according to claim 6, comprising: defining corresponding thresholds for the mean deviation and / or standard deviation of pixel offsets for each pair of frames in the plurality of frames, Evaluate whether the average deviation and / or standard deviation of the pixel offsets of each pair of frames in the plurality of frames exceeds corresponding thresholds of the average deviation and / or standard deviation of the pixel offsets of each pair of frames in the plurality of frames.

8. The method according to claim 7, comprising: The stopped state of the vehicle is determined when the average deviation and / or standard deviation of the pixel offsets of each pair of frames in the plurality of frames is evaluated to be below a corresponding threshold of the average deviation and / or standard deviation of the pixel offsets of each pair of frames in the plurality of frames.

9. A ground vehicle comprising a control unit configured to perform the method as claimed in any one of claims 1 to 8.

Images