Method for warning a vehicle of a stationary obstacle - Patents.com

Abstract

To provide a method of warning a vehicle about a stationary object.SOLUTION: A method includes, when a vehicle moves at a first speed, extracting, by a light scanning device, an infrared image extraction device, and an image extraction device, a first light scanning image, a first infrared image, and a first surrounding image of around one side of the vehicle, providing first obstacle information of a first obstacle in the first image according to a fusion algorithm and an optical flow method, and in response to the location information of the first obstacle, a vehicle computer determining and generating a warning message if a relative distance is less than a distance threshold and obstacle volume is greater than a volume threshold.SELECTED DRAWING: Figure 1A

Description

[Background technology]

[0001] Advanced driver assistance systems (ADAS) are developed to assist drivers and can be divided into three main parts: vehicle sensors, vehicle processors, and actuators. ADAS uses vehicle sensors to sense signals outside the vehicle. In addition to millimeter wave radar and Lidar, the vehicle sensors also include thermal sensors and pressure sensors. The vehicle sensors send sensing data to a vehicle processor, such as an Electronic Control Unit (ECU). The latter can then generate warning information for the driver according to the sensing data to avoid danger.

[0002] Furthermore, when the driver is unable to react, vehicle sensors can intervene in the driver's driving behavior, allowing actuators to slow the vehicle, apply emergency brakes, or change the direction of the vehicle to protect the driver.

[0003] In addition, to protect the driver, radar detection technology has been developed to detect the surroundings of the vehicle. Unfortunately, the radar cannot inform the vehicle whether the object around the vehicle is a fixed obstacle or a moving object. When an object that does not affect the vehicle is close, the warning unit is often activated to frequently issue warning messages. These frequent warnings make the driver uncomfortable.

[0004] The detection of obstacles around a vehicle for surveillance has been greatly improved. However, during the movement of a vehicle, in addition to the dangers caused by other vehicles, there are still things that affect the safety of the vehicle. For example, pedestrians, animals, and moving objects can be considered as obstacles for the vehicle. All of these will induce emergency situations when the vehicle is moving. This effect is most noticeable on the crowded streets of cities.

[0005] In case of emergency, dash cams exist to record color images. However, allowing the unexpected to happen and then making a decision is not a sufficient solution. A better solution is to allow the driver to avoid the emergency in advance. Often, modern dash cams are installed at the front and rear of the vehicle. There are still blind spots on the sides. Therefore, it is required to integrate side recording and detection technology to help the driver avoid the side blind spots. Furthermore, it is required that the detection of the side blind spots can predict the potential danger and notify the driver to protect him.

[0006] Current vehicles can, for example, detect potential hazards while driving through intersections or even while parked.

[0007] Unfortunately, current detection methods are mainly applied to cross roads, adjacent lanes, and parking scenarios. They are limited to detecting moving objects relative to the vehicle instead of stationary obstacles such as safety zones or parked motorcycles. As a result, accidents may occur due to false positives of stationary obstacles when the vehicle is moving.

[0008] Therefore, how to accurately and appropriately judge stationary obstacles to avoid minor accidents involving obstacles has become a major challenge in the field. Summary of the Invention

[0009] The objective of the present disclosure is to provide a method for warning a vehicle of stationary objects, by scanning objects on one side of the vehicle, obtaining corresponding object images, filtering the object images, and obtaining filtered images that affect the vehicle, and generating a warning message for a user according to the speed and distance between the corresponding object and the vehicle, so that the user can take preventive measures in advance to avoid danger.

[0010] In order to achieve the above object, the present disclosure provides a method for warning a vehicle of a stationary object, the method being applicable to a vehicle, the vehicle including a vehicle computer, an optical scanning device, an infrared image extracting device, and an image extracting device, the vehicle computer being electrically connected to the optical scanning device, the infrared image extracting device, and the image extracting device. The vehicle computer executes the following steps when the vehicle moves: the optical scanning device, the infrared image extractor, and the image extractor extract a first optical scanning image, a first infrared image, and a first surrounding image of the surroundings on one side of the vehicle; the vehicle computer receives and combines the first optical scanning image, the first infrared image, and the first surrounding image according to a fusion algorithm to give a first image; the vehicle computer analyzes and obtains first obstacle information of a first obstacle in the first image according to an image optical flow method, the first obstacle information including a relative distance to the vehicle and an obstacle volume of the first obstacle; and when the relative distance is smaller than a distance threshold and the obstacle volume is larger than a volume threshold, the vehicle computer labels the actual obstacle on a display unit of the vehicle computer and generates a warning message.

[0011] According to one embodiment of the present disclosure, in the process in which the vehicle computer receives and combines the first optical scanning image, the first infrared image, and the first surrounding image in accordance with the fusion algorithm to provide the first image, the vehicle computer includes a processing unit, the processing unit includes an image processing program, and the image processing program is used to execute the fusion algorithm and the image optical flow method.

[0012] According to one embodiment of the present disclosure, in the step of the vehicle computer analyzing and obtaining first obstacle information of a first obstacle in the first image according to an image optical flow method, the first obstacle information includes a relative distance to the vehicle in the first image and an obstacle volume of the first obstacle, and the vehicle computer further executes a step of the vehicle classifying the actual obstacle according to the obstacle volume, and when the obstacle volume is smaller than the volume threshold, the vehicle computer determines that the actual obstacle is an invalid obstacle.

[0013] According to one embodiment of the present disclosure, the warning message is a voice message and / or a picture message.

[0014] According to one embodiment of the present disclosure, when the relative distance is less than a distance threshold and the obstacle volume is greater than a volume threshold, the vehicle computer labels the actual obstacle on a display unit of the vehicle computer and generates a warning message, and the volume threshold is set to 400 or more. cubic Smaller than a centimeter.

[0015] According to one embodiment of the present disclosure, when the relative distance is less than a distance threshold and the obstacle volume is greater than a volume threshold, the vehicle computer labels the actual obstacle on the vehicle computer's display unit and generates a warning message, in the process where the distance threshold is less than 200 meters. [Brief description of the drawings]

[0016] [Figure 1A] 1 illustrates method steps for stationary object warning according to one embodiment of the present disclosure. [Figure 1B] 1 illustrates a process for determining invalid obstacles according to one embodiment of the present disclosure. [Figure 2A] FIG. 2 shows a schematic diagram of image extraction in a method for stationary object warning according to an embodiment of the present disclosure; [Figure 2B] FIG. 2 shows a schematic diagram of image processing in a method for stationary object warning according to an embodiment of the present disclosure. [Figure 2C] FIG. 1 shows a schematic diagram of obstacle selection in a method for stationary object warning according to an embodiment of the present disclosure. [Figure 2D] FIG. 2 shows a schematic diagram of an actual obstacle warning in a method for warning stationary objects according to an embodiment of the present disclosure; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] In order to further understand and appreciate the configuration and characteristics as well as the effectiveness of the present disclosure, a detailed description of the present disclosure is provided below along with embodiments and accompanying drawings.

[0018] Prior art detection methods are mainly applied to cross roads, adjacent lanes, and parking scenarios, and they are limited to detecting moving objects relative to the vehicle instead of stationary obstacles such as safety zones or parked motorcycles, etc. As a result, accidents may occur due to false positives of stationary obstacles when the vehicle is moving.

[0019] In order to solve the problem that the prior art cannot warn of stationary obstacles, the present disclosure provides a method for warning a vehicle of stationary objects, which includes scanning objects on one side of the vehicle, obtaining corresponding object images, filtering the object images, and obtaining filtered images that affect the vehicle. Depending on the speed of the corresponding object and the vehicle and the distance between them, the method generates a warning message for the user, so that the user can take preventive measures to avoid danger in advance.

[0020] In the following description, various embodiments of the present disclosure are described with reference to figures for describing the present disclosure in detail. Nevertheless, the concept of the present disclosure can be embodied in various forms. These embodiments are not used to limit the scope and scope of the present disclosure.

[0021] First, please refer to FIG. 1A and FIG. 2A. FIG. 1A shows steps of a method for warning stationary objects according to an embodiment of the present disclosure, and FIG. 2A shows a schematic diagram of image extraction in the method for warning stationary objects according to an embodiment of the present disclosure. As shown in the figure, the embodiment is applied to a vehicle 1. The vehicle 1 includes a vehicle computer 12, an optical scanning device 14, an infrared image extracting device 16, and an image extracting device 18. The vehicle computer 12 is electrically connected to the optical scanning device 14, the infrared image extracting device 16, and the image extracting device 18. The vehicle computer 12 executes the following steps:

[0022] Step S10: As the vehicle moves, the optical scanning device, the infrared image extracting device, and the image extracting device extract a first optical scanning image, a first infrared image, and a first surrounding image of the surroundings on one side of the vehicle.

[0023] Step S20: The vehicle computer receives and combines the first optical scanning image, the first infrared image, and the first surrounding image according to a fusion algorithm to provide a first image.

[0024] Step S30: The vehicle computer analyzes and obtains the first obstacle information of the first obstacle in the first image according to the image optical flow method.

[0025] Step S40: When the relative distance is smaller than the distance threshold and the obstacle volume is larger than the volume threshold, the vehicle computer labels the actual obstacle on the display unit of the vehicle computer and generates a warning message.

[0026] In step S10 of this embodiment, as the vehicle 1 moves, the optical scanning device 14, the infrared image extraction device 16, and the image extraction device 18 extract a first optical scanning image 141, a first infrared image 161, and a first surrounding image 181 of the surroundings 90 on one side of the vehicle 1.

[0027] In this embodiment, the vehicle computer 12 includes a processing unit 121. The processing unit 121 may be a server, a notebook computer, a tablet computer, or any electronic device with image processing capabilities. The processing unit 12 of the vehicle computer 12 executes an image processing program 123. The image processing program 123 is used to receive data generated by the optical scanning device 14, the infrared image extracting device 16, and the image extracting device 18, and to perform subsequent image processing.

[0028] The optical scanning device 14 according to the present embodiment is a lidar or laser scanner. The image extracting device 18 according to the present embodiment is a color image extracting unit, for example, an automotive CMOS sensor. The infrared image extracting device 16 according to the present embodiment is an infrared camera. According to the present embodiment, the reason for using the infrared image extracting device 16 is that near-infrared light has a higher transmittance through biological skins such as glass, plastic, and polymers than visible light. Visible light cannot penetrate the above-mentioned materials, so the image appears opaque. Conversely, in the case of near-infrared imaging, the surface layer of the object cannot block the near-infrared light, so the image appears transparent. This allows the near-infrared light to penetrate below the surface of the object to expose the image. According to the present embodiment, a more accurate model of the surroundings outside the vehicle can be provided by comparing the stereoscopic image formed by the optical scanning device 14 and the image extracting device 18 with the IR image provided by the infrared image extracting device 16.

[0029] The vehicle computer 12 according to the present embodiment is installed in the vehicle 1. The optical scanning device 14, the infrared image extracting device 16, and the image extracting device 18 are arranged on one side of the vehicle 1. The vehicle computer 12 is electrically connected to the optical scanning device 14, the infrared image extracting device 16, and the image extracting device 18. The image extracting angle of the infrared image extracting device 16 and the image extracting device 18 is from 120° to 170°. This is for extracting an image of the surroundings or front of the vehicle 1 within 200 meters. The image is, for example, but not limited to, a safety zone, a parked motorcycle, or a tree.

[0030] Now, please refer to Fig. 2B, which shows a schematic diagram of image processing in a method for warning stationary objects according to an embodiment of the present disclosure, in step S20 of the present embodiment. As shown in the figure, the vehicle computer 1 receives and combines the first optical scanning image 141, the first infrared image 161, and the first surrounding image 181 according to a fusion algorithm to give a first image 1231.

[0031] According to this embodiment, the above-mentioned fusion algorithm uses a characteristic function f(x, y) expressed by equation (1). When x and y satisfy a certain event, the value of the characteristic function is equal to 1.

number

[0032] The corresponding hidden state of an observable is determined by the context (observations or states). A characteristic function is derived to select a marginal characteristic (combination of observations or states). In other words, the characteristic (combination of observations) is used to replace the observation to avoid the constraint (restriction) in the independence assumption for the Naive Bayes of the generation model HMM (Hidden Markov Model).

[0033] Let D = {(x (1) ,y (1) ),(x (2) ,y (2) ),···,(x (N) ,y (N) )}, empirical expectation (shown in equation (2)), and model expectation (shown in equation (3)), training a maximum entropy model is equivalent to optimizing the constraints.

number

number

[0034] Assuming that the empirical expectation is equal to the model expectation, there exist multiple sets C that satisfy the constraints for any characteristic function fi, as expressed in the following equation (4).

number

[0035] The maximum entropy principle states that the only reasonable probability distribution that can be derived from incomplete information (e.g., a limited number of training data) should satisfy maximum entropy given constraints. That is, the maximum entropy distribution is the optimal distribution given the constraints. This makes the maximum entropy function model a constrained optimization problem for a convex function, as shown below.

number

number

number

[0036] Usually, the Lagrange duality principle is used to transform the original equations to solve the unconstrained maximum.

number

number

[0037] Partially differentiate the Lagrangian function with respect to p and set it equal to zero. Solving the equation gives us

number

number

[0038] The Maximum Entropy Markov Model (MEMM) is as follows:

number

[0039] distribution

number

number

[0040] Discrete probability distribution {p1, p2, , p n} and maximum information entropy is found. n}. The maximum entropy equation is

number

[0041] Since this is a probability distribution, for each x i In p i The sum of must be equal to 1,

number

[0042] The use of Lagrange multipliers gives the maximum entropy angle.

number

number

[0043] This means that for k=1, ,n,

number

number

[0044] These

number

number

[0045] This equation depends only on λ, so for all

number

number

number

[0046] This gives the following equation:

number

[0047] This results in a uniform distribution with the maximum entropy

number

[0048] Using the above formula (20), the maximum entropy distribution is given. As shown in FIG. 2B, in the image fusion process, the image overlapping regions of the first optical scanning image 141, the first infrared image 161, and the first peripheral image 181 are removed to generate a first image 1231.

[0049] Next, please refer to FIG. 2C. FIG. 2C shows a schematic diagram of selecting an obstacle in a method for warning a stationary object according to an embodiment of the present disclosure. As shown in FIG. 2C, in step S30 of this embodiment, the vehicle computer 12 analyzes and obtains first obstacle information 125 of a first obstacle 1231A in a first image 1231 by an image optical flow method. The first obstacle information 125 includes a relative distance 1221 to the vehicle 1 and an obstacle volume 1225 of the first obstacle 1231A.

[0050] Also, according to this embodiment, it can be determined whether the obstacle is stationary or not. Referring again to FIG. 2A-FIG. 2C, the optical scanning device 14, the infrared image extracting device 16, and the image extracting device 18 can further extract the second optical scanning image 143, the second infrared image 163, and the second surrounding image 183 at another time. By using these images, the vehicle computer 12 can obtain the second image 1233. The vehicle computer 12 uses the optical flow method to analyze the second obstacle 1233A in the second image 1233 and gives the second obstacle information 127. The first obstacle 1231A is the same as the second obstacle 1233A. Since the vehicle 1 is moving, the difference between the first obstacle 1231A and the second obstacle 1233A is only the relative distance.

[0051] Therefore, the presence or absence of the first obstacle 1231 in the first image 1231 can be determined by using the difference between the relative distance between the first obstacle 1231A and the second obstacle 1233A and the speed of the vehicle 1.

[0052] The above optical flow method aims to calculate the position information of the first obstacle 1231A and the second obstacle 1233A using the Lucas-Kanade optical flow algorithm.

[0053] First, we use the image partial differentiation method. We expand the constraint equation using Taylor's rule to obtain the following equation.

number

[0054] Here, HOT stands for high-order terms, which can be ignored when the displacement is small. According to this formula,

number

number

number

[0055] Vx, Vy, Vz are the x, y, z components of the optical flow vector of I(x, y, z, t).

number

number

number

number

number

[0056] Moreover, equation (25) is rewritten as follows:

number

[0057] Since equation (24) contains three unknowns (Vx, Vy, Vz), the following algorithm is used to solve for the unknowns.

[0058] First, assume that the optical flow (Vx, Vy, Vz) is constant in a small cube of size m*m*m, where m>1. From voxels 1...n, n=m3, we obtain the following set of equations:

number

[0059] All the above equations contain three unknowns and form an overdetermined set of equations. In other words, there is redundancy in the equations. The set of equations can be expressed as follows:

number

[0060] The above formula is expressed as follows:

number

[0061] To solve the redundancy problem of the overdetermined equation, equation (29) is

number

number

number

[0062] By substituting the result of equation (32) into equation (24), the first obstacle information 125 of the first obstacle 1231A and the second obstacle information 127 of the second obstacle 1233A can be estimated. By using the first obstacle information 125 and the second obstacle information 127, the relative distance 1221 between the first obstacle 1231A and the vehicle 1 can be obtained.

[0063] Finally, please refer to Fig. 2D. Fig. 2D shows a schematic diagram of an actual obstacle warning in a method for warning stationary objects according to an embodiment of the present disclosure. As shown in Fig. 2D, in step S40 of this embodiment, if the relative distance 1221 of the first obstacle 1231A is smaller than the distance threshold and the obstacle volume 1225 is larger than the volume threshold, the vehicle computer 12 displays the first obstacle 1231A on the display image 1291 of the display unit 129 of the vehicle computer 12, and generates a warning message 1295.

[0064] In step S40, please refer to Fig. 1B. Fig. 1B shows a process of determining invalid obstacles according to one embodiment of the present disclosure. As shown in the figure, the process further includes the following steps:

[0065] Step S41: The vehicle classifies the actual obstacles according to the obstacle volume.

[0066] Step S43: If the obstacle volume is smaller than the volume threshold, the vehicle computer determines the actual obstacle as an invalid obstacle.

[0067] In steps S41 to S43, the vehicle computer 12 classifies the first obstacle 1231A. If the volume of the first obstacle 1231A is smaller than the volume threshold, the vehicle computer 12 determines that the first obstacle 1231A is an invalid obstacle. If the first obstacle 1231A is an invalid obstacle, the vehicle computer 12 does not list the first obstacle 1231A as an attention target.

[0068] In step 40, the warning message 1295 is a voice message or / and a picture message. The distance threshold is 200 meters. The speed threshold is less than or equal to zero. The volume threshold is 400. cubic It is in centimeters.

[0069] For example, the warning message may display the first obstacle 1231A on the display unit 129. Additionally, it may include an audible alarm.

[0070] In other words, when the distance between the first obstacle 1231A and the vehicle 1 is less than 200 meters, a warning message 1295 labels the first obstacle 1231A on the display unit 129 of the vehicle computer 12.

[0071] If the distance is less than 100 meters, in addition to labeling the first obstacle 1231A on the display unit 129 of the vehicle computer 12, the warning message 1295 further provides a warning sound (e.g., a short beep) to alert the driver of the vehicle 1.

[0072] Further, if the distance between the first obstacle 1231A and the vehicle 1 is less than 50 meters, in addition to labeling the first obstacle 1231A on the display unit 129 of the vehicle computer 12, the vehicle computer 12 also provides an intermittent warning sound (e.g., multiple beeps).

[0073] Finally, if the distance between the first obstacle 1231A and the vehicle 1 is less than 50 meters, in addition to labeling the first obstacle 1231A on the display unit 129 of the vehicle computer 12, the vehicle computer 12 will emit a continuous warning sound (e.g., four consecutive beeps) to alert the driver.

[0074] The advantage of this embodiment is that the lidar (optical scanning device 14, infrared image extractor 16, image extractor 18) is used to utilize the 3D surrounding information within 200 meters of the vehicle 1. By using the 3D surrounding information, the dynamic behavior of various objects within 200 meters of the vehicle 1, as well as their speed, acceleration, size and direction, can be obtained in advance so that the driver can avoid the objects. In addition, within 10 meters, images and detailed information are further provided to obtain the omnidirectional information outside the vehicle 1 and a more accurate surrounding model.

[0075] As shown in the above embodiment, the present disclosure provides a method for warning a vehicle of stationary objects, which scans objects on one side of the vehicle to obtain corresponding object images, and filters the object images to obtain filtered images that affect the vehicle. According to the speeds of the corresponding objects and the vehicle, and the distances between them, the method generates a warning message for a user, so that the user can take preventive measures in advance to avoid danger.

[0076] Therefore, the present disclosure complies with legal requirements due to its novelty, non-obviousness, and usefulness. However, the above description is merely an embodiment of the present disclosure and is not used to limit the scope of the present disclosure. Any equivalent changes or modifications made based on the shape, configuration, features, or intent described in the claims of the present disclosure are included in the claims of the present disclosure.

Claims

1. 1. A method for warning a vehicle of a stationary obstacle, comprising: The method is applicable to a vehicle, the vehicle includes a vehicle computer, an optical scanning device, an infrared image extractor, and an image extractor for extracting a visible light image; the vehicle computer is electrically connected to the optical scanning device, the infrared image extracting device, and the image extracting device; The vehicle computer includes: extracting a first optical scanning image, a first infrared image, and a first peripheral image with the optical scanning device, the infrared image extractor, and the image extractor as the vehicle moves; receiving and combining, by the vehicle computer, the first optical scanning image, the first infrared image, and the first surroundings image in accordance with a fusion algorithm to provide a first image; The vehicle computer analyzes and obtains first obstacle information of a first obstacle in the first image according to an image optical flow method, the first obstacle information including a relative distance to the vehicle and an obstacle volume of the first obstacle; when the relative distance is less than a distance threshold and the obstacle volume is greater than a volume threshold, the vehicle computer determines that the first obstacle is a stationary obstacle, and the vehicle computer labels the stationary obstacle on a display unit of the vehicle computer and generates a warning message; A method for performing.

2. the step of the vehicle computer receiving and combining the first optical scanning image, the first infrared image, and the first surrounding image in accordance with the fusion algorithm to provide the first image; the vehicle computer includes a processing unit; the processing unit includes an image processing program; The method of claim 1 , wherein the image processing program is used to perform the fusion algorithm and the image optical flow method.

3. In the step of the vehicle computer analyzing and obtaining first obstacle information of a first obstacle in the first image according to an image optical flow method, The first obstacle information includes a relative distance to the vehicle in the first image and an obstacle volume of the first obstacle, The vehicle computer includes: classifying the first obstacle according to the obstacle volume by the vehicle; 2. The method of claim 1, further comprising the step of: when the obstacle volume is less than the volume threshold, the vehicle computer determining that the first obstacle is an invalid obstacle.

4. The method for warning a vehicle of a stationary obstacle as claimed in claim 1 , wherein the warning message is an audio message and / or an image message.

5. When the relative distance is smaller than a distance threshold and the obstacle volume is larger than a volume threshold, the vehicle computer determines that the first obstacle is a stationary obstacle, The method of claim 1 , wherein the volume threshold is less than 400 cubic centimeters.

6. When the relative distance is smaller than a distance threshold and the obstacle volume is larger than a volume threshold, the vehicle computer determines that the first obstacle is a stationary obstacle, The method of claim 1 , wherein the distance threshold is less than 200 meters.

Images