Vehicle sensor fusion system and method thereof

Abstract

This invention proposes a vehicle perception fusion system and method, which is applied to a vehicle moving at a certain speed. First, an infrared sensing module, a depth sensing module, and an image sensing module are used to acquire a first infrared image, a first depth image, and a first sensing image, as well as a second infrared image, a second depth image, and a second sensing image on one side of the vehicle, and transmit them to a host computer to obtain a first image and a second image. After differential operation, obstacle information is obtained to determine whether the obstacle is located in a first detection area close to the vehicle or a second detection area outside it. When the obstacle is determined to be in the first detection area, a first warning message is generated, thereby allowing the driver to avoid the danger of the obstacle.

Description

Technical Field

[0001] This invention relates to a sensing system and method, and more particularly to a vehicle sensing fusion system and method. Prior Technology

[0002] Traditional Advanced Driver Assistance Systems (ADAS) are developed to assist drivers and can be basically divided into three main parts: onboard sensors, onboard processors, and actuators. ADAS uses onboard sensors to detect signals outside the vehicle, including not only millimeter-wave radar and LiDAR, but also thermal and pressure sensors. This sensor data is transmitted back to the onboard processor, such as the Electronic Control Unit (ECU), which generates warning information sufficient for the driver to recognize, thus avoiding dangerous road conditions. Even when the onboard sensors cannot react in time, they can directly intervene in the driver's driving behavior and activate actuators to achieve functions such as vehicle deceleration, emergency braking, or vehicle swerving, protecting the driver.

[0003] Furthermore, to protect drivers, manufacturers have developed radar detection technology to detect the vehicle's surroundings. However, radar cannot distinguish between fixed and moving obstacles around the vehicle. Moreover, whenever an object is detected approaching the vehicle, it triggers the vehicle's warning unit to frequently issue warning messages, which also causes confusion for drivers. Although there have been many improvements to the detection of the vehicle's surroundings to achieve vehicle-wide monitoring, drivers who ignore any obstacle while the vehicle is in motion will cause accidents, especially when driving on streets. For example, streetlights, vehicles overtaking, pedestrians crossing the street, traffic islands, traffic lights or signs at intersections, and roadside signs are all obstacles that can easily lead to accidents if drivers ignore them.

[0004] While manufacturers have developed color image capture technologies such as dashcams to record the moment an accident occurs and to determine liability afterward, this recording method is not the best solution to prevent accidents. The real solution lies in enabling drivers to prevent accidents in advance. Currently, vehicle radar is only installed on the front and rear sides of the vehicle. Newer vehicles will further integrate side imaging equipment and detection technology to help drivers avoid emergencies in side blind spots. The vehicle will also need to further detect side blind spots to anticipate dangers on the side of the vehicle and notify the driver to protect the driver.

[0005] However, drivers need reaction time while the vehicle is in motion and need to pay attention to obstacles. In particular, with the widespread application of modern automated driving assistance technology, it is no longer just about intervening in the driver's driving behavior to protect the driver, but also about providing the driver with the reaction time needed to avoid obstacles in advance.

[0006] To address the aforementioned problems, this invention provides a vehicle perception fusion system and method. It acquires a first infrared image, a first depth image, and a first sensing image from one side of the vehicle, and acquires a second infrared image, a second depth image, and a second sensing image at different times to obtain corresponding first and second images. Furthermore, it extracts obstacle images through differential calculations to obtain obstacle information, thereby determining whether the obstacle is located near a first detection area or outside a second detection area of ​​the vehicle. When the obstacle is determined to be located within the first detection area and its size exceeds a size threshold, the host computer generates a corresponding first warning message to prevent the driver from encountering unexpected situations. Summary of the Invention

[0007] One objective of this invention is to provide a vehicle perception fusion system and method. This system acquires corresponding images from an infrared sensing module, a depth sensing module, and an image sensing module on one side of the vehicle, and fuses these images into a first image and a second image. An obstacle image is then extracted through differential calculation to obtain obstacle information. The relative distance between the vehicle and the obstacle is obtained from this obstacle information to determine whether the obstacle is located in a first detection area close to the vehicle or in a second detection area outside the first detection area. When the obstacle is determined to be located in the first detection area, the main unit generates a corresponding first warning message to help the driver avoid unexpected situations.

[0008] To achieve the above objectives, this invention discloses a vehicle perception fusion method applied to a vehicle moving at a certain speed. The vehicle is equipped with a host computer, an infrared sensing module, a depth sensing module, and an image sensing module. The host computer is electrically connected to the infrared sensing module, the depth sensing module, and the image sensing module. The vehicle perception fusion method of this invention first uses the infrared sensing module to capture a first infrared image and a second infrared image based on one side of the vehicle. The depth sensing module captures a first depth image and a second depth image based on that side of the vehicle. Then, the image sensing module uses the image sensing module to capture a first infrared image and a second infrared image based on the vehicle's... The first side acquires a first sensing image and a second sensing image, and the first infrared image, the first depth image, and the first sensing image, as well as the second infrared image, the second depth image, and the second sensing image, are acquired at different times and transmitted to the host computer respectively. Then, the host computer executes a fusion algorithm to obtain a first image by fusing the first infrared image, the first depth image, and the first sensing image, and to obtain a second image by fusing the second infrared image, the second depth image, and the second sensing image. The first image and the second sensing image are then fused together. Both images include a first image region and a second image region. The first image region corresponds to a first detection region, and the second image region corresponds to a second detection region. The first detection region is located outside one of the vehicle's outer edges, and the second detection region is located outside one of the first detection regions. Next, the host computer performs an image optical flow method to obtain obstacle information based on the difference between the first and second images. Then, the host computer uses the obstacle information to obtain the relative distance between the obstacle and one of the vehicle's outer edges, and judges the distance based on the relative distance and a distance threshold. The obstacle is located within the first or second detection area of ​​the vehicle. Further, when the host determines that the obstacle is located within the first detection area based on the relative distance being less than or equal to a distance threshold, the host obtains a movement vector, a size, and an acceleration vector of the obstacle based on the obstacle information. When the host determines that the obstacle is sufficient to threaten the vehicle based on the size being greater than a size threshold, the host generates a first warning message corresponding to the obstacle based on the vehicle's speed, the obstacle's movement vector, size, and acceleration vector. Therefore, this invention provides a hazard prediction for one side of the vehicle during high-speed travel and generates corresponding warning messages. This can be applied to allow the driver assistance system to intervene in driving control based on the warning messages and simultaneously notify the driver, or to warn the driver of obstacles in advance to avoid accidents.

[0009] This invention provides an embodiment in which the steps further include: when the host obtains the relative distance based on the obstacle information and the relative distance is greater than the distance threshold, and determines that the obstacle is located in the second detection area based on the obstacle image being located in the second image area, the host obtains the movement vector of the obstacle based on the obstacle information, and the host generates a second warning message corresponding to the obstacle being located in the second detection area based on the vehicle's movement speed and the obstacle's movement vector. Therefore, this invention further provides a way to reduce the amount of data access and computation of obstacle information when the vehicle is not close to an obstacle.

[0010] The present invention provides an embodiment in which the host performs the difference operation based on the first image and the second image to obtain an obstacle image, and obtains obstacle information based on the obstacle image.

[0011] This invention provides an embodiment in which, during the process of generating a first warning message for an obstacle using the host based on the vehicle's moving speed, the obstacle's movement vector, its size, and its acceleration vector, the movement vector corresponds to the obstacle's relative speed and direction of movement relative to the vehicle. When the moving speed is less than the relative speed, the host further controls the vehicle to avoid the obstacle. Therefore, this invention further provides a method for a vehicle to actively avoid an obstacle when approaching it, for example, by braking to a stop before the obstacle or accelerating to avoid the obstacle.

[0012] The present invention provides an embodiment in which, when the host determines that the relative distance is less than or equal to the distance threshold and the size is greater than a size threshold, the host generates a first warning message corresponding to the obstacle based on the vehicle's moving speed, the obstacle's moving vector, the size, and the acceleration vector. Furthermore, the host obtains the obstacle's positioning information based on the obstacle information and determines whether the obstacle is a concave road surface based on a road surface threshold and the positioning information. When the positioning information is less than the road surface threshold, the host further determines that the obstacle is the concave road surface.

[0013] The present invention provides an embodiment in which, in the step of using the host to obtain the relative distance between the obstacle and one of the vehicles based on the obstacle information, and determining whether the obstacle is located in a first detection area or a second detection area of ​​one of the vehicles based on the relative distance and a distance threshold value, the distance threshold value is 10 meters to 20 meters, and the size threshold value is 1 square centimeter to 10 square centimeters, thereby distinguishing the first detection area and the second detection area, and distinguishing the threat level of one of the obstacles.

[0014] The present invention provides an embodiment in which, during the process of using the host to execute a fusion algorithm to obtain a first image based on the first infrared image, the first depth image, and the first sensing image, and to obtain a second image based on the second infrared image, the second depth image, and the second sensing image, the host further modifies the first infrared image, the first depth image, and the first sensing image and fuses them into the first image, and modifies the second infrared image, the second depth image, and the second sensing image and fuses them into the second image, and the host corrects the first image and the second image respectively, so that the first image and the second image are more consistent with the target sensed by the infrared sensing module, the depth sensing module, and the image sensing module.

[0015] The present invention further provides a vehicle perception fusion system applied to a vehicle moving at a certain speed. The vehicle perception fusion system includes a host, an infrared sensing module, a depth sensing module, and an image sensing module. The host is disposed inside the vehicle, and the infrared sensing module, the depth sensing module, and the image sensing module are disposed on one side of the vehicle and electrically connected to the host. The infrared sensing module, the depth sensing module, and the image sensing module capture a first infrared image, a first depth image, and a first sensing image from one side of the vehicle, and capture a second infrared image, a second depth image, and a second sensing image at different times, and then transmit them to the host computer. The host computer executes a fusion algorithm to obtain a first image based on the first infrared image, the first depth image, and the first sensing image, and obtains a second image based on the second infrared image, the second depth image, and the second sensing image. Both the first image and the second image include a first image region and a second image region. The first image region corresponds to a first detection region, and the second image region corresponds to a second detection region. The first detection region is located between the vehicle and the second detection region. The host computer executes an image optical flow method to obtain an image based on the difference operation between the first image and the second image. An obstacle image is obtained, and obstacle information is acquired based on the obstacle image. When the host obtains the relative distance between the obstacle and the vehicle based on the obstacle information, and the relative distance is less than or equal to a distance threshold, and the obstacle is located in the first detection area based on the obstacle image being in the first image area, the host obtains the obstacle's movement vector, size, and acceleration vector based on the obstacle information. When the relative distance is less than or equal to the distance threshold and the size is greater than a size threshold, the host generates a first warning message corresponding to the obstacle being located in the first detection area based on the vehicle's movement speed, the obstacle's movement vector, size, and acceleration vector. Therefore, the host predicts whether the obstacle will affect the vehicle's movement direction based on the first warning message, which can be used to notify the driver assistance system to intervene or to notify the driver.

[0016] The present invention provides another embodiment in which, when the host obtains the relative distance between the obstacle and the vehicle based on the obstacle information and the relative distance is greater than the distance threshold, and determines that the obstacle is located in the second detection area based on the obstacle image being located in the second image area, the host obtains the movement vector of the obstacle based on the obstacle information, and the host generates a second warning message corresponding to the obstacle being located in the second detection area based on the vehicle's movement speed and the obstacle's movement vector. Therefore, the present invention further provides a way to reduce the amount of data access and computation of obstacle information when the vehicle is not close to an obstacle.

[0017] The present invention provides another embodiment in which the host performs the difference operation based on the first image and the second image to obtain an obstacle image, and obtains obstacle information based on the obstacle image.

[0018] The present invention provides another embodiment, wherein the distance threshold is 10 meters to 20 meters and the size threshold is 1 square centimeter to 10 square centimeters.

[0019] The present invention provides another embodiment in which the movement vector corresponds to the relative velocity of the obstacle with respect to one of the vehicles and the direction of movement of one of the obstacles. When the movement velocity is less than the relative velocity, the host further controls the vehicle to avoid the obstacle. Therefore, the present invention further provides a vehicle actively avoiding an obstacle when it approaches it, for example: braking to a stop before the obstacle, or accelerating to avoid the obstacle.

[0020] The present invention provides another embodiment in which the host obtains a location message of the obstacle based on the obstacle information and determines whether the obstacle is located on a recessed road surface based on a road surface threshold value and the location message. When the location message is less than the road surface threshold value, the host further determines that the obstacle is located on the recessed road surface.

[0021] The present invention provides an embodiment in which the host computer uses the fusion algorithm to modify the first infrared image, the first depth image and the first sensing image respectively and fuse them into the first image, and modifies the second infrared image, the second depth image and the second sensing image respectively and fuses them into the second image, and the host computer corrects the first image and the second image respectively. Simple Explanation of the Diagram

[0022] Figures 1A to 1B: These are flowcharts of one embodiment of the present invention; Figures 2A to 2J: These are schematic diagrams of a vehicle perception fusion system according to one embodiment of the present invention; Figure 3A: It is a schematic diagram of the detection area according to one embodiment of the present invention; Figure 3B: It is a schematic diagram of a perspective projection method according to one embodiment of the present invention; Figure 4A: This is a schematic diagram of an obstacle located in the first detection area according to one embodiment of the present invention; and Figure 4B: It is a schematic diagram of an obstacle located in the second detection area according to one embodiment of the present invention. Implementation

[0023] To enable your review committee to have a better understanding of the features and effects of this invention, the following examples and accompanying descriptions are provided:

[0024] In view of the fact that conventional radar systems and dashcams fail to provide obstacle prediction for vehicles, the present invention proposes a vehicle perception fusion system and method to solve the problem that conventional technology makes it difficult for drivers to avoid obstacles.

[0025] The following will further explain the characteristics provided by the vehicle perception fusion system and method disclosed in this invention, as well as the associated system:

[0026] First, please refer to Figures 1A and 1B, which are flowcharts of one embodiment of the present invention. As shown in the figures, the vehicle perception fusion method of the present invention includes the following steps:

[0027] Step S10: Using the infrared sensing module, the depth sensing module, and the image sensing module, a first infrared image, a first depth image, and a first sensing image are obtained respectively based on the first detection area and the second detection area on one side of the vehicle, and a second infrared image, a second depth image, and a second sensing image are obtained respectively, and then transmitted to the host computer respectively;

[0028] Step S12: Execute a fusion algorithm using the host computer to obtain a first image based on the first infrared image, the first depth image, and the first sensing image, and to obtain a second image based on the second infrared image, the second depth image, and the second sensing image;

[0029] Step S14: The host computer performs an image optical flow method to perform a difference operation between the first image and the second image to obtain an obstacle image, and obtains obstacle information based on the obstacle image;

[0030] Step S16: Use the host computer to obtain the relative distance between the obstacle and the vehicle based on obstacle information;

[0031] Step S18: The host computer determines that the obstacle is located in the first detection area based on the obstacle image being located in the first image area;

[0032] Step S20: Using the host computer, obtain the obstacle's movement vector, size, and acceleration vector based on obstacle information;

[0033] Step S22: The host computer generates a first warning message based on the vehicle's moving speed, the obstacle's moving vector, size, and acceleration vector.

[0034] Please refer to Figures 2A to 2J, which are schematic diagrams of a vehicle perception fusion system according to one embodiment of the present invention. As shown in Figures 2A to 2J, the vehicle perception fusion system 1 used in conjunction with the vehicle perception fusion method of the present invention includes a host 10, a depth sensing module 20, an infrared sensing module 25, and an image sensing module 30. In this embodiment, the host 10 is exemplified by an automotive computer with a computing processing unit 12, but it is not limited to this. It can also be a server, a laptop computer, a tablet computer, or an electronic device with image processing capabilities. The computing processing unit 12 can be a system on a chip (SOC), a microprocessor (μP), a microcontroller (MCU), a programmable logic controller (PLC), a central processing unit (CPU), or a graphics processing unit (GPU). The host 10 executes a computation program 122 through the computing processing unit 12. The computation program 122 supports DepthWise convolution operations, OpenNI operations, OpenCV operations, and custom function operations. The depth sensing module 20 in this embodiment is a structured light sensing module, which senses by emitting a matrix of light spots. The image resolution of the depth sensing module 20 in this embodiment is QVGA (320x240): 60 frames per second (FPS).

[0035] Continuing from the above, the infrared sensing module 25 in this embodiment is a near-infrared (NIR) image sensor, a short-wave infrared image sensor, or a far-infrared (LWIR) image sensor; the image sensing module 30 in this embodiment is a common light image sensing module, such as an automotive CMOS image sensor. The image resolution of the image sensing module 30 in this embodiment is SXGA (1280x1024). The horizontal viewing angle of the depth sensing module 20, the infrared sensing module 25, and the image sensing module 30 in this embodiment is 40 to 70 degrees, and the vertical viewing angle is 40 to 60 degrees.

[0036] The host unit 10 is disposed within a vehicle V. A depth sensing module 20, an infrared sensing module 25, and an image sensing module 30 are disposed on one side of the vehicle V. The host unit 10 is electrically connected to the depth sensing module 20, the infrared sensing module 25, and the image sensing module 30. In this embodiment, the image capturing angle range of the image sensing module 30 is 120 to 180 degrees, and it captures sensing images, such as object images, within a range of 10 to 300 meters around the vehicle V. Further details are provided below.

[0037] In step S10, as shown in Figure 2A, the present invention utilizes a depth sensing module 20 to perform structured light sensing on the environment 90 on one side of the vehicle V, or even optical sensing within 10 to 300 meters around the vehicle V, to generate a first depth image 202 based on the sensing results. Furthermore, an infrared sensing module 25 and an image sensing module 30 are used to capture a first infrared image 252 and a first sensing image 302 on the environment 90 on one side of the vehicle V, respectively, and these are transmitted together. Image 302 is transmitted to host 10, particularly to the computing unit 12 of host 10; and as shown in Figure 2B, depth sensing module 20, infrared sensing module 25 and image sensing module 30 capture a second depth image 204, a second infrared image 254 and a second sensing image 304 at different times from the environment 90 on one side of the vehicle V, and transmit the second depth image 204, the second depth image 204 and the second sensing image 304 to host 10, particularly to the computing unit 12 of host 10.

[0038] As shown in Figure 3A, the depth sensing module 20, the infrared sensing module 25, and the image sensing module 30 obtain corresponding first depth images 202, first infrared images 252, and first sensing images 302, as well as second depth images 204, second infrared images 254, and second sensing images 304, based on a first detection area A1 and a second detection area A2 corresponding to the vehicle V. The first detection area A1 is located between the vehicle V and the second detection area A2, that is, the first detection area A1 is closer to the vehicle V, while the second detection area A2 is located outside the first detection area A1. For example, the first detection area A1 is an inner circle area close to the vehicle V, and the second detection area A2 is an outer circle area adjacent to the first detection area A1.

[0039] Furthermore, as shown in Figures 2A and 2B, the obstacle 92 sensed by the depth sensing module 20, infrared sensing module 25, and image sensing module 30 will change as the vehicle V moves. Therefore, the first depth image 202, the first infrared image 252, and the first sensing image 302, as well as the second depth image 204, the second infrared image 254, and the second sensing image 304 captured at different times, will have obstacles 92 with slightly different positions. However, the interval between the first group of first depth images 202, first infrared images 252, and first sensing images 302 and the second group of second depth images 204, second infrared images 254, and second sensing images 304 of the present invention is less than 0.5 seconds, so that the host 10 can give the driver of the vehicle V reaction time to react quickly to the obstacle 92.

[0040] Furthermore, the aforementioned first detection area A1 and second detection area A2 cover blind spot locations, which correspond to one side of the vehicle V and conform to the blind spot area specified in the ISO 17387 standard for intelligent transportation system certification. The depth sensing module 20 is specifically designed for visual blind spot locations that the vehicle V cannot visually reach, that is, blind spot locations outside the driver's visual field. Even if the vehicle V has left and right rearview mirrors, the depth sensing module 20, infrared sensing module 25, and image capturing device 30 are still needed to capture images that cannot visually reach. Moreover, the Advanced Driver Assistance System (ADAS) also requires more sophisticated image capture to more accurately identify whether there are obstacles on one side of the vehicle V, such as people, vehicles, bus stop signs, traffic signs, or traffic signals, or even any obstacles in the visual location where blind spots frequently occur, such as the A-pillar inside the vehicle.

[0041] As shown in Figure 3B, using perspective projection, the projected image points of image point P0 used by the infrared sensing module 25 and the image sensing module 30 to sense the image are divided into a first image point P1 and a second image point P2. The coordinates (x, y) of the first image point P1 are located in the first surface region DM1, and the coordinates (x′, y′) of the second image point P2 are located in the second surface region DM2. Therefore, the relative relationship between the infrared sensing module 25 and the image sensing module 30 in capturing the first image point P1 and the second image point P2 is as follows: Formula (1) Formula (II)

[0042] Where (x, y) is the first image point P1 and (x', y') is the second image point P2; m0, m1, ..., m7 are the relevant focal length, rotation angle and scaling parameters of the infrared sensing module 25 and the image sensing module 30. They can be expanded into a complex array of image point pairs, and then the optimal values ​​of m1 to m7 are obtained by nonlinear minimization operation using the Levenberg-Marquardt algorithm, which are used as the optimal capturing focal length of the image sensing module 30.

[0043] Referring again to Figures 1 and 2C, in step S12, the host 10 executes a calculation program 122 through the processing unit 12 to perform a fusion algorithm 124. This algorithm receives the first depth image 202 generated by the depth sensing module 20, the first infrared image 252 generated by the infrared sensing module 25, and the first sensing image 302 generated by the image sensing module 30, and performs image fusion processing to generate a first image IMG1. Then, it continues to receive the second depth image 204 generated by the depth sensing module 20, the second infrared image 254 generated by the infrared sensing module 25, and the second sensing image 304 generated by the image sensing module 30, and performs image fusion processing to generate a second image IMG2. Since this embodiment uses an environment 90 that includes an obstacle 92, both the first image IMG1 and the second image IMG2 include an obstacle image 922.

[0044] The processing unit 12 executes the processing program 122 to perform preprocessing on the first depth image 202, the first infrared image 252 and the first sensing image 302, as well as the second depth image 204, the second infrared image 254 and the second sensing image 304 in batches. This highlights the obstacle image 922 corresponding to the obstacle 92 on the first image IMG1 and the second image IMG2. The processing unit 12 also performs image stitching and color grayscale correction on the first image IMG1 and the second image IMG2 to provide subsequent spatial recognition.

[0045] The fusion algorithm 124 first introduces the feature function f(x,y) as shown in equation (iii). f(x,y) is a binary function, which means that when x and y satisfy a certain fact, the feature function value is 1. Formula (3)

[0046] In real-world numerical computation environments, the hidden state corresponding to a given observation is determined by the context (observation, state). Introducing feature functions allows us to freely select features (combinations of observations or states). Essentially, features (combinations of observations) replace observations, avoiding the limitations of the observation independence assumption in generative models (e.g., Hidden Markov Models (HMMs), naive Bayes).

[0047] Given training data of size T, D={(x,y)}, we obtain an empirical expectation and a model expectation. Formula (IV) Formula (5)

[0048] Assuming the empirical expectation equals the model expectation, then there exists a set C of conditional probability distributions for any feature function fi that satisfy this constraint, thus: Formula (VI)

[0049] In step S14, as shown in Figure 2D, the host 10 executes an image optical flow method 126 through the processing unit 12 to perform a difference operation 1262 based on the first image IMG1 and the second image IMG2 to obtain an obstacle image 922, and obtain obstacle information 924 of the obstacle 92 based on the obstacle image 922. Both the first image IMG1 and the second image IMG2 include a first image region IMA1 and a second image region IMA2. The obstacle information 924 includes the relative distance D between the obstacle 92 and the vehicle V. Therefore, step S16 is then executed. The first image region IMA1 corresponds to the first detection region A1, and the second image region IMA22 corresponds to the second detection region A2. That is, the first image IMG1 and the second image IMG2 have different image regions due to different depths of field, thereby corresponding to different detection regions of the environment 90. This embodiment uses the first detection region A1 and the second detection region A2 as examples.

[0050] In step S16, as shown in Figure 2E, the host 10 obtains a relative distance D between the obstacle 92 and the vehicle V through the processing unit 12 based on the obstacle information 924. The host 10 obtains the positioning information 926 of the obstacle 92 through the processing unit 12 based on the obstacle information 924. This information is used to obtain the relative distance D between the obstacle 92 and the vehicle V. In particular, the relative distance D between the obstacle 92 and the vehicle V is obtained by using the vehicle V's moving speed SPD in conjunction with the positioning information 926. The positioning information 926 of the obstacle 92 comes from the depth of field corresponding to the obstacle 92. That is, the host 10 obtains the positioning information 926 of the obstacle 92 based on the depth of field corresponding to the obstacle 92 through the processing unit 12, thereby obtaining the relative distance D between the obstacle 92 and the vehicle V. Therefore, as shown in Figure 2F, this embodiment uses the image space of the first image IMG1 as a reference to determine the image area corresponding to the obstacle image 922 in the first image IMG1. Thus, the host 10, through the processing unit 12, determines, based on a distance threshold value DTH and the relative distance D between the obstacle 92 and the vehicle V, that the obstacle image 922 is located in the first image area IMA1 or the second image area IMA2 of the first image IMG1, and therefore determines that the obstacle 92 is located in the first detection area A1 or the second detection area A2 of the vehicle V. However, this invention further allows the host 10 to use the image space of the second image IMG2 as a reference to determine the image area corresponding to the obstacle image 922 in the second image IMG1, for example, determining that the obstacle image 922 is located in the first image area IMA1 or the second image area IMA2 of the second image IMG2. Furthermore, the host 10 further obtains a horizontal height value 928 based on the positioning information 926 of the obstacle 92 through the processing unit 12.

[0051] When there are multiple obstacle images 92, the obstacle information 924 obtained by the host 10 through the processing unit 12 is further used to distinguish the obstacle images 922. In this embodiment, the first image IMG1 is used as an example. When the relative distance D corresponding to the depth of field is the same, the host 10 uses the processing unit 12 to binarize the first image IMG1, so as to perform data analysis on the image data of the first image IMG1 to distinguish the obstacle 92. The binarization equation is as follows: Formula (VII)

[0052] in, Binarization threshold Input image, : All pixel items, : The grayscale value of the pixel coordinate.

[0053] In addition, the equation for distinguishing obstacles is as follows: Formula (8)

[0054] Among them, the obstacle image 922 obtained by differential calculation of the first image IMG1 and the second image IMG2 is assumed to have a sample size of... K cluster centers are randomly selected as For each sample i, its cluster center is calculated. , , The class in which sample i is closest to cluster K, where K is the number of clusters. J represents the predicted cluster center, and J is the centroid of each cluster center.

[0055] If the K-group decreases by one point, the equation becomes as follows: , Formula (9) ð ð ð Formula (10) Formula (XI)

[0056] in The class in which sample i is closest to cluster K, where K is the number of clusters. J represents the predicted cluster center, and J is the centroid of each cluster center.

[0057] In particular, the vector corresponding to the relative distance D can be close to 0 relative to the distance threshold value DTH, indicating that the obstacle 92 is close to the distance threshold value DTH. The distance threshold value DTH is preset in the calculation program 122 and can be 10 meters to 20 meters. In particular, the distance threshold value DTH can be 10 meters to 15 meters. This determines which warning message should be provided to the driver. Even if the obstacle 92 is located in the first detection area A1 of the vehicle V, the driver still has a buffer distance to react in time.

[0058] Further, as shown in step S18 and Figure 2G, when the host 10 determines, through the processing unit 12, that the obstacle image 922 is located in the first image region IMA1 of the first image IMG1 based on a distance threshold value DTH and the relative distance D between the obstacle 92 and the vehicle V, that is, as shown in Figure 4A, the host 10 determines that the obstacle 92 is located in the first detection region A1 of the vehicle V. Then, as shown in step S20 and Figure 2G, the obstacle information 924 includes not only the relative distance D between the obstacle 92 and the vehicle V, but also a movement vector OBV1, an acceleration vector OBV2, and a size OBV3 of the obstacle 92. Therefore, the host 10 obtains the movement vector OBV1, acceleration vector OBV2, and size OBV3 of the obstacle 92 through the processing unit 12 based on the obstacle information 924.

[0059] In this embodiment, the movement vector OBV1 corresponds to the relative velocity of obstacle 92 relative to vehicle V and the direction of movement of obstacle 92. The first image IMG1 and the second image IMG2 are point cloud image data, and the obstacle image 922 obtained by the difference operation 1262 between the first image IMG1 and the second image IMG2 is also point cloud image data. Therefore, the image processing process of image optical flow method 126 performed by host 10 through processing unit 12 in this embodiment is based on point cloud image processing technology as the basic principle of image processing. The calculation of movement vector OBV1 by processing unit 12 is equivalent to the calculation of relative velocity of obstacle 92 relative to vehicle V and direction of movement of obstacle 92 by processing unit 12. At the same time, through the above formulas (viii) to (xi), host 10 obtains the centroid of obstacle image 922 and thus obtains the size OBV3 of obstacle 92 through the calculation of formulas (viii) to (xi) by processing unit 12.

[0060] In addition, the host 10 further uses the computing unit 12 to compare a road threshold value HTH with the horizontal height value 928 corresponding to the positioning information 926 to determine whether the obstacle 92 is located on a recessed road surface. When it is less than the road threshold value HTH, the host 10 further uses the computing unit 12 to determine that the obstacle 92 is located on a recessed road surface.

[0061] As shown in Figures 2H and 4A, when the relative distance D is less than the distance threshold DTH and the size OBV3 is greater than or equal to the size threshold STH, the host 10 determines that the obstacle 92 is located in the first detection area A1 and that the obstacle 92 will cause an accident to the vehicle V based on the obstacle image 922 being located in the first image area IMA1. Continuing in step S22, the host 10 uses the processing unit 12 to generate a first warning message M1 corresponding to the obstacle 92 based on the vehicle V's moving speed SPD, the obstacle 92's moving vector OBV1, and the acceleration vector OBV2. For example, the vehicle V projects a message on the windshield asking the driver to pay attention to a side obstacle or a vehicle approaching from the side.

[0062] In step S22, referring again to Figure 2H, the movement vector OBV1 corresponds to the relative speed of obstacle 92 with respect to the vehicle and the direction of movement of obstacle 92. When the movement speed SPD of vehicle V is less than the relative speed of obstacle 92, the host 10 further generates a braking warning message XM through the arithmetic processing unit 12.

[0063] However, when the relative distance D is less than the distance threshold DTH and the size OBV3 is less than the size threshold STH, the host 10 determines that the obstacle 92 located in the first detection area A1 is not enough to threaten the vehicle V, that is, the obstacle 92 will not cause the vehicle V to have an accident. Therefore, the host 10 switches from step S20 back to step S10, and the depth sensing module 20, infrared sensing module 25 and image sensing module 30 continue to capture the corresponding images.

[0064] Referring again to Figures 1A and 1B, the vehicle perception fusion method of the present invention further includes:

[0065] Step S24: The host computer determines that the obstacle is located in the second detection area based on the obstacle image being located in the second image area;

[0066] Step S26: Obtain the movement vector of the obstacle using the host computer based on obstacle information; and

[0067] Step S28: The host computer generates a second warning message based on the vehicle's speed and the obstacle's movement vector.

[0068] Referring again to Figure 1A, when the host 10 determines that the relative distance D is greater than the distance threshold value DTH, it proceeds from step S16 to a first process node A, as shown in Figure 1B. The first process node A then proceeds to step S24, as shown in Figures 2I and 4B. The host 10 determines that the obstacle 92 is located in the second detection area A2 based on the obstacle image 922 being located in the second image area IMA2. That is, when the host 10 determines that the obstacle 92 is located in the second detection area A2 based on the obstacle image 922 being located in the second image area IMA2, it proceeds to step S26. In step S26... As shown in Figure 2I, the host 10 obtains the movement vector OBV1 of obstacle 92 based on obstacle information 924 through the processing unit 12. Continuing in step S28, as shown in Figure 2J, the host 10 generates a second warning message M2 corresponding to obstacle 92 based on the vehicle V's movement speed SPD and the obstacle 92's movement vector OBV1 through the processing unit 12. The warning level of the second warning message M2 is lower than that of the first warning message M1. In other words, the host 10 regards the second detection area A2 as a non-warning area and the first detection area A1 as a warning area.

[0069] The computation program 122 described in the above embodiments includes Sobel edge detection in its image processing, and the Sobel edge detection algorithm is as follows:

[0070] Sobel edge detection:

[0071] In the image, each pixel and its neighbors are represented in a matrix (Pixel) using a nine-grid layout, labeled P1, P2, P3, P4…P9, as shown in equation (12). Formula (12) Formula (13) Formula (XIV) Formula (15) Formula (16) Formula (17) Formula (18)

[0072] The aforementioned image optical flow method 126 uses the Lucas–Kanade Optical Flow algorithm to estimate obstacles. First, it performs a difference operation 1262 on the first image IMG1 and the second image IMG2 using image difference to obtain the corresponding obstacle image 922. The difference operation 1262 is obtained by using Taylor's formula to solve the image constraint equations. Formula (19)

[0073] Wherein, HOT represents a higher-order equation, which can be ignored when the movement is sufficiently small. Since the interval between the first image IMG1 and the second image IMG2 in the above embodiment is less than 1 second, the movement of the obstacle 92 corresponding to the obstacle image 922 obtained by the difference operation in the first image IMG1 and the second image IMG2 can be ignored. From equation (19), we can obtain: Formula (20) or Formula (21) And obtain: Formula (22)

[0074] Vx, Vy, and Vz are the components of x, y, and z in the optical flow vector of I(x,y,z,t). , , and This is the difference of the image at point (x,y,z,t) in the corresponding direction of movement over time t. Therefore, equation (22) is transformed into the following equation: Equation (23): IxVx+IyVy+IzVz = -It

[0075] Equation (23) can be further written as follows: Formula (24)

[0076] Since there are three unknowns (Vx, Vy, Vz) in equation (23), the unknowns are calculated by the continuation algorithm:

[0077] First, assume that the flow (Vx, Vy, Vz) is a constant within a small window of size m*m*m (m>1). Then, from primitives 1...n, n = m3, we can obtain the following set of equations: Formula (25)

[0078] The above equations all contain three unknowns, forming a system of equations. Furthermore, this system is overdetermined, meaning it contains redundancy. The system can be represented as: Formula (26)

[0079] Notation: Formula (27)

[0080] To solve this overdetermined problem, equation (XXI) is obtained using the least squares method: Formula (28)

[0081] or Formula (29)

[0082] get: Formula (30)

[0083] Substituting the result of equation (30) into equation (22) allows us to estimate the acceleration vector of one of the target objects and the relative distance between the target object and one of the vehicles, which are used to classify and predict the movement of the target objects.

[0084] The maximum entropy principle states that the only reasonable probability distribution derived from incomplete information (such as a finite amount of training data) should have the maximum entropy value under the constraints provided by this information—that is, the distribution with the maximum entropy is optimal in the set of conditional probabilities. Therefore, the maximum entropy model becomes a constrained optimization problem of convex functions. Formula (31) Formula (32) Formula (33)

[0085] The Lagrange duality principle is typically used to transform the original expression into an unconstrained extremum solution: Formula (34) Formula (35)

[0086] Taking the partial derivative of the Lagrange function with respect to p and setting it equal to 0, we can solve the equation by omitting the integers from 1 to n, and obtain the following expression: Formula (36) Formula (37)

[0087] Maximum Entropy Markov Model (MEMM) Formula (38)

[0088] use The distribution replaces the two conditional probability distributions in the HMM, representing the probability of obtaining the current state from the previous state given the observations; that is, predicting the current state based on the previous state and the current observations. Each such distribution function... Both are exponential models that follow maximum entropy.

[0089] Suppose we find a point on the discrete probability distribution The probability distribution of the minimum upper point to be found, along with the maximum information entropy. The formula for maximum entropy: Formula (39)

[0090] This is the sum of probabilities from a probability distribution. At each point Must equal 1: Formula (40)

[0091] By using Lagrange multipliers to find the angle of maximum entropy, Spanning all discrete probability distributions superior The following formula is obtained: Formula (41)

[0092] Equation (41) gives a system equation. , so that: Formula (42)

[0093] By performing these differentiation equations, we obtain the following expression: Formula (43)

[0094] This indicates that all They are equal (because equation (43) depends on λ). By using constraints, we obtain the following equation: Formula (44)

[0095] Therefore, we obtain the result from equation (44). Formula (45)

[0096] Therefore, a uniform distribution is a distribution with maximum entropy, and there is no difference between the upper distributions. Formula (46)

[0097] In summary, the vehicle perception fusion system and method of the present invention provide a host computer that acquires corresponding images at different times from an infrared sensing module, a depth sensing module, and an image sensing module on one side of the vehicle. A fusion algorithm is then used to obtain a first image and a second image. The host computer then performs an image optical flow method to obtain an obstacle image based on the difference between the first image and the second image, and obtains obstacle information based on the obstacle image. Finally, the host computer uses the obstacle information to obtain the relative distance between the obstacle and one side of the vehicle, and determines the obstacle based on the relative distance and a distance threshold. If the object is located in the first detection area or the second detection area of ​​the vehicle, and further, when the relative distance is less than or equal to the distance threshold value and the obstacle image is located in the first image area, the host obtains a movement vector, a size, and an acceleration vector of the obstacle based on the obstacle information. When the relative distance is less than or equal to the distance threshold value and the size is greater than a size threshold value, the host generates a first warning message corresponding to the obstacle based on the vehicle's movement speed, the obstacle's movement vector, the size, and the acceleration vector.

[0098] Furthermore, when the host obtains the relative distance between the obstacle and the vehicle based on the obstacle information, and the relative distance is greater than the distance threshold, and determines that the obstacle is located in the second detection area based on the obstacle image being located in the second image area, the host obtains the movement vector of the obstacle based on the obstacle information, and generates a second warning message corresponding to the obstacle being located in the second detection area based on the vehicle's movement speed and the obstacle's movement vector. Therefore, the present invention further provides a way to reduce the amount of data access and computation of obstacle information when the vehicle is not close to the obstacle, so as to provide a more lightweight second warning message. Moreover, the host obtains the obstacle's location information based on the obstacle information and determines whether the obstacle is located on a concave road surface based on a road surface threshold and the location information. When the location information is less than the road surface threshold, the host further determines that the obstacle is located on a concave road surface.

[0099] Therefore, this invention is indeed novel, inventive, and industrially applicable, and undoubtedly meets the requirements for patent application under the Patent Law of our country. Thus, we hereby file an invention patent application in accordance with the law, and earnestly pray that the Bureau will grant the patent as soon as possible.

[0100] However, the above description is only a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes and modifications made to the shape, structure, features and spirit described in the claims of the present invention should be included in the scope of the claims of the present invention.

[0101] 1: Vehicle Perception Fusion System 10: Host 12: Processing Unit 122: Calculation Program 124: Fusion Algorithm 126: Image Optical Flow Method 1262 Difference Operations 20: Depth Sensing Module 202: First Depth Image 204: Second Depth Image 25: Infrared sensing module 252: First Infrared Image 254: Second Infrared Image 30: Image Sensing Module 302: First Sensing Image 304: Second Sensing Image 90: Environment 92: Obstacles 922: Obstacle Images 924: Obstacle Information 926: Location Message A1: First Detection Area A2: Second Detection Area D: Relative distance DM1: First region DM2: Second region DTH: Distance threshold value HTH: Road sill value IMG1: First Image IMG2: Second Image IMA1: First Image Region IMA2: Second Image Region M1: First Warning Message M2: Second Warning Message OBV1: Movement Vector OBV2: Acceleration Vector P0: Image point P1: First image point P2: Second image point RTH: Road Threshold Value SPD: Movement Speed V: Vehicle x1: First X-axis x2: Second X-axis XM: Brake Warning Message y1: First X-axis y2: Second X-axis S10-S28: Steps

Claims

1. A vehicle perception fusion method, applied to a vehicle moving at a speed, the vehicle being equipped with a host, an infrared sensing module, a depth sensing module, and an image sensing module, the host being electrically connected to the infrared sensing module, the depth sensing module, and the image sensing module, the vehicle perception fusion method comprising the steps of: acquiring a first infrared image, a first depth image, and a first sensing image respectively from one side of the vehicle using the infrared sensing module, the depth sensing module, and the image sensing module, and acquiring a second infrared image, a second depth image, and a second sensing image respectively at different times, and transmitting them respectively to the host; executing a fusion algorithm using the host to obtain a first image based on the first infrared image, the first depth image, and the first sensing image, and to obtain a second image based on the second infrared image, the second depth image, and the second sensing image, wherein... Both the first image and the second image include a first image region and a second image region. The first image region corresponds to a first detection region, and the second image region corresponds to a second detection region. The first detection region is located outside the vehicle, and the second detection region is located outside the first detection region. The host computer performs an image optical flow method to perform a difference operation based on the first image and the second image to obtain obstacle information. The host computer obtains the relative distance between the obstacle and the vehicle based on the obstacle information, which is used to determine whether the obstacle is located in the first detection region or the second detection region. When the host computer determines that the obstacle is located in the first detection region based on the relative distance being less than or equal to a distance threshold, the host computer obtains the obstacle's movement vector, the obstacle's size, and the obstacle's acceleration vector based on the obstacle information. When the host determines that the obstacle is sufficient to threaten the vehicle based on the size being greater than or equal to a size threshold, the host generates a first warning message based on the vehicle's moving speed, the obstacle's moving vector, the size, and the acceleration vector, corresponding to the obstacle being located in the first detection area.

2. The vehicle perception fusion method as described in claim 1 further includes: when the host determines that the obstacle is located in the second detection area based on the relative distance being greater than the distance threshold, the host obtains the movement vector of the obstacle based on the obstacle information; and the host generates a second warning message based on the vehicle's movement speed and the obstacle's movement vector, corresponding to the obstacle being located in the second detection area.

3. The vehicle perception fusion method as described in claim 1, wherein in the step of using the host to perform an image optical flow method to perform a difference operation based on the first image and the second image to obtain obstacle information of an obstacle, the host performs the difference operation based on the first image and the second image to obtain an obstacle image, and obtains the obstacle information based on the obstacle image.

4. The vehicle perception fusion method as described in claim 1, wherein in the step of generating a first warning message corresponding to the obstacle using the host based on the vehicle's moving speed, the obstacle's movement vector, its size, and its acceleration vector, the movement vector corresponds to the obstacle's relative speed to the vehicle and the obstacle's moving direction, and when the moving speed is less than the relative speed, the host further controls the vehicle to avoid the obstacle.

5. The vehicle perception fusion method as described in claim 1, wherein in the step of generating a first warning message corresponding to an obstacle by the host based on the vehicle's moving speed, the obstacle's moving vector, the size, and the acceleration vector when the host determines that the relative distance is less than or equal to the distance threshold and the size is greater than a size threshold, the host further obtains a location message of the obstacle based on the obstacle information and determines whether the obstacle is located on a recessed road surface based on a road surface threshold and the location message; when the location message is less than the road surface threshold, the host further determines that the obstacle is located on the recessed road surface.

6. The vehicle perception fusion method as described in claim 1, wherein in the step of generating a first warning message corresponding to an obstacle based on the vehicle's moving speed, the obstacle's moving vector, the size, and the acceleration vector when the host determines that the size is greater than a size threshold, the distance threshold is 10 meters to 20 meters, and the size threshold is 1 square centimeter to 10 square centimeters.

7. The vehicle perception fusion method as described in claim 1, wherein in the step of using the host to execute a fusion algorithm to obtain a first image based on the first infrared image, the first depth image and the first sensing image, and to obtain a second image based on the second infrared image, the second depth image and the second sensing image, the host performs preprocessing on the first infrared image, the first depth image and the first sensing image and the second infrared image, the second depth image and the second sensing image in batches, thereby highlighting an obstacle image corresponding to the obstacle on the first image and the second image, and performing image stitching and color grayscale correction on the first image and the second image respectively.

8. A vehicle perception fusion system applied to a vehicle moving at a speed, the vehicle perception fusion system comprising: an infrared sensing module disposed on one side of the vehicle, the infrared sensing module capturing a first infrared image and a second infrared image based on one side of the vehicle; a depth sensing module disposed on the same side of the vehicle and adjacent to the infrared sensing module, the depth sensing module capturing a first depth image and a second depth image based on the same side of the vehicle; and an image sensing module disposed on the same side of the vehicle and adjacent to the depth sensing module and the infrared sensing module, the image sensing module capturing a first sensing image and a second sensing image based on the same side of the vehicle; A host computer, disposed within the vehicle and electrically connected to the infrared sensing module, the depth sensing module, and the image sensing module, receives the first infrared image, the first depth image, and the first sensed image, as well as the second infrared image, the second depth image, and the second sensed image. The host computer executes a fusion algorithm to generate a first image based on the first infrared image, the first depth image, and the first sensed image. The host computer generates a second image based on the second infrared image, the second depth image, and the second sensed image. The host computer performs an image optical flow method to perform a difference operation on the first image and the second image to obtain obstacle information. The first image and the second image each include a first image region and a second image region. The first image region corresponds to a first detection region, and the second image region corresponds to a second detection region. The first detection region is located outside the vehicle, and the second detection region is located outside the first detection region. The host obtains the relative distance between the obstacle and the vehicle based on the obstacle information. When the host determines that the obstacle is located in the first detection area based on the relative distance being less than or equal to a distance threshold, the host obtains the movement vector, size, and acceleration vector of the obstacle based on the obstacle information. When the host determines that the size is greater than a size threshold and that the obstacle is sufficient to threaten the vehicle, the host generates a first warning message corresponding to the obstacle being located in the first detection area based on the vehicle's movement speed, the obstacle's movement vector, size, and acceleration vector.

9. The vehicle perception fusion system as described in claim 8, wherein, When the host obtains the relative distance between the obstacle and the vehicle based on the obstacle information and the relative distance is greater than the distance threshold, and determines that the obstacle is located in the second detection area, the host obtains the movement vector of the obstacle based on the obstacle information, and the host generates a second warning message corresponding to the obstacle being located in the second detection area based on the vehicle's movement speed and the obstacle's movement vector.

10. The vehicle perception fusion system as described in claim 8, wherein the host performs the difference operation based on the first image and the second image to obtain an obstacle image, and obtains obstacle information based on the obstacle image.

11. The vehicle perception fusion system as described in claim 8, wherein the distance threshold is 10 meters to 20 meters and the size threshold is 1 square centimeter to 10 square centimeters.

12. The vehicle perception fusion system as described in claim 8, wherein, The movement vector corresponds to the relative velocity of the obstacle with respect to one of the vehicles and the direction of movement of the obstacle. When the movement velocity is less than the relative velocity, the host further controls the vehicle to avoid the obstacle.

13. The vehicle perception fusion system as described in claim 8, wherein, The host further obtains the location information of the obstacle based on the obstacle information and determines whether the obstacle is located on a recessed road surface based on a road surface threshold value and the location information. When the location information is less than the road surface threshold value, the host further determines that the obstacle is located on the recessed road surface.

14. The vehicle perception fusion system as described in claim 8, wherein the host performs preprocessing on the first infrared image, the first depth image and the first sensing image, as well as the second infrared image, the second depth image and the second sensing image in batches, thereby highlighting one of the obstacle images corresponding to the obstacle on the first image and the second image, and performing image stitching and color grayscale correction on the first image and the second image respectively.

Images