Mobile vehicle sensor fusion system and method thereof

Abstract

This invention proposes a mobile vehicle perception fusion system and method. When a vehicle is moving at a certain speed, a light scanning unit and an image capturing unit first acquire a first scan image of a first detection area and a second detection area of ​​the vehicle, as well as a first environmental image, and transmit them to a host computer to obtain a first image. An obstacle image is then obtained from the first image to obtain obstacle information. The relative distance between the vehicle and the obstacle is then obtained to determine whether the obstacle is located in the first detection area close to the vehicle or in the second detection area outside the vehicle. By determining whether the obstacle is located in the first detection area, a corresponding first auxiliary message is generated, thereby allowing the driver to avoid ignoring danger based on the first auxiliary message.

Description

Technical Field

[0001] This invention relates to a vehicle sensing system and method, and more particularly to a mobile vehicle sensing fusion system and method. Prior Technology

[0002] Traditional Advanced Driver Assistance Systems (ADAS) are developed to assist drivers and can be basically divided into three main parts: onboard sensors, onboard processors, and actuators. ADAS uses onboard sensors to detect signals outside the vehicle, including not only millimeter-wave radar and LiDAR, but also thermal and pressure sensors. This sensor data is transmitted back to the onboard processor, such as the Electronic Control Unit (ECU), which generates warning information sufficient for the driver to recognize, thus avoiding dangerous road conditions. Even when the onboard sensors cannot react in time, they can directly intervene in the driver's driving behavior and activate actuators to achieve functions such as vehicle deceleration, emergency braking, or vehicle swerving, protecting the driver.

[0003] Furthermore, to protect drivers, manufacturers have developed radar detection technology to detect obstacles around vehicles. However, radar cannot identify whether obstacles around a vehicle are fixed or moving. Whenever an obstacle is detected approaching the vehicle, the warning unit will still frequently issue warning messages, causing confusion for drivers. Although numerous improvements have been made to the detection of obstacles around moving vehicles to achieve obstacle monitoring, drivers who ignore any obstacle while the vehicle is in motion will cause accidents, especially in urban streets. Examples include streetlights, vehicles overtaking, pedestrians crossing the street, traffic islands, corner signs or traffic lights, and street signs. Ignoring these can easily lead to accidents.

[0004] While manufacturers have developed color image capture technologies such as dashcams to record the situation in case of accidents for post-accident assessment, this recording method is not the solution. The real solution lies in enabling drivers to prevent accidents from happening in the first place. Currently, vehicle radar is only installed on the front and rear sides of the vehicle. Newer vehicles will further integrate side imaging equipment and detection technology to help drivers avoid emergencies in side blind spots. Furthermore, it is necessary to anticipate dangers and notify the driver based on side blind spot detection to protect the driver.

[0005] However, drivers need reaction time while the vehicle is in motion and need to pay attention to obstacles. Especially with the widespread application of modern autonomous driving assistance technology, it is no longer just about intervening in the driver's driving behavior to protect the driver, but also about quickly anticipating the dangers around the vehicle.

[0006] To address the aforementioned problems, this invention provides a mobile vehicle perception fusion system and method. It acquires a first scanned image and a first environmental image from one side of the vehicle to obtain a corresponding first image, and further extracts obstacle images from these images to obtain obstacle information. This allows for further determination that the obstacle is located either near a first detection area of ​​the vehicle or outside a second detection area. When the obstacle is determined to be located within the first detection area based on a distance threshold and the relative distance of the obstacle information, corresponding first auxiliary information is generated based on the obstacle information to prevent the driver from encountering unexpected situations. Summary of the Invention

[0007] One objective of this invention is to provide a mobile vehicle perception fusion system and method. This system acquires a first scanned image and a first environmental image from one side of the vehicle, and fuses them into a first image. From this first image, obstacle images are extracted, and obstacle information is obtained. The relative distance between the vehicle and the obstacle is then determined from the obstacle information. Based on a distance threshold, it is determined whether the obstacle is located in a first detection area close to the vehicle or in a second detection area outside the first detection area. When the obstacle is determined to be in the first detection area, corresponding first auxiliary information is generated based on the obstacle information to help the driver avoid unexpected situations.

[0008] To achieve the above objectives, this invention discloses a mobile vehicle perception fusion method applied to a vehicle moving at a certain speed. The vehicle is equipped with a host computer, an optical scanning unit, and an image capturing unit. The host computer is electrically connected to the optical scanning unit and the image capturing unit. The mobile vehicle perception fusion method of this invention first uses the optical scanning unit to capture a first scan image based on a first detection area and a second detection area of ​​the vehicle. Then, the image capturing unit uses the first detection area and the second detection area on one side of the vehicle to obtain a first environmental image. The optical scanning unit and the image capturing unit transmit the first scan image and the first environmental image together to the host computer. The first detection area is located between the vehicle and the second detection area. Next, the host computer executes a fusion algorithm to obtain a first image based on the first scan image and the first environmental image through fusion calculation. The first image includes a first image region and a second image region. The image area corresponds to the first detection area, and the second image area corresponds to the second detection area. Next, the host computer executes an image optical flow method to obtain an obstacle image based on the first image, and obtains obstacle information of an obstacle based on the obstacle image. Then, the host computer obtains a movement vector and an acceleration vector of the obstacle, the relative distance between the obstacle and the vehicle based on the obstacle information, and determines whether the obstacle is located in the first detection area or the second detection area of ​​the vehicle based on the relative distance and a distance threshold value. The movement vector corresponds to the relative speed of the obstacle relative to the vehicle and the direction of movement of the obstacle. Further, when the relative distance is less than or equal to the distance threshold value and the obstacle is located in the first image area based on the obstacle image, the host computer generates a first warning message corresponding to the obstacle based on the vehicle's movement speed, the obstacle's movement vector and acceleration vector. Therefore, the present invention provides a hazard prediction for one side of the vehicle during travel and generates corresponding auxiliary information, which can be used to allow the driver assistance system to intervene in driving control based on the auxiliary information and notify the driver at the same time, or to warn the driver to be aware of obstacles in advance to avoid accidents.

[0009] The present invention provides an embodiment in which, in the step of generating a first warning message corresponding to an obstacle based on the moving speed of the vehicle, the moving vector of the obstacle, and the acceleration vector of the obstacle, when the moving speed is less than the relative speed, the host further generates a braking warning message to remind the driver to brake immediately to avoid an accident. In addition, when the moving speed is greater than the relative speed, the host further drives the vehicle to brake, that is, if the driver cannot react in time to brake, the vehicle brakes immediately to avoid the vehicle from hitting the obstacle.

[0010] The present invention provides an embodiment in which, in the steps of using the host to obtain the relative distance between the obstacle and one of the vehicles based on the obstacle information and determining that the obstacle is located in a first detection area or a second detection area of ​​one of the vehicles based on the moving speed, the relative distance and a distance threshold value, the host extracts a positioning message corresponding to the obstacle based on the obstacle information, which is used to obtain the relative distance between the obstacle and the vehicle.

[0011] The present invention provides an embodiment in which, in the step of using the host to obtain the relative distance between the obstacle and one of the vehicles based on the obstacle information and determining whether the obstacle is located in a first detection area or a second detection area of ​​one of the vehicles based on the relative distance and a distance threshold value, the distance threshold value is 5 meters to 10 meters, that is, thereby distinguishing the first detection area and the second detection area.

[0012] The present invention provides an embodiment in which, when the relative distance is greater than the distance threshold and the obstacle is determined to be located in the second detection area based on the obstacle image being located in the second image area, the host computer generates a second warning message corresponding to the obstacle based on the moving speed of the vehicle and the moving vector of the obstacle. The warning level of the second warning message is lower than that of the first warning message, indicating a lower level of warning. Furthermore, the host computer consumes less computational resources due to the fewer parameters it processes.

[0013] The present invention further provides a mobile vehicle perception fusion system, comprising a vehicle moving at a certain speed, the mobile vehicle perception fusion system including a host, an optical scanning unit, and an image capturing unit, wherein the host is disposed within the vehicle, and the optical scanning unit and the image capturing unit are disposed on one side of the vehicle and electrically connected to the host. The optical scanning unit and the image capturing unit capture a first scan image and a first environmental image based on a first detection area and a second detection area of ​​the vehicle and transmit them to the host, wherein the first detection area is located between the vehicle and the second detection area; the host executes a fusion algorithm and obtains a first image based on the first scan image and the first environmental image, the first image including a first image area and a second image area, the first image area corresponding to the first detection area, and the second image area corresponding to the second detection area; the host executes an image optical flow method to obtain an obstacle image based on the first image, and obtains obstacle information based on the obstacle image, utilizing the... The host obtains the movement vector and acceleration vector of one of the obstacles, the relative distance between the obstacle and one of the vehicles based on the obstacle information, and determines whether the obstacle is located in the first detection area or the second detection area based on the relative distance and a distance threshold value. When the relative distance is less than or equal to the distance threshold value and the obstacle image is located in the first image area, the host generates a first warning message corresponding to one of the obstacles based on the vehicle's movement speed, the obstacle's movement vector and acceleration vector. Therefore, the host predicts whether the obstacle will affect the vehicle's movement direction based on the first warning message, which can be used to notify the driver assistance system to intervene or to notify the driver.

[0014] The present invention provides another embodiment in which the distance threshold is 5 to 10 meters.

[0015] The present invention provides another embodiment in which, when the moving speed is less than the relative speed, the host further generates a braking warning message to remind the driver to brake immediately in order to avoid an accident. In addition, when the moving speed is greater than the relative speed, the host further drives the vehicle to brake, that is, if the driver does not have time to react and brake, the vehicle brakes immediately to avoid the vehicle from hitting the obstacle.

[0016] The present invention provides another embodiment in which the host extracts a positioning message corresponding to the obstacle based on the obstacle information, which is used to obtain the relative distance between the obstacle and the vehicle.

[0017] The present invention provides an embodiment in which, when the relative distance is greater than the distance threshold and the obstacle is determined to be located in the second detection area based on the obstacle image being located in the second image area, the host generates a second warning message corresponding to the obstacle based on the moving speed of the vehicle and the moving vector of the obstacle, wherein the warning level of the second warning message is lower than that of the first warning message. Simple Explanation of the Diagram

[0018] Figure 1: It is a flowchart of one embodiment of the present invention; Figures 2A to 2G: These are schematic diagrams of a mobile vehicle perception fusion system according to one embodiment of the present invention; Figure 3A: It is a schematic diagram of the detection area according to one embodiment of the present invention; Figure 3B: It is a schematic diagram of a perspective projection method according to one embodiment of the present invention; Figure 4A: This is a schematic diagram of an obstacle located in the first detection area according to one embodiment of the present invention; and Figure 4B: It is a schematic diagram of an obstacle located in the second detection area according to one embodiment of the present invention. Implementation

[0019] To enable your review committee to have a better understanding of the features and effects of this invention, the following examples and accompanying descriptions are provided:

[0020] In view of the fact that conventional radar systems and dashcams fail to provide obstacle prediction for vehicles, the present invention proposes a mobile vehicle perception fusion system and method to solve the problem that conventional technology makes it difficult for drivers to avoid obstacles.

[0021] The following will further explain the characteristics and associated systems provided by the mobile vehicle perception fusion system and method disclosed in this invention:

[0022] First, please refer to Figure 1, which is a flowchart of one embodiment of the present invention. As shown in the figure, the mobile vehicle perception fusion method of the present invention includes the following steps:

[0023] Step S10: Using the optical scanning unit and the image capturing unit, a first scan image and a first environmental image are obtained respectively based on the first detection area and the second detection area on one side of the vehicle, and then transmitted to the host computer;

[0024] Step S12: Execute a fusion algorithm using the host computer to obtain a first image based on the first scan image and the first environmental image;

[0025] Step S14: Utilize the host computer to execute the image optical flow method to obtain obstacle images based on the first image, and obtain obstacle information based on the obstacle images;

[0026] Step S16: Using the host computer, obtain the movement vector and acceleration vector of the obstacle, and the relative distance between the obstacle and the vehicle based on obstacle information;

[0027] Step S18: Using the host computer, determine whether the obstacle is located in the vehicle's first or second detection area based on the distance threshold and the relative distance between the obstacle and the vehicle; and

[0028] Step S20: The host computer generates a first warning message for the corresponding obstacle based on the vehicle's moving speed, the obstacle's moving vector, and the acceleration vector.

[0029] Please refer to Figures 2A to 2G, which illustrate the mobile vehicle perception fusion system 1 used in conjunction with the mobile vehicle perception fusion method of the present invention. The system includes a host computer 10, an optical scanning unit 20, an infrared image capturing unit 25, and an image capturing unit 30. In this embodiment, the host computer 10 is exemplified as a vehicle computer with a processing unit 12, but it is not limited to this. It can also be a server, laptop computer, tablet computer, or any electronic device with image processing capabilities. The processing unit 12 can be a system-on-a-chip (SoC), a microprocessor (μP), a microcontroller (MCU), a programmable logic controller (PLC), a central processing unit (CPU), or a graphics processing unit (GPU). In this embodiment, the optical scanning unit 20 is a LiDAR device or a laser scanner. A laser scanner can achieve the same effect as LiDAR through multiple laser scans. In this embodiment, the infrared image capturing unit 25 is a near-infrared (NIR) image sensor, a short-wave infrared image sensor, or a far-infrared (LWIR) image sensor; the image capturing unit 30 in this embodiment is a general visible light image capturing unit, such as an automotive CMOS image sensor; wherein the host 10 is disposed inside a vehicle V, the light scanning unit 20 and the image capturing unit 30 are disposed on one side of the vehicle V, the host 10 is electrically connected to the light scanning unit 20 and the image capturing unit 30, and the image capturing angle range of the image capturing unit 30 in this embodiment is 120 to 180 degrees, and it captures environmental images within a range of 10 to 30 meters around the vehicle V, such as object images, as detailed in the following description.

[0030] In step S10, as shown in Figure 2A, the present invention utilizes the optical scanning unit 20 to perform optical scanning on the environment 90 on one side of the vehicle V, or even within 10 to 50 meters around the vehicle V, to generate a first optical scanning image 202 based on the scanning results. Furthermore, the infrared image capturing unit 25 and the image capturing unit 30 capture a first infrared image 252 and a first environmental image 302 on the environment 90 on one side of the vehicle V. As shown in Figure 3A, the optical scanning unit 20, the infrared image capturing unit 25, and the image capturing unit 30 obtain the corresponding first optical scanning image 202, first infrared image 252, and first environmental image 302 based on a first detection area A1 and a second detection area A2 corresponding to the vehicle V. The first detection area A1 is located between the vehicle V and the second detection area A2, meaning the first detection area A1 is closer to the vehicle V, while the second detection area A2 is located outside the first detection area A1. For example: The first detection area A1 is the inner circle area close to the vehicle V, and the second detection area A2 is the outer circle area adjacent to the first detection area A1.

[0031] Furthermore, as shown in Figure 2A, the aforementioned first detection area A1 and second detection area A2 cover blind spot locations, which correspond to one side of the vehicle V and conform to the blind spot area specified in the ISO 17387 standard for intelligent transportation system certification. The light scanning unit 20 is specifically designed for visual blind spot locations that the vehicle V cannot visually reach, that is, blind spot locations outside the driver's visual field. Even if the vehicle V has left and right rearview mirrors, the auxiliary light scanning unit 20, infrared image capturing unit 25, and image capturing device 30 are still needed to capture images that cannot visually reach. Moreover, the Advanced Driver Assistance System (ADAS) also requires more sophisticated image capturing to more accurately identify whether there are obstacles on one side of the vehicle V, such as people, vehicles, bus stop signs, traffic signs, or traffic signals, or even any obstacles in the visual location where blind spots frequently occur, such as the A-pillar inside the vehicle.

[0032] As shown in Figure 3B, using perspective projection, the infrared image capturing unit 25 and the image capturing unit 30 divide the image point P0 of the environmental image into a first image point P1 and a second image point P2. The coordinates (x, y) of the first image point P1 are located in the first surface region DM1, and the coordinates (x′, y′) of the second image point P2 are located in the second surface region DM2. Therefore, the relative relationship between the infrared image capturing unit 25 and the image capturing unit 30 in capturing the first image point P1 and the second image point P2 is as follows: Formula (1) Formula (II)

[0033] Where (x, y) is the first image point P1, (x', y') is the second image point P2; m0, m1, ..., m7 are the relevant focal length, rotation angle, and scaling parameters of the infrared image capturing unit 25 and the image capturing unit 30. These parameters can be expanded into a complex array of image point pairs, and then the optimal values ​​of m1 to m7 can be obtained by nonlinear minimization using the Levenberg-Marquardt algorithm, which serve as the optimal capturing focal length of the image capturing unit 30, for example, from 10mm to 100mm.

[0034] Referring again to Figures 1 and 2A, in step S12, the host 10 executes a program P using the processing unit 12 and memory 14 to perform a fusion algorithm P1. This algorithm receives the first optical scan image 202 generated by the optical scanning unit 20, the first infrared image 252 generated by the infrared image capturing unit 25, and the first environmental image 302 generated by the image capturing unit 30, and performs image fusion processing to generate a first image IMG1. Since this embodiment uses an environment 90 that includes an obstacle OB, the first image IMG1 will contain an obstacle image OBI. The processing unit 12, by executing the program P, performs preprocessing on the first optical scan image 202, the first infrared image 252, and the first environmental image 302, thus highlighting the obstacle image OBI corresponding to the obstacle OB on the first image IMG1. Furthermore, it performs image stitching and color grayscale correction on the first image IMG1 to provide subsequent spatial recognition.

[0035] The fusion algorithm P1 first introduces the feature function f(x,y) as shown in equation (III). f(x,y) is a binary function, which means that when x and y satisfy a certain fact, its feature function value is 1. Formula (3)

[0036] In real-world numerical computation environments, the hidden state corresponding to a given observation is determined by the context (observation, state). Introducing feature functions allows us to freely select features (combinations of observations or states). Essentially, features (combinations of observations) replace observations, avoiding the limitations of the observation independence assumption in generative models (e.g., Hidden Markov Models (HMMs), naive Bayes).

[0037] We can obtain an empirical expectation and a model expectation based on training data D={(x,y)} of size T. Formula (IV) Formula (5)

[0038] We assume that the empirical expectation is equal to the model expectation. Then there exists a set C of conditional probability distributions for any feature function fi that satisfy this constraint, and thus: Formula (VI)

[0039] In step S14, as shown in Figure 2B, the host 10 executes an image optical flow method L through the processing unit 12 to obtain the obstacle image OBI based on the first image IMG1, and obtains the obstacle information INFO of the obstacle OB based on the obstacle image. The obstacle information INFO includes a movement vector OBV1 and an acceleration vector OBV2 of the obstacle OB, and a relative distance R between the obstacle OB and the vehicle V. Therefore, in step S16, as shown in Figure 2C, the host 10 obtains the movement vector OBV1 and acceleration vector OBV2 of the obstacle OB, and the relative distance R between the obstacle OB and the vehicle V based on the obstacle information INFO through the processing unit 12. The movement vector OBV1 corresponds to the relative velocity of obstacle OB relative to vehicle V and the direction of movement of obstacle OB. The first image IMG1 is point cloud image data, and the obstacle image OBI is also point cloud image data. Therefore, the image processing in this embodiment is based on point cloud image processing technology. Calculating the movement vector OBV is equivalent to calculating the relative velocity of obstacle OB relative to vehicle V and the direction of movement of obstacle OB. Furthermore, the host 10 extracts the positioning information 122 of the corresponding obstacle OB through the processing unit 12 based on the obstacle information INFO. This information is used to obtain the relative distance R between obstacle OB and vehicle V. In particular, the relative distance R between obstacle OB and vehicle V is obtained by combining the vehicle V's movement speed SPD with the positioning information 122.

[0040] In step S18, as shown in Figure 2D, the host 10 uses the processing unit 12 to determine, based on a distance threshold value TH and the relative distance R between the obstacle OB and the vehicle V, whether the obstacle image OB is located in the first image region IMA1 or the second image region IMA2 of the first image IMG1. Therefore, it is determined that the obstacle OB is located in the first detection region A1 or the second detection region A2 of the vehicle V. The vector corresponding to the relative distance R can be close to 0 relative to the distance threshold value TH, indicating that the obstacle OB is close to the distance threshold value TH. The distance threshold value TH is preset in the calculation program P and can be 5 meters to 50 meters. In particular, the distance threshold value TH can be 5 meters to 10 meters.

[0041] As shown in Figures 2E and 4A, when the relative distance R is less than the distance threshold TH, and the host 10 determines that the obstacle OB is located in the first detection area A1 based on the obstacle image OBI being located in the first image area IMA1, in step S20, the host 10 uses the processing unit 12 to generate a first warning message M1 corresponding to the obstacle OB based on the vehicle V's moving speed SPD, the obstacle OB's movement vector OBV1, and the acceleration vector OBV2. In addition to generating the first warning message M1, the host 10 can further consider the vehicle V's moving speed SPD and the relative speed of the obstacle OB to remind the driver. The movement vector OBV1 corresponds to the relative speed of the obstacle OB relative to the vehicle and the obstacle OB's direction of movement. When the vehicle V's moving speed SPD is less than the obstacle OB's relative speed, the host 10 uses the processing unit 12 to further generate a braking warning message XM. For example, it may remind the driver to pay attention to side obstacles or side oncoming vehicles, or even combine the braking warning message XM to remind the driver to brake.

[0042] Additionally, as shown in Figures 2F and 4A, when the moving speed SPD of vehicle V is greater than the relative speed of obstacle OB, the driver inside vehicle V is further judged by the host 10 through the processing unit 12 to be unable to react in time under this state. In order to avoid an unavoidable emergency accident involving vehicle V and its driver, the host 10 further generates a braking control message SM through the processing unit 12, thereby controlling the vehicle V to brake. However, the host 10 still generates the first warning message M1 corresponding to obstacle OB through the processing unit 12.

[0043] In addition, the host 10 can also perform the following steps through the computing unit 12.

[0044] Referring again to Figure 1, the mobile vehicle perception fusion method of the present invention further includes:

[0045] Step S22: The host computer generates a second warning message for the corresponding obstacle based on the vehicle's moving speed and the obstacle's moving vector.

[0046] When the relative distance R is greater than the distance threshold TH and the host 10 determines that the obstacle OB is located in the second detection area A2 based on the obstacle image OBI being located in the second image area IMA2, in step S22, as shown in Figures 2G and 4B, the host 10 uses the computing unit 12 to generate a second warning message M2 corresponding to the obstacle OB based on the vehicle V's moving speed SPD and the obstacle OB's moving vector OBV1. The warning level of the second warning message M2 is lower than that of the first warning message M1.

[0047] The Sobel edge detection algorithm used in the image processing performed by program P is as follows:

[0048] Sobel edge detection:

[0049] In the image, each pixel and its neighboring pixels are represented in a matrix (Pixel) using a nine-grid layout, labeled P1, P2, P3, P4…P9, as shown in equation (VII). Formula (VII) Formula (8) Formula (9) Formula (10) Formula (XI) Formula (12) Formula (13)

[0050] R is the turning radius of vehicle V, L is the wheelbase, d1 is the front wheel spacing, d2 is the rear wheel spacing, α is the angle between the midpoint of the front and rear axles of vehicle V and the center of the turning circle, a is the radius of motion of the inner rear wheel centerline, b is the radius of motion of the inner front wheel centerline, and m is the inner wheel difference of the non-trailer vehicle.

[0051] The aforementioned image optical flow method L utilizes the Lucas–Kanade Optical Flow algorithm for obstacle estimation. It first employs image differencing to derive the image constraint equations using the Taylor formula. Formula (XIV)

[0052] Where HOT represents a higher-order equation, which can be ignored when the shift is sufficiently small. From this equation, we can obtain: Formula (15) or Formula (16) And obtain: Formula (17)

[0053] Vx, Vy, and Vz are the components of x, y, and z in the optical flow vector of I(x,y,z,t). , , and This is the difference of the image at point (x,y,z,t) in the corresponding direction, so equation (xVII) is transformed into the following equation. I xV x+I yV y+I zV z= -I t Formula (18)

[0054] Equation (18) can be further written as follows: Formula (19)

[0055] Since there are three unknowns (Vx, Vy, Vz) in equation (18), the unknowns are calculated by the continuation algorithm:

[0056] First, assume that the flow (Vx, Vy, Vz) is a constant within a small window of size m*m*m (m>1). Then, from primitives 1...n, n = m3, we can obtain the following set of equations: Formula (20)

[0057] The above equations all contain three unknowns, forming a system of equations. Furthermore, this system is overdetermined, meaning it contains redundancy. The system can be represented as: Formula (21)

[0058] Notation: Formula (22)

[0059] To solve this overdetermined problem, equation (xvii) is obtained using the least squares method: or (Form 23) Formula (24)

[0060] get: Formula (25)

[0061] Substituting the result of equation (25) into equation (17) allows us to estimate the acceleration vector of one of the target objects and the relative distance between the target object and one of the vehicles. This is used to classify and predict the movement of the target objects.

[0062] The maximum entropy principle states that the only reasonable probability distribution derived from incomplete information (such as a finite amount of training data) should have the maximum entropy value under the constraints provided by this information—that is, the distribution with the maximum entropy is optimal in the set of conditional probabilities. Therefore, the maximum entropy model becomes a constrained optimization problem of convex functions. Formula (26) Formula (27) Formula (28)

[0063] The Lagrange duality principle is typically used to transform the original expression into an unconstrained extremum solution: Formula (29) Formula (30)

[0064] Taking the partial derivative of the Lagrange function with respect to p and setting it equal to 0, solving the equation, and omitting N steps of integer transformation, yields the following expression: Formula (31) Formula (32)

[0065] Maximum Entropy Markov Model (MEMM) Formula (33)

[0066] use The distribution replaces the two conditional probability distributions in the HMM, representing the probability of obtaining the current state from the previous state given the observations; that is, predicting the current state based on the previous state and the current observations. Each such distribution function... Both are exponential models that follow maximum entropy.

[0067] Suppose we find a point on the discrete probability distribution The probability distribution of the minimum upper point to be found, along with the maximum information entropy. The formula for maximum entropy: Formula (34)

[0068] This is the sum of probabilities from a probability distribution. At each point Must equal 1: Formula (35)

[0069] We use Lagrange multipliers to find the angle of maximum entropy. Spanning all discrete probability distributions superior We require: Formula (36)

[0070] It gives a system equation. , so that: Formula (37)

[0071] By performing these differentiation equations, we obtain Formula (38)

[0072] This indicates that all They are equal (because they depend only on λ). This is achieved through the use of constraints. Formula (39)

[0073] Therefore, we obtain Formula (40)

[0074] Therefore, a uniform distribution is a distribution with maximum entropy, and there is no difference between the upper distributions. Formula (41)

[0075] In summary, the mobile vehicle perception fusion system and method of the present invention provides a host computer that acquires object images of a plurality of obstacles on one side of the vehicle, and classifies them by the relative distance between the target objects and one side of the vehicle. The host computer then performs predictive calculations on the obstacles corresponding to the selected images to obtain a predicted movement path. This predicted path is then calculated against the vehicle's actual travel route data to generate a warning message. Furthermore, the host computer can further adjust the travel data based on the obstacles to prevent dangerous situations from occurring.

[0076] Therefore, this invention is indeed novel, inventive, and industrially applicable, and undoubtedly meets the requirements for patent application under the Patent Law of our country. Thus, we hereby file an invention patent application in accordance with the law, and earnestly pray that the Bureau will grant the patent as soon as possible.

[0077] However, the above description is only a preferred embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes and modifications made to the shape, structure, features and spirit described in the claims of the present invention should be included in the scope of the claims of the present invention.

[0078] 1: Mobile Vehicle Perception Fusion System 10: Host 12: Processing Unit 122: Location Message 20: Optical Scanning Unit 202: First light scan image 25: Infrared Image Acquisition Unit 252: First Infrared Image 30: Image Capture Unit 302: First Environmental Image 90: Environment A1: First Detection Area A2: Second Detection Area DM1: First region DM2: Second region IMG1: First Image IMA1: First Image Region IMA2: Second Image Region L: Image Optical Flow Method M1: First Warning Message M2: Second Warning Message OB: Obstacle OBI: Obstacle Imaging OBV1: Movement Vector OBV2: Acceleration Vector P: Calculation program P1: Fusion Algorithm P 0: Image point P1: First image point P2: Second image point R: Relative distance SPD: Movement Speed TH: Distance threshold value V: Vehicle x 1: First X-axis x 2: Second X-axis XM: Brake Warning Message y1: First X-axis y2: Second X-axis S10-S22: Steps

Claims

1. A mobile vehicle perception fusion method, applied to a vehicle moving at a moving speed, the vehicle being equipped with a host, an optical scanning unit, an infrared image capturing unit, and an environmental image capturing unit, the host being electrically connected to the optical scanning unit, the infrared image capturing unit, and the image capturing unit, the mobile vehicle perception fusion method comprising the steps of: using the optical scanning unit, the infrared image capturing unit, and the image capturing unit to acquire a first scan image, a first infrared image, and a first environmental image respectively based on a first detection area and a second detection area of ​​the vehicle, the first scan image and the first environmental image being transmitted together to the host, the first detection area being located between the vehicle and the second detection area; The host computer executes a fusion algorithm to obtain a first image based on the first scan image, the first infrared image, and the first environmental image. The first image includes a first image region and a second image region, which exhibit different depths of field. The first image region corresponds to the first detection region, and the second image region corresponds to the second detection region. The host computer executes an image optical flow method to obtain an obstacle image based on the first image and obtain obstacle information based on the obstacle image. The host computer obtains a movement vector of the obstacle, an acceleration vector of the obstacle, and a relative distance between the obstacle and the vehicle based on the obstacle information. The host computer determines whether the obstacle is located in the first detection area or the second detection area of ​​the vehicle based on the relative distance and a distance threshold value; and when the relative distance is less than or equal to the distance threshold value and the obstacle image is located in the first image area, the host computer generates a first warning message corresponding to the obstacle based on the vehicle's moving speed, the obstacle's moving vector and acceleration vector.

2. The mobile vehicle perception fusion method as described in claim 1, wherein in the step of generating a first warning message corresponding to the obstacle by using the host based on the vehicle's moving speed, the obstacle's movement vector and acceleration vector, the movement vector corresponds to the obstacle's relative speed and the obstacle's direction of movement; when the moving speed is less than the relative speed, the host further generates a braking warning message; when the moving speed is greater than the relative speed, the host further drives the vehicle to brake.

3. The mobile vehicle perception fusion method as described in claim 1, wherein in the step of using the host to obtain the relative distance between the obstacle and the vehicle based on the obstacle information and determining that the obstacle is located in a first detection area or a second detection area of ​​the vehicle based on the moving speed, the relative distance and a distance threshold value, the host extracts a positioning message corresponding to the obstacle based on the obstacle information, which is used to obtain the relative distance between the obstacle and the vehicle.

4. The mobile vehicle perception fusion method as described in claim 1, wherein in the steps of obtaining the relative distance between the obstacle and one of the vehicles based on the obstacle information using the host and determining that the obstacle is located in a first detection area or a second detection area of ​​one of the vehicles based on the relative distance and a distance threshold value, the distance threshold value is 5 meters to 10 meters.

5. The mobile vehicle perception fusion method as described in claim 1 further includes: when the relative distance is greater than the distance threshold and the obstacle is located in the second detection area based on the obstacle image being located in the second image area, the host computer generates a second warning message corresponding to the obstacle based on the vehicle's moving speed and the obstacle's moving vector, wherein the warning level of the second warning message is lower than that of the first warning message.

6. A mobile vehicle perception fusion system applied to a vehicle moving at a certain speed, the mobile vehicle perception fusion system comprising: a host computer disposed within the vehicle; a light scanning unit disposed on the vehicle and electrically connected to the host computer, the light scanning unit capturing a first scan image based on a first detection area and a second detection area of ​​the vehicle and transmitting it to the host computer, the first detection area being located between the vehicle and the second detection area; and an image capturing unit disposed on the vehicle and electrically connected to the host computer, the image capturing unit being adjacent to the light scanning unit, the image capturing unit capturing a first environmental image based on the first detection area and the second detection area on one side of the vehicle and transmitting it to the host computer, the host computer executing a fusion algorithm and capturing a first image based on the first scan image and the first environmental image, the first image including a first image area and a second image area, the first image area corresponding to the first detection area, and the second image area corresponding to the second detection area; wherein... The host computer executes an image optical flow method to obtain an obstacle image based on the first image, and obtains obstacle information based on the obstacle image. The host computer uses the obstacle information to obtain the obstacle's movement vector and acceleration vector, the relative distance between the obstacle and the vehicle, and determines whether the obstacle is located in the first detection area or the second detection area based on the relative distance and a distance threshold value. When the relative distance is less than or equal to the distance threshold value and the obstacle image is located in the first image area, the host computer uses the vehicle's movement speed, the obstacle's movement vector and acceleration vector to generate a first warning message corresponding to the obstacle.

7. The mobile vehicle perception fusion system as described in claim 6, wherein the distance threshold is 5 to 10 meters.

8. The mobile vehicle perception fusion system as described in claim 6, wherein, The movement vector corresponds to the relative velocity of the obstacle with respect to one of the vehicles and the direction of movement of the obstacle. When the movement speed is less than the relative velocity, the host further generates a braking warning message. When the movement speed is greater than the relative velocity, the host further drives the vehicle to brake.

9. The mobile vehicle perception fusion system as described in claim 6, wherein, The host extracts a location message corresponding to the obstacle based on the obstacle information, which is used to obtain the relative distance between the obstacle and the vehicle.

10. The mobile vehicle perception fusion system as described in claim 6, wherein when the relative distance is greater than the distance threshold and the obstacle is determined to be located in the second detection area based on the obstacle image being located in the second image area, the host generates a second warning message corresponding to the obstacle based on the vehicle's moving speed and the obstacle's moving vector, wherein the warning level of the second warning message is lower than that of the first warning message.

Images