Passive Infrared Pedestrian Detection and Avoidance System

Abstract

A passive infrared pedestrian detection and avoidance system and method for improving vehicle operation on roadways, specifically for recognizing potential pedestrian / vehicle collision risks for a moving vehicle and adjusting the vehicle's position and operation accordingly, includes at least one passive infrared sensor array mounted on a vehicle in operative communication with an image processor connected to the vehicle's operation system. The system uses thermal imaging and thermal image processing to detect the presence of a human within or crossing the vehicle's lane of travel. The image processor analyzes the detection of the human thermal signature and identifies the direction and speed at which the detected human thermal signature is moving to assess the potential threat to the pedestrian or biker and determine whether responsive action in the vehicle's operation is necessary to avoid a collision. [Selected Figure] Figure 1

Description

Technical Field

[0001] The present invention generally relates to improving the driving of autonomous vehicles as well as manually operated vehicles, and more specifically, particularly in the interaction between a vehicle and a pedestrian, identifying situations where a moving vehicle may pose a danger to the pedestrian, and adjusting the position and driving of the vehicle accordingly to prevent dangerous interactions between the vehicle and the pedestrian, and further to a system and method for warning of an imminent collision with a pedestrian and a biker.

Background Art

[0002] Many companies are developing commercial and personal autonomous vehicles on existing roads for various applications, such as, but not limited to, personal taxi services, delivery services, etc. According to the present invention, an autonomous vehicle is a vehicle that can travel without a human driver. Such a vehicle is designed to be able to travel using an in-vehicle computer and sensor system designed to drive, steer, brake, and even operate the vehicle as if a human driver were present. Similar to the networks of taxis, buses, and delivery vans, it is assumed that many autonomous vehicles will soon be available, allowing users to request autonomous vehicles to pick up and drop off passengers, pick up, transport, and deliver goods according to their requests. Alternatively, users can own a personal autonomous vehicle and use it for daily tasks such as commuting to work, running errands, taking children to school, or using it during travel.

[0003] Current self-driving vehicles in the development and testing stages generally utilize multiple systems so that the vehicle can be fully operated even without a human driver. First, a standard GPS system is used to set the vehicle's route. The GPS system determines the optimal route for the vehicle to take, considering the starting and ending points of the journey, as well as other factors such as traffic conditions, road closures, weather conditions, priority routes, and toll roads. However, for safe and efficient driving, self-driving vehicles also require a system to recognize the dynamic state along the route while the vehicle is in motion. Such a system, called an extended GPS system, utilizes a series of technologies such as cameras, sensors, radar, lidar (LIDAR), and lasers to provide a three-dimensional view of the area around the moving vehicle. Such a system can identify other vehicles around the vehicle, detect obstacles and dangers approaching the vehicle from the surrounding area, in front, or from all directions (front, rear, left, and right), identify the position of the edge of the road or driving lane, upcoming corners, uphill or downhill slopes, and determine the general road conditions in front of, behind, and around the vehicle.

[0004] Self-driving vehicles also require a centralized processing system inside the vehicle to process the information provided by the GPS system and the extended GPS system and utilize the processed information for the vehicle's driving. Such commonly used systems generally include a computer area network (CAN) bus inside the vehicle that communicates with the GPS system, the extended GPS system, and other driving systems for self-driving vehicles to perform coordinated operations.

[0005] Non-self-driving vehicles also utilize similar technologies to assist the human driver. For example, various forms of cruise control have been used in vehicles for decades. Recently, systems that can autonomously parallel park the vehicle have been installed. Currently, many of the latest vehicles are equipped with systems that assist the driver when the vehicle starts to drift out of the lane on the highway, apply the brakes when the vehicle gets too close to the vehicle in front, or warn the driver when there is an object on the road ahead. SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

[0006] Until the guidance system installed in a self-driving vehicle matches or exceeds the perception and analysis / decision-making abilities of a human driver, there are numerous situations in daily life that frustrate the ability of a fully self-driving vehicle to respond or react appropriately and dynamically to its surrounding conditions. Furthermore, until a self-driving vehicle can rely on existing driving systems and sensors without problems for safe and efficient driving and can essentially eliminate all risks, the general public will not fully trust the true self-driving of such vehicles. In fact, in tests of "real" self-driving vehicles, it has been shown that due to the inability of existing guidance systems and sensors to detect, recognize, and / or respond appropriately and timely to situations caused by vehicles crossing lanes, pedestrians, or bikers, there have been numerous cases of guidance failures and accidents, and "incorrect" instructions have been displayed with a high probability. Even when numerous guidance systems and sensors are installed in both self-driving and manually driven vehicles, they fail to identify and avoid the interaction in the encounter with pedestrians and / or bikers for various reasons.

[0007] Various optical-based automotive and / or autonomous driving guidance systems and sensors (e.g., video, LIDAR, etc.) perform well under ideal visual conditions, but their performance can degrade to unusable levels during and immediately after rain, snow, fog, etc., or in adverse environmental conditions such as when it is dark outside or in areas with little road lighting, or when pedestrians blend into the surroundings due to clothing, skin color, weather, blinding sunlight, etc. In addition, while existing levels of "in-vehicle" sensors, cameras, devices, and interfaces can somewhat change the characteristics in the operation of autonomous vehicles (e.g., braking for unexpected obstacles and / or other vehicles, maneuvering the vehicle when it deviates from the lane, etc.), currently, there are significant unique problems in assisting autonomous vehicles to recognize pedestrian traffic, further identify when such pedestrians will perform problematical actions towards the vehicle, and automatically take actions to prevent such vehicles from performing such problematical actions. Therefore, it is necessary to improve existing optical-based guidance systems and sensors so that the driving of autonomous vehicles is safe and efficient under all conditions.

[0008] Furthermore, it has been proven that it is extremely difficult to make a vehicle drive autonomously under all conditions because it is essentially difficult to make a vehicle make real-time decisions considering situations that are different from normal and difficult to anticipate and predict. Even under ideal ambient conditions, a vehicle's CAN bus needs to interface with a significant number of sensors, and it requires a great deal of analytical processing power just for the vehicle to recognize, actually detect, confirm, and appropriately respond to the presence of objects that are predicted and / or different from normal or difficult to predict during driving. Since the main objective of the present invention is first to detect "humans," it is impossible to store all the expressions indicating the countless existing "humans" in an in-vehicle database, and thus conventional optical means often fail. As an example of this, there are a very large number of variations, namely, a human pushing themselves in a wheelchair, a wheelchair user being pushed by another person, a human seen from the side standing next to a bicycle, a human seen from the side with a backpack, etc., and a video processor that attempts to recognize "humans" from among these. Furthermore, even if a vehicle can recognize an object or a situation, it is still difficult to make the vehicle understand what an appropriate response process is. Since there are almost an infinite number of potential situations that require an appropriate response from an autonomous vehicle, considering the limitations of in-vehicle processing power and the size of the in-vehicle database and the very short actual time available for a response, it can be seen that this is again not impossible but extremely difficult. Considering that there are a very large number of unknown or difficult-to-predict situations, there is still a need for a better method of guiding an autonomous vehicle, and the first thing to do is to improve the speed, efficiency, and effectiveness with which a vehicle can detect and recognize unknown, difficult-to-anticipate and predict situations and whether guidance adjustment is necessary.

[0009] Therefore, there is a need for an improved system for the operation of an autonomous vehicle that can recognize unknown, difficult-to-anticipate and predict situations or conditions on the road and overcome the drawbacks and limitations of existing dynamic guidance systems. Further, regardless of whether the human is a pedestrian, runner, or biker, a system that can support the safe and efficient operation of a vehicle in response to unknown, difficult-to-anticipate or predict situations involving humans or situations where a human intrudes into the predicted driving path of a moving vehicle and collides with the vehicle is needed, which utilizes an infrared image sensor designed for the detection and analysis of specific "humans".

Means for Solving the Problem

[0010] The present invention generally relates to a passive infrared-based pedestrian detection and avoidance system for improving the operation of autonomous or manually driven vehicles on the road. According to an embodiment of the present invention, an excellent method is provided by this system to assist a vehicle or a driver in detecting and recognizing potential interactions between a moving vehicle, such as a vehicle traveling on a predicted path on the road, and a pedestrian or biker.

[0011] Central to the operation of the system according to the present invention is the operation communication with an image processor connected to the vehicle's driving system, such as via a central CAN bus unit that constantly communicates with various vehicle sensors, such as a forward-looking IR sensor. At least one forward-looking passive infrared (IR) image sensor attached to the vehicle is preferably used immediately in real time for analysis and output processing. During operation, the system according to the present invention immediately detects specially selected and filtered thermal wavelengths indicative of human characteristics, such as whether a human is walking, running, or cycling, already in the lane, crossing, or about to enter the lane, and further provides means to immediately respond to such intrusion to an autonomous or manually driven vehicle. The image processor searches for and analyzes the detection of thermal characteristics indicative of a human, identifies in which direction and at what speed the detected human thermal characteristics are moving, evaluates the current or future threat to the vehicle or pedestrian, and further determines whether a response action to avoid mutual interaction between the two is required to be initiated for the driving of the vehicle. This system can also identify whether a vehicle is headed towards a stationary human already in the travel lane. This information can be used, for example, to provide a "head-up" display on the windshield of a manually driven vehicle or other graphic or warning display at other locations, or as data input to the driving system and / or navigation system of an autonomous vehicle. Although preferably used in an autonomous vehicle, the system of the present invention can also be used in a human-driven vehicle as an alarm and / or automatic assistance system to assist the driver.

[0012] In one embodiment of the present invention, a method for recognizing a human who is likely to move laterally and a human who already exists in the driving lane in order to assist in driving a moving vehicle and adjusting the driving of the vehicle accordingly includes detecting thermal characteristics of a human on the side of the road or on the road using a thermal image via an infrared sensor array or the like attached to the vehicle, and identifying whether the detected thermal characteristics of the human are at a specific position or are moving in a direction where a collision accident may occur based on the directions and speeds of both the vehicle and the pedestrian. This method further includes determining whether a response operation is required for the driving of the vehicle, and performing a response operation on the driving of the vehicle when it is recognized that such an operation is necessary.

[0013] In another embodiment of the present invention, a system for recognizing the danger of a human moving laterally with respect to a moving vehicle and adjusting the driving of the vehicle accordingly includes a thermal image sensor assembly for detecting thermal characteristics of a human on the side of the road using a thermal image. This system further includes an image processor that operates in communication with the thermal image sensor assembly to identify whether the thermal characteristics of the human are moving in a direction towards the road and to determine whether a response operation is required for the driving of the vehicle. This system further includes a central vehicle control system that operates in communication with the image processor to perform a response operation on the driving of the vehicle when it is recognized that such a response operation is necessary.

[0014] According to the present invention, the response to the aforementioned vehicle can take the form of providing a direct input to the "driving assistance" steering system of the vehicle and / or automatically activating the braking system of the vehicle. This system is designed to assist other sensors and guidance systems on the vehicle, especially when the surrounding conditions are severe, and can improve the autonomous guidance system of an autonomous driving vehicle.

[0015] The system according to the present invention can function in both manually driven vehicles and autonomous driving vehicles.

[0016] As described above, using an image processor, it is also possible to detect the presence of a person on the side of the road. More importantly, it is possible to detect that such a person is approaching from the side of the road. More preferably, the image processor using the system according to the present invention can identify the direction and speed at which an object detected on the side of the road is moving, and adjust the driving of the vehicle accordingly.

[0017] The objects, features, and advantages of the present invention will become apparent from the embodiments and the description of their features illustrated by the accompanying drawings.

Brief Description of the Drawings

[0018]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Modes for Carrying Out the Invention

[0019] Referring to FIGS. 1 and 2, the first operation mode of the passive infrared pedestrian avoidance system according to the present invention will be described. As shown in FIG. 1, vehicle 100 generally travels within travel lane 102 on road 104. The system, indicated by reference numeral 10, comprises at least one forward-looking passive IR image sensor or sensor array, indicated as reference numeral 106, mounted outwardly at the front of vehicle 100, which, as vehicle 100 moves, can detect any stationary or moving object 108, including the heat profile of a human, within travel lane 102 of the vehicle or moving towards the predicted path of the vehicle by IR sensor 106.

[0020] In a preferred embodiment of the present invention, as shown in FIG. 1, a single forward-looking IR sensor 106 is attached to the vehicle 100, preferably to the front portion of the vehicle 100, more preferably to the center of the front portion of the vehicle 100, whereby, with respect to both sides of the vehicle 100, moving objects 108 at the left and right ends of the road 104 in front of the vehicle 100, as well as any moving or stationary object 108 already on the path of the vehicle can be detected. Such a forward-looking IR sensor 106 can dynamically change the number and area of the pixels of the sensor to be analyzed according to the forward speed of the vehicle. In the case of high speed, the image processor 110 associated with the IR sensor 106 prioritizes the central region of the sensor 106, and as the speed of the vehicle decreases, the number and area of the pixels to be analyzed can be increased to substantially widen the region of interest of the sensor. In the case of low speed, the analysis region generally covers a relatively close range in front of the vehicle 100, that is, a range of about 10 to 20 feet in front of the vehicle 100 from the IR sensor 106 within about ±45 degrees from the center line of the vehicle 100, and images on the left and right sides of the road 104 or the driving lane 102 in front of the moving vehicle 100 can be acquired. Optimally, each IR sensor 106 has a relatively large pixel array in order to identify laterally moving objects according to the present invention, for example, to achieve at least 640×480. The number and area of the sensor pixels of the thermal reception window can also be dynamically changed according to the ambient temperature, weather conditions, road conditions, or the movement of the object with respect to the vehicle.

[0021] In an alternative configuration of the thermal image sensor assembly provided in the vehicle 100, a plurality of forward-looking IR sensors 106 can be attached to the vehicle 100, preferably at the front end of the vehicle 100. Such IR sensors 106 can operate functionally independently of each other, overlapping and / or redundantly. In embodiments of the present invention, as shown in FIGS. 5 and 6, the vehicle 100 may include a dedicated right-side IR sensor 106R directed towards the right end of the road / driving lane in front of the vehicle 100 for forward viewing and a dedicated left-side IR sensor 106L directed towards the left end of the road / driving lane in front of the vehicle 100 for forward viewing. In such embodiments, the sensors 106R and 106L can be arranged at the front end of the vehicle 100 or on the sides of the vehicle 100 facing forward so as to be able to detect objects in front of the moving vehicle 100. More preferably, the sensors 106R and 106L take into account the movement and speed of the vehicle 100, and further consider the appropriate response actions that the vehicle 100 can take when a threat moving laterally is detected, and can be provided on the sides of the vehicle 100 in a direction capable of predictively detecting an approaching object 108 (for example, a pedestrian and a person on a bicycle as shown in the figure).

[0022] In yet another embodiment, the vehicle 100 may include a combination of front and side IR sensors 106, each preferably being a forward-looking type, in order to optimize the detection area of the system. Also in this case, such IR sensors 106 can operate functionally independently of each other, overlapping and / or redundantly. By using a plurality of IR sensors 106, it is possible to help compensate for varying vehicle speeds and quickly recognize actual and potential threats regardless of the speed of the vehicle 100, the speed of the object 108, the ambient lighting conditions, the road conditions, the weather conditions, etc.

[0023] The descriptions of the IR sensor and the IR sensor array according to the present invention are intended to be replaceable with each other, and each embodiment according to the present invention relates to a potential threat to the vehicle 100, such as a pedestrian walking in front of the vehicle 100, and monitors the side of the road / driving lane in front of the moving vehicle 100. It covers the situation of using a single sensor and a sensor array or a set of sensors that operate to do so.

[0024] Referring to the schematic diagram shown in FIG. 2, the IR sensor 106 is operatively communicating with an image processor 110, such as a video processor, connected to the vehicle 100's driving system via a central CAN bus unit 112 or the like. The CAN bus 112 preferably communicates constantly with various vehicle sensors, such as the IR sensor 106, based on the detected data for analysis and output processing, preferably immediately in real time. The image processor 110 removes objects that deviate from a narrow or pre-defined thermal acceptance window that exhibits a normal human thermal profile (e.g., about 98.6° Fahrenheit) before analyzing whether the thermally detected object is within or about to enter the road 104. The acceptance window can be dynamically expanded or contracted and / or the thermal center point can be changed according to ambient temperature or weather conditions, road conditions, lighting conditions, vehicle speed, etc.

[0025] When in operation, a method for recognizing the risk of lateral movement towards a moving vehicle 100 and adjusting the driving of the vehicle 100 accordingly first involves detecting an object 108 related to the thermal characteristics of a human on the side of the road 104, on the way towards the road 104, or on the road 104 in the predicted path of the vehicle 100, using a thermal image (e.g., IR sensor 106 and image processor 110). When detecting and selecting an object 108 that meets the conditions, i.e., a potential "threat", the system 10 first identifies the number of eligible pixels that meet the target criteria so as to be dynamically corrected by the forward speed of the vehicle. The rate of change of the target pixels with respect to the forward speed of the vehicle determines how close the vehicle 100 is to the potential object 108. Next, the system 10 identifies whether the object 108 is already within the driving lane 102 of the road, or is moving in the direction towards the driving lane 102 of the road, and also at what speed, and determines whether a response action is required for the driving of the vehicle.

[0026] While the system 10 is tracking the movement of the object 108 that meets the thermal characteristics of a human with respect to the vehicle 100, the image processor 110 can also operate in an alternative mode, i.e., when the object 108 that meets the conditions is not moving with respect to the vehicle 100 and the vehicle 100 is moving towards the object 108 that meets the conditions of being stationary, the image processor 110 counts the number of adjacent pixels ("eligible pixels") within a group that meets the human thermal profile criteria, and regards the increase in the number of those eligible pixels as movement towards the object 108 that meets the conditions. The above-mentioned threshold value of the number of eligible pixels required to start the response action can be dynamically changed by the image processor 110 according to the speed of the vehicle and / or the position of the eligible pixels on the sensor array.

[0027] When it is determined that a response action is necessary, System 10 starts response actions for vehicle operation, such as providing an adjustment input to the vehicle's steering system 114, activating the vehicle's braking system 116, sounding an audible alarm 118 related to the vehicle 100 such as activating the vehicle's horn, and / or activating a visual alarm 120 related to the vehicle 100 such as turning on the vehicle's headlights. In the case of a manually operated vehicle 100, an in-vehicle audible alarm, tactile feedback, and / or an alarm display on the windshield in front of the driver may be additionally provided.

[0028] The step of identifying whether a human thermal feature is moving in a direction that potentially approaches the vehicle's predicted travel path, for example, based on the direction and speed of the vehicle 100, first involves identifying whether the human thermal feature is moving or stationary at its first detection, then, if it is moving, identifying whether the direction of the human thermal feature is generally orthogonal to the road 104, and further identifying whether the movement is towards the vehicle's travel lane 102 on the road. The step of determining whether a response action is necessary includes identifying the speed and direction of the movement of the target 108 and / or identifying whether the target 108 is near or already within the vehicle's predicted path. For example, System 10 can indicate that adjacent eligible thermal pixels have increased and reached a numerical threshold representing a proximity response threshold. In this regard, other factors such as ambient temperature, weather conditions, road conditions, vehicle speed, etc. can be analyzed to change the threshold of the response algorithm, as will be described in more detail below.

[0029] Although described herein for use in connection with autonomous or self-driving vehicles, the system 10 of the present invention can also be used in a manually-operated vehicle having a semi-autonomous mode or as a redundant backup to a human driver. For example, the detected human thermal signature and proposed corrective actions can be provided, for example, as a "head-up" display or superimposed display 122 on a manually-operated vehicle 100 or as data inputs to the vehicle 100's navigation system and automatic braking systems 114 and 116. The driver may manually adjust the position and speed of the vehicle, or the vehicle 100 can automatically adjust the position, speed, and brakes of the vehicle based on a continuous monitoring of lateral movement threats to the operation of the vehicle. When the system 10 detects a response input from the driver, while the driver is concentrating on overall vehicle control, the system 10 can concentrate on warning the detected target via the audible and visual signal systems 118 and 120.

[0030] As described above, the image processor 110 can be used to detect the presence of a human thermal profile lateral to the road 104 and, more importantly, to detect that such a human is crossing the road 104. More preferably, the image processor 110 using the system 10 according to the present invention can not only detect the presence of a target object 108 lateral to the road / travel lane using a thermal image, but also identify in which direction and at what speed the object 108 is moving and accordingly adjust the operation of the vehicle 100. When identifying whether a response action is required for the safe operation of the vehicle 100 and what response actions are available, the system 10 takes into account the speed and position of the vehicle 100 relative to the detected human thermal signature, the position of the vehicle 100 on the road 104 (e.g., outer or inner lane on a multi-lane road), the position of the vehicle 100 within a particular travel lane 102, the response time of the system, and ambient weather conditions that may affect the braking force required to respond appropriately to the situation, etc., to predict whether the detected human thermal signature may collide with the vehicle 100.

[0031] According to the present invention, a simultaneous analysis mode as shown in FIG. 3 is available for the autonomous vehicle 100. During operation, the system 10 uses its own algorithm to first detect the position of an object 108 having a human thermal signature (e.g., a pedestrian, a person by the roadside, a biker, a wheelchair with a person), i.e., an object showing a thermal signature of the “nominal” human body temperature of approximately 98.6° Fahrenheit. Subsequently, the algorithm determines whether the detected human thermal signature within the aforementioned thermal range (indicating a pedestrian or a biker, etc.) is already located within the vehicle's driving area, is moving within it, or is moving towards it, or whether it is moving in a safe manner within a designated bicycle lane or along the side of the road, etc. If both the temperature and the movement criteria that are opposite to each other are simultaneously met, response actions such as activating the vehicle's brake system 116 by the logic circuit output and / or biasing the steering control system 114 to turn the vehicle 100 away from an object 108 that it is about to collide with or may collide with are automatically initiated in the autonomous vehicle 100. Further, when the mode is activated, the system 10 can also activate the vehicle's horn 118 and lights 120 in a characteristic manner to visually and auditorily alert or warn distracted pedestrians (such as those looking at mobile devices), pedestrians with visual impairments, and children running behind an object on the road. In a further implementation of the system 10, a warning signal may be transmitted to area mobile devices equipped to receive such safety signals via Bluetooth, V2P, or other signal transmission means.

[0032] During operation, the image processor 110 specifically detects and identifies whether there is movement in the orthogonal or diagonal direction of the thermal characteristics of a person who is likely to enter and / or has entered the road on the path of the moving vehicle 100, or whether there are thermal characteristics of a person stationary within the road 104 or the driving lane 102. At the same time, it typically does not respond to the movement of "body temperature" that is completely parallel, such as that of a biker or pedestrian walking or running along the sidewalk or the edge or shoulder of the road. Regarding these objects moving in parallel, unless they are already in or likely to enter the driving lane or the collision area of the pedestrian / vehicle, the output from the image processor to the vehicle's steering and braking systems is not supplied, avoiding untimely or premature activation of the braking system 116 or untimely or premature steering adjustment by the steering system 114. For the detection of thermal objects other than human thermal characteristics, since they are detected by other detection systems such as LIDAR and proximity sensors that make appropriate responses based on criteria other than the human thermal profile, they usually do not respond.

[0033] Referring to FIG. 3, an IR lateral detection process using the system 10 according to the present invention is shown. In a preferred embodiment, the system 10 is activated and operates only when the vehicle 100 is in operation, and more preferably when the vehicle 100 is moving. In block 200, the vehicle 100 is in operation and the system 10 is preferably activated along with the operation of the vehicle. In block 202, the system 10 queries the vehicle's operating system to identify whether the vehicle 100 is in operation. If not, the system 10 continues to repeatedly query about operation until the vehicle is actually in operation, as shown in block 204. When the system 10 is notified that the vehicle 100 is in operation, the output of the IR sensor 106 and the operating algorithm that have already been activated to search for and / or detect the eligible target 108 is analyzed, and in block 206, permission is given to send information to the vehicle's operating system. As described above, the vehicle 100 includes a single IR sensor 106 disposed at the center of the front of the vehicle 100, and can detect a forward object 108 approaching the vehicle's path from the front directly in front of the vehicle 100 and from both sides of the vehicle 100, or, in cooperation with two IR sensors 106R and 106L disposed on each side of the vehicle 100, monitor both sides of the road 104 in front of the moving vehicle 100 to detect the object 108.

[0034] If a thermal feature is detected, the system 10 determines, at block 208, whether the detected thermal object exhibits the characteristics of a human thermal profile by passing it through a dynamic eligibility thermal gate centered nominally at 98.6° Fahrenheit. If not, the system 10 does not react to the object and continues, at block 204, to search for and determine the eligibility of other detected thermal features. However, if the object 108 is within a predetermined human thermal range, it is classified as a valid detected human thermal feature, and the system 10 determines, as shown at block 210, whether the aforementioned human thermal feature is moving. If no movement is detected and the object 108 is outside the travel lane 102 or the vehicle's path (block 212), the system 10 does not notify the vehicle 100 to perform a response action and continues to monitor changes in the movement state of these and other human thermal feature targets (block 204). If no movement is detected and the object 108 is within the travel lane 102, the system 10 notifies the vehicle 100 to take response actions such as maintaining its current position or adjusting the vehicle's operation as needed (block 226). If no movement is detected and the object 108 is safely adjacent to or within the shoulder or verge of the travel lane 102, the system does not perform a response action. Further, if movement is detected, the system 10 determines, at block 214, the direction of movement, e.g., whether the movement of the human thermal feature is parallel to the current and predicted travel path of the vehicle, away from the vehicle's travel path, or towards the vehicle's predicted travel path. In this regard, the system 10 first determines, at block 216, whether the movement is parallel to the vehicle 100 (e.g., indicating a person walking in a dedicated lane alongside the road or riding a bicycle). If such movement is parallel, the system 10 determines that the human thermal feature does not pose an imminent danger of encroaching on the travel path of the vehicle 100 and returns to monitoring changes in this and other thermal features (block 204).System 10 continues to track the movement of the detected human thermal signature (block 218) and determines whether the direction of that movement shifts to a direction that may be considered a threat, e.g., whether a person on a bicycle crosses road 104 and enters the travel lane 102, until the object 108 moves out of the field of view of the IR sensor 106. If such movement is detected, System 10 re-evaluates the actual threat level (block 220) and determines whether a response action is required, as described below.

[0035] At this stage where a human thermal signature has been detected and System 10 is determining whether there is movement of the detected human thermal signature, System 10 can prevent itself from being misactivated by a temperature-qualified object moving laterally at a low speed outside the lateral angular acceptance window by using a dynamic lateral angular acceptance window. That is, the lateral angular acceptance window can exclude potential threats that are very far from road 104 or that would not pose a danger when, computationally, vehicle 100 passes by the detected object 108, according to the current speed of vehicle 100 and the speed of the detected human 108. The angular window is defined from the centerline of the moving vehicle 100 and is, for example, ±45 degrees from the centerline of the vehicle. More preferably, the width of the lateral acceptance window can be adjusted in proportion to the speed of vehicle 100, such that it is narrower at low speeds and wider at high speeds.

[0036] When it is detected that the movement is generally orthogonal to road 104, at block 220, system 10 determines whether the movement is towards road 104 or away from road 104, and evaluates the threat level to the detected human thermal feature. If it is determined that the movement is away from road 104 or outside the predicted path of the vehicle (block 222), since the threat level is low, system 10 does not react and continues to monitor other eligible thermal features and movements (block 204). If it is determined that the movement is towards road 104 and is likely to cause a vehicle collision with pedestrian 108 (block 224), at block 226, system 10 initiates one or more avoidance response actions, such as, but not limited to, activating the vehicle's braking system 116, providing and activating a direct input to the vehicle's "driver assistance" steering system 114, and / or providing visual and auditory warnings such as activating and illuminating the vehicle's lights 120 and sounding the vehicle's horn 118 rhythmically. Also, if it is determined that the movement is away from road 104 (block 222), system 10 can consider the detected human thermal feature not to be a threat and discard it, but continue to track the movement of the human thermal feature in case the speed and / or direction of the movement changes, and if a change occurs, the threat level can be re-evaluated.

[0037] To more accurately evaluate the detected human thermal feature moving horizontally within the horizontal acceptance window, the algorithm analyzes the current position, direction of travel, forward and horizontal speeds of the detected human thermal feature relative to the vehicle's current position, direction of travel, forward and horizontal speeds, and determines whether their movement vectors will cause a collision.

[0038] After initiating the avoidance response operation as shown in block 226, system 10 communicates with vehicle 100 to return to normal driving mode, resumes monitoring the thermal characteristics indicated by block 228, and essentially starts the process over again to be vigilant against the next unknown, difficult-to-anticipate and predict situations.

[0039] As the vehicle speed increases during driving, the braking speed / braking force increases proportionally so as to be able to appropriately respond to the occurrence of threats in an optimal manner. As a result, as shown in the flowchart of FIG. 4, system 10 dynamically adapts the timing and activation of the vehicle's response system in consideration of both the real-time speed of the vehicle and the forward and lateral speeds of the detected human thermal characteristics. For example, when vehicle 100 is traveling at high speed, system 10 initiates response operations (such as braking and steering adjustments) with faster responsiveness and braking force more quickly than when traveling at low speed. Proactive adjustment of the timing of the system's response operations and its action response level can also be similarly performed in cases of bad weather, such as when snowfall or freezing is detected by the vehicle's temperature sensors and rain sensors, or in other situations where there is a possibility that the brakes / steering may not function properly. Preferably, these sensor inputs adjust the system's response, enabling an earlier (and gentler) start of the response operation, and if possible, avoiding severe response operations (such as a "panicky" immediate stop) on wet or frozen roads. That is, when system 10 recognizes a road condition where it is dangerous to perform a severe braking response operation, it can automatically correct and / or intervene in the detection and analysis of potential threats to perform an earlier activation and a gentler adjustment of the vehicle's speed and steering.

[0040] Referring to FIG. 4, the vehicle speed bias (block 250), as well as the temperature bias (block 264) and the object movement speed bias (block 262), are incorporated into the process of analyzing the outputs of the IR sensor 106 and the operating algorithm that are already activated and searching for and / or detecting the eligible target 108 (block 206). The threat detection process described in FIG. 3 is represented by block 240 in FIG. 4. When a threat is detected and a response action needs to be taken in response to the operation of the vehicle 100, such an adjustment of the operation of the vehicle (block 226) incorporates the vehicle speed bias (block 250) and the temperature / precipitation bias (block 252), which will affect the immediate adjustment of the vehicle. Further, as shown in FIG. 4, the adjustment to the operation of the vehicle can be made in one or more forms of activation of the brakes (block 254), visual warnings (block 256), auditory warnings (block 258), and intervention and adjustment of the vehicle's steering (block 260).

[0041] The system 10 of the present invention is designed to assist other sensors and guidance systems in difficult ambient conditions, and thus can improve the guidance system of an autonomous vehicle.

[0042] As described above, any IR sensors 106R and 106L respectively arranged on both sides of the vehicle 100 while being directed to look forward assist in improving the detection of potential threats on the sides of the road 104 in front of the vehicle 100, as shown in FIGS. 5 and 6. That is, the system 10 can recognize an object on the side of the previous road 104, identify whether it is moving towards the road 104 and at what speed, and determine whether a response action is required before the vehicle 100 approaches the predicted position of the detected human thermal characteristics.

[0043] Unlike many optical-based guidance systems that become less effective at night, especially in sections where the lighting on road 104 is insufficient, the system 10 of the present invention may function more efficiently at night because the thermal contrast between the surrounding background and detectable targets, particularly humans, is expected to increase. By using a cooling element for the infrared imaging device, the temperature discrimination ability of the sensor can also be further improved.

[0044] The foregoing description of the embodiments of the present invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the forms disclosed. Obvious modifications and variations are possible in light of the above disclosure. The described embodiments were chosen to best explain the principles of the invention and its practical application, and that those skilled in the art may utilize the invention in various embodiments including various modifications suitable for the particular purposes contemplated.

Claims

1. A method for recognizing the danger of a pedestrian occurring between a stationary or laterally moving pedestrian and a moving vehicle and accordingly adjusting the operation of the vehicle to prevent the danger, comprising: Detecting thermal characteristics generated by a human in the side or running part of a road using passive thermal radiation detection; Identifying whether the thermal characteristics of the human are within or moving towards the predicted path of the vehicle; Determining whether a response operation for the operation of the vehicle is necessary; Starting the response operation for the operation of the vehicle when such an operation is necessary, and comprising: The detection of the thermal characteristics generated by the human passes a thermal reception window using the detected thermal radiation data if the thermally detected object is within a predetermined temperature range and removes others; When thermal radiation passing through the thermal reception window is detected, recognize a pedestrian based on the thermal radiation data; Identifying whether the thermal characteristics of the human are within or moving towards the predicted path of the vehicle includes: When the thermal characteristics of the human are detected, first identifying whether the thermal characteristics of the human are moving; Identifying whether the movement is orthogonal to the road; Identifying whether the movement is towards or away from the road, and being characterized by comprising: Method.

2. The detection of the thermal characteristics of a human in the side or running part of the road includes using a thermal radiation sensor assembly provided on the vehicle, and is characterized by this. The method according to claim 1.

3. The thermal radiation sensor assembly includes at least one infrared sensor array attached to the vehicle, and is characterized by this. The method according to claim 2.

4. The at least one infrared sensor array includes a forward-looking thermal sensor mounted at the front of the vehicle, characterized in that The method according to claim 3.

5. The at least one infrared sensor array A left infrared sensor mounted on the left side of the vehicle for detecting the thermal radiation of the road on the left side of the vehicle and / or its surrounding area, A right infrared sensor mounted on the right side of the vehicle for detecting the thermal radiation of the road on the right side of the vehicle and / or its surrounding area, characterized in that it includes The method according to claim 3.

6. The pre-defined thermal reception window is dynamically changeable so as to compensate for the influence of ambient temperature conditions on the thermal radiation range of the human, characterized in that The method according to claim 1.

7. Determining whether the response action is necessary includes identifying the moving speed of the thermal characteristics of the human, characterized in that The method according to claim 1.

8. Determining whether the response action is necessary includes identifying the moving speed of the vehicle, characterized in that The method according to claim 1.

9. Determining whether the response action is necessary includes identifying whether the thermal characteristics of the human are within the predicted path of the vehicle, characterized in that The method according to claim 1.

10. The response action for the operation of the vehicle is (1) providing an adjustment input to the steering system of the vehicle, and (2) activating the braking system of the vehicle; (3) emitting an audible warning related to the vehicle; (4) activating a visual warning related to the vehicle, and comprising at least one of the above. The method according to claim 1.

11. A method for recognizing the danger of a pedestrian occurring between a stationary or laterally moving pedestrian and a moving vehicle and adjusting the driving of the vehicle accordingly to prevent the danger, detecting thermal characteristics generated by a human in the side or driving part of the road using passive thermal radiation detection; identifying whether the thermal characteristics of the human are within the predicted path of the vehicle or moving towards the predicted path; determining whether a response operation for the driving of the vehicle is necessary; and starting the response operation for the driving of the vehicle when such an operation is necessary. The detection of the thermal characteristics generated by the human uses the detected thermal radiation data to pass the thermally detected object if it is within a predetermined temperature range and remove others. When thermal radiation passing through the thermal reception window is detected, a pedestrian is recognized based on the thermal radiation data. Determining whether the response operation for the driving of the vehicle is necessary for the thermal characteristics of a stationary human within the predicted path of the vehicle includes using an image processor that searches for a plurality of adjacent qualified pixels indicating the thermal characteristics of the human from a thermal sensor array. Method.

12. Determining whether the response operation is necessary for the thermal characteristics of a stationary human within the predicted path of the vehicle includes using an image processor that searches for a change in the number of adjacent qualified pixels indicating the thermal characteristics of the human from a thermal sensor array. The method according to claim 11.

13. The method according to claim 12, characterized in that the rate of change of the number of adjacent eligible pixels indicating the thermal characteristics of a human is correlated with the approaching speed of a vehicle towards the thermal characteristics of a stationary human. The method according to claim 12.

14. A system for recognizing the danger of a pedestrian to a moving vehicle and adjusting the driving of the vehicle accordingly, a thermal sensor array assembly that uses passive thermal radiation detection to detect thermal characteristics generated by a human on the side of a road or within a predicted path of a vehicle on the road, an image processor that operates in communication with the thermal sensor array assembly to identify whether the thermal characteristics of the human are within the predicted path of the vehicle or are moving towards the predicted path, and to determine whether a response action is required for the driving of the vehicle, a central vehicle control system that operates in communication with the image processor to initiate the response action for the driving of the vehicle when such an action is required, comprising: The detection of the thermal characteristics generated by the human uses the detected thermal radiation data to pass those for which the thermally detected object is within a predetermined temperature range and remove the others, When thermal radiation passing through the thermal reception window is detected, a pedestrian is recognized based on the thermal radiation data, When the image processor receives a signal from the thermal sensor array assembly that recognizes the thermal characteristics of the human, it first identifies whether the thermal characteristics of the human are moving, whether the movement of the thermal characteristics of the human is orthogonal to the road, and whether the movement of the thermal characteristics of the human is towards or away from the road, characterized by: System.

15. The thermal sensor array assembly according to claim 14, characterized in that it includes at least one infrared sensor array attached to the vehicle. The system according to claim 14.

16. The at least one infrared sensor array includes a forward-looking thermal sensor attached to the front of the vehicle, characterized in that The system according to claim 15.

17. The at least one infrared sensor array a left infrared sensor attached to the left side of the vehicle for detecting the thermal radiation of the road and / or its surrounding area on the left side of the vehicle; and a right infrared sensor attached to the right side of the vehicle for detecting the thermal radiation of the road and / or its surrounding area on the right side of the vehicle, characterized in that The system according to claim 15.

18. The predefined thermal reception window is dynamically changeable to compensate for the influence of ambient temperature conditions on the human thermal radiation range, characterized in that The system according to claim 14.

19. When determining whether a response operation by the vehicle is necessary, the image processor identifies the moving speed of the human thermal characteristics in relation to the thermal sensor array assembly, characterized in that The system according to claim 14.

20. When determining whether a response operation by the vehicle is necessary, the system identifies the moving speed of the vehicle, characterized in that The system according to claim 14.

21. When determining whether a response operation by the vehicle is necessary, the image processor identifies whether the human thermal characteristics are within the predicted path of the vehicle in relation to the thermal sensor array assembly, characterized in that The system according to claim 14.

22. The response operation for the driving of the vehicle is (1) providing an adjustment input to the steering system of the vehicle; (2) activating the braking system of the vehicle; (3) emitting an audible warning related to the vehicle; (4) activating a visual warning related to the vehicle, characterized by including at least one of the above. The system according to claim 14.

Images