Method and system for processing reports received from vehicles

Abstract

This disclosure provides a method and system for processing reports received from a vehicle. The method includes receiving a first report from a first navigation system of a first vehicle, the first report being generated in response to an alarm issued after a first hazard is detected in the environment of the first vehicle, the first report including a first location and a first time of the detected first hazard, the first location being determined based on analysis of image data collected by the first navigation system; receiving a second report from a second navigation system of a second vehicle, the second report being generated in response to an alarm issued after a second hazard is detected in the environment of the second vehicle, the second report including a second location and a second time of the detected second hazard; analyzing the first report and the second report to determine which identify the associated hazard; integrating the first report and the second report into a comprehensive report; and processing the comprehensive report to identify the root causes and contributing factors of the associated hazard.

Description

[0001] This application is a divisional application of the invention patent application filed on May 18, 2016, with application number 201680014477.2 (international application number PCT / US2016 / 033026) and entitled "Safety System for Detecting and Warning of Potential Collisions for Vehicles".

[0002] This application claims priority to U.S. Provisional Application No. 62 / 162,838, filed May 18, 2015, and U.S. Provisional Application No. 62 / 261,759, filed December 1, 2015. This application also claims priority to UK Application No. GB1511316.0, filed June 29, 2015. The entire contents of all the foregoing applications are incorporated herein by reference. Technical Field

[0003] This disclosure relates to a safety system for detecting and warning of potential collisions with objects in the vehicle's path, as part of a driver assistance system. Background Technology

[0004] Driver assistance systems (DAS) typically include lane departure warning (LDW), automatic high beam control (AHC), pedestrian recognition, forward collision warning (FCW), and pedestrian detection. Vehicle detection, pedestrian detection, and traffic sign recognition algorithms can share a common general structure: initial candidate detection, followed by more computationally intensive classification of that initial candidate. Various structure-from-motion algorithms can also be used to perform obstacle detection.

[0005] Lane Departure Warning (LDW) systems are designed to warn of unintentional lane departure. The warning is issued when the vehicle crosses or is about to cross lane markings. Driver intent is determined based on the use of turn signals, changes in steering wheel angle, vehicle speed, and brake activation.

[0006] Traffic sign recognition (TSR) modules are typically designed to detect speed limit signs and end-of-speed-limit signs in highway, rural road, and urban environments. Preferably, partially occluded, slightly distorted, and rotated traffic signs are detected. Systems implementing TSR may or should ignore the following signs: signs on trucks / buses, exit road numbers, minimum speed signs, and embedded signs. TSR modules focusing on speed limit signs may not have specific detection range requirements, as the speed limit sign only needs to be detected before it leaves the image.

[0007] The core technology of forward collision warning (FCW) systems and headway distance monitoring is vehicle detection. A key component of a typical FCW algorithm is the estimation of the distance to a single camera and the estimation of the scale change from time-to-contact / collision (TTC), as disclosed in, for example, U.S. Patent No. 7,113,867.

[0008] Structure of Motion (SfM) is a method for recovering the three-dimensional information of a scene that has been projected onto the back focal plane of a camera. The structural information derived from the SfM algorithm can take the form of a set of projection matrices, one for each image frame, representing the relationship between specific two-dimensional points in the image plane and their corresponding three-dimensional points. The SfM algorithm relies on tracking specific image features from image frame to image frame to determine structural information about the scene. Summary of the Invention

[0009] The various systems and methods disclosed herein can be performed by a system that can be installed in a vehicle to provide object detection near the vehicle. The system includes a camera operatively attached to a processor. The camera is externally mounted at the rear of the vehicle. The camera's field of view is along the side of the vehicle, substantially in the direction of travel. Multiple image frames are captured from the camera. The yaw of the vehicle can be input, or the yaw can be calculated from the image frames. In response to the yaw of the vehicle, a corresponding portion of the image frames is selected. The image frames are processed to detect objects in the selected portion of the image frames. The yaw is measurable or determinable by processing the image frames, or by processing inputs from gyroscope devices, turning signals, steering angles, and sensors attached to the vehicle's steering column. As the absolute value of the yaw increases, a larger area of ​​the image frames is processed to increase the effective horizontal field of view, and thus to detect obstacles at a wider horizontal angle measured from the side of the vehicle.

[0010] For a camera mounted on the right side of the vehicle, when the vehicle turns right, a larger area of ​​the image frame is processed to increase the effective horizontal field of view and to detect obstacles at a wider horizontal angle measured from the right side of the vehicle.

[0011] For a camera mounted on the left side of the vehicle, when the vehicle turns left, a larger area of ​​the image frame is processed to increase the effective horizontal field of view, and obstacles are detected at a wider horizontal angle measured from the left side of the vehicle. The object can be determined to be a moving obstacle, and it can be verified that the object will be in the vehicle's expected path to determine a potential collision between the vehicle and the moving obstacle. The driver is warned of the potential collision with the moving obstacle. The warning can be issued as an audible warning and / or a visual warning. The warning can be issued from the side of the vehicle equipped with a camera to view moving obstacles. The object can be a cyclist, motorcyclist, vehicle, pedestrian, child riding a toy car, person pushing a stroller, or wheelchair user. Another driver assistance system: lane detection, structural barrier recognition, and / or traffic sign recognition can be used to identify stationary objects, and if all objects detected in the immediate vicinity of the vehicle are stationary, a warning to the driver can be suppressed. The stationary objects can include structural barriers, lane markings, traffic signs, trees, walls, and / or poles.

[0012] Another aspect of this disclosure provides a method for processing reports received from a vehicle, comprising: receiving a first report from a first navigation system of a first vehicle, wherein the first report is generated by the first navigation system in response to an alarm issued after detecting a first hazard in the environment of the first vehicle based on analysis of image data collected by the first navigation system, and wherein the first report includes a first location of the detected first hazard and a first time associated with the detected first hazard, and the first location is determined by the first navigation system at least in part based on analysis of image data collected by the first navigation system; receiving a second report from a second navigation system of a second vehicle, wherein the second report is generated by the second navigation system in response to an alarm issued after detecting a second hazard in the environment of the second vehicle based on analysis of image data collected by the second navigation system, and wherein the second report includes a second location of the detected second hazard and a second time associated with the detected second hazard; analyzing the first report and the second report to determine that the first report and the second report identify associated hazards; integrating the first report and the second report into a comprehensive report; and processing the comprehensive report to identify the root causes and contributing factors of the associated hazards.

[0013] Another aspect of this disclosure provides a system for processing reports received from a vehicle, the system including a server operable to: receive a first report from a first navigation system of a first vehicle, wherein the first report is generated by the first navigation system in response to an alarm issued after detecting a first hazard in the environment of the first vehicle based on analysis of image data collected by the first navigation system, and wherein the first report includes a first location of the detected first hazard and a first time associated with the detected first hazard, and the first location is determined by the first navigation system at least in part based on analysis of image data collected by the first navigation system; receive a second report from a second navigation system of a second vehicle, wherein the second report is generated by the second navigation system in response to an alarm issued after detecting a second hazard in the environment of the second vehicle based on analysis of image data collected by the second navigation system, and wherein the second report includes a second location of the detected second hazard and a second time associated with the detected second hazard; analyze the first report and the second report to determine that the first report and the second report identify associated hazards; integrate the first report and the second report into a comprehensive report; and process the comprehensive report to identify the root causes and contributing factors of the associated hazards. Attached Figure Description

[0014] The disclosed embodiments are described herein by way of example only with reference to the accompanying drawings, in which:

[0015] Figure 1A and 1B A side view and a top view of a vehicle according to one aspect of the disclosed embodiments are shown respectively.

[0016] Figure 2A A block diagram of a system according to one aspect of the disclosed embodiments is shown.

[0017] Figure 2B A view of the vehicle interior according to the features of the disclosed embodiment is shown.

[0018] Figure 3 A flowchart of a method according to one aspect of the disclosed embodiments is shown.

[0019] Figure 4A and 4B A road scene scenario diagram according to one aspect of the disclosed embodiments is shown.

[0020] Figure 5A A front view of the driver from inside the vehicle, showing features according to the disclosed embodiment, is shown.

[0021] Figure 5B , 5CFigures 5D and 5E illustrate examples of central warning displays featuring the characteristics of the disclosed embodiments.

[0022] Figure 5F A right and / or left warning display featuring characteristics according to the disclosed embodiments is shown.

[0023] Figure 6 A flowchart of a method according to an aspect of the disclosed embodiments is shown.

[0024] The above and / or other aspects will become apparent from the following detailed description when considered in conjunction with the accompanying drawings. Detailed Implementation

[0025] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings, wherein the same reference numerals always denote the same elements. Embodiments are described below with reference to the accompanying drawings.

[0026] As is well known, drivers of buses or heavy vehicles have limited visibility in the immediate vicinity of the vehicle. In particular, pillars separating the front window from the side of the bus partially obstruct the driver's view. Furthermore, the driver has a limited field of view at the side of the vehicle, which can be partially compensated for by the arrangement of side mirrors. As a result, the driver may be unaware of potential obstacles, such as cyclists or pedestrians moving into a "blind spot" that is not easily seen by the driver. The disclosed embodiments can provide a warning to the driver when a moving obstacle encroaches on the side of the vehicle, especially when turning, to avoid a collision between the unseen moving obstacle and the side of the vehicle. According to a feature of the disclosed embodiments, when the vehicle turns, the region of interest in the image frame and the effective horizontal field of view, or angle of interest, measured from the side of the bus change. When the vehicle turns right, the angle of interest on the right side increases, and when the bus travels straight after the turn, the horizontal angle of interest decreases again. The system is configured to process the image frame and detect potential obstacles in the region of interest of the image frame. When a vehicle is traveling in a straight line, reducing the area of ​​interest is important for reducing false positive warnings to the driver when the bus is traveling in a straight line, and the likelihood of a collision is lower compared to when it is turning.

[0027] Bus drivers approaching a bus stop are typically more attentive to the traffic lane they are attempting to merge into—for example, the left side of a bus in the US and the right side in the UK—than to any obstacles that might be adjacent to a pedestrian walkway. If a pedestrian, cyclist, or child riding a toy car accidentally enters the bus's path from the pedestrian walkway, it could result in personal injury or death. Similarly, a bus driver may not be aware of a pedestrian or cyclist moving into front of the bus when looking at an oncoming vehicle in their rearview mirror. Therefore, according to another feature of the disclosed embodiment, audible and / or visual warnings are given to the driver to bring them to the attention of identified obstacles near the bus path that they would otherwise not be aware of.

[0028] In some embodiments, the disclosed systems and methods may be implemented to navigate an autonomous vehicle. As used throughout this disclosure, the term "autonomous vehicle" refers to a vehicle capable of implementing at least one navigation change without driver input. A "navigation change" refers to a change in one or more of the vehicle's steering, braking, or acceleration. For autonomous driving, the vehicle does not need to be fully autonomous (e.g., fully operational without a driver or driver input). Rather, an autonomous vehicle includes those vehicles capable of operating under driver control during certain time periods and operating without driver control during other time periods. An autonomous vehicle may also include vehicles that control only certain aspects of vehicle navigation, such as steering (e.g., maintaining the vehicle's path between road constraints), while leaving other aspects to the driver (e.g., braking). In some cases, an autonomous vehicle may handle some or all aspects of the vehicle's braking, speed control, and / or steering.

[0029] Now refer to the appendix Figure 1A and 1B , attached Figure 1A and 1BSide and top views of a vehicle 18 according to one aspect of the disclosed embodiments are shown. Vehicle 18 may be a public transport bus, school bus, long-distance coach, heavy-duty truck (HGV), or passenger vehicle (e.g., a car, such as a sedan). In some embodiments, vehicle 18 may be longer than a typical vehicle (e.g., a limousine, truck, etc.). Two cameras 12R and 12L are mounted on the left and right sides of vehicle 18, respectively, near the rear portion of vehicle 18. Cameras 12R and 12L are mounted such that their respective fields of view generally encompass the direction of travel of vehicle 18, as shown by dashed lines. Camera 12C is shown mounted at the front of vehicle 18, its field of view generally centered on the direction of travel of vehicle 18, as shown by dashed lines. Cartesian coordinate axes X, Y, and Z are shown. The Z-axis is a long horizontal axis along the length of vehicle 18. The Y-axis is the vertical axis of vehicle 18. Rotation about the Y-axis defines the yaw of vehicle 18 and / or camera 12. Yaw is controlled by the steering mechanism of vehicle 18. The X-axis is horizontal along the width of vehicle 18.

[0030] Still referencing Figure 2A , Figure 2A A block diagram of system 16 is shown, which includes a camera 12, illustrating one of the various features of the disclosed embodiments, namely camera 12C, 12R, or 12L. Different examples of system 16 may be attached to a left camera 12L, a right camera 12R, and a central camera 12C, which are not shown individually for simplicity. Processor 14 receives and processes image frames 15 from camera 12. Storage device or memory 20 is bidirectionally connected to processor 14. Memory 20 can be used to store algorithms known in the art for camera-based driver assistance systems (DAS) that provide pedestrian and / or other classification-based object (e.g., bicycle, motorcycle) detection / recognition and / or general obstacle detection / recognition. Output from processor 14 is shown attached to display 32. Input from a yaw sensor to processor 14 is shown. System 16 can be installed in, for example, Figure 1A and 1B In multiple locations within the vehicle 18 shown. Alternatively, multiple cameras 12 (e.g., side cameras 12R and 12L and a central camera 12C) can be connected to a single processor 14, and each can provide multiple sets of image frames to the single processor 14.

[0031] Now refer to Figure 2B , Figure 2BView 30, showing features from inside vehicle 18 according to the disclosed embodiment, illustrates the rear of driver 36 when operating vehicle 18. To the left of driver 36 is display 32L, used to provide driver 36 with visual and / or audible warnings regarding the detection and / or identification of objects visible to left camera 12L, and whether such objects may collide with vehicle 18. The detection / identification of the objects is provided by one of processors 14L / 14R that processes image frames 15L / 15R captured by cameras 12L / 12R.

[0032] In some embodiments, instead of providing a warning to driver 36, system 16 may instead cause the vehicle (e.g., vehicle 18) to take action. For example, system 16 may analyze collected data to determine whether the vehicle (e.g., vehicle 18) should take a certain action, and then automatically take the determined action without human intervention. For example, in response to recognizing a potential collision between an object and the vehicle, system 16 may automatically control the vehicle's braking, acceleration, and / or steering (e.g., by sending control signals to one or more of the throttle control system, braking system, and steering system). In other embodiments, system 16 may provide a visual and / or audible warning to the driver and cause the vehicle to take action.

[0033] Similarly, to the right of driver 36 is display 32R, which provides driver 36 with a possible warning of a potential collision with an object to driver 36's right and / or prompts the vehicle to take actions such as automatically controlling the vehicle's braking, acceleration, and / or steering (e.g., by sending control signals to one or more of the throttle adjustment system, braking system, and steering system). In the driver 36's forward field of vision is display 32C, which provides visual and / or audible notification of objects that may be detected in front of driver 36 in image frames 15C captured by camera 12C. Displays 32L, 32C, and 32R are positioned to provide a display to driver 36 that follows driver 36's field of vision while driving vehicle 18. For example, when driver 36 is driving straight forward, display 33C is positioned in his / her overall field of vision, while if turning left or right, displays 33L and 33R are positioned in his / her overall field of vision, respectively.

[0034] Still referencing Figure 3 , Figure 3 A flowchart illustrating a method 301 according to one aspect of the disclosed embodiment is shown, the method using a system 16, the system 16 including components installed as... Figure 1A and 1BMultiple cameras 12C, 12R, and 12L are shown on vehicle 18. Cameras 12R, 12L, and 12C independently provide image frames 15R, 15L, and 15C, respectively, which are captured by processors 14C, 14L, and 14R (step 303). Yaw information 4 can be input (step 305) to processors 14C, 14L, and 14R. Yaw information 4 can be derived from any or a combination of the following: yaw sensor (e.g., gyroscope device), use of turning signals or changes in steering wheel angle via the CAN bus of vehicle 18, or a sensor attached to the steering column of vehicle 18 that provides a measurement of the steering column angular position / speed when vehicle 18 turns left or right. Yaw information 4 can alternatively or additionally be derived from processing image motion in image frame 15. For example, in some embodiments, yaw information 4 can be derived by analyzing at least two image frames. The analysis may include identifying an object in at least two image frames and determining that the object has shifted or moved. This determination may involve comparing, for example, the position of the object relative to the center point or edge of at least two image frames.

[0035] In step 307, the processor, in response to yaw information 4, sets the region of interest in image frame 15. For example, if vehicle 18 is turning right, processor 14R is configured to use the wider portion of image frame 15R (nominally in horizontal image coordinates) for subsequent detection and classification (step 309). If vehicle 18 is turning left, processor 14L is configured to use the wider portion of image frame 15L for subsequent detection and classification (step 309). If vehicle 18 is traveling in a straight line, processors 14R and 14L are configured to use the narrower portion (in horizontal image coordinates). In decision block 309, algorithms stored in memory 20R / 20L and / or contained in the circuitry of processors 14C / 14R / 14L are used to detect and / or identify obstacles, such as pedestrians, motorcycles, bicycles, and general obstacles, whose images are at least partially within the region of interest in response to yaw information. If such an object is detected and / or identified, driver 36 is warned (step 313). Otherwise, method 301 continues capturing multiple image frames (step 303). Advantageously, the region of interest in image frames 15R / 15L is set in response to yaw information (step 307) to reduce false positive object detection / classification near vehicle 18 and avoid frequent warnings to driver 36 when object detection / classification is not expected in the region of interest (decision box 309) (step 313).

[0036] In some embodiments, in addition to using yaw information or as an alternative to using yaw information, system 16 may determine the portion (e.g., a wider or narrower portion) of one or more image frames to use, based at least on information stored in a local or remote database. For example, the stored information may include information collected during previous navigation in a particular area or location. This information may include areas with potential obstacles such as pedestrians, and this information may specify portions of image frames to be used in these areas. In some embodiments, portions of image frames for subsequent detection and classification may be selected based on stored map data and / or based on the vehicle's position determined via GPS data. For example, based on the vehicle's position, system 16 may determine to use a portion of one or more image frames for subsequent detection and classification.

[0037] Now refer to the appendix Figure 4A and 4B , attached Figure 4A and 4B Road scene scenarios 40a and 40b, respectively, are shown according to aspects of the disclosed embodiments. Figure 4A The image illustrates, by way of example, a vehicle 18 equipped with camera 12R traveling on a straight road. Objects 54a and 54b are shown, which could be cyclists traveling in a bike lane. The dashed lines emanating from camera 12R indicate the range of the horizontal effective field of view. Outside the horizontal effective field of view, although images of objects 54a and 54b can be imaged by camera 12R, algorithm 309 does not detect / classify objects 54a and 54b. Object 54b may be partially obscured by the right front pillar of vehicle 18. However, since vehicle 18 is traveling in a straight line, there is virtually no chance of collision with objects 54a and 54b, for example, stationary in the bike lane or also traveling in a straight line.

[0038] The presence of stationary objects, such as structural barrier 86, can be determined using image frames from the right camera 12R by employing the Structure of Motion (SfM) algorithm. SfM can recover 3D information from the stationary structural barrier 86. In the absence of other obstacles, such as moving or stationary obstacles 54a / 54b, the presence of the stationary barrier 86 suppresses warnings to the driver 36 to avoid unnecessary false positives.

[0039] If structural barrier 86 can be a pedestrian walkway, a positive warning can be issued using the SfM algorithm. Vehicle 18 turns at an intersection adjacent to a pedestrian walkway, for example, turning right from the rightmost lane, potentially encroaching on the pedestrian walkway even if its right front wheels are not on the walkway. Pedestrians standing at the corner may be hit. If a bus or truck turns too sharply, a warning is issued if a pedestrian is detected on the pedestrian walkway. The curvature of the pedestrian walkway at the intersection can be indicated by detection and classification using the SfM algorithm. If vehicle 18 makes a right turn at the intersection, the region of interest in image frame 15R in response to yaw information can be increased (step 307) to warn driver 36 that the current right turn has the potential for vehicle 18 to encroach on the pedestrian walkway from the side.

[0040] Now also refer to the appendix showing route scenario 40b. Figure 4B Vehicle 18 is shown turning on a right curve, and detection and classification algorithm 309 is configured to use a wider horizontal angle, as indicated by the dashed line projected by camera 12R. The wider horizontal angle corresponds to a wider portion of the image in the horizontal image coordinates of image frame 15. Obstacles 54a and 54b are shown again. Obstacle 54b may be partially obscured by the right front pillar of vehicle 18. If vehicle 18 turns right, it is unclear whether, for example, the moving obstacle cyclist 54b intends to travel in a straight line near the intersection, or whether cyclist 54b intends to turn right alongside vehicle 18. Even if cyclist 54b intends to turn right, the road condition may be such that cyclist 54b may unintentionally slide towards vehicle 18 during the right turn. In this case, detecting / identifying the cyclist within the effective field of view of the right camera 12R could trigger a warning ( Figure 3 (Step 313).

[0041] In decision block 309, in response to yaw information, an algorithm stored in memory 20C / 20R / 20L and / or contained in the circuitry of processor 14C / 14R / 14L can be used to detect and / or identify obstacles. If an object is detected and / or identified, the driver 36 is warned (step 313).

[0042] Now refer to the appendix Figure 5A , attached Figure 5A A driver's front view 50 from inside a vehicle 18 shows features according to a disclosed embodiment. The vehicle 18 may be equipped with multiple systems 16 ( Figure 2A Each system has image frames 15 captured by a processor 14 from camera 12. The processor 14 processes the image frames 15 and identifies or detects obstacles that may collide with vehicle 18.

[0043] An alarm display 32L is shown on the left side of the steering wheel of vehicle 18. Alarm display 32L is connected to the output of processor 14L, which captures image frames 15L from camera 12L. The alarm output of processor 14L can be displayed by display 32L to provide a warning to the driver 36 of vehicle 18. The warning may be an obstacle identified by processor 14L within the field of view of left camera 12L, wherein the obstacle may collide with vehicle 18. On the right side of vehicle 18, there may be another display 32R connected to the output of processor 14R. The alarm output of processor 14R can be displayed by display 32R to provide a visual warning to the driver 36 of vehicle 18. The warning may be caused by an obstacle identified by processor 14R within the field of view of right camera 12R, wherein the obstacle may collide with vehicle 18.

[0044] Warning display 32C is displayed at the center of the bottom of the windshield above the dashboard of vehicle 18. Processor 14C processes image frames 15C captured from central camera 12C and identifies obstacles that vehicle 18 may collide with along its collision path. The output of processor 14C is connected to display 32C to provide a warning to driver 36 of vehicle 18. The warning output of processor 14C can be visually displayed on display 32C. The warning may be an obstacle identified by processor 14C that is visible to central camera 12C.

[0045] Three identified objects are marked with rectangular boxes: two are cyclists, and the other is a pedestrian. Since the three identified objects are in front of vehicle 18, depending on which of cameras 12L, 12R, or 12C views the identified obstacle, an appropriate warning can be audibly issued and / or displayed on one or more of displays 32R, 32L, and / or 32C. Similarly, the audible warning depends on which camera views the identified obstacle. If the identified obstacle is on the right side of vehicle 18, a speaker or buzzer located on the driver's right side can emit a warning sound; this speaker or buzzer is optionally housed with the right display 32R. Similarly, if the identified obstacle is on the left side of vehicle 18, a speaker or buzzer located on the driver's left side can emit a warning sound; this speaker or buzzer is optionally housed with the left display 32L. If the detected obstacle is identified as being in front of and in the center of the vehicle 18, a speaker or buzzer located near the driver may emit a warning sound, the speaker or buzzer being optionally housed in conjunction with the center display 32C.

[0046] Now refer to the appendix Figure 5B , 5C 5D and 5E, with Figure 5B , 5CExamples of display 32C are shown in more detail in 5D and 5E according to the features of the disclosed embodiments. Figure 5B , 5C Figures 5D and 5E illustrate four possible warnings that can be displayed visually from the monitor 32C.

[0047] refer to Figure 5B The display 32C provides the driver 36 of vehicle 18 with a warning of an impending rear-end collision with a car, truck, or motorcycle moving at any speed. The warning is indicated by a vehicle icon 54 within a flashing red circle, and / or accompanied by an audible warning. If a collision with a pedestrian is imminent, a pedestrian icon 52 may similarly flash red and / or be accompanied by an audible warning. The pedestrian collision warning may be activated only at speed limits appropriate to the area where vehicle 18 is traveling, i.e., for example, within a built-up area where pedestrians are present.

[0048] refer to Figure 5C The display 32C provides a headway warning accompanied by a subsequent time of 0.8 seconds. As used herein, "headway" is defined as the time interval between arrival at the current position of an obstacle in front of vehicle 18, and it can be calculated by dividing the distance to the obstacle by the speed of vehicle 18. If the headway drops below a predetermined value in seconds, a visible and / or audible warning may be displayed to the driver 36.

[0049] If the obstacle is a vehicle ahead or an oncoming vehicle, vehicle icon 54 can be used to alert the driver and / or provide an audible warning along with the vehicle's time distance display.

[0050] refer to Figure 5D The display 32C provides a Lane Departure Warning (LDW), indicated by a dashed line in the circular portion of the display 32C. Figure 5D In this case, the dashed line is located to the right of the circle to indicate to the driver 36 that vehicle 18 has deviated from the lane on the right side without using a turn signal. Similarly, a dashed line located to the left of the circle portion of display 32C can be used to warn of lane departure on the left-hand side. Lane departure warnings can be displayed as flashing dashed lines and / or accompanied by an audible warning. Lane departure warning (LDW) can be activated when vehicle 18 is traveling faster than a threshold speed. LDW can be activated differently depending on whether the road is an urban road, highway, or rural road. For example, on urban roads with bus stops and pedestrians, vehicle 18 will frequently cross lane markings, and lane departure warnings can be activated or suppressed differently.

[0051] Now refer to Figure 5EIf the driver 36 exceeds the indicated speed limit, the display 32C provides a visual and / or audible warning. In this case, an example of an indicated speed limit is shown on the display 32C as 20 miles per hour or 20 kilometers per hour.

[0052] Now refer to the appendix Figure 5F , attached Figure 5F The display 30L / 30R is shown in more detail according to the features of the disclosed embodiments. The warning provided on the display 30L / 30R is a result of an obstacle identified by the processor 14L / 14R. As described below, the processor 14L / 14R processes image frames 15L / 15R captured by the camera 12L / 12R, and can issue warnings from the processor 14L / 14R and display the warnings on the corresponding display 32L / 32R.

[0053] Displays 30L / 30R include a pedestrian icon 52. A flashing or solid yellow pedestrian icon 52 can warn the driver 36 of the vehicle 18, who is turning left or right, of a potential collision with an obstacle. The flashing or solid yellow pedestrian icon 52 indicates to the driver 36 that a pedestrian or cyclist is nearby but not within the collision trajectory of the vehicle 18. However, a flashing or solid red pedestrian icon 52 on displays 30L / 30R and / or an audible sound can warn the driver 36 of the vehicle 18 that a pedestrian or cyclist is within the collision trajectory of the vehicle 18 when the vehicle 18 is turning left or right.

[0054] Now refer to Figure 6 The following is a flowchart of method 601 according to an aspect of the disclosed embodiments. In the following discussion, reference is made to camera 12L, processor 14L, and display 32L. However, the following method steps are similar to those of the other systems 16: cameras 12R, 12C; processors 14R, 14C; and displays 32R, 32C. Method 601 can be executed in parallel by each of the systems 16. In step 603, a plurality of image frames 15 captured by processor 14L from camera 12L are processed so that candidate images are detected and / or identified as obstacles in the field of view of camera 12L in step 605. In step 607, a determination is made as to whether the obstacle detected and / or identified in step 605 is in the driving trajectory of vehicle 18 such that vehicle 18 is likely to collide with the obstacle. If the driving trajectory of vehicle 18 is likely to collide with the obstacle, a visual and / or audible warning is provided to the driver 36 of vehicle 18 via display 32L (step 609).

[0055] Alternatively, as described above, in some embodiments, instead of providing a warning to driver 36, system 16 may cause the vehicle (e.g., vehicle 18) to take action. For example, system 16 may analyze collected data to determine whether the vehicle (e.g., vehicle 18) should take a certain action, and then automatically take the determined action without human intervention. For example, in response to recognizing a potential collision with the vehicle, system 16 may automatically control the vehicle's braking, acceleration, and / or steering (e.g., by sending control signals to one or more of the throttle control system, braking system, and steering system). In other embodiments, system 16 may provide a visual and / or audible warning to the driver and cause the vehicle to take action.

[0056] Many vehicles have blind spots, which hinder a vehicle operator's ability to notice hazards in certain areas around the vehicle. In dense urban environments, vulnerable road users (VRUs), including pedestrians and cyclists, frequently share the road with vehicles. Blind spots, and the inability of operators to detect VRUs within their blind spots, are a serious problem that can lead to severe consequences. Blind spots can be a particularly serious problem for drivers of large vehicles such as trucks (or freight vehicles) and public transportation vehicles, especially in urban environments.

[0057] Accordingly, in other embodiments, the disclosed systems and methods select at least a portion of one or more image frames in response to identifying a potential risk of blind spots. The disclosed systems and methods can then process the selected portion of the one or more image frames to detect, for example, objects within the selected portion of the image frames. Furthermore, in some embodiments, as described below, portions of image frames can be selected based on areas where potential blind spots and / or associated hazards are known, for example, based on data collected by the system from previous drivers at a particular location (e.g., based on reports discussed in more detail below). Even further, in some embodiments, the selection of at least a portion of one or more image frames can be based on the vehicle's yaw rate and collected data (e.g., reports, etc.).

[0058] For example, in some embodiments, the disclosed systems and methods can be configured to detect VRUs. Optionally, the systems and methods can be configured to detect VRUs while ignoring inanimate objects. In some embodiments, the system may include a master camera and one or more slave cameras. The master camera may be configured to provide an advanced driver assistance system (which may be implemented on a processor) to an operator in addition to detecting VRUs. The slave cameras may be configured to receive input from the master camera and be able to monitor one or more blind spots around the vehicle. The processor may process images from the one or more cameras and detect potential hazards in the blind spots. Optionally, the processor may be configured to issue an alert when a hazard is detected in a blind movement area. Optionally, the processor may be configured to initiate an operation in the vehicle or one of the vehicle's systems in response to the detection of a hazard in a blind spot of the vehicle.

[0059] For example, the system (e.g., system 16) may include a camera control unit configured to adjust the effective coverage of at least one camera mounted on or in a vehicle for monitoring an area around the vehicle. Further, for example, the camera control unit may be configured to adjust the effective field of view of at least one camera. In yet another example, the camera control unit can be configured to rotate the at least one camera to adjust its field of view in a desired direction as needed. Optionally, the camera control unit may be implemented in a master camera, and the master camera may be configured to control and adjust at least one slave camera. According to another example, the master camera may be configured to control the detection angle of the slave cameras such that they are configured to detect VRUs in a specific area (e.g., effective FOV) around the vehicle. In some embodiments, the system may include one, two, three, four, or more cameras. For convenience, and by way of non-limiting example, the cameras of the system are sometimes referred to as a “camera array” in the following description. A processor may be configured to control a first camera in a camera array based on corresponding adjustments applied or to be applied to a second camera in the camera array by the processor. In another example, two or more cameras in a camera array can be adjusted collaboratively and consistently. Alternatively, camera adjustments can be performed in conjunction with dynamically changing positions or the projected positions of one or more vehicle-related blind spots.

[0060] In some embodiments, the disclosed systems and methods can display a map of hazards detected by a smart camera and / or alerts issued by an in-vehicle smart camera. The term "smart camera" refers to a camera that captures images, video sequences, or environments, and one or more processors (and possibly other computer hardware) that process the images to provide information about objects appearing in the images. Various smart cameras are known and used in driver assistance systems and autonomous vehicle systems. According to examples of the currently disclosed subject matter, the method can include: receiving and storing multiple reports of hazards detected by a smart camera and / or alerts issued by an in-vehicle smart camera, each of the multiple reports including at least one location information, wherein the location information is the location in global coordinates of the corresponding hazard detected by the smart camera and / or the corresponding alert issued by one or more in-vehicle smart cameras; receiving display parameters and providing indications on a map of hazards detected by a smart camera and / or alerts issued by an in-vehicle smart camera based on the display parameters. In another example, the method can include integrating two or more reports of hazards detected by a smart camera and / or alerts issued by an in-vehicle smart camera based on the display parameters.

[0061] To further illustrate, the display parameter could be the region of interest (ROI), which indicates which part of the map should be displayed to the user on the screen. As a further example, the map indication of hazards detected by a smart camera and / or alerts issued by an in-vehicle smart camera is an indication of hazards and / or alerts detected within the ROI. As a further example, the ROI can be dynamically adjusted, and the display of detected hazards and / or alerts can also be dynamically and individually changed. For example, the display parameter could be the map scale. In another example, hazards detected by a smart camera and / or alerts issued by an in-vehicle smart camera can be integrated according to the map scale. Thus, for example, when the map scale is relatively large, detected hazards and / or alerts from relatively large areas can be combined, and the corresponding indication of the integrated detected hazards and / or alerts can be displayed on the map at a location, such as the center of all detected hazards and / or alerts; while when the map scale is relatively small, detected hazards and / or alerts from relatively small areas are combined, and the indication of the integrated detected hazards and / or alerts is displayed on the map. It should be understood that this feature enables at least one locational aspect of detected hazards and / or issued alarms to be displayed dynamically. Thus, for example, a user can start with a map of the entire metropolitan area to view the overall layout of detected hazards and / or issued alarms, and by zooming in, the user can gradually (although it could also be in a non-gradual manner) understand the specific area where a large number of alarms were triggered in a single location.

[0062] For example, when multiple detected hazards and / or alerts are reported for a single location or a specific area, an indication can be displayed at the corresponding location on the map, and this indication can be associated with the number or density of the reported detected hazards and / or alerts. In another example, the indication can also be associated with the severity of the detected hazards and / or alerts. In yet another example, conditions can be clarified by displaying different types or different information on the map in association with corresponding detected hazards and / or alerts, for example, by a user-set threshold or by providing predefined thresholds. In yet another example, the indication on the map can relate to the reported detected hazards and / or alerts, or it can be associated with the corresponding type of each detected hazard and / or alert. In yet another example, detected hazards and / or alerts can be aggregated. In yet another example, a counter displaying multiple events or reports (e.g., the total number of detected hazards and / or alerts) associated with a location or area on the map can be updated based on the number of detected hazards and / or alerts of the same type. The size of the area associated with the counter, and optionally the location of the area, can be associated with the map scale or with an area shown on the map. In a digital map, the scale and reference area can be dynamically adjusted, and the display of detected hazards and / or issued alarms will also change as the scale changes.

[0063] In some embodiments, the disclosed systems and methods can generate signatures of hazards detected by smart cameras and / or alarms issued by in-vehicle smart cameras. As described above, one or more cameras and at least one processor can be mounted on a vehicle. The processor can process images from the cameras and detect hazards around the vehicle, which can trigger alarms and / or operations in one of the vehicle or vehicle systems. For example, the processor can be configured to issue a report related to a hazard detected near the vehicle. The processor can transmit the report to a back-end server, or the processor can store the report locally (e.g., in the vehicle's memory), and the back-end server can be configured to "recall" such a report. Optionally, the report can include the location of the hazard and the time when the hazard was detected or when an incident associated with the hazard occurred (e.g., when an incident was captured by an in-vehicle camera). Optionally, the report can include information about the type of hazard detected. In addition to, or as an alternative to, hazard indication types, the report can include one or more images associated with the detected hazard, including images of an incident or potential incident that triggered an alarm, or operational images of the road and / or surrounding environment and infrastructure, and possible information about the camera used to capture the images.

[0064] In some embodiments, the server may be configured to process incoming reports from in-vehicle smart cameras. Optionally, the server may be configured to identify relevant reports, such as those associated with public or relevant safety hazards, and the server may generate a signature for a particular safety hazard. Each signature may be associated with one or more reports of a safety hazard. Furthermore, for example, the disclosed systems and methods can identify similar safety hazards associated with dangers detected by smart cameras installed on vehicles. In one example, a matching algorithm may be used to determine whether two different sets of data produce results within a predetermined threshold or range when compared. Thus, for example, when the area or point of an object that triggered hazard detection and / or reporting in the first set of reports is equivalent to or sufficiently close (according to some approximation threshold) to a location or set of locations of the vehicle relative to a corresponding location or area in the second set of reports, an indication of the similarity of safety hazards between the first and second sets of reports can be inferred.

[0065] In another example, image processing, geometric representation, schematics, etc., can be used to determine similarity, such that similarity is determined when two environments have equivalent or sufficiently close layouts (according to some similarity threshold). The layout can be the overall layout of a street or even just the location of a single object. For example, in a location where a relatively large number of hazards are detected or alarms are issued, it can be determined that a sign on a sidewalk obstructs the view of a bus traveling past the sign, as well as the view of pedestrians crossing the street near the sign. This inappropriate signage leads to a large number of hazard detections or alarms. The report indicates the point relative to a vehicle where a hazard was detected or an alarm was issued, and the location of an inanimate object relative to a bus, or the sign in this particular example. Sometimes, similar situations in different locations (e.g., different streets, different cities, or even different countries) lead to similar hazard detection or alarm patterns, and the report may have already indicated that an inanimate object is located in a similar position relative to a vehicle.

[0066] In some embodiments, the system may include a memory capable of storing multiple security vulnerability entries, each security vulnerability entry including a signature. Each signature may include structured data characterizing a potential security vulnerability. The system may also include a processor configured to receive a first security vulnerability entry, extract a first signature from the first security vulnerability entry, and generate a search query based on the first signature. According to one example, the search query may include at least one acceptable range based on at least a first data item in the first signature, and the processor may be configured to retrieve from the memory at least a second signature including at least a second data item within a predetermined range. In another example, the processor may be configured to retrieve any signature within the acceptable range.

[0067] In some embodiments, the disclosed systems and methods may select and / or suggest a response to a first security hazard based on the similarity between a signature associated with a first security hazard and a signature associated with a second security hazard, wherein the suggested response to the first security hazard is a response applied to the second security hazard. The response may be a simulated or virtual response, or an actual physical measure taken to attempt to mitigate or eliminate the hazard.

[0068] The system may include, for example, a memory and a processor. The memory may retain safety hazard entries. Optionally, the safety hazard entries may include signatures of detected hazards and / or issued alarms reported by in-vehicle smart cameras. For example, a signature may be associated with multiple hazard detections and / or multiple issued alarms, wherein the multiple detected hazards and / or issued alarms are determined to be associated with each other. The multiple hazard detections and / or multiple issued alarms associated with the signature may originate from multiple vehicles (each vehicle has an in-vehicle smart camera). For example, a detected hazard and / or issued alarm associated with a particular signature may be a hazard or alarm that has been determined to be associated with a specific cause or with multiple related causes. Optionally, some or all of the data may be structured so that it can be searched, evaluated, and co-processed with data from other safety hazard entries (e.g., combined, compared, etc.).

[0069] As a further example, the safety hazard entry may be associated with a specific response or with related responses. Optionally, a safety hazard entry may be associated with a single response, or in some cases, and according to some examples of the subject matter currently disclosed, a safety hazard entry may be associated with more than one response (e.g., with two, three, or more responses). Optionally, the response data may be structured so that it can be searched, evaluated, and co-processed with data from other safety hazard entries (e.g., combined, comparative, etc.). The response data may be related to changes taking place on and / or around the road. Optionally, in association with the response data, indications of the effectiveness of potential responses, as well as indications of possible other effects and trade-offs, such as: assuming any parking areas are removed to provide the response, assuming increased congestion as a result, assuming new hazards occur or existing hazards are exacerbated (and to what extent), etc.

[0070] Optionally, the report can be processed to determine the root cause or multiple causes and / or contributing factors or circumstances that contribute to the security hazard. In another example, the root cause or multiple causes and / or contributing factors or circumstances that contribute to the security hazard can be manually provided by the user. For example, the processor can be configured to provide a structured signature based on the processing of reports from in-vehicle smart cameras. According to examples in the currently disclosed subject matter, the reaction report can also be stored in memory. The reaction report can be associated with one or more security hazards. In one example, the reaction report can be associated with a security hazard signature, and therefore any security hazard entry whose signature matches or is sufficiently similar to the signature associated with the reaction report can be associated with the reaction report.

[0071] For example, by providing a signature of a security vulnerability to be resolved, a solution to the security vulnerability to be resolved can be provided or selected based on the similarity between its signature and the signatures of one or more previously resolved security vulnerabilities. The use of the term "previously resolved" should not be construed as limiting the examples in this disclosure to solutions that are necessarily successful in resolving the potential problem, and in many cases the solutions may be partially successful, and in other cases there may be compromises for a given solution, and in various cases the compromises may be more or less attractive or even unacceptable, and indeed in some examples of the subject matter disclosed herein, more than one solution can be returned for a given signature, and the processor can be configured to select from the multiple solutions returned for a given signature to provide one or more solutions based on compromises, or to select one or more solutions that do not contradict one or more constraints associated with the corresponding signature of the unresolved security vulnerability of the subject.

[0072] Various methods can be performed using a vehicle-mountable system to provide object detection near the vehicle. The system includes multiple cameras operatively attached to one or more processors. A right camera is externally mounted on the right rear side of the vehicle or facing the right rear side, and / or a left camera is externally mounted on the right rear side of the vehicle or facing the right rear side. The cameras' fields of view are substantially in the direction of the vehicle's travel. A right warning device may be mounted on the vehicle to the driver's right, and a left warning device may be mounted on the vehicle to the driver's left. The right and left warning devices may be connected to one or more processors.

[0073] Multiple image frames can be captured from the camera. The image frames can be processed to detect moving objects within a portion of the image frames. It can be determined that the potential collision between the vehicle and the moving object has a probability greater than a threshold, and in response to the probability of a potential collision with the moving object, a warning can be issued to the driver of the vehicle by emitting an audible warning or displaying a visual warning. When the detected object is viewed through the right camera, a warning is issued from a right warning device mounted on the driver's right side of the vehicle, while when the detected object is viewed through the left camera, a warning is issued from a left warning device mounted on the driver's left side of the vehicle.

[0074] The system may also include a central camera mounted in front of the driver and a central processing unit connected to the central camera, as well as a forward warning device connected to the central processing unit, which, if necessary, warns of an impending collision with a moving obstacle in the direction of travel of the vehicle.

[0075] The system may also include the use of other installed driver assistance systems, such as lane detection, structural barrier recognition, and / or traffic sign recognition, to identify stationary objects. If the other driver assistance systems determine that all objects detected near the vehicle are stationary, any warnings to the driver can be suppressed.

[0076] This paper provides various methods for using a vehicle-mountable system to provide object detection near a vehicle. The system includes a camera operatively attached to a processor. The camera is mounted at the rear of the vehicle. The camera's field of view is along the side of the vehicle, substantially in the direction of travel. Multiple image frames are captured from the camera. Intersections are detected when the vehicle turns. If the vehicle turns too sharply at the intersection, a potential intrusion of a vehicle onto the pedestrian crossing is determined by the vehicle at the intersection. The outline of the pedestrian crossing shoulder at the intersection is detected by using a motion recovery structure algorithm that processes image motion over time with image frames. Obstacles, such as objects or pedestrians, can be detected on the pedestrian crossing. If a bus is determined to be intruding onto the pedestrian crossing, a warning can be issued, and / or if the bus is determined not to be intruding onto the pedestrian crossing, the warning can be disabled.

[0077] The terms “sidewalk” and “road surface” are used interchangeably in this document. The terms “object” and “obstacle” are used interchangeably in this document. In the context of the rear of a vehicle, the term “rear” refers to the rear half of the vehicle near the back of the bus or the rear passenger door. The terms “right” and “left” are used from the driver’s perspective facing forward (essentially in the direction of travel). The term “corresponding” as used in this document refers to matching image points in different image frames that are found to be the same object point. The indefinite articles “one” and “a” are used in this document, and terms such as “one processor,” “one camera,” and “one image frame” have the meaning of “one or more,” i.e., “one or more processors,” “one or more cameras,” or “one or more image frames.”

[0078] While exemplary embodiments have been shown and described, it should be understood that these examples are not limiting. Rather, it should be understood that changes can be made to these embodiments, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for processing reports received from a vehicle, the method comprising: A first report is received from the first navigation system of the first vehicle, wherein the first report is generated by the first navigation system after detecting a first stationary object and a first non-stationary object in the environment of the first vehicle based on the analysis of image data collected by the first navigation system; A second report is received from the second navigation system of the second vehicle, wherein the second report is generated by the second navigation system after detecting a second stationary object and a second non-stationary object in the environment of the second vehicle based on the analysis of image data collected by the second navigation system; Analyze the first report and the second report to determine the common object of the first report and the second report; Integrate the first and second reports into a comprehensive report; and The comprehensive report is processed to identify causes associated with security risks, wherein the security risks are associated with the common object.

2. The method according to claim 1, further comprising: Based on the processing of the comprehensive report, a structured signature is provided to identify the cause.

3. The method according to claim 2, further comprising: Based on the processing of the comprehensive report, the reaction report associated with the structured signature is stored in memory.

4. The method according to claim 1, wherein, The cause is identified as being associated with a specific cause or with multiple related causes.

5. The method according to claim 1, wherein, The processed report includes one or more captured images associated with the detected object.

6. The method according to claim 1, wherein, The first stationary object and the second stationary object each include at least one of the following: a sidewalk, a lane marking, a traffic sign, a tree, a wall, and a pole.

7. The method according to claim 1, wherein, The first non-stationary object and the second non-stationary object each include at least one of the following: a cyclist, a motorcyclist, a vehicle, a pedestrian, a child riding a toy car, and a wheelchair user.

8. The method according to claim 1, wherein, The first report includes the first location of the first stationary object and the first non-stationary object.

9. The method according to claim 8, wherein, The first location is determined by the first navigation system based at least in part on the analysis of image data collected by the first navigation system.

10. The method according to claim 9, wherein, The first location is further determined based on GPS signals associated with the location of an alarm issued after the detection of the first stationary object and the first non-stationary object.

11. The method according to claim 1, wherein, The second report includes the second location of the second stationary object and the second non-stationary object.

12. The method according to claim 11, wherein, The second position is determined by the second navigation system based at least in part on the analysis of image data collected by the second navigation system.

13. The method according to claim 12, wherein, The second location is further determined based on GPS signals associated with the location of the alarm issued after the detection of the second stationary object and the second non-stationary object.

14. The method according to claim 1, wherein, The common objects identified in the first report and the second report include: determining that the environments of the first vehicle and the second vehicle have related layouts.

15. The method according to claim 14, wherein, Determining that the environments of the first vehicle and the second vehicle have related layouts includes comparing the differences between the layouts of the first environment and the layouts of the second environment with a threshold.

16. The method according to claim 15, wherein, The layout of the first environment is determined based on the position of the first stationary object, and the layout of the second environment is determined based on the position of the second stationary object.

17. A system for processing reports received from a vehicle, the system comprising a server, the server being operable to: A first report is received from the first navigation system of the first vehicle, wherein... The first report is generated by the first navigation system after detecting a first stationary object and a first non-stationary object in the environment of the first vehicle based on the analysis of image data collected by the first navigation system; A second report is received from the second navigation system of the second vehicle, wherein the second report is generated by the second navigation system after detecting a second stationary object and a second non-stationary object in the environment of the second vehicle based on the analysis of image data collected by the second navigation system; Analyze the first report and the second report to determine the common object of the first report and the second report; Integrate the first and second reports into a comprehensive report; and The comprehensive report is processed to identify causes associated with security risks, wherein the security risks are associated with the common object.

18. The system according to claim 17, wherein, The first stationary object and the second stationary object each include at least one of the following: a sidewalk, a lane marking, a traffic sign, a tree, a wall, and a pole.

19. The system according to claim 17, wherein, The first non-stationary object and the second non-stationary object each include at least one of the following: a cyclist, a motorcyclist, a vehicle, a pedestrian, a child riding a toy car, and a wheelchair user.

20. The system according to claim 17, wherein, The common objects identified in the first report and the second report include: determining that the environments of the first vehicle and the second vehicle have related layouts.

Images