Blind spot view enhancement for vehicles

Abstract

The invention relates to blind spot view enhancement for a vehicle. Systems and methods for vehicle blind spot object tracking are provided. Embodiments include performing object detection using a machine learning model to detect an object based on video data captured by one or more cameras associated with a vehicle; predicting an intent associated with the object based on one or more features associated with the object in the video data; applying a collision prediction algorithm based on the object, the intent associated with the object, and the one or more measured attributes of the vehicle in order to predict proximity between the vehicle and the object in a given direction; and after determining the intent to move the vehicle in the given direction, generating an alert for presentation within the vehicle based on the predicted proximity between the vehicle and the object.

Description

[0001] Related patent applications

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 651,794, filed on May 24, 2024, entitled “BLIND SPOT VIEW ENHANCEMENTS FORVEHICLES”. Background Technology

[0003] This disclosure relates to vehicles. More specifically, this disclosure relates to blind spot view enhancement for vehicles. Summary of the Invention

[0004] The embodiments disclosed herein advantageously provide systems and methods for enhancing blind spot views of vehicles.

[0005] In some implementations, a method for blind spot object tracking may include: performing object detection using a machine learning model based on video data captured by one or more cameras associated with the vehicle to detect objects; predicting an intent associated with the object based on one or more features associated with the object in the video data; applying a collision prediction algorithm based on the object, the intent associated with the object, and one or more measurement attributes of the vehicle to predict the proximity between the vehicle and the object in a given direction; and, after determining the intent to move the vehicle in the given direction, generating an alert to be displayed inside the vehicle based on the predicted proximity between the vehicle and the object. Attached Figure Description

[0006] Figure 1A and Figure 1B An illustration of an example vehicle according to an embodiment of this disclosure is depicted.

[0007] Figure 2 A block diagram of example components of a vehicle according to an embodiment of the present disclosure is presented.

[0008] Figure 3 A flowchart depicts the functionality associated with blind spot object tracking according to an embodiment of this disclosure.

[0009] Figure 4 Another flowchart depicts the functionality associated with blind spot object tracking according to an embodiment of this disclosure.

[0010] Figure 5 Another flowchart depicts the functionality associated with blind spot object tracking according to an embodiment of this disclosure.

[0011] Figure 6 Another flowchart depicts the functionality associated with blind spot object tracking according to an embodiment of this disclosure.

[0012] Figure 7 An example display screen associated with blind spot object tracking according to an embodiment of this disclosure is depicted.

[0013] Figure 8 Another example display screen associated with blind spot object tracking according to an embodiment of this disclosure is depicted. Detailed Implementation

[0014] Vehicles can provide object tracking and alerting capabilities to help drivers identify and avoid objects on or near the road, such as other vehicles, pedestrians, cyclists, animals, etc. In some cases, vehicles can provide this functionality, particularly for the driver's "blind spots," such as areas on or near the left and right rear sides of the vehicle.

[0015] According to the techniques described herein, enhanced blind spot object tracking can be performed using machine learning and / or other predictive algorithms. These algorithms enable the blind spot object detection system to consider not only the vehicle's current position and one or more detected external objects, but also the vehicle's predicted future position and one or more detected external objects. Through this predictive capability, aspects of this disclosure achieve an intelligent, dynamic, and proactive blind spot monitoring process, allowing alerts and / or other guidance related to detected objects to be generated and presented to the driver of the vehicle when, or even before, the driver signals an intention to move the vehicle in a given direction (e.g., change lanes).

[0016] In some implementation schemes, the following applies: Figure 3More specifically, a machine learning model already trained for object detection can be used to analyze video streams captured via the vehicle's left and / or right cameras. Multi-object tracking (MOT) algorithms can then be used to track objects detected using such a machine learning model to estimate their distance and trajectory. Furthermore, predictive algorithms can be used to predict the intent associated with the detected objects, such as whether the object is likely to move in a specific direction, at a specific speed, or in a specific location at a specific time. Additionally, data about the vehicle, such as its speed and / or route data (e.g., associated with a satellite-based navigation system), can be analyzed to predict whether the vehicle is likely to move in a specific direction, at a specific speed, or in a specific location at a specific time. A collision prediction engine can use these predictions about the vehicle and detected objects to determine whether the vehicle and detected objects are likely to be close to each other in a given direction, and this proximity prediction can be used to generate an alert to be displayed inside the vehicle. The alert may include, for example, displaying a camera view on the instrument cluster screen and / or using color, object outlines, visual indicators, text, sound, haptic feedback, etc., to provide the driver with information relevant to vehicle operation, such as potential lane changes. See below for reference. Figure 7 and Figure 8 An example describing this type of alert.

[0017] In some respects, as follows: Figure 4 More specifically, lane-specific location predictions can be generated by performing panoramic segmentation on images captured using vehicle cameras to detect lane boundaries and comparing the detected lane boundaries with segmentation masks associated with objects detected in the images. This lane-specific prediction can be combined with other blind-spot object tracking techniques to appropriately generate alerts. Furthermore, as discussed below... Figure 5 More specifically, in some implementations, images from one or more cameras (such as a rear-view camera of the vehicle) can be used to proactively predict the proximity between the vehicle and a detected object, to present a blind spot warning to the driver, for example, before the driver signals an intention to change lanes in a particular direction (e.g., by activating a turn signal).

[0018] The predictive capabilities, object tracking, and preemptive monitoring techniques described in this paper enable improved contextual awareness and sophisticated early warning capabilities, allowing vehicle drivers to receive timely alerts about the potential proximity of objects, particularly in the driver's blind spot.

[0019] Although the embodiments of this disclosure are presented with respect to electric vehicles, this disclosure is not limited to electric vehicles and can be incorporated into any combustion engine vehicle.

[0020] Figure 1AExample vehicle 100 is shown. (e.g.) Figure 1A As shown, vehicle 100 has multiple external cameras 102 and one or more front displays 104. Each of these external cameras 102 can capture a specific view or perspective of the exterior of vehicle 100. The images or videos captured by the external cameras 102 can then be displayed on one or more displays in vehicle 100, such as one or more front displays 104, for the driver to view.

[0021] refer to Figure 1B The vehicle 100 may include a chassis 106, which includes a frame 108 that provides the main structural components of the vehicle 100. The frame 108 may be formed by one or more beams or other structural components, or it may be integrated with the vehicle body (i.e., a monocoque construction).

[0022] In embodiments where vehicle 100 is a battery electric vehicle (BEV) or possibly a hybrid vehicle, a large battery 110 is mounted to the chassis 106 and may occupy a significant portion (e.g., at least 80%) of the area within the frame 108. For example, battery 110 may store 100 to 200 kWh. Battery 110 may be a lithium-ion battery or other types of rechargeable battery. The battery may be substantially planar in shape.

[0023] Power from battery 110 can be supplied to one or more drive units 112. Each drive unit 112 may consist of an electric motor and possibly a gear train providing gear reduction. In some embodiments, a single drive unit 112 drives either the front or rear wheels of vehicle 100. In another embodiment, two drive units 112 are present, each driving either the front or rear wheels of vehicle 100. In yet another embodiment, four drive units 112 are present, each driving one of the four wheels of vehicle 100.

[0024] Power from battery 110 can be supplied to drive unit 112 by power electronics 114 in each drive unit 112. Power electronics 114 may include inverters configured to convert direct current (DC) from battery 110 into alternating current (AC) supplied to the motor of drive unit 112. Power electronics 114 also facilitate the use of the drive unit's motor as a generator to provide regenerative braking. Power electronics 114 also facilitate the transfer of regenerative current to battery 110.

[0025] A drive unit 112 is coupled to two or more wheel hubs 116 to which wheels can be mounted. Each wheel hub 116 includes a corresponding brake 118, such as a disc brake as illustrated. Each wheel hub 116 is further coupled to a frame 108 via a suspension 120. The suspension 120 may include metal or pneumatic springs for absorbing shocks. The suspension 120 may be implemented as a pneumatic or hydraulic suspension capable of adjusting the ground clearance of the chassis 106 relative to a supporting surface. The suspension 120 may include a damper, wherein the characteristics of the damper are fixed or electronically adjustable.

[0026] exist Figure 1B In the implementation scheme and in the discussion below, vehicle 100 is a battery electric vehicle. However, the systems and methods disclosed herein can be used in any type of vehicle, including vehicles powered by an internal combustion engine (ICE), a hybrid powertrain, a hydrogen fuel cell powertrain, or other types of powertrains that may have a portion that idles during some operating modes. For example, the front or rear differential of an all-wheel-drive vehicle. In another example, in a hybrid powertrain, according to the methods described herein, the idle drive unit including the electric motor can be heated using waste heat from the ICE.

[0027] Figure 2 Examples Figure 1A Example components of vehicle 100. (e.g.) Figure 2 As shown, vehicle 100 includes a camera 102, one or more front displays 104, a user interface 200, one or more sensors 202, a motion sensor 204, and a positioning system 206. The one or more sensors 202 may include ultrasonic sensors, radio detection and ranging (RADAR) sensors, light detection and ranging (LIDAR) sensors, or other types of sensors. The positioning system 206 may be implemented as a Global Positioning System (GPS) receiver. The user interface 200 allows a user (such as a driver or passenger in vehicle 100) to provide input.

[0028] The components of vehicle 100 may include one or more temperature sensors 208. Temperature sensors 208 may include sensors configured to sense ambient air temperature, battery 110 temperature, power electronics 114 temperature, temperature of each drive unit 112 and / or each motor of each drive unit 112 temperature, temperature of coolant fluid entering or leaving the coolant system, oil temperature within drive unit 112 temperature, or temperature of any other component of vehicle 100.

[0029] The vehicle 100 may include a friction braking system 210. The friction braking system 210 may include any components of a hydraulic braking system, such as a rotor, brake pads, calipers, caliper pistons, and a master cylinder connected to the caliper pistons and the brake pedal via brake lines. The friction braking system 210 may further include a pump and / or valves for automatically applying hydraulic pressure to the caliper pistons. The friction braking system 210 may be implemented as a drum braking system or any friction braking system known in the art.

[0030] Control system 214 executes instructions to perform at least some of the actions or functions of vehicle 100, including those related to... Figures 3 to 6 The described function. For example, such as Figure 2 As shown, the control system 214 may include one or more electronic control units (ECUs) configured to perform at least some of the actions or functions of the vehicle 100, including regarding... Figures 3 to 6 The functions described. In some implementations, each ECU in the ECU is dedicated to a specific set of functions. Each ECU may be a computer system, and each ECU may include features referenced below. Figures 3 to 6 The functionality described.

[0031] Certain features of the implementation scheme described herein can be controlled by a telematics control module (TCM) ECU. The TCM ECU can provide a wireless vehicle communication gateway to support functionalities such as (by way of example and not limitation) over-the-air (OTA) software updates, vehicle-to-internet communication, vehicle-to-computing device communication, in-vehicle navigation, vehicle-to-vehicle communication, vehicle-to-landscape features (e.g., automatic toll collection sensors, automatic toll booths, power distributors at charging stations), or automatic calling functionality.

[0032] Certain features of the implementation described herein can be controlled by a Central Gateway Module (CGM) ECU. The CGM ECU serves as the vehicle's communication hub, connecting various ECUs, sensors, cameras, microphones, motors, displays, and other vehicle components, and transmitting data to and from these components. The CGM ECU may include a network switch providing connectivity via a Controller Area Network (CAN) port, a Local Interconnect Network (LIN) port, and an Ethernet port. The CGM ECU can also function as the master controller for different vehicle modes (e.g., road driving mode, parking mode, off-road mode, towing mode, camping mode), thereby controlling certain vehicle components associated with placing the vehicle in one of these vehicle modes.

[0033] In various implementations, the CGM ECU collects sensor signals from one or more sensors of the vehicle 100. For example, the CGM ECU may collect data from camera 102, sensor 202, motion sensor 204, positioning system 206, and temperature sensor 208. The sensor signals collected by the CGM ECU are then transmitted to the appropriate ECU to perform, for example, reference... Figures 3 to 6 Describe the operations and functions.

[0034] The control system 214 may also include one or more additional ECUs, such as (by way of example and not limitation): Vehicle Dynamics Module (VDM) ECU, Experience Management Module (XMM) ECU, Vehicle Entry / Exit System (VAS) ECU, Near Field Communication (NFC) ECU, Body Control Module (BCM) ECU, Seat Control Module (SCM) ECU, Door Control Module (DCM) ECU, Rear Zone Control (RZC) ECU, Autonomous Control Module (ACM) ECU, Autonomous Safety Module (ASM) ECU, Driver Monitoring System (DMS) ECU, and / or Winch Control Module (WCM) ECU.

[0035] If vehicle 100 is an electric vehicle, one or more ECUs may provide functionality related to the vehicle's battery pack, such as a Battery Management System (BMS) ECU, a Battery Power Isolation (BPI) ECU, a Balanced Voltage and Temperature (BVT) ECU, and / or a Thermal Management Module (TMM) ECU. In various implementations, the XMM ECU sends data to the TCM ECU (e.g., via Ethernet, etc.). Additionally or alternatively, the XMM ECU may send other data (e.g., audio data from microphone 216, etc.) to the TCM ECU.

[0036] The ECU may include one or more ECUs configured to control the friction braking system 210. For example, the ECU may include a traction control module, a stability control system, an automatic emergency braking (AEB) module, an anti-lock braking system (ABS), an adaptive cruise control (ACC) module, and / or an advanced driver assistance system (ADAS). The traction control module controls braking and acceleration to control wheel slip according to any method known in the art. The traction control module may also control the torque applied to each wheel, i.e., torque vectoring control. According to any method known in the art, the stability control system controls braking and acceleration to prevent vehicle 100 from rolling over. According to any method known in the art, the AEB module stops vehicle 100 in a controlled manner in response to a predicted collision. The ABS adjusts the braking to maintain traction. The ACC maintains the vehicle's speed while also maintaining a prescribed following distance relative to other vehicles. The ADAS controls the steering, acceleration, and braking of vehicle 100 according to any autonomous driving method known in the art to reach the destination.

[0037] Figure 3 A flowchart 300 depicts a representation of a function associated with blind spot object tracking according to an embodiment of this disclosure.

[0038] In some implementations, video stream 302 indicates via Figure 1A and Figure 2 Video data captured by one or more cameras in camera 102 (e.g., including one or more series of images). For example, video stream 302 may include video data captured via... Figures 1A to 2 Video data captured by the left-side camera 102 and right-side camera 102 of vehicle 100.

[0039] Object detection 304 can be performed on video stream 302 to detect one or more objects present in video stream 302. For example, object detection 304 may involve using a machine learning model that has been trained for object detection. In one implementation, object detection 304 involves using data from... The YOLOv8 object detection model is used. In other implementations, any suitable object detection model can be used at object detection 304, such as a neural network trained for a computer vision task (e.g., a deep learning model). In one example, object detection 304 involves feeding an image from video stream 302 as input to a machine learning model, and in response to the input, receiving indications of one or more objects present in one or more images (such as bounding boxes and / or category labels of such one or more objects) as output from the machine learning model.

[0040] One or more objects detected at object detection 304 can be tracked at multi-object tracking (MOT) 306. For example, MOT 306 may involve performing a simple online real-time tracking (SORT) or DeepSORT algorithm, or another suitable MOT algorithm. MOT 306 typically involves tracking one or more detected objects over time. For example, MOT 306 may accept a list of detected objects associated with confidence scores and bounding boxes output by object detection 304, and may output information about all currently "tracked" objects (such as identifiers of unique objects detected across multiple image frames), such that the progress of each unique object is tracked in association with an assigned identifier across multiple image frames. At trajectory and distance estimation 310, the corner radar sensor from the vehicle (e.g., Figure 2Signals from one or more sensors in sensor 202 can be fused with camera input (e.g., represented by the output of MOT 306, which is based on video stream 302) to generate a distance and trajectory vector for each detected object. For example, the distance vector of the tracked object output by trajectory and distance estimation 310 can indicate the estimated distance between the tracked object and the vehicle, and the trajectory vector output by trajectory and distance estimation 310 can indicate the estimated trajectory of the tracked object.

[0041] The driver intent prediction 308 can predict the intent associated with an object detected at object detection 304. For example, driver intent prediction 308 can involve analyzing visual signals associated with one or more objects detected at object detection 304, such as turn signals, hazard lights, object poses, etc., to predict the likely location and when such movement might occur for a given object (e.g., vehicle, cyclist, pedestrian, animal, etc.). For example, driver intent prediction 308 can involve using machine learning models, such as computer vision models, time series models, barrier functions, deep learning models, etc., and such machine learning models can output indications of intent associated with a given detected object (e.g., whether the object is likely to move into a specific lane, accelerate, decelerate, turn, etc.), such as based on visual indicators present in the image of the detected object and / or based on location, speed, direction, and / or one or more other detected attributes associated with the detected object. It should be noted that although the term "driver intent" is used, driver intent prediction 308 can also involve predicting the intent of objects that are not necessarily controlled by a driver, such as pedestrians, animals, autonomous vehicles, etc.

[0042] At collision prediction 312, the outputs of trajectory and distance estimation 310 and / or driver intent prediction 308 can be used with vehicle-related information (e.g., vehicle data 314 and real-time route information 316) to determine the probability of an approach occurring between the vehicle and each detected object, such as in a given direction. Vehicle data 314 may include, for example, the vehicle's speed, steering wheel angle, acceleration, direction, etc. (e.g., it may be transmitted via one or more sensors associated with the vehicle, such as...) Figure 2 (Captured by one or more sensors in sensor 202). Real-time route information 316 may include, for example, information about the vehicle's route, such as from a satellite-based navigation system associated with the vehicle, such as... Figure 2 The positioning system 206 (e.g., the route can be based on input from the driver, such as via...) Figure 2The user interface 200 configures the vehicle's destination and is based on the detected vehicle's position relative to the route. Collision prediction 312 may involve logic and / or one or more machine learning models, which can be used to analyze outputs from trajectory and distance estimation 310, driver intent prediction 308, vehicle data 314, and real-time route information 316 to predict whether a detected object is now or likely (e.g., at a given time) to be within a given distance of the vehicle in a given direction. For example, collision prediction 312 may involve predicting whether a collision or near-collision is possible between the vehicle and a detected object if the vehicle moves into a specific lane within a given amount of time. In some implementations, collision prediction 312 is based on monitoring the movement of the detected object using optical flow, block matching, and / or Kalman filtering techniques relative to video data (e.g., at MOT 306, trajectory and distance estimation 310, and / or collision prediction 312).

[0043] The rendering engine 318 can generate one or more alerts based on the output from the collision prediction 312 to be rendered within the vehicle. For example, the rendering engine 318 can generate an alert indicating the predicted proximity (or probability of collision) between the vehicle and a given detected object in a given direction. Such alerts can be presented in… Figure 2 The blind spot view is provided to the user within the user interface 200, such as when displayed on the dashboard inside the vehicle. For example, a live view from a side camera of the vehicle can be displayed in the direction in which the driver has signaled an intention to move (or the driver has been predicted to move). In some implementations, one or more detected objects predicted to be of interest (e.g., due to proximity and / or the possibility of a collision) can be highlighted (e.g., with a specific color, such as a bounding box applied to a given object) and / or otherwise indicated within the display (e.g., within a live camera view). In one example, the bounding box of a detected object (e.g., with a given color (e.g., red) and / or with other characteristics (e.g., flashing)) is overlaid on the live camera view displayed to the user to draw the object's attention to the driver. In other examples, the alert may involve the use of color, text, graphics, visual indicators, sound, haptic feedback, etc. See below for reference. Figure 7 and Figure 8 Example of describing such an alarm.

[0044] It should be noted that the process described in flowchart 300 can be performed on a continuous basis, at regular intervals, and / or when one or more conditions occur (e.g., when an intention to change lanes or turn is predicted), and not necessarily only when the driver of the vehicle activates a turn signal in a given direction. For example, vehicle data 314 and / or real-time route information 316 can be used to predict whether the driver of the vehicle is likely to change lanes or turn, even if the driver has not yet activated a turn signal. Therefore, in some embodiments, an alert can be proactively presented to the driver, such as before the driver activates a turn signal, immediately after the driver activates a turn signal, and / or immediately after a potential approach or collision probability is predicted.

[0045] Figure 4 Another flowchart 400 depicts the functionality associated with blind spot object tracking according to an embodiment of this disclosure. For example, the functionality described with respect to flowchart 400 may be combined with or independent of the functionality described above. Figure 3 The function described in flowchart 300 is executed to determine whether to generate an alert related to the detected object.

[0046] In some implementations, the functionality associated with blind spot object tracking may include function blocks 402, 404, 406, 408, 410, 412, 414, 416, 418, 420, 422, and 424. In other implementations, these function blocks may be arranged in a different order, some function blocks may be omitted, and other function blocks may be added, etc.

[0047] At point 402, a user (e.g., a driver) can activate a turn signal. For example, a driver can activate a left turn or right turn signal. Alternatively, the driver's intention to change lanes or turn can be predicted (e.g., based on vehicle data and / or route data).

[0048] At point 404, camera frames can be retrieved from the direction associated with the turn signal. For example, this could be done from one or more cameras on one side of the vehicle (e.g., Figure 1A The image frame of the camera 102) corresponds to the direction of the activated turn signal.

[0049] At position 406, a sliding window of frames (e.g., camera frames acquired at position 404) can be fed into a convolutional neural network (CNN)-based model and / or a recurrent neural network (RNN)-based model to detect one or more lanes. For example, an ensemble model including CNNs and RNNs may have been trained via a supervised learning process in response to a sliding window provided with camera frames (e.g., via...). Figure 1AOne or more cameras 102 capture images and output indications of one or more lanes (e.g., bounding boxes for each detected lane). CNNs can be used to detect objects (e.g., lanes) in image frames, and RNNs can be used to perform sequence analysis of detected objects across multiple consecutive image frames.

[0050] At position 408, a frame (e.g., a camera frame acquired at position 404) can be fed into a region-based CNN (R-CNN) model, such as Mask R-CNN, to perform panoptic segmentation. R-CNNs are typically trained to perform a two-step object detection process involving identifying regions in an image that may contain objects and classifying the objects based on features extracted from those regions. Mask R-CNN extends the concept of R-CNN by detecting objects in an image while simultaneously generating a high-quality segmentation mask for each instance, such as by including a branch for predicting object masks that runs in parallel with an existing branch for bounding box recognition (present in standard R-CNNs). Panoptic segmentation involves identifying the category to which each pixel in an image belongs (e.g., trackable objects like cars or people might be one category, while untrackable objects like the sky or sidewalks might be another), while distinguishing different instances of the same category. For example, panoptic segmentation can determine whether each pixel belongs to one category or another based on the output from an R-CNN (such as Mask R-CNN), such as using one or more object bounding boxes and / or segmentation masks determined by such a model.

[0051] At 410, extract the vehicle (and / or other object), bounding box, and segmentation mask. For example, the results of analyzing the camera frame using an R-CNN such as Mask R-CNN and / or performing panoramic segmentation at 408 can be used to identify one or more vehicles (and / or other objects), bounding boxes, and segmentation masks from the camera frame.

[0052] At 412, it is determined whether a vehicle was detected. For example, if the results of 408 and / or 410 indicate that a vehicle (and / or other object) was detected in one or more camera frames, the operation can proceed to 414, where the intersection-over-union ratio (IoU) between the segmentation mask and the lane boundary is calculated. If the results of 408 and / or 410 indicate that no vehicle (and / or other object) was detected in one or more camera frames, the process can end (or return to 402 and / or another function block or process).

[0053] At 414, the IoU between the segmentation mask (e.g., extracted at 408 and / or 410) and the lane boundary (e.g., detected at 406) is calculated. IoU is a measure of overlap between two boxes or regions, such that a higher amount of overlap results in a higher IoU value, and a lower amount of overlap results in a lower IoU value. Typically, the IoU between two boxes is equal to the intersection area between the two boxes divided by the union area of ​​the two boxes.

[0054] At 416, it is determined whether the IoU calculated at 414 exceeds a threshold. For example, the threshold can be a configurable value and can represent a value such that, above which the IoU between the bounding box of the detected object and the lane boundary might indicate that the detected object is within the lane represented by the lane boundary.

[0055] At point 418, the blind spot monitoring (BSM) process was determined (e.g., as mentioned above). Figure 3 The process described in flowchart 300 (and / or another similar process) determines whether a vehicle (and / or another object) is detected. If the IoU exceeds a threshold at 416 and a vehicle or other object is detected by the BSM process at 418, the operation proceeds to 422, where an alarm is generated, such as highlighting the vehicle or other object (e.g., in a live camera view displayed inside the vehicle) and / or otherwise informing the driver that changing lanes may be dangerous and / or drawing the driver's attention to the detected vehicle or other object. The operation may also proceed to 422 based on some other combination of the results of 416 and 418, for example, if the IoU exceeds the threshold and the BSM does not detect a vehicle, or if the BSM detects a vehicle and the IoU does not exceed the threshold.

[0056] If it is determined at 418 that the BSM has not detected a vehicle, the operation can proceed to 420, where it is determined whether a BSM error has been detected and the IoU exceeds a threshold (e.g., at 416). If both conditions are true at 420, the operation can proceed to 424, where an alert is generated, such as highlighting a vehicle or other object (e.g., in a live camera view displayed inside the vehicle) and / or otherwise informing the driver that changing lanes may be dangerous and / or drawing the driver's attention to detected vehicles or other objects. A BSM error can be detected if, for example, the BSM process returns an error condition, such as if the IoU exceeds a given threshold (e.g., indicating a high confidence level) while the BSM indicates that no object has been detected.

[0057] In some cases, an alarm may not be generated if the IoU does not exceed a threshold and / or if the BSM does not detect a vehicle or other object, and / or if a certain confidence level associated with one or both determinations does not exceed a threshold. It should be understood that various techniques can be used to utilize the results of comparing the IoU with a threshold and determining whether the BSM detects a vehicle or other object to determine whether an alarm should be generated. More generally, according to certain implementations, such as those described above... Figure 3 The BSM process described and such as about Figure 4 The calculated IoU between the segmentation mask and the lane boundary can be used to determine whether to generate an alert related to the detected object.

[0058] Figure 5 Another flowchart 500 depicts a representation of a function associated with blind spot object tracking according to an embodiment of this disclosure. For example, flowchart 500 could represent the functionality described above regarding... Figure 3 and Figure 4 The described combination of functions and / or independent of the above regarding Figure 3 and Figure 4 The described function performs functions such as using one or more rear-facing cameras of the vehicle (e.g., Figure 1A Video data captured by one of the cameras in camera 102.

[0059] At the pedestrian / cyclist behavior analysis section 510, functions related to analyzing the behavior of objects such as pedestrians and / or cyclists can be performed. For example, pedestrian movement analysis 512 may involve determining the movement of one or more pedestrians and / or cyclists detected in camera frames captured via a rear-view camera from a vehicle (e.g., based on the use of one or more object detection machine learning models, as described above). Figure 3 or Figure 4 The model described. Applications of correlating Global Positioning System (GPS) and map data in pedestrian crossings and traffic lights 514 may involve correlating route and / or other location / map / GPS data (e.g., indicating the presence of pedestrian crossings and / or traffic lights) with objects detected in camera frames, such as pedestrians (e.g., the use of machine learning models based on one or more object detection, as described above). Figure 3 or Figure 4 The described model can be used to, for example, determine the presence and / or location of one or more crosswalks and / or traffic lights relative to one or more objects (such as pedestrians). Cyclist trajectory analysis and its association with map data of bike lanes 516 can involve analyzing the trajectories of one or more cyclists detected in camera frames (e.g., based on machine learning models using one or more object detection, such as those described above). Figure 3 or Figure 4 The model is described, and it is determined whether such a trajectory corresponds to location / map / GPS data indicating the presence and / or location of one or more bike lanes.

[0060] More generally, pedestrian / cyclist behavior analysis 510 may involve object detection / tracking based on images and / or location / GPS / map data to determine the location where objects such as pedestrians and / or cyclists may move to and / or the time when such movement may occur relative to vehicles and / or relative to crosswalks, traffic lights and / or bike lanes.

[0061] At driver intent prediction 520, functions related to analyzing the behavior of objects such as vehicles can be performed to predict the driver's intent. For example, the association application 522 of turning signals and vehicle direction prediction could involve making turning signals detected on the vehicle in a camera frame (e.g., based on the use of one or more object detection machine learning models, as mentioned above) Figure 3 or Figure 4 The described model is correlated with the predicted direction of the vehicle to, for example, determine whether the predicted direction is consistent with a detected turning signal. Behavioral analysis of vehicle type and trajectory 524 can involve analyzing the behavior of vehicles detected in camera frames based on the detected vehicle type (e.g., delivery vehicle, postal vehicle, taxi or ride-sharing vehicle, emergency vehicle, etc.) and the vehicle's trajectory (e.g., based on using one or more object detection machine learning models, such as those described above). Figure 3 or Figure 4 The described model, such as determining the location a vehicle might move to and / or the time when such movement might occur based on trajectories and expected behavior associated with a specific vehicle type. Speed ​​analysis 526 may involve analyzing the detection speed of a vehicle detected in a camera frame (e.g., based on the use of one or more object detection machine learning models, such as those described above). Figure 3 or Figure 4 The described model, such as determining how fast a detected vehicle is moving based on the position of the detected vehicle on multiple camera frames and / or based on other information (such as radar information captured via one or more sensors associated with the vehicle).

[0062] More generally, driver intent prediction 520 may involve object detection / tracking in an image (e.g., detecting vehicles and vehicle attributes such as turn signals, direction, trajectory, vehicle type, etc.) and / or determining the position of a vehicle detected in a camera frame relative to the vehicle's possible movement and / or the time when such movement may occur, based on data captured using other sensors.

[0063] In cross-correlation to determine the impact on the vehicle's trajectory (straight line or lane change) and to determine which side view would be most helpful 530, the results of pedestrian / cyclist behavior analysis 510 and driver intent prediction 520 can be correlated with the vehicle's trajectory (and / or the predicted location of the vehicle's movement, such as based on vehicle data and / or route data, as described above). It can be determined whether any detected object (e.g., pedestrian, cyclist, vehicle, etc.) is likely to be located in or near the location the vehicle might move to (e.g., in a given direction), and this determination can be used to determine whether showing the driver a left-side camera view or a right-side camera view would be helpful. For example, if it is predicted that the vehicle will move one lane to the right in the near future, and it has already been predicted (e.g., at driver intent prediction 520) that the detected vehicle is within a given distance of the location the vehicle might move to in the adjacent lane to the right of the vehicle, then (e.g., via...) Figure 2 The user interface 200 and / or on the dashboard actively displays a view of the right-side camera inside the vehicle, for example, before the driver activates the right turn signal.

[0064] When displaying the left or right blind spot view based on decision 540, a left camera view or a right camera view (e.g., a live feed from the left or right camera) can be displayed inside the vehicle, such as to warn the driver of the presence of an object in the driver's blind spot. As described above, the alert can include using overlays, colors, text, visual indicators, flashing visual content, audio, haptic feedback, etc., to warn the driver of the presence of a detected object predicted to be located in or near a position where the vehicle is predicted to move (e.g., within a threshold distance). See below for reference. Figure 7 and Figure 8 Example of describing such an alarm.

[0065] Figure 6 A method 600 is described that represents a function associated with blind spot object tracking according to an embodiment of the present disclosure. For example, method 600 may be provided by... Figure 1A One or more components of vehicle 100 (such as Figure 2 The control system 214 is executed by one or more ECUs.

[0066] Method 600 may include, at 610, using a machine learning model to perform object detection based on video data captured by one or more cameras associated with the vehicle, in order to detect objects.

[0067] Method 600 may include, at 620, predicting an intent associated with an object based on one or more features associated with an object in the video data.

[0068] Method 600 may include, at 630, applying a collision prediction algorithm based on the object, the intent associated with the object, and one or more measured attributes of the vehicle, in order to predict the proximity between the vehicle and the object in a given direction.

[0069] Method 600 may include, at 640, generating an alarm for presentation inside the vehicle based on a predicted proximity between the vehicle and an object, after determining the intention to move the vehicle in a given direction.

[0070] In some implementations, the collision prediction algorithm is further based on the vehicle's route data, and the vehicle's route data is based on a configured route associated with a satellite-based navigation system.

[0071] Some implementations further include estimating the distance associated with the object and its trajectory relative to the vehicle based on performing multi-object tracking (MOT) using computer vision techniques after the object is detected, wherein a collision prediction algorithm is applied based on the distance and trajectory.

[0072] In some implementations, one or more measurement attributes include one or more of the vehicle's speed, the vehicle's steering wheel angle, or the vehicle's direction.

[0073] In some implementations, collision prediction algorithms are applied to further monitor object movement based on optical flow, block matching, or Kalman filtering techniques performed on video data.

[0074] Some implementations further include: providing a sliding window of frames from video data as input to a convolutional neural network (CNN) and a recurrent neural network (RNN) to detect one or more driving lanes; extracting a segmentation mask of an object from the frame; and comparing the boundaries of one or more driving lanes with the segmentation mask of the object to predict whether the object is in the adjacent lane of the vehicle, wherein generating an alert is further based on the prediction that the object is in the adjacent lane of the vehicle.

[0075] In some implementations, comparing the boundaries of one or more lanes with a segmentation mask of an object includes calculating the intersection-over-union (IoU) ratio of the boundaries of one or more lanes with the segmentation mask of the object, and comparing the IoU with a threshold.

[0076] Some implementations further include analyzing data captured using a rear-facing camera associated with the vehicle to predict the trajectory of an object, wherein an alert is generated based on the predicted trajectory of the object.

[0077] In some implementations, the driver's intention to move the vehicle in a given direction is determined based on one or more of the following: activation of the vehicle's turn signal; the vehicle's route data.

[0078] In some implementations, generating an alarm includes generating a graphical indicator of the object to be displayed via a screen inside the vehicle.

[0079] Figure 7 An example display screen 700 associated with blind spot object tracking according to an embodiment of this disclosure is depicted. For example, display screen 700 may be in Figure 1A Front display 104 and / or Figure 2 User interface 200 and / or Figure 1A An example of a display screen shown on the dashboard of vehicle 100, including a right-side blind spot view. For example, display screen 700 may include alerts related to detected objects as described above, such as those displayed in response to determining that a detected object is predicted to be in or near a location where the vehicle is predicted to move (e.g., due to a predicted and / or signaled lane change).

[0080] Display 700 includes a camera view 710, which includes a rear-facing camera from the right side of the vehicle (e.g., Figure 1A The camera view 710 may be determined to be displayed based on real-time feeds from one of the cameras in camera 102. For example, it may be determined to display the camera view 710 based on the determination that the vehicle may move to the right or one or more lanes (e.g., based on the driver activating a right turn signal and / or based on vehicle data and / or route data).

[0081] The display screen 700 further includes a graphic depiction 702 of the vehicle and a graphic depiction 706 of an object (in this case, a motorcycle) detected to the right of the vehicle and slightly rearward (e.g., in the driver's right blind spot). A graphic indicator 704 may be displayed to warn the driver of the presence of the object represented by the graphic depiction 706, such as including a specific color (e.g., red), shape (e.g., arc), movement (e.g., flashing), etc., for example, near the graphic depiction 706 of the detected object displayed on the display screen 700. For example, the graphic indicator 704 may be displayed based on an automatic determination (e.g., using one or more of the techniques described above) that the detected object represented by the graphic depiction 706 is now or may soon be within a threshold distance of the location where the vehicle is predicted to move (e.g., one lane to the right).

[0082] Figure 8 Another example display screen 800 associated with blind spot object tracking according to an embodiment of this disclosure is depicted. For example, display screen 780 may be in Figure 1A Front display 104 and / or Figure 2 User interface 200 and / or Figure 1AAn example of a display screen shown on the dashboard of vehicle 100, including a left blind spot view. For example, display screen 800 may include alerts related to detected objects as described above, such as those displayed in response to determining that a detected object is predicted to be in or near a location where the vehicle is predicted to move (e.g., due to a predicted and / or signaled lane change).

[0083] Display 800 includes a camera view 810, which includes a view from the vehicle's left-side rear-facing camera (e.g., Figure 1A The camera view 810 may be determined to be displayed based on real-time feeds from one of the cameras in camera 102. For example, based on the determination that the vehicle may move to the left by one or more lanes (e.g., based on the driver activating a left turn signal and / or based on vehicle data and / or route data).

[0084] The display screen 800 further includes a graphic depiction 802 of the vehicle and a graphic depiction 806 of an object (in this case, a different vehicle) detected to the left and slightly rear of the vehicle (e.g., in the driver's left blind spot). A graphic indicator 804 may be displayed to warn the driver of the presence of the object represented by the graphic depiction 806, such as including a specific color (e.g., red), shape (e.g., arc), movement (e.g., flashing), etc., for example, near the graphic depiction 806 of the detected object displayed on the display screen 800. For example, the graphic indicator 804 may be displayed based on an automatic determination (e.g., using one or more of the techniques described above) that the detected object represented by the graphic depiction 806 is now or may soon be within a threshold distance of the location where the vehicle is predicted to move (e.g., one lane to the left).

[0085] Figure 7 and Figure 8 The display screen depicted herein is included as an example, and according to the techniques described herein, many other types of displays and / or other forms of alerts can be used to warn a driver of the presence of a detected object in a given direction (such as a lane to the left or right of the vehicle).

[0086] Various embodiments of this disclosure have been described for illustrative purposes. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to explain the principles of the embodiments, their practical application, or technical improvements to technologies found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0087] In the foregoing, reference has been made to the embodiments presented in this disclosure. However, the scope of this disclosure extends beyond the specifically described embodiments. Rather, any combination of features and elements is contemplated for implementing and practicing the intended embodiments, regardless of whether different embodiments are involved. Furthermore, while the embodiments disclosed herein may achieve advantages over other possible solutions or over the prior art, the embodiments may achieve some advantages or no particular advantages. Therefore, the aspects, features, embodiments, and advantages discussed herein are merely illustrative.

[0088] The various aspects of this disclosure may take the form of a completely hardware implementation, a completely software implementation (including firmware, resident software, microcode, etc.), or a combination of software and hardware implementations, all of which may be collectively referred to herein as “circuit,” “module,” or “system.”

[0089] Various aspects of this disclosure are described by narrative text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in implementations of a computer program product (CPP). Regarding any flowchart, depending on the technology involved, operations may be performed in a different order than that shown in a given flowchart. For example, again depending on the technology involved, two operations shown in consecutive flowchart frames may be performed in reverse order, as a single integrated step, concurrently, or in a manner that at least partially overlaps in time.

[0090] A Computer Program Product Implementation (“CPP Implementation” or “CPP”) is a term used in this disclosure to describe any set of one or more storage media (also referred to as “medium”) collectively included in a set of one or more storage devices, which collectively include machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A “storage device” is any tangible device capable of holding and storing instructions for use by one or more computer processing devices. Without limitation, a computer-readable storage medium can be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Specific types of storage devices including these media include: magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punched cards or pits / platforms formed in the main surface of the disk), or any suitable combination of the foregoing. As used in this disclosure, computer-readable storage medium refers to a non-transitory storage device rather than the transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, optical pulses transmitted through fiber optic cables, and electrical signals transmitted through wires and / or other transmitting media. As those skilled in the art will understand, data typically moves at some incidental points in time during the normal operation of the storage device (such as during access, defragmentation, or garbage collection), but the storage device remains non-transitory during these processes because the data remains non-transitory while stored.

[0091] While the foregoing relates to embodiments of this disclosure, other and further embodiments may be devised without departing from the basic scope of this disclosure, the scope of which is defined by the appended claims.

Claims

1. A method for tracking objects in a vehicle's blind spot, the method comprising: Based on video data captured by one or more cameras associated with the vehicle, a machine learning model is used to perform object detection in order to detect objects; Predict the intent associated with the object based on one or more features associated with the object in the video data; A collision prediction algorithm is applied based on the object, the intent associated with the object, and one or more measured attributes of the vehicle in order to predict the proximity between the vehicle and the object in a given direction. as well as After determining the intention to move the vehicle in the given direction, an alarm is generated based on the predicted proximity between the vehicle and the object to be displayed inside the vehicle.

2. The method according to claim 1, wherein, The collision prediction algorithm is further applied based on the vehicle's route data, wherein the vehicle's route data is based on a configured route associated with a satellite-based navigation system.

3. The method according to claim 1, wherein the method further comprises: After the object is detected, the distance and trajectory associated with the object relative to the vehicle are estimated based on multi-object tracking (MOT) performed using computer vision techniques, wherein the collision prediction algorithm is applied based on the distance and the trajectory.

4. The method according to claim 3, wherein, The one or more measurement attributes include one or more of the vehicle's speed, the vehicle's steering wheel angle, or the vehicle's direction of travel.

5. The method according to claim 4, wherein, The collision prediction algorithm is further applied to perform optical flow, block matching, or Kalman filtering techniques on the video data to monitor the movement of the object.

6. The method according to claim 1, wherein the method further comprises: A sliding window of frames from the video data is provided as input to a convolutional neural network (CNN) and a recurrent neural network (RNN) to detect one or more driving lanes; Extract the segmentation mask of the object from the frame; as well as The boundaries of the one or more driving lanes are compared with the segmentation mask of the object to predict whether the object is in the adjacent lane of the vehicle, wherein the alarm is generated further based on the prediction that the object is in the adjacent lane of the vehicle.

7. The method according to claim 6, wherein, Comparing the boundaries of the one or more lanes with the segmentation mask of the object includes: calculating the intersection-over-union (IoU) ratio of the boundaries of the one or more lanes with the segmentation mask of the object, and comparing the IoU with a threshold.

8. The method of claim 1, the method further comprising analyzing data captured using a rear-facing camera associated with the vehicle to predict the trajectory of the object, wherein generating the alert is further based on the predicted trajectory of the object.

9. The method according to claim 1, wherein, The determination of the intention to move the vehicle in the given direction is based on one or more of the following: The activation of the vehicle's turning signal; or The vehicle's route data.

10. The method according to claim 1, wherein, Generating the alarm includes generating a graphical indicator of the object for display via a screen inside the vehicle.

11. A vehicle, the vehicle comprising: One or more cameras; The display is located inside the vehicle. One or more processors; and The memory stores instructions that, when executed by the one or more processors, cause the one or more processors to: Based on video data captured by the one or more cameras, a machine learning model is used to perform object detection in order to detect objects; Predict the intent associated with the object based on one or more features associated with the object in the video data; A collision prediction algorithm is applied based on the object, the intent associated with the object, and one or more measured attributes of the vehicle in order to predict the proximity between the vehicle and the object in a given direction. as well as After determining the intention to move the vehicle in the given direction, an alarm is generated based on the predicted proximity between the vehicle and the object to be displayed via the display.

12. The vehicle according to claim 11, wherein, The collision prediction algorithm is further applied based on the vehicle's route data, wherein the vehicle's route data is based on a configured route associated with a satellite-based navigation system.

13. The vehicle according to claim 11, wherein, The instructions, when executed by the one or more processors, further cause the one or more processors to: After the object is detected, the distance and trajectory associated with the object relative to the vehicle are estimated based on multi-object tracking (MOT) performed using computer vision techniques, wherein the collision prediction algorithm is applied based on the distance and the trajectory.

14. The vehicle according to claim 13, wherein, The one or more measurement attributes include one or more of the vehicle's speed, the vehicle's steering wheel angle, or the vehicle's direction of travel.

15. The vehicle according to claim 14, wherein, The collision prediction algorithm is further applied to perform optical flow, block matching, or Kalman filtering techniques on the video data to monitor the movement of the object.

16. The vehicle according to claim 11, wherein, The instructions, when executed by the one or more processors, further cause the one or more processors to: A sliding window of frames from the video data is provided as input to a convolutional neural network (CNN) and a recurrent neural network (RNN) to detect one or more driving lanes; Extract the segmentation mask of the object from the frame; as well as The boundaries of the one or more driving lanes are compared with the segmentation mask of the object to predict whether the object is in the adjacent lane of the vehicle, wherein the alarm is generated further based on the prediction that the object is in the adjacent lane of the vehicle.

17. The vehicle according to claim 16, wherein, Comparing the boundaries of the one or more lanes with the segmentation mask of the object includes: calculating the intersection-over-union (IoU) ratio of the boundaries of the one or more lanes with the segmentation mask of the object, and comparing the IoU with a threshold.

18. The vehicle according to claim 11, wherein, When executed by the one or more processors, the instructions further cause the one or more processors to analyze data captured using a rear-facing camera associated with the vehicle in order to predict the trajectory of the object, wherein the generation of the alert is further based on the predicted trajectory of the object.

19. The vehicle according to claim 11, wherein, The determination of the intention to move the vehicle in the given direction is based on one or more of the following: The activation of the vehicle's turning signal; or The vehicle's route data.

20. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to: Based on video data captured by one or more cameras associated with the vehicle, a machine learning model is used to perform object detection in order to detect objects; Predict the intent associated with the object based on one or more features associated with the object in the video data; A collision prediction algorithm is applied based on the object, the intent associated with the object, and one or more measured attributes of the vehicle in order to predict the proximity between the vehicle and the object in a given direction. as well as After determining the intention to move the vehicle in the given direction, an alarm is generated based on the predicted proximity between the vehicle and the object to be displayed inside the vehicle.

Images