Enhanced vehicle blind spot detection with transparent trailer

Abstract

The invention provides enhanced vehicle blind spot detection with a transparent trailer. A system for object detection of a trailer. In one example, the system includes a camera configured to capture an image of the trailer, where the image includes an object other than the trailer. The system also includes a sensor configured to capture sensor data, a display configured to display an image of the angle from the camera, and a controller on the vehicle. The controller includes an electronic processor configured to: receive image data from the camera, receive sensor data from the sensor, determine a blind spot, analyze the sensor data for radar data associated with the object, calculate a position of the object relative to the blind spot and the trailer, and transmit the calculated position to the electronic processor. An object detection algorithm is used to determine that the object is within the blind spot, and in response to determining that the object is within the blind spot, an enhanced image is generated.

Description

Technical Field

[0001] Embodiments, examples, aspects and features described herein relate to a system for vehicle blind spot detection.

[0002] Blind spots in vehicles are areas around the vehicle that the driver cannot directly see while in control. Blind spots exist in a variety of vehicles, such as cars, buses, trucks, and agricultural equipment. In passenger cars and light trucks, blind spots may occur in front of the driver, for example, when the A-pillars (also called windshield pillars) block the driver's view. Blind spots often occur behind the driver, for example, in areas that cannot be seen via side-view mirrors, or in areas where the view is blocked by vehicle structure (such as B-pillars or C-pillars). Summary of the invention

[0003] Many vehicles include radar systems for detecting vehicle blind spots. Additionally, many vehicles include trailer towing capabilities. Blind spot detection systems do not always cover the additional blind spots introduced by a trailer. It would be desirable to enhance a vehicle blind spot detection system so that additional and modified blind spots that exist when a vehicle is towing a trailer are detected. Thus, the embodiments described herein provide, among other things, systems and methods for detecting vehicle blind spots that exist when a vehicle is towing a trailer.

[0004] Some examples provide a system for object detection of a trailer connected to a vehicle. In some cases, the system includes a camera located at the rear end of the vehicle, wherein the camera is configured to capture an image of the trailer. The image includes objects other than the trailer. The system also includes: a sensor configured to capture sensor data about the object; and a display configured to display an image from the angle of the camera. The system also includes a controller on the vehicle. The controller includes an input / output interface, a memory, and an electronic processor, and the electronic processor is configured to: receive image data from the camera, receive sensor data from the sensor, and determine a blind spot. The controller is further configured to: analyze the sensor data for radar data associated with the object, calculate the position of the object relative to the blind spot, use an object detection algorithm to determine that the object is within the blind spot, and in response to determining that the object is within the blind spot, generate an enhanced image including a representation of the object for display presentation.

[0005] In some cases, the enhanced image includes proximity information of the object relative to the trailer. The enhanced image includes a visual representation of the blind spot. The enhanced image may include more than one blind spot. In some examples, the enhanced image includes more than one object other than the trailer. In some examples, the sensor is configured to capture sensor data about more than one object. In some cases, the electronic processor is further configured to calculate the position of the more than one object relative to the blind spot and the trailer, and use an object detection algorithm to determine that one of the more than one objects is within the blind spot. The electronic processor may be further configured to analyze the sensor data for radar data associated with more than one object and calculate the position of all of the more than one objects relative to the blind spot and the trailer.

[0006] In some cases, the electronic processor is further configured to generate an enhanced image including more than one object for presentation on the display. In some cases, the electronic processor is further configured to generate an enhanced image including all of the more than one object for presentation on the display. In some cases, the electronic processor is further configured to generate an enhanced image including proximity information of the more than one object for presentation on the display. In some cases, the electronic processor is further configured to generate an enhanced image including more than one object and more than one blind spot.

[0007] Some examples provide a method for object detection of a trailer connected to a vehicle. The method includes generating an image of the trailer by a camera located at a rear end of the vehicle, wherein the image includes objects other than the trailer. The method also includes: generating sensor data about the object by a sensor; receiving the image data by an electronic processor; receiving the sensor data by the electronic processor; and analyzing the sensor data by the electronic processor for radar data associated with the object. The method further includes: determining a blind spot by the electronic processor; calculating a position of the object relative to the blind spot and the trailer by the electronic processor; determining, via the electronic processor, that the object is within the blind spot using an object detection algorithm; and in response to determining that the object is within the blind spot, generating, via the electronic processor, an augmented image including a representation of the object for presentation via a display.

[0008] In some cases, the augmented image includes more than one blind spot. In some cases, the augmented image includes more than one object other than the trailer, and the sensor is configured to capture sensor data about the more than one object. In some cases, the method further includes calculating, via the electronic processor, the position of the more than one object relative to the blind spot and the trailer, and determining, via the electronic processor, using an object detection algorithm, that one of the more than one objects is within the blind spot. In some cases, the method further includes analyzing, via the electronic processor, the sensor data for radar data associated with the more than one object, and calculating, via the electronic processor, the position of all of the more than one object relative to the blind spot and the trailer.

[0009] In some cases, a system for object detection of a trailer connected to a vehicle includes: a camera located at a rear end of the vehicle, the camera configured to capture an image of the trailer, the image including an object other than the trailer; and a sensor configured to capture sensor data about the object, the sensor data including the proximity of the object to the trailer. The system further includes: a display configured to display the image from the angle of the camera; and a controller on the vehicle, the controller including an input / output interface, a memory, and an electronic processor. The electronic processor is configured to: receive image data from the camera, receive sensor data from the sensor, determine a blind spot, and analyze the sensor data against radar data associated with the object. The electronic processor is further configured to: calculate the position of the object relative to the blind spot and the trailer, determine that the object is within the blind spot using an object detection algorithm, determine whether the object has exceeded a proximity threshold, and in response to determining that the object is within the blind spot, generate an augmented image including a representation of the object for presentation via the display, and in response to determining that the object has exceeded the proximity threshold, control an aspect of the vehicle.

[0010] In some cases, in response to determining that the object has exceeded the proximity threshold, the controller displays a proximity alert on the display. In some cases, in response to determining that the object has exceeded the proximity threshold, the controller controls the brakes of the vehicle.

[0011] Other aspects, features, examples, and embodiments will become apparent by consideration of the detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is an illustration of a system for monitoring a trailer blind spot according to some aspects.

[0013] Figure 2A is a pictorial representation of a trailer including trailer blind spots according to some aspects.

[0014] Figure 2B is a pictorial representation of a trailer including trailer blind spots according to some aspects.

[0015] Figure 2C is a pictorial representation of a trailer including trailer blind spots according to some aspects.

[0016] Figure 2D is a pictorial representation of a trailer including trailer blind spots according to some aspects. Figure 3 is a graphic representation of radar detection according to some aspects.

[0017] Figure 4A Based on some aspects Figure 1 Illustration of an enhanced display of a system.

[0018] Figure 4B Based on some aspects Figure 1 Illustration of an enhanced display of a system.

[0019] Figure 5 is a flow chart of a process for monitoring a trailer blind spot according to some aspects.

[0020] Figure 6 is a flow chart of a process for monitoring a trailer's blind spot. DETAILED DESCRIPTION

[0021] The articles "a", "an", and "the" should not be construed as meaning "one" or "only one", unless the context of their use clearly indicates otherwise. Instead, these articles should be construed as meaning "at least one" or "one or more". Similarly, when the terms "the" or "the" are used to refer to a noun previously introduced by the indefinite article "a" or "an", "the" and "the" mean "at least one" or "one or more", unless the usage clearly indicates otherwise.

[0022] In addition, it should be understood that, unless explicitly described to the contrary, the illustrated components may be combined or divided into separate software, firmware and / or hardware. For example, the logic and processing described herein may be distributed among multiple electronic processors, rather than being located in a single electronic processor and being performed by a single electronic processor. Similarly, even if the embodiments described or illustrated herein have a single such device or element, one or more memory modules and communication channels or networks may be used. In addition, regardless of how they are combined or divided, hardware and software components may be located on the same computing device, or may be distributed among multiple different devices. Therefore, in the claims, if a device, method or system is claimed, for example, such as including a controller, a control unit, an electronic processor, a computing device, a logic element, a module, a memory module, a communication channel or a network, or other elements configured in some way, for example, to perform multiple functions, then the claim or claim element should be interpreted as meaning one or more such elements, any of which is configured as claimed, for example, to implement any one or more of the multiple functions described, so that one or more elements perform multiple functions together as a set.

[0023] Figure 1A system for monitoring trailer blind spots according to some aspects is illustrated. System 100 includes a vehicle 105 and a trailer 110 attached to vehicle 105 by a hook 112. Vehicle 105 has an onboard controller 115. In the example shown, controller 115 includes an electronic processor 120, an input / output interface 125, and a memory 130. In some examples, electronic processor 120 is implemented as a microprocessor with a separate memory (e.g., memory 130) as shown. In other examples, electronic processor 120 can be implemented as a microcontroller (wherein memory 130 is on the same chip). In other examples, electronic processor 120 can be implemented using multiple processors. In addition, electronic processor 120 can be partially or fully implemented as, for example, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and the like, and memory 130 may not be required or memory 130 may be modified accordingly.

[0024] In some examples, the memory 130 includes a non-transitory computer-readable medium that stores instructions received and executed by the electronic processor 120 to perform the methods described herein. The memory 130 may include, for example, a program storage area and a data storage area. The program storage area and the data storage area may include a combination of different types of memory, such as read-only memory and random access memory. In the example shown, the memory 130 stores software used during operation of the vehicle 105. In one specific case, the memory 130 stores an object detection algorithm 135 (also referred to as algorithm 135).

[0025] The input / output interface 125 may include one or more input mechanisms and one or more output mechanisms, such as general purpose input / output (GPIO), analog input, digital input, and the like.

[0026] In some examples, the illustrated components may be combined or divided into separate software, firmware, and / or hardware. For example, logic and processing may be distributed among multiple electronic processors and memories rather than being located within and executed by a single electronic processor. Regardless of how they are combined or divided, hardware and software components may be located on the same computing device, or may be distributed among different computing devices connected by one or more networks or other suitable communication links.

[0027] The vehicle 105 also includes one or more sensors 140. The sensor 140 may include a radar sensor, a LIDAR sensor, or the like. In some examples, the individual sensors 140 include internal processing hardware and software, and the sensor 140 is configured to generate sensor data about an object detected by (or via) the sensor 140. For example, when an object is within the sensing range of the sensor 140, the sensor may generate proximity data about the object. The sensor 140 may also generate other data, such as data corresponding to the size, dimensions, speed, or other distinguishing features of the object. The vehicle 105 also includes a camera 145, which is configured to capture an image of the trailer 110 connected to the vehicle 105. For example, in one embodiment, the camera is mounted at the rear of the vehicle, close to the trailer hitch, and tilted downward to capture a view of the area behind the vehicle. In some cases, additional cameras are mounted to the vehicle or trailer. The vehicle 105 also includes a display 150, which is configured to display an image captured by (or via) the camera 145 or an image enhanced by the algorithm 135. Communication bus 155 electrically connects camera 145, sensor 140, display 150, and controller 115 to each other. In some cases, the bus is a controller area network (CAN) bus. In other cases, the bus is a FlexRay TM Communications Bus. In still other cases, an Ethernet network or other suitable bus is implemented.

[0028] Figure 2A 200 is a diagram of a trailer 110 connected to a vehicle 105. Blind spots 205 are identified in the image, in which case blind spots 205 are areas that cannot be seen by the driver via the side mirrors. A trailer may have many blind spots, which are caused by the size of the trailer, the position of the driver relative to the trailer, the angle of the turn of the vehicle or trailer, or the like. The blind spots 205 in the diagram 200 are marked via cones 210, but it should be understood that the mark is used to indicate potential blind spots that occur when the vehicle 105 and the trailer 110 are in a stationary position. When the vehicle is in motion (i.e., being driven), the blind spots are dynamic, and a specific blind spot may be larger or smaller than the blind spot 205, or may appear in many different positions (including the position behind the vehicle 105). When the vehicle crosses the terrain, potential blind spots are generated. For example, when the vehicle makes a right turn, the blind spot 205 may appear at the left rear of the trailer. The size of the trailer may also lead to the possibility of blind spots. For example, a wide trailer may block a larger area of ​​the driver's field of vision than a narrow trailer. In addition, trailers with large loads, such as flatbed trailers configured to tow another vehicle, may create additional blind spots. Blind spots may also occur in front of a vehicle. For example, a blind spot may be located in front of a vehicle with a large, elevated cab, or in a vehicle with a working element in front of it, such as a forklift.

[0029] Figure 2B FIG. 2 is a top-down diagram 250 of vehicle 105 and trailer 110. Blind spot 205 is located to the left rear of trailer 110, similar to Figure 1 The position of the vehicle 105 and the trailer in the vehicle 105. The field of view 215 is shown in the diagram 250 to indicate the position of the blind spot 205. Figure 2C The illustration shows a dynamic movement of the vehicle, in this case a right turn 220. As the vehicle turns right, the driver's view 215 is obstructed by the trailer. Figures 2A-2C Blind spots are illustrated from the perspective of the driver of vehicle 105 , but it should be understood that blind spots may appear from additional perspectives, such as from a passenger of vehicle 105 , camera 145 , sensor 140 , or any other location on vehicle 105 or trailer 110 .

[0030] Figure 2D 270 is a top-down illustration of a vehicle 105 and a trailer 110 according to some aspects and examples. As previously mentioned, in some cases, the vehicle includes a camera 145. In some examples, the camera 145 is a first camera 275 located near the hitch 112. The vehicle 105 may include a second camera 280 located behind the trailer 110. The first camera 275 may be a back-up camera, a rear-facing dashboard camera, a safety camera, or the like. The first camera 275 includes a field of view 285. The field of view 285 generally extends from the hitch to the rear in a direction opposite to the front of the vehicle 105. In Figure 2D , the horizontal range of the field of view is shown. In some examples, the field of view 285 of the first camera 275 is limited by the presence of the trailer 10 and extends in two relatively limited areas or limited degrees of view. In the case where the vehicle 105 is disconnected from the trailer 110, the field of view 285 can extend to 180 degrees (in the horizontal plane).

[0031] In some cases, the trailer 110 also includes a second camera 280. The second camera 280 can be configured similarly to the first camera 275 and also includes a field of view 290. The field of view 290 generally extends rearward from the trailer 110, and in some examples, extends to 180 degrees. When the trailer 110 is hooked to the vehicle 105, such as shown in the top-down illustration 270, the field of view 285 of the first camera 275 may be partially blocked by portions of the trailer 110. This blockage creates a blind spot, such as a blind spot 295. Similar to the blind spot 205 as previously described, objects within the blind spot 295 cannot be seen by the camera (275, 280) and may not be seen by the driver. As previously described, the blind spot may be dynamic and change size, shape or position, for example, according to the movement of the vehicle 105 and the trailer 110.

[0032] The images captured by the cameras (145, 275, 280) are displayed on the display 150 of the vehicle. For example, when the trailer 110 is not connected to the vehicle 105, the camera 275 can display a rearward view of objects behind the vehicle 105. When the trailer 110 is connected, the rearward view is blocked by the trailer 110, thereby producing the previously described blind spot 295. Among other things, the systems and methods described herein also help to increase the driver's awareness of objects within these blind spots, which indicates to the driver the objects detected in the blind spot. In some examples, the images captured by the first camera 275 and the images captured by the second camera 280 enhance each other to form a single image to be displayed on the display 150.

[0033] Figure 3 3 is an illustration of a radar map 300 generated by one of the sensors when the sensor 140 is a radar sensor. The radar map 300 includes representations of objects around the vehicle 105. The sensor 140 can (alone or in conjunction with other processing resources) detect various objects around the vehicle and plot them at locations on the radar map 300. The sensor 140 then generates sensor data (also referred to as radar data 305) for processing by (or via) the electronic processor 120 and analysis by the algorithm 135. The algorithm 135 then determines whether the radar data 305 is an object, such as the object 310 displayed on the radar map 300. The sensor 140 is also configured to detect the proximity of the object 310 to the trailer 110 and the vehicle 105. In some examples, the sensor 140 is configured to target adjacent lanes of travel of the vehicle 105. For example, the sensor can be configured to target the nearest lanes of travel to the left and right of the vehicle 105. In some cases, one sensor is configured to monitor one lane, while another sensor is configured to monitor another lane. Some sensors are configured to monitor the lane currently occupied by vehicle 105. For example, object 310 is a vehicle detected by sensor 140 and classified as a vehicle by algorithm 135. Algorithm 135 may classify object 310 as an object other than a vehicle. In such a case, the enhanced image includes a different representation of the object. Objects detected by the camera may include fences, cones, curbs, other vehicles, road surface, or any other object. Each type of object may include a unique enhanced representation, or certain objects may include a classification of objects. For example, pedestrians, cyclists, and animals may be a classification of an object. Another type of classification of objects may include stationary objects, such as lamp posts, street lights, or lamps. The process by which algorithm 135 classifies objects and enhances the images captured by camera 145 is described below and in Figure 5 Medium picture.

[0034] Figure 4A4 is an illustration of an enhanced display of a system for monitoring trailer blind spots according to some aspects. Illustration 400 includes a camera view 405 of an image from camera 145. As previously described, the system can combine multiple image captures from more than one camera (e.g., cameras 145, 275, 280) for display. The captured images can be overlaid, displayed side by side, or otherwise presented. Image 410 includes vehicle 105, trailer 110, road 415, and has been enhanced by algorithm 135 to include object 310 obscured by the trailer. As seen in image 410, trailer 110 would normally obstruct the vision of object 310. As the vehicle travels along road 415, each frame captured by camera 145 is processed by electronic processor 120 and analyzed by algorithm 135 to generate an enhanced image. In addition, sensor 140 generates radar data including information about detected objects, and the radar data is used to determine the location, velocity, and size of the detected objects.

[0035] The algorithm processes the object 310 and the location of the vehicle 105 and the trailer 110, and determines whether the object 310 is obscured by the trailer 110. If the algorithm 135 determines that the object 310 is obscured, the algorithm 135 enhances the image. In some cases, the object 310 is classified to generate an enhanced coverage of the blind spot area. For example, two different objects can be classified similarly (e.g., classified as vehicles) to be displayed on the enhanced image. The enhanced image may include a virtual object, such as a green cube, which represents the object 310 and illustrates that the object 310 is visually obscured. In some examples, the image may include a picture of the object 310 superimposed on the location of the blind spot 205. In some examples, the enhanced image includes a warning that explains the location of all possible blind spots. The enhanced image is then displayed by the display 150.

[0036] In some examples, the enhanced image may include a symbol, a general 3D model of an object, or a detailed model of an object generated by artificial intelligence. For example, a generative AI model may generate a vehicle image based on the classification of an object as a vehicle. The generative AI model may be based on one or more images captured by a camera (145, 275, 280) combined with radar data generated by sensor 140. The generative AI model may be created based on an image of the object (before the object enters the blind spot) and using techniques such as NeRF (neural radiant field). In some cases, the generative AI model may be based on a training network using radar sensor inputs such as radar spectral data or high-precision reflection positions. These can be used to directly generate 3-D models, or they can be used to classify objects by using pre-stored 3-D models for each object classification.

[0037] In some examples, the object 310 detected by the sensor 140 can be associated with the object 310 seen by the camera 145 outside the blind spot, and then subsequently tracked when the object 310 enters the blind spot. In some cases, this detection in the image can be performed by the camera (145, 275) when the vehicle 105 overtakes the object 310 and the object 310 is in the adjacent lane. In some cases, this can be performed by the camera 280 mounted on the rear of the trailer when the vehicle approaches from behind. Static objects (such as road markings, parked vehicles, vegetation, or the like) captured by the camera (145, 275, 280) or sensor 140 can be rendered in the enhanced image based on the driving distance. For example, an object closer to the vehicle 105 may appear larger than an object farther from the vehicle 105. The rendering and enhancement of static objects does not need to use a model like a dynamic object, but instead uses an aligned video image with a transformation to take into account the increased distance. The size, orientation, and pose of the 3-D model projected into the image may depend on the distance measurement and estimated size of the object 310. In some cases, these calculations are performed during the classification process. In other cases, they are performed by algorithms as detailed herein.

[0038] Also visible in diagram 400 are controls for display 150. Display 150 may include an interface 420 for controlling camera 145 or an aspect of display 150. In some examples, display 150 includes options to cycle through multiple cameras, turn augmented images on or off, expand or shrink images, change field of view, or the like. In some examples, display 150 is part of the onboard computer and can be turned on or off. In other examples, display 150 is configured as a separate monitor and is dedicated only to displaying augmented images.

[0039] Figure 4B is an illustration of an enhanced display of a system for monitoring a trailer blind spot as previously described. Illustration 450 includes a display 150 of a vehicle 105 connected to a controller 115 via a communication bus 155. As previously described, the display 150 may include an interface 420 having a plurality of buttons 425 for selecting various options of the display 150. The illustration includes an augmented image 430 displayed on the display 150.

[0040] Figure 55 is a flow chart of a process for monitoring a trailer blind spot according to some aspects. Process 500 begins at step 505 with vehicle 105 towing trailer 110. The process continues to step 510, where sensor 140 captures radar data 305 and camera 145 attached to the vehicle records image 410 of trailer 110. In some cases, radar data 305 may include radar spectrum data in addition to position data. In addition, the relative speed of the vehicle may be detected by sensor 140. In some examples, at step 505, other cameras capture different images from alternative angles, such as images captured from camera 275 and / or 280. At step 515 of process 500, camera 145 sends image 410 to controller 115, and sensor 140 sends radar data 305 to controller 115.

[0041] The process continues to step 520, where the electronic processor 120 of the controller 115 runs the algorithm 135. Each image captured by the camera 145 and all radar data 305 captured by the sensor 140 are run through the algorithm 135. The algorithm 135 associates the radar data 305 with the image. All objects 310 detected by the sensor 140 are analyzed by the algorithm 135. The algorithm 135 then superimposes a representation of the object 310 obscured by the trailer 110 onto the image 410, thereby generating an enhanced image that includes the object 310 located within the blind spot. The enhanced image is then displayed on the display 150. In some cases, the enhanced image also includes proximity information. For example, the sensor 140 detects the proximity of the object 310 and includes the proximity information in the radar data 305 sent to the controller 115. The controller can then calculate the distance between the object 310 and the vehicle 105 or the trailer 110. This proximity information can be displayed on the display 150 along with the enhanced image. The proximity information may include speed information of object 310, a distance between object 310 and vehicle 105, or a visual representation of the proximity information. For example, the distance between object 310 and vehicle 105 may be represented by color-coded directional arrows, arrows of different lengths, or the like.

[0042] At step 525 of the process, the enhanced image is displayed on the display 150. The enhanced image may include an object 310, an overlay representation of the object 310, or other elements configured to be displayed on the display 150, as previously described. The process proceeds to step 530, where the controller determines whether the proximity between the objects 310 in the blind spot 205 has exceeded a proximity threshold. The proximity threshold may include a distance between 10 centimeters and 5 meters. If the object 310 has not exceeded the proximity threshold, the process returns to step 505, where the vehicle is operating. However, if the proximity threshold has been exceeded, the process continues to step 535, where the controller 115 is configured to control one aspect of the vehicle. The controller 115 may generate a warning output by the input / output interface 125, warning the driver of the vehicle that the trailer 110 and the object 310 are in danger of colliding. For example, the display 150 may include a proximity alert indicating that the object 310 in the blind spot has exceeded the proximity threshold. In some cases, the controller 115 can control the brakes of the vehicle to slow down, stop, control steering within a lane, or otherwise avoid a collision of the trailer 110 with the object 310. In some examples, there are multiple proximity thresholds. For example, there can be a first proximity threshold that, when exceeded, generates a warning for the driver, and a second proximity threshold that is different from the first proximity threshold that, when exceeded, causes the controller to control the vehicle. In some cases, the second proximity threshold is a closer distance between the obstacle and the trailer than the first proximity threshold.

[0043] Figure 6 6 is a flow chart of a process 600 for monitoring a trailer blind spot. Similar to the previously described process 500, the process 600 includes a vehicle 105 towing a trailer 110. The process begins at step 605, where the sensor 140 captures radar data 305 and the camera (145, 275, 280) records an image 410. Similar to the process 500, in some cases, in addition to the position data, the radar data 305 may also include radar spectrum data, and the relative speed of the vehicle may be detected by the sensor 140. At step 610 of the process 600, the image 410 and the radar data 305 are analyzed using object detection and object localization techniques. In some cases, object detection refers to the process of identifying one or more objects 310 in the image 410 and classifying them (such as "car", "person" or "tree"). This step can be performed by the algorithm 135, or can be performed by another machine learning technique trained on a large dataset of labeled images. In some examples, object detection may use a variety of techniques, such as sliding windows, region-based convolutional neural networks (R-CNN), or single-shot detectors (SSD), to locate object 310 in image 410.

[0044] Object localization, on the other hand, involves not only identifying the presence of object 310 in image 410, but also identifying its location within image 410. This step may also be performed by algorithm 135, but may also be performed by defining a bounding box around object 310, which is a rectangular area that tightly surrounds object 310. The coordinates of the bounding box may then be used to accurately locate object 310 in image 410. Process 600 continues with step 615, where electronic processor 120 determines whether object 310 is within the blind spot (205, 295). If electronic processor 120 determines that object 310 is not within the blind spot (205, 295), image 410 is displayed on display 150, as described in step 630. On the other hand, if electronic processor 120 determines that object 310 is within the blind spot (205, 295), the process continues to step 620, where an object model is generated.

[0045] As previously described, an object model is a representation of an object 310 or a group of related objects. In some cases, object detection and object localization are trained to identify and locate specific objects in an image or video stream. The training can involve a large dataset of images, where each image includes metadata, such as the location, size, dimensions, vector, or other attributes of the object 310. The object model can learn to recognize and distinguish features of the object 310 from the background and other objects in the image 410. In some cases, the training of the object model is part of the process 600. In other cases, the training of the object model is performed before the process 600 and then used during the process 600.

[0046] The process continues with step 625, where the object model is overlaid on the image 410. As previously described, the overlaid model can be generated based on previous image capture from the camera (145, 275, 280), or can be a previously generated 3D model based on the classification of the object 310. Once the object model is overlaid on the image 410, at step 630, the image is displayed on the display 150. The image displayed on the display can include multiple objects or partially enhanced objects. For example, if the object 310 is partially within the blind spot (205, 295), but is partially detected by the sensor 140 or the camera (145, 275, 280), the object 310 may only be partially enhanced when displayed. The process continues to step 635, where the electronic processor analyzes the sensor data collected by the sensor 140 to determine whether the object 310 has exceeded the proximity threshold. The electronic processor may additionally or alternatively analyze the sensor data to determine whether the object 310 has exceeded the collision time (TTC) threshold. If the electronic processor determines that the object 310 has exceeded the proximity threshold, then in step 640, the electronic processor continues to control the vehicle 105. The electronic processor may control the braking of the vehicle 105, the acceleration of the vehicle 105, the steering of the vehicle 105, or any other method of vehicle control. Additionally or alternatively, the electronic processor may control the display 150 to warn the driver of the vehicle 105 that the threshold has been exceeded.

[0047] Thus, various implementations of the systems and methods described herein provide, among other things, techniques for detecting and monitoring vehicle blind spots.Other features and advantages of the present invention are set forth in the following claims.

[0048] In the foregoing description, specific examples have been described. However, it will be appreciated by those skilled in the art that various modifications and variations may be made without departing from the scope of the invention as set forth in the following claims. Therefore, the description and drawings are to be viewed in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present teachings.

[0049] Benefits, advantages, solutions to problems, and any element(s) that may cause any benefit, advantage, or solution to occur or become more apparent are not to be construed as key, required, or essential features or elements of any or all the claims. The invention is defined solely by the appended claims including any amendments made during the pendency of this application and all equivalents of those claims as issued.

[0050] Furthermore, in this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "having," "including," "containing," "containing," "including," or any other variation thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises, has, includes, contains a list of elements includes not only those elements, but may also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0051] Without further constraints, an element preceded by "comprises...a", "has...a", "includes...a", or "contains...a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, or contains the element. The terms "substantially", "essentially", "approximately", "about", or any other version thereof are defined as close to, as understood by one of ordinary skill in the art, and in one non-limiting example, the terms are defined as within 10%, within 5%, within 1% in another example, and within 0.5% in another example. The term "coupled", as used herein, is defined as connected, but not necessarily directly, and not necessarily mechanically. A device or structure that is "configured" in a certain way is configured in at least that way, but may also be configured in ways not listed.

Claims

1. A system for object detection of a trailer connected to a vehicle, the system comprising: a camera located at a rear end of the vehicle, the camera configured to capture an image of the trailer, the image including objects other than the trailer; a sensor configured to capture sensor data about the object, a display configured to display an image from the perspective of the camera, A controller on a vehicle, the controller comprising an input / output interface, a memory, and an electronic processor, the electronic processor being configured to: Receive image data from the camera, receiving sensor data from sensors, Identify blind spots, analyzing the sensor data for radar data associated with the object, Calculate the position of the object relative to the blind spot, Use object detection algorithms to determine if an object is within the blind spot, and In response to determining that the object is within the blind spot, an augmented image including a representation of the object is generated for presentation on a display.

2. The system of claim 1, wherein the augmented image includes proximity information of the object relative to the trailer.

3. The system of claim 1, wherein the augmented image includes a visual representation of the blind spot.

4. The system of claim 1, wherein the enhanced image includes more than one blind spot.

5. The system of claim 1, wherein the augmented image includes more than one object other than the trailer. The system of claim 1 , wherein the sensor is configured to capture sensor data regarding more than one object.

7. The system of claim 5, wherein the electronic processor is further configured to calculate the position of more than one object relative to the blind spot and the trailer and to use an object detection algorithm to determine that one of the more than one objects is within the blind spot.

8. The system of claim 6, wherein the electronic processor is further configured to analyze the sensor data for radar data associated with more than one object and calculate the position of all of the more than one object relative to the blind spot and the trailer.

9. The system of claim 7, wherein the electronic processor is further configured to generate an augmented image including more than one object for presentation on the display.

10. The system of claim 8, wherein the electronic processor is further configured to generate an enhanced image including all of the more than one objects for presentation on the display.

11. The system of claim 8, wherein the electronic processor is further configured to generate an augmented image including proximity information of more than one object for presentation on the display.

12. The system of claim 8, wherein the electronic processor is further configured to generate an enhanced image that includes more than one object and more than one blind spot.

13. A method for object detection of a trailer connected to a vehicle, the method comprising: generating an image of the trailer via a camera located at a rear end of the vehicle, the image including objects other than the trailer; generating sensor data about the object via the sensor, receiving image data via an electronic processor, receiving sensor data via an electronic processor, analyzing the sensor data via an electronic processor for radar data associated with the object, The electronic processor determines the blind spot. The electronic processor calculates the object's position relative to the blind spot and the trailer. using an object detection algorithm via an electronic processor to determine that an object is within the blind spot, and In response to determining that the object is within the blind spot, an augmented image including a representation of the object is generated via the electronic processor for presentation on the display. The method of claim 13 , wherein the enhanced image includes more than one blind spot.

15. The method of claim 13, wherein the augmented image includes more than one object other than the trailer, and the sensor is configured to capture sensor data regarding the more than one object.

16. The method of claim 15, wherein the method further comprises calculating, via the electronic processor, the position of more than one object relative to the blind spot and the trailer, and determining, via the electronic processor using an object detection algorithm, that one of the more than one object is within the blind spot.

17. The method of claim 15, wherein the method further comprises analyzing the sensor data via the electronic processor for radar data associated with more than one object, and calculating, via the electronic processor, the position of all of the more than one object relative to the blind spot and the trailer.

18. A system for object detection of a trailer connected to a vehicle, the system comprising: a camera located at a rear end of the vehicle, the camera configured to capture an image of the trailer, the image including objects other than the trailer; a sensor configured to capture sensor data regarding the object, the sensor data including a proximity of the object to the trailer, a display configured to display an image from the perspective of the camera, A controller on a vehicle, the controller comprising an input / output interface, a memory, and an electronic processor, the electronic processor being configured to: Receive image data from the camera, receiving sensor data from sensors, Identify blind spots, analyzing the sensor data for radar data associated with the object, Calculate the object's position relative to the blind spot and trailer, Use object detection algorithms to determine if objects are within the blind spot, Determine if the object has crossed a proximity threshold, and wherein, in response to determining that the object is within the blind spot, generating an augmented image including a representation of the object for presentation via the display, and Therein, in response to determining that the object has exceeded a proximity threshold, an aspect of the vehicle is controlled.

19. The system of claim 18, wherein in response to determining that the object has exceeded the proximity threshold, the controller displays a proximity alert on the display.

20. The system of claim 18, wherein in response to determining that the object has exceeded the proximity threshold, the controller controls brakes of the vehicle.

Images