Camera surveillance system including integrated distance sensor

Abstract

The camera surveillance system includes a first mirror replacement assembly including a housing supporting a rear-facing camera and a rear-facing distance sensor; a controller in communication with the mirror replacement assembly, the controller including a processor and a memory, the memory storing instructions for identifying at least one object in an image feed from the camera, determining the angular position of the object using image analysis, identifying the distance of the at least one object from the mirror replacement assembly using the distance sensor, and fusing the distance and angular position of the object into a single data set; and a display connected to the controller and configured to display a mirror replacement view including at least a portion of the image feed.

Description

[Technical field]

[0001] The present disclosure relates to a camera surveillance system (CMS) for use in commercial trucks, and more particularly, to a camera surveillance system that includes an integrated distance sensor.

[0002] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 293188, filed December 23, 2021. [Background technology]

[0003] Mirror replacement systems, and camera systems that supplement the mirror view, are utilized in commercial vehicles to enhance the vehicle operator's ability to view the surrounding environment. Camera surveillance systems (CMS) utilize one or more cameras to provide an enhanced field of view to the vehicle operator. In some instances, mirror replacement systems cover a wider field of view than a traditional mirror or include views that are not fully available via a traditional mirror.

[0004] The camera surveillance system utilizes an object detection and image analysis system to identify objects in the image feed from the mirror-replacement camera. The image analysis may be further used to determine information about the objects, which may be provided to a vehicle controller for use by automated or semi-automated driver assistance systems, driver warning systems, and any other similar systems on the vehicle. Summary of the Invention

[0005] In one exemplary embodiment, a camera surveillance system includes a first mirror replacement assembly including a housing supporting a rear-facing camera and a rear-facing distance sensor; a controller in communication with the mirror replacement assembly, the controller including a processor and a memory, the memory storing instructions for identifying at least one object in an image feed from the camera and determining an angular position of the object using image analysis, identifying a distance of at least one object from the mirror replacement assembly using the distance sensor, and fusing the distance and the angular position of the object into a single data set; and a display connected to the controller and configured to display a mirror replacement view including at least a portion of the image feed.

[0006] In another example of the above camera surveillance system, the distance sensor is a time-of-flight sensor.

[0007] In another example of any of the above camera surveillance systems, the time-of-flight sensor includes a radar sensor.

[0008] In another example of any of the above camera surveillance systems, the time-of-flight sensor includes a lidar distance sensor.

[0009] In another example of any of the above camera surveillance systems, the controller is further configured to determine a velocity of the identified object using the distance sensor.

[0010] In another example of any of the above camera surveillance systems, the controller is further configured to communicate the combined angular position and distance to at least one advanced driver assistance system.

[0011] In another example of any of the above camera surveillance systems, the first mirror replacement assembly is disposed within a first camera wing of a commercial vehicle.

[0012] Another example of any of the above camera surveillance systems further includes a second mirror replacement assembly substantially identical to the first mirror replacement assembly.

[0013] In another example of any of the above camera surveillance systems, the first mirror replacement assembly is located on a driver's side of the vehicle and the second mirror replacement assembly is located on a passenger's side of the vehicle.

[0014] In another example of any of the above camera surveillance systems, the controller is configured to use a nearest neighbor matching algorithm to fuse the distances and the angular positions of the objects into the single data set.

[0015] In another example of any of the above camera surveillance systems, the nearest neighbor matching algorithm is configured to correlate the angular position of the at least one object and the distance of the at least one object with tracked objects within a maximum cost number of the at least one object.

[0016] In another example of any of the above camera surveillance systems, the maximum cost number is a weighted sum of the range difference, the velocity difference, and the angle difference between the at least one object and the tracked object.

[0017] An exemplary method for improving object detection in a mirror replacement system includes a controller receiving at least one object detection from a rear-facing time-of-flight sensor in a vehicle wing, the controller receiving image data from a rear-facing camera in the wing and identifying the at least one object in the image data using an image analysis process, and fusing an angular position of the at least one object from the image data with at least one distance from the time-of-flight sensor to generate a fused tracked object.

[0018] In another example of the above method for improving object detection in a mirror replacement system, the time-of-flight sensor is one of a radar sensor and a lidar sensor.

[0019] In another example of any of the above methods for improving object detection in a mirror replacement system, fusing the angular position of the at least one object from the image data with at least one distance from the time-of-flight sensor to generate a fused tracked object includes fusing object velocity data from the time-of-flight sensor.

[0020] In another example of any of the above methods for improving object detection in a mirror replacement system, fusing the angular position of the at least one object from the image data with at least one distance from the time-of-flight sensor is performed by the controller using a nearest neighbor matching algorithm.

[0021] In another example of any of the above methods for improving object detection in a mirror replacement system, the nearest neighbor matching algorithm is configured to correlate the angular position of the at least one object and the distance of the at least one object with tracked objects within a maximum cost number of the at least one object.

[0022] In another example of any of the above methods for improving object detection in a mirror replacement system, the maximum cost number is a weighted sum of the range difference, velocity difference, and angle difference between the at least one object and the tracked object. [Brief description of the drawings]

[0023] The present disclosure can be further understood by reference to the following detailed description taken in conjunction with the accompanying drawings.

[0024] [Figure 1A] FIG. 1 is a schematic front view of a commercial truck equipped with a camera surveillance system (CMS) used to provide at least Class II and Class IV views.

[0025] [Figure 1B]FIG. 1 is a schematic top view of a commercial truck equipped with a camera surveillance system providing Class II, Class IV, Class V and Class VI views.

[0026] [Diagram 2] FIG. 1 is a schematic top perspective view of a vehicle cab including a display and an in-cabin camera.

[0027] [Figure 3A] FIG. 2 shows the field of view of a distance sensor of a commercial vehicle.

[0028] [Figure 3B] FIG. 2 illustrates the field of view of a camera on a commercial vehicle.

[0029] [Figure 4] FIG. 3C is a schematic diagram illustrating nearest neighbor fusion of the sensor field of view of FIG. 3A and the camera field of view of FIG. 3B.

[0030] [Diagram 5] 1 illustrates a process for fusing sensor and camera-based object detection from a single mirror replacement assembly.

[0031] [Figure 6] 2 illustrates a schematic example of nearest neighbor matching of multiple tracked objects;

[0032] The embodiments, examples and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or their respective individual features, may be taken independently or in any combination. Features described in relation to one embodiment are applicable to all embodiments, except where such features are incompatible. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] A schematic diagram of a commercial vehicle 10 is shown in Figures 1A and 1B. The vehicle 10 includes a vehicle cab or tractor 12 for towing a trailer 14. Although commercial trucks are contemplated in this disclosure, the invention may be applied to other types of vehicles. The vehicle 10 incorporates a camera surveillance system (CMS) 15 (Figure 2) that includes driver and passenger side camera arms 16a, 16b mounted on the exterior of the vehicle cab 12. If desired, the camera arms 16a, 16b may also include conventional mirrors integrated therewith, although the CMS 15 may be used to replace the mirrors entirely. In additional examples, multiple camera arms may be included on each side, with each arm housing one or more cameras and / or mirrors.

[0034] Each camera arm 16a, 16b includes a base that is fixed to, for example, the cab 12. A pivot arm is supported by the base and may be articulated relative thereto. At least one rear-facing camera 20a, 20b is disposed within the camera arm. Each of the exterior cameras 20a, 20b has an exterior field of view (FOV) that includes at least one of a Class II view and a Class IV view (FIG. 1b), which are legally defined views in the commercial trucking industry. EX1 , FOV EX2 If desired, multiple cameras may be used on each camera arm 16a, 16b to provide these views. Each arm 16a, 16b may also provide a housing that encloses electronics configured to provide various features of the CMS 15.

[0035] First and second video displays 18a, 18b are positioned on the driver's and passenger's sides, respectively, within the vehicle cab 12 on or near the A-pillars 19a, 19b and display Class II and Class IV views on each side of the vehicle 10, which provide rear-facing views along the vehicle 10 captured by exterior cameras 20a, 20b.

[0036] If Class V and Class VI view video is also required, a camera housing 16c and camera 20c may be located at or near the front of the vehicle 10 to provide these views (FIG. 1b). A third display 18c located in the cab 12 near the top center of the windshield may be used to display to the driver the Class V and Class VI views toward the front of the vehicle 10.

[0037] If Class VIII view video is required, camera housings can be positioned on the sides and rear of the vehicle 10 to provide a field of view that includes some or all of the Class VIII zone of the vehicle 10. As illustrated, the Class VIII view includes a view that surrounds the immediate vicinity of the trailer and a rear close-up view of the vehicle that includes the rear of the trailer. In one example, the rear close-up view of the vehicle is generated by a rear-facing camera positioned at the rear of the vehicle and may include both an immediate rear close-up view and a traditional rear view (e.g., a field of view extending rearward to the horizon provided by a rearview mirror in a vehicle without a trailer). In such an example, the third display 18c may include one or more frames that display a Class VIII view. Alternatively, additional displays may be added near the first, second, and third displays 18a, 18b, 18c to provide a dedicated display that provides a Class VIII view. The displays 18a, 18b, 18c face a driver area 24 in the cab 22 where the driver is seated in the driver's seat 26.

[0038] Each of the camera arms 16a, 16b also includes a mirror replacement assembly that includes a distance sensor 50, such as a radar sensor, lidar sensor, or other time-of-flight sensor, and a corresponding camera 20a, 20b. The distance sensor 50 and the camera 20a, 20b face rearward and have a field of view at least as large as that provided by a conventional rear-facing mirror as seen by an occupant at the driver's position. Each camera arm 16a, 16b includes a substantially identical mirror replacement assembly. As used herein, substantially identical mirror replacement assemblies refer to mirror replacement assemblies that have the same systems and functions while allowing for slight changes in orientation and relative placement that may be necessary to account for driver / passenger side positioning or similar position-related changes.

[0039] The distance sensor 50 emits radio (radar) or light (lidar) waves that reflect off objects and return to the distance sensor 50. The distance sensor 50 measures the time of flight between when the wave is emitted and when it is returned to the sensor. Based on this time of flight, the distance sensor 50, or a connected controller, can detect how far an object is from the distance sensor 50. Because of the way the distance sensor 50 operates, the distance to an object and the rate of change of that distance relative to the sensor (i.e., the object's velocity) can be determined quickly and accurately using the sensor signal and minimal calculations. While the distance sensor 50 can identify the distance of an object from the distance sensor 50, it is relatively difficult to determine the angular position of the object based on the sensor signal alone.

[0040] FIG. 3A shows a schematic representation of a field of view 310 generated by a distance sensor 312. An object 320 is located within the field of view 310 and is detected by the distance sensor 312. A distance 322 from the object 320 to the distance sensor 312 can be determined relatively easily from data generated by the distance sensor 312 alone. However, it is difficult or impossible to determine an exact angular position from the data of the distance sensor 312 alone. The angular position is a position on an arc 324 around the distance sensor 312, the arc 324 having a radius equal to the detected distance 320. Although only a single object 320 is shown in the example of FIG. 3A for ease of explanation, it will be understood that the distance sensor 312 can identify the distances from the distance sensor 312 to, and approximately corresponding angular positions of, multiple distinct objects 320 distributed around the field of view 310 within a single sensing operation, and this detection can be achieved using conventional techniques.

[0041] In addition to the distance sensor 312 detection, the controller in the CMS 15 identifies objects using object detection analysis on the image feed from the camera in the mirror replacement assembly. With continued reference to FIGS. 1-3A, FIG. 3B shows a schematic of object detection using image-based analysis in the CMS controller. First, the CMS analyzes the image in the field of view 410 of the camera 412 and identifies the object 402 using conventional object identification techniques that can be either rule-based, neural network-based, or a combination of rules and neural networks. After identifying the object, the image analysis process can further detect the angular position 422 of the object 402 relative to the camera 412 and help distinguish between multiple detected objects. Unlike the distance sensor example of FIG. 3A, the image-based analysis is difficult to determine the exact distance between the object 402 and the camera 412, and therefore difficult to determine the change in distance, but at the same time, is suitable for determining the exact angular position of the object.

[0042] With continued reference to Figures 3A and 3B, Figure 4 shows a schematic of the fusion of the range data from Figure 3A with the angular position data from Figure 3B using a nearest neighbor matching fusion technique to combine the range data and angular position data of each detected object. In the alternative, an alternative fusion technique (e.g., the Hungarian algorithm) may be used. The nearest neighbor matching method identifies the detected position of the radar object 320 and the detected position of the detected object 402 based on the image. In addition to the detected position, at least one tracked object 520 is stored in the controller that runs the fusion algorithm. The tracked object 520 includes the last known fused range and angular position of the tracked object 520.

[0043] Each radar object 320 and each image detection object 402 is correlated with the closest tracking object 520 as a "nearest neighbor". If both the radar object 320 and the image detection object 402 are correlated with a single tracking object 520, the tracking object 520 updates its range and angular position data such that the range data of the radar object 320 and the angular position data of the image detection object 402 become the new range and angular position data of the tracking object 520. In some examples, such as the example of FIG. 6, multiple tracking objects 620 can be utilized to fuse multiple object detections. While the example of FIG. 6 is shown to include three tracking objects 620, it is understood that any number of tracking objects can be utilized.

[0044] If the tracked object 520 is not within the maximum cost number of the radar object 320 and image-based detected object 420 pair, a new tracked object 520 is created using its angular position from the image-based detected object 402 and its distance from the radar object 320. The maximum cost number is a weighted sum of the range difference, velocity difference and angle difference between at least one object and the tracked object, ensuring that the closest tracked object is not correlated with a different object.

[0045] If a tracked object 520 is the closest tracked object to multiple detections of the same type (e.g., a single tracked object 520 is the closest tracked object 520 to two separate radar detections 320), the tracked object 520 is correlated with the closest of the two separate radar detections 320, and the other radar detection 320 is paired with the second-closest tracked object 520.

[0046] In some exemplary systems, where computational power is expensive, the number of available tracked objects 520 may be limited to at most 16 tracked objects. In such cases, radar and image based object detections beyond a certain distance from the vehicle are ignored. In addition to providing a fused position of the tracked object 520, the fused data may in some examples include historical data that provides a position history, allowing the movement of the tracked object 520 to be followed.

[0047] 3A and 3B, Fig. 5 is a flow chart illustrating a method for fusing data from each of the range sensor 312 and the camera 412, thereby identifying the range, velocity and position of each identified object. Initially, objects are identified by the range sensor 312 in "identify objects by range sensor" step 410, and simultaneously, objects are identified based on image analysis in "identify objects by image analysis" step 420. Objects identified using the image-based analysis of step 420 are transformed at least in part from their two-dimensional location in the image to a three-dimensional location relative to the camera 412 in "convert image coordinates to three-dimensional coordinates" step 430.

[0048] After converting the image coordinates to three-dimensional coordinates, the controller correlates the distance detections from the distance sensor 312 with the image detections from the camera 412 in a "Correlate Distance Detections and Image Detections" step 440. In one example, the correlation is accomplished using the nearest neighbor matching method described above. The correlation provides a one-to-one mapping of each object identified by the image analysis to each object identified by the distance sensor 312 using the tracked objects 520. The tracked objects 520 may alternatively be referred to as fused objects since they track fused data.

[0049] Once combined by the fusion algorithm, the distance, relative velocity, and angular position for the object are output by a process in step 450 "Output Range, Velocity, Angular Position for Object". In some examples, the information is output from the controller to a separate vehicle system via a vehicle communication bus (such as a CAN bus). In other examples, the information is output from this process to one or more additional processes in the CMS controller. As an example, the CMS controller may use the distance and angular position of a given object to generate human machine interface elements displayed to a vehicle operator, determine the position of a distance-based indicator in a mirror replacement display, etc. In an alternative example, the information may be provided to an advanced driver assistance system (ADAS), such as a blind spot warning, lane change assist, rear cross traffic warning, or any similar system.

[0050] By locating the distance sensor 50 within the housing of the camera arms 16a, 16b and pointing the field of view of the distance sensor 50 and the cameras 20a, 20b in the same rearward direction, the CMS controller can utilize fusion algorithms to achieve highly accurate data regarding objects detected at the sides and rear of the commercial vehicle. Additionally, although described above with respect to individual objects, it will be appreciated that the disclosed systems and processes can be used to simultaneously detect and analyze multiple separate objects without requiring additional modifications.

[0051] In addition to improving the ability to fuse the data using fusion algorithms by locating the distance sensor 312 and the camera 412 in approximately the same location, the inclusion of the distance sensor within the camera wings 16a, 16b allows a single heater to provide de-icing for both the camera 412 and the distance sensor 312. Additionally, including both within the same housing allows a single alignment system to be utilized to maintain alignment of the camera 412 and the distance sensor 312.

[0052] Although exemplary embodiments have been disclosed, a person of ordinary skill in this art would recognize that certain modifications would come within the scope of the following claims, and for that reason the following claims should be studied to determine their true scope and content.

Claims

1. a first mirror replacement assembly including a housing supporting a rear-facing camera and a rear-facing distance sensor; a controller in communication with the first mirror replacement assembly, the controller including a processor and a memory, the memory storing instructions for identifying at least one object in an image feed from the camera and determining an angular position of the object using image analysis, identifying a distance of the at least one object from the first mirror replacement assembly using the distance sensor, and fusing the distance and the angular position of the object into a single data set; a display coupled to the controller and configured to display a mirror-replacement view including at least a portion of the image feed; A camera surveillance system comprising:

2. The camera surveillance system of claim 1 , wherein the distance sensor is a time-of-flight sensor.

3. The camera surveillance system of claim 2 , wherein the time-of-flight sensor comprises a radar sensor.

4. The camera surveillance system of claim 2 , wherein the time-of-flight sensor comprises a lidar distance sensor.

5. The camera surveillance system of claim 1 , wherein the controller is further configured to determine a velocity of an object identified using the distance sensor.

6. The camera surveillance system of claim 1 , wherein the controller is further configured to communicate the combined angular position and distance to at least one advanced driver assistance system.

7. The camera surveillance system of claim 1 , wherein the first mirror replacement assembly is disposed within a first camera wing of a commercial vehicle.

8. The camera surveillance system of claim 1 , further comprising a second mirror replacement assembly substantially identical to the first mirror replacement assembly.

9. the first mirror replacement assembly is located on a driver's side of the vehicle; 9. The camera surveillance system of claim 8, wherein the second mirror replacement assembly is located on a passenger side of the vehicle.

10. The camera surveillance system of claim 1 , wherein the controller is configured to fuse the distances and the angular positions of the objects into the single data set using a nearest neighbor matching algorithm.

11. The camera surveillance system of claim 10 , wherein the nearest neighbor matching algorithm is configured to correlate the angular position of the at least one object and the distance of the at least one object with tracked objects within a maximum cost number of the at least one object.

12. The camera surveillance system of claim 11 , wherein the maximum cost number is a weighted sum of the range difference, the velocity difference, and the angle difference between the at least one object and the tracked object.

13. 1. A method for improving object detection in a mirror replacement system, comprising: a controller receiving at least one object detection from a rear-facing time-of-flight sensor in a vehicle wing; the controller receiving image data from a rear-facing camera in the vehicle wing and using an image analysis process to identify the at least one object in the image data; and fusing the angular position of the at least one object from the image data with at least a distance from the time-of-flight sensor to generate a fused tracked object. A method comprising:

14. The method of claim 13 , wherein the time-of-flight sensor is one of a radar sensor and a lidar sensor.

15. 14. The method of claim 13, wherein fusing the angular position of the at least one object from the image data with at least a distance from the time-of-flight sensor to generate a fused tracked object includes fusing object velocity data from the time-of-flight sensor.

16. The method of claim 13 , wherein fusing the angular position of the at least one object from the image data with at least a distance from the time-of-flight sensor is performed by the controller using a nearest neighbor matching algorithm.

17. The method of claim 16 , wherein the nearest neighbor matching algorithm is configured to correlate the angular position of the at least one object and the distance of the at least one object with tracked objects within a maximum cost number of the at least one object.

18. The method of claim 17 , wherein the maximum cost number is a weighted sum of range difference, velocity difference, and angle difference between the at least one object and the tracked object.

Images