Vehicle camera lens shield

Abstract

A method for handling vehicle camera lens occlusion includes obtaining an image from a vehicle camera, generating a first camera lens smudge map from pixel texture richness and chromatographic levels extracted from the image, obtaining a sequential feed image of a plurality of frames from the vehicle camera, and generating a second camera lens smudge map from the sequential feed image. Generating a second camera lens smudge map from the at least one highlighted region extracted from the plurality of frames, combining the first camera lens smudge map and the second camera lens smudge map to generate a combined camera lens smudge map, and generating a second camera lens smudge map from the combined camera lens smudge map, at least one of automatic lens cleaning or camera lens occlusion signals is initiated according to the combined camera lens smudge map.

Description

[0001] introduce

[0002] The information provided in this section is for the purpose of generally presenting the context of the present disclosure. From the description in this section, the work of the presently named inventors and aspects of the description that may not constitute prior art at the time of filing should neither be explicitly nor implicitly admitted to be prior art to the present disclosure.

[0003] The present disclosure generally relates to vehicle camera lens occlusion, including detecting vehicle camera lens occlusion to control automatic lens cleaning and other vehicle control features.

[0004] Some vehicles include cameras, such as a rear camera that provides a camera feed to the driver when backing up. Other vehicle systems may also use the camera image to sense the surrounding environment and control various vehicle functions, such as controlling an audible alarm when an object is detected near the rear of the vehicle. Sometimes, dirt or other debris may collect on the lens of a vehicle camera, negatively affecting systems that rely on accurate images captured from the vehicle camera. Summary of the invention

[0005] A method for processing vehicle camera lens occlusion includes: obtaining an image from a vehicle camera, generating a first camera lens dirtiness map based on pixel texture richness and color spectrum levels extracted from the image, obtaining a plurality of frames of sequential feed images from the vehicle camera, generating a second camera lens dirtiness map based on at least one prominent area extracted from the plurality of frames, combining the first camera lens dirtiness map and the second camera lens dirtiness map to generate a combined camera lens dirtiness map, and initiating at least one of automatic lens cleaning or a camera lens occlusion signal based on the combined camera lens dirtiness map.

[0006] In other features, initiating automatic lens cleaning includes at least one of dispensing a cleaning solution onto a lens of a vehicle camera or moving a wiper across a surface of the lens.

[0007] In other features, initiating the camera lens block signal includes alerting at least one vehicle motion control system of the lens blockage of the vehicle camera to inhibit generation of at least one of automatic vehicle braking or an audible warning based on input received from the vehicle camera.

[0008] In other features, the method includes extracting a texture sub-region in the image, discretizing an image blur level based on a pixel texture richness of the texture sub-region, and extracting a color spectrum sub-region of the image, wherein a first camera lens dirtiness map is generated by combining the image blur level and the color spectrum sub-region.

[0009] In other features, the method includes extracting and tracking at least one salient region in a plurality of frames, and detecting at least one occlusion region of a vehicle camera based on the at least one salient region, wherein a second camera lens dirtiness map is generated based on the at least one occlusion region.

[0010] In other features, the method includes obtaining at least one vehicle motion signal and generating a timeline of pixel statistics in multiple frames based on the at least one vehicle motion signal, wherein at least one salient region is tracked based on the timeline of pixel statistics in the multiple frames, and wherein at least one occluded region is detected based on the timeline of pixel statistics in the multiple frames.

[0011] In other features, tracking the at least one salient region includes identifying locations in a plurality of frames where objects detected during motion of the vehicle repeatedly disappear.

[0012] In other features, the at least one vehicle motion signal includes at least one of a steering wheel rotation signal and a wheel speed sensor signal. In other features, combining the first camera lens dirtiness map and the second camera lens dirtiness map includes applying a weighted average to the first camera lens dirtiness map and the second camera lens dirtiness map to generate a combined camera lens dirtiness map.

[0013] In other features, obtaining the image includes obtaining the image from a raw camera feed of a vehicle camera, and generating the first camera lens dirtiness map includes generating an information map based on the image from the raw camera feed, and determining the effective perception area based on the information map. In other features, the vehicle camera includes a rear backup camera.

[0014] A vehicle camera lens occlusion detection system comprises: a vehicle camera mounted on a vehicle, the vehicle camera comprising a lens, and the vehicle camera is configured to capture images; and a vehicle control module communicating with the vehicle camera, the vehicle control module being configured to receive a plurality of frames of sequentially fed images from the vehicle camera, extract and track at least one protruding area in the plurality of frames, detect at least one occlusion area of ​​the vehicle camera based on the at least one protruding area, generate a camera lens dirtiness map based on the at least one occlusion area, and initiate at least one of automatic lens cleaning or a camera lens occlusion signal based on the camera lens dirtiness map.

[0015] In other features, the vehicle control module is configured to obtain at least one vehicle motion signal and generate a timeline of pixel statistics in multiple frames based on the at least one vehicle motion signal, wherein at least one salient region is tracked based on the timeline of pixel statistics in the multiple frames, and wherein at least one occluded region is detected based on the timeline of pixel statistics in the multiple frames.

[0016] In other features, the vehicle control module is configured to track the at least one salient region by identifying locations in a plurality of frames where objects detected during vehicle motion repeatedly disappear.

[0017] In other features, the at least one vehicle motion signal includes at least one of a steering wheel rotation signal and a wheel speed sensor signal. In other features, initiating automatic lens cleaning includes at least one of dispensing a cleaning solution onto a lens of a vehicle camera or moving a wiper across a surface of the lens.

[0018] In other features, initiating the camera lens block signal includes alerting at least one vehicle motion control system of a lens blockage of a vehicle camera to inhibit generation of at least one of automatic vehicle braking or an audible warning based on input received from the vehicle camera. In other features, the vehicle camera includes a rear backup camera.

[0019] A vehicle camera lens occlusion detection system includes: a vehicle camera mounted on a vehicle, the vehicle camera including a lens, and the vehicle camera is configured to capture an image; and a vehicle control module communicating with the vehicle camera, the vehicle control module being configured to receive the image from the vehicle camera, extract a texture sub-region of the image, discretize an image blur level according to pixel texture richness of the texture sub-region, extract a color spectrum sub-region of the image, generate a camera lens dirtiness map according to the pixel texture richness of the texture sub-region and the color spectrum sub-region of the image, and initiate at least one of automatic lens cleaning or a camera lens occlusion signal according to the camera lens dirtiness map.

[0020] In other features, initiating automatic lens cleaning includes at least one of dispensing a cleaning solution onto a lens of a vehicle camera or moving a wiper across a surface of the lens, and initiating a camera lens occlusion signal includes alerting at least one vehicle motion control system of a lens occlusion of a vehicle camera to inhibit generation of at least one of automatic vehicle braking or an audible warning based on input received from the vehicle camera.

[0021] Other areas of applicability of the present disclosure will become apparent from the detailed description, claims and drawings.The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present disclosure can be more fully understood from the detailed description and accompanying drawings, in which:

[0023] Figure 1 is a diagram of an example vehicle including a vehicle control module configured to detect a vehicle camera lens occlusion.

[0024] Figure 2is a flow chart depicting one exemplary process for detecting lens occlusion of a vehicle camera.

[0025] Figure 3 is a flow chart depicting an exemplary process for detecting lens occlusion from a single camera image.

[0026] Figure 4 is a flow chart depicting an exemplary process for detecting lens occlusions based on a sequential feed of multiple vehicle camera lens frames.

[0027] Figure 5A and 5B is an exemplary illustration of an image information map and an effective perception area of ​​a clear rear view camera lens.

[0028] Fig. 6A and 6B is an exemplary illustration of the image information map and effective perception area of ​​a fully occluded rear-view camera lens.

[0029] Fig. 7A and 7B is an exemplary illustration of the image information map and effective perception area of ​​a partially occluded rear-view camera lens.

[0030] Fig. 8A and 8B are exemplary illustrations of a clear rear view camera lens and a partially obstructed rear view camera lens.

[0031] Among the drawings, reference numerals may be repeated to identify similar and / or identical elements. DETAILED DESCRIPTION

[0032] Vehicle cameras are sometimes obscured by dirt, water, or other debris. Some exemplary embodiments described herein implement a lightweight computing method to detect an obstruction area or range of a vehicle camera lens and provide an obstruction detection signal to support vehicle camera viewing functions, cleaning functions, sensing features, etc. The obscured area may include an obstruction of at least a portion of a vehicle camera lens caused by particles from dirt, water, snow, ice, mud, etc., which may be generally referred to as a lens dirt area or a lens dirtiness map.

[0033] One exemplary detection algorithm includes two components for detecting occluded vehicle camera lens areas, although other exemplary embodiments may use each component alone, or combine (one or more) components with other detection features. For example, in the case of a stationary vehicle (or when a single camera image is acquired), texture richness and color spectrum levels may be extracted from the camera image and used to estimate occlusion of the vehicle camera lens.

[0034] In various embodiments, the vehicle camera depth of field may be greater than zero (such as approximately 20 centimeters, etc.), and the ground and normal natural scenes around the vehicle are rich in texture. Therefore, the vehicle control module may be configured to detect areas with high levels of blurriness and lack of color in the captured camera image frames as areas of occlusion of the vehicle camera lens.

[0035] In some example embodiments, the vehicle control module may be configured to estimate camera lens occlusion when the vehicle is moving based on tracking the protruding area across multiple camera frames in conjunction with the vehicle motion signal. For example, when the vehicle motion signal indicates known motion of the vehicle, the vehicle control module may track the protruding area across multiple camera frames and / or track the protruding area between multiple cameras as a strong indicator of camera lens occlusion (e.g., dirt occlusion).

[0036] Reference now Figure 1 , the vehicle 10 includes front wheels 12 and rear wheels 13. Figure 1 , the drive unit 14 selectively outputs torque to the front wheels 12 and / or the rear wheels 13 through drive lines 16, 18, respectively. The vehicle 10 may include different types of drive units. For example, the vehicle may be an electric vehicle, such as a battery electric vehicle (BEV), a hybrid vehicle or a fuel cell vehicle, a vehicle including an internal combustion engine (ICE), or other types of vehicles.

[0037] Some examples of drive unit 14 may include any suitable electric motor, a power inverter, and a motor controller configured to control a power switch in the power inverter to adjust motor speed and torque during propulsion and / or regeneration. During propulsion or regeneration, the battery system provides power to or receives power from the electric motor of drive unit 14 via the power inverter.

[0038] Although in Figure 1 The vehicle 10 includes a drive unit 14, but the vehicle 10 may have other configurations. For example, two separate drive units that can drive the front wheels 12 and the rear wheels 13, one or more separate drive units that can drive separate wheels, etc. It is understood that other vehicle configurations and / or drive units may also be used.

[0039] The vehicle control module 20 may be configured to control the operation of one or more vehicle components (e.g., drive unit 14) (e.g., by commanding torque settings of the electric motor of drive unit 14). The vehicle control module 20 may receive inputs for controlling vehicle components, such as signals received from a steering wheel, an accelerator pedal, a brake pedal, etc. The vehicle control module 20 may monitor the vehicle's telematics for safety purposes (e.g., vehicle speed, vehicle position, vehicle braking and acceleration, etc.).

[0040] The vehicle control module 20 may receive signals from any suitable component for monitoring one or more aspects of the vehicle, including one or more vehicle sensors (e.g., cameras, microphones, pressure sensors, wheel position sensors, position sensors such as global positioning system (GPS) antennas, etc.). Some sensors may be configured to monitor the current motion of the vehicle, the acceleration of the vehicle, steering torque, etc.

[0041] like Figure 1 As shown, the vehicle 10 includes a rear camera 24, an optional front camera 26, and an optional side camera 28. In various embodiments, the vehicle 10 may also include more or less (or no) of any of these vehicle cameras. Each vehicle camera may include any suitable camera lens and image capture components, and may optionally be associated with a laser, a lidar sensor, etc.

[0042] The vehicle camera may be used to detect objects around the vehicle 10. In some exemplary embodiments, the vehicle camera may be configured to detect a closest in-transit vehicle (CIPV) (e.g., another vehicle ahead of the current driving path of the vehicle 10), a vulnerable road user (VRU) (e.g., a pedestrian or cyclist), may be implemented as a backup camera for controlling automatic braking or an audible alarm when the vehicle is in reverse, etc.

[0043] Data obtained from the vehicle camera can be used to control automatic vehicle motion control features, such as automatic acceleration and braking of the vehicle based on sensing objects around the vehicle 10. In some cases, the lens of the vehicle camera may be obscured (due to dirt, water, snow, ice, etc.). If the vehicle control module 20 cannot receive a clear image from the vehicle camera, it may negatively affect the vehicle control function.

[0044] In various embodiments, the automatic lens cleaning system may be controlled by a signal from the vehicle control module 20. For example, a cleaning solution may be released or sprayed onto the camera lens in response to a control signal from the vehicle control module 20, a wiper or other component may be moved across the lens surface in response to a control signal, a heater may be activated to melt snow or ice on the lens, and the like.

[0045] The vehicle control module 20 can communicate with other devices via a wireless communication interface, which may include one or more wireless antennas for transmitting and / or receiving wireless communication signals. For example, the wireless communication interface can communicate via any suitable wireless communication protocol including but not limited to vehicle-to-everything (V2X) communication, Wi-Fi communication, wireless area network (WAN) communication, cellular communication, personal area network (PAN) communication, short-range wireless communication (e.g., Bluetooth), etc. The wireless communication interface can communicate with a remote computing device via one or more wireless and / or wired networks. With respect to vehicle-to-vehicle (V2X) communication, the vehicle 10 may include one or more V2X transceivers (e.g., V2X signal transmission and / or reception antennas).

[0046] The vehicle 10 may also include a user interface. The user interface may include any suitable display (such as on a dashboard, console, or elsewhere), a touch screen or other input device, a speaker for generating audio, etc. In some exemplary embodiments, the vehicle control module 20 may be configured to provide a visual notification of the recommended evasive steering strategy on the user interface, and / or generate an audio alert of the recommended evasive steering strategy.

[0047] Figure 2 is a flow chart depicting an exemplary process for detecting lens occlusion of a vehicle camera. The process may be performed by, for example, Figure 1 The process begins at 204 by obtaining an image from a vehicle camera, such as by Figure 1 The image captured by the rear camera 24 of the vehicle. In some example embodiments, the image may be a single static frame captured by the vehicle camera.

[0048] At 208, the vehicle control module is configured to generate a first camera lens dirtiness map based on the extracted image features. For example, one or more features such as blurriness level, pixel texture richness area (e.g., sub-area of ​​the image), color spectrum level (e.g., sub-area with various color spectrums), etc. can be extracted (e.g., analyzed, determined, obtained, etc.) from the image.

[0049] Reference below Figure 3 Additional details of generating the first camera lens dirtiness map are further described.A camera lens dirtiness map may generally refer to a map of predicted occluded regions of a camera lens, which may be caused by particles of dirt, mud, water, snow, ice, etc.

[0050] At 212, the vehicle control module is configured to obtain a sequential feed of multiple frames from a vehicle camera. For example, the vehicle control module may receive video from a rear camera 24 that includes multiple sequential frames. The multiple sequential frames may include any suitable number of frames (such as at least ten frames, at least 100 frames, etc.), or may include multiple frames within a fixed time period (such as one second of frames, ten seconds of frames, one minute of frames, etc.). Frames may be selected at periodic intervals, such as selecting one of every five image frames received from the vehicle camera, one of every twenty image frames received from the vehicle camera, and so on.

[0051] At 216, control is configured to generate a second camera lens dirtiness map based on features extracted from the plurality of frames. Figure 4 Additional details of generating a second camera lens dirtiness map based on features extracted from a plurality of frames are further described.

[0052] At 220, the vehicle control module is configured to combine the first camera lens dirtiness map and the second camera lens dirtiness map. In this way, lens occlusion can be accurately detected by combining analysis of features extracted from a single static image in conjunction with features determined from a sequence of frames (e.g., a video) to analyze the motion of an object throughout the frames.

[0053] In other exemplary embodiments, only analysis of a single static image may be used to detect lens occlusion (e.g., Figure 3 ), or the occlusion of the lens may be detected using only analysis of a sequence of multiple frames (e.g., as described and shown in Figure 4 ).

[0054] At 224, the vehicle control module is configured to determine whether the combined image indicates that a lens occlusion exists. For example, if at least one occluded region is identified (e.g., due to an occlusion or invalid sensing region in the processed image map data), the vehicle control module can determine that at least a portion of the lens is occluded. Figure 2 FIG. 4 shows the determination of whether there is lens occlusion based on the combined image, but in other exemplary embodiments, the determination of lens occlusion may also be based on the analysis of a single static image (for example, Figure 3 ), or the lens occlusion may be based solely on an analysis of a sequence of multiple frames (e.g., as described and shown in Figure 4 ) as described and shown in .

[0055] If control determines at 224 that there is no lens obstruction, control returns to 204 to obtain other images from the vehicle camera. The lens obstruction check may be performed at a specific periodic interval, such as every second, every minute, every hour, etc.

[0056] If control determines at 224 that the combined image indicates a lens obstruction, control proceeds to 228 to initiate a camera lens obstruction signal. The camera lens obstruction signal may indicate to other vehicle systems that at least a portion of the vehicle camera image is currently obstructed, and therefore other systems may not be able to reliably use the vehicle camera image for control functions. For example, a rear brake system may not rely solely on the camera image to determine whether an object is present behind the vehicle to control automatic braking or audible alarm generation.

[0057] At 232, control determines whether the vehicle is equipped with an automatic lens cleaning system. If so, control may initiate automatic vehicle camera lens cleaning at 236. For example, the vehicle control module may send a control signal to spray a cleaning solution on the vehicle camera lens, may activate wipers to scrub the vehicle camera lens surface, etc.

[0058] Figure 3 is a flow chart depicting an exemplary process for detecting lens occlusion based on a single camera image. The process may be performed by, for example, Figure 1 The vehicle control module 20 is executed. At 304, the process begins by obtaining an image from a vehicle camera.

[0059] At 308, the vehicle control module is configured to extract a texture sub-region in the image. Then, at 312, control discretizes the image blur level according to the pixel texture richness of the texture sub-region. For example, the image texture may include one or more metrics calculated via image processing to quantify the perceived texture of the image. The pixel image texture may provide information about the spatial arrangement of color or brightness in the image or a selected region of the image, such as by using segmentation or classification to identify, for example, spatial frequency, average gray level, etc.

[0060] At 316 , the vehicle control module is configured to extract a color spectrum sub-region of the image. For example, the color spectrum information can be obtained by performing image processing on the image to obtain the location of color values, the range of color values ​​used in the image, color patterns or matches in the image, etc.

[0061] At 320, the vehicle control module is configured to generate a camera lens dirtiness map based on the pixel texture richness and the color spectrum level. The camera lens dirtiness map can be used alone to determine whether lens occlusion exists, or can be used in conjunction with a camera lens dirtiness map based on, for example, Figure 2 The image sequence is combined with a camera lens dirtiness map.

[0062] Figure 4 is a flow chart depicting an exemplary process for detecting lens occlusion based on a sequential feed of multiple vehicle camera lens frames. The process may be performed by, for example, Figure 1The process is performed by the vehicle control module 20. At 404, the process starts by obtaining a plurality of frames of sequential feed images from a vehicle camera.

[0063] At 408, control obtains at least one vehicle motion signal. For example, the vehicle control module may receive a steering wheel rotation value, a wheel speed sensor value, etc. At 412, the vehicle control module is configured to generate a timeline of pixel statistics in a plurality of frames based on the vehicle motion sensor. For example, the vehicle control module may obtain statistics for a plurality of frames via image processing and associate the image statistics with the motion of the vehicle detected at different points in time.

[0064] The vehicle control module is configured to extract and track at least one salient region in the plurality of frames at 416. For example, the vehicle control module may be configured to process the plurality of frames to determine an expected object path in the frames. If a rock is detected in the rear camera footage as the vehicle moves forward, the vehicle control module would expect the rock to move further away and smaller in the image as the vehicle moves forward.

[0065] If the vehicle control module detects that the identified object (e.g., a rock) disappears from the image faster than expected, it can determine that there is a lens occlusion at the disappeared area. If the unexpected disappearance of multiple objects at the same location in multiple frames continues, the vehicle control module can provide increased confidence that there is a lens occlusion at the same location.

[0066] At 420, control detects at least one occlusion region of the vehicle camera based on the at least one protrusion region. For example, as described above, if the vehicle control module identifies an area in the image where a tracked object unexpectedly disappears when the vehicle moves, control can determine that an occlusion region exists at that area.

[0067] At 424, the vehicle control module is configured to generate a camera lens dirtiness map based on at least one occlusion region. The camera lens dirtiness map can be used alone to determine whether a lens occlusion exists, or can be used in conjunction with a camera lens dirtiness map based on the following example: Figure 2 The camera lens dirtiness map is combined with a single static image.

[0068] Figure 5A and 5B is an exemplary illustration of an image information diagram and an effective perception area of ​​a clear rear-view camera lens. Figure 5A As shown, in a clear camera view, image information map 501 includes a normal amount of pixel detail.

[0069] like Figure 5BAs shown, the corresponding effective perception area 503 indicates a large area of ​​effective perception of the vehicle camera when the lens is clear. The effective perception area 503 can be determined by performing image processing on the pixel data of the image information graph 501.

[0070] Fig. 6A and 6B is an exemplary illustration of the image information diagram and effective perception area of ​​a completely occluded rear-view camera lens. Fig. 6A As shown in , in a fully occluded lens view, the image information graph 601 includes substantially no pixel details. Figure 6B As shown, when the lens is completely blocked, the corresponding effective sensing area 603 indicates that the vehicle camera has almost no effective sensing area.

[0071] Fig. 7A and 7B is an exemplary illustration of the image information diagram and effective perception area of ​​a partially obstructed rear-view camera lens. Fig. 7A As shown in , in a partially obstructed lens view, the image information map 701 includes some areas with a normal amount of pixel detail, and some areas with substantially no pixel detail. Figure 7B As shown, when the lens is partially blocked, the corresponding effective perception area 703 indicates some areas that are effectively perceived by the vehicle camera, but other areas that are not effectively perceived by the vehicle camera.

[0072] The vehicle control module can be configured to determine whether there is a lens occlusion based on information from the processed effective perception area. For example, the vehicle control module can determine that the vehicle lens is completely clear when analyzing the effective perception area 503, can determine that the vehicle lens is completely blocked when analyzing the effective perception area 603, and can determine the location of the partial lens occlusion based on the analysis of the effective perception area 703.

[0073] Fig. 8A and 8B are exemplary illustrations of a clear rear view camera lens and a partially obstructed rear view camera lens. Fig. 8A 801 shows the effective perception area when the vehicle's lens is clear. In contrast, Figure 8B The effective sensing area 803 includes an identified occlusion area 804. The vehicle control module can be configured to identify the area of ​​lens occlusion and send a lens occlusion signal to other vehicle systems that rely on the image from the vehicle camera or a specific area of ​​the image.

[0074] Some of the aspects explained above are summarized below by means of numbered examples.

[0075] Example 1. A method for processing vehicle camera lens occlusion, the method comprising:

[0076] Obtaining images from a vehicle camera;

[0077] generating a first camera lens dirtiness map based on pixel texture richness and color spectrum levels extracted from the image;

[0078] Obtaining a plurality of frames of sequential feed images from the vehicle camera;

[0079] generating a second camera lens dirtiness map based on at least one salient region extracted from the plurality of frames;

[0080] combining the first camera lens dirt map and the second camera lens dirt map to generate a combined camera lens dirt map; and

[0081] At least one of automatic lens cleaning or a camera lens occlusion signal is initiated based on the combined camera lens dirtiness map.

[0082] Example 2. The method of Example 1, wherein initiating automatic lens cleaning comprises at least one of dispensing a cleaning solution onto a lens of the vehicle camera or moving a wiper across a surface of the lens.

[0083] Example 3. A method according to Example 1, wherein the initiation of the camera lens occlusion signal includes alerting at least one vehicle movement control system of the lens occlusion of the vehicle camera to inhibit the generation of at least one of automatic vehicle braking or an audible warning based on input received from the vehicle camera.

[0084] Example 4. The method according to Example 1, further comprising:

[0085] extracting a texture sub-region of the image;

[0086] discretizing the image blur level according to the pixel texture richness of the texture sub-region; and

[0087] extracting a chromatographic sub-region of the image,

[0088] Wherein, the first camera lens dirtiness map is generated by combining the image blur level and the color spectrum sub-area.

[0089] Example 5. The method according to Example 1, further comprising:

[0090] extracting and tracking at least one salient region in the plurality of frames; and

[0091] detecting at least one blocked area of ​​the vehicle camera according to the at least one protruding area,

[0092] The second camera lens dirtiness map is generated according to the at least one occlusion area.

[0093] Example 6. The method according to Example 5, further comprising:

[0094] obtaining at least one vehicle motion signal; and

[0095] generating a timeline of pixel statistics in the plurality of frames based on the at least one vehicle motion signal,

[0096] wherein the at least one salient region is tracked based on a timeline of pixel statistics in the plurality of frames, and

[0097] Wherein, at least one occlusion area is detected according to a time line of pixel statistics in the plurality of frames.

[0098] Example 7. The method of Example 5, wherein tracking at least one salient region comprises identifying locations in the plurality of frames where objects detected during vehicle motion repeatedly disappear.

[0099] Example 8. The method of Example 6, wherein the at least one vehicle motion signal comprises at least one of a steering wheel rotation signal and a wheel speed sensor signal.

[0100] Example 9. The method of Example 1, wherein combining the first camera lens dirtiness map and the second camera lens dirtiness map comprises applying a weighted average to the first camera lens dirtiness map and the second camera lens dirtiness map to generate the combined camera lens dirtiness map.

[0101] Example 10. The method according to Example 1, wherein:

[0102] The obtaining of the image comprises obtaining the image from a raw camera feed of a camera of the vehicle; and

[0103] Generating the first camera lens dirtiness map includes generating an information map based on images from an original camera feed, and determining an effective perception area based on the information map.

[0104] Example 11. The method of Example 1, wherein the vehicle camera comprises a rear reversing camera.

[0105] Example 12. A vehicle camera lens occlusion detection system, comprising:

[0106] a vehicle camera mounted on the vehicle, the vehicle camera comprising a lens and configured to capture an image; and

[0107] a vehicle control module in communication with the vehicle camera, the vehicle control module being configured to:

[0108] receiving a plurality of frames of sequential feed images from the vehicle camera;

[0109] extracting and tracking at least one salient region in the plurality of frames;

[0110] detecting at least one blocked area of ​​the vehicle camera based on the at least one protruding area;

[0111] generating a camera lens dirtiness map according to the at least one occlusion region; and

[0112] At least one of automatic lens cleaning or a camera lens blocking signal is initiated according to the camera lens dirtiness map.

[0113] Example 13. The vehicle camera lens occlusion detection system according to Example 12, wherein the vehicle control module is configured to:

[0114] obtaining at least one vehicle motion signal; and

[0115] generating a timeline of pixel statistics in the plurality of frames based on the at least one vehicle motion signal,

[0116] wherein the at least one salient region is tracked based on a timeline of pixel statistics in the plurality of frames, and

[0117] Wherein, at least one occlusion area is detected according to a time line of pixel statistics in the plurality of frames.

[0118] Example 14. A vehicle camera lens occlusion detection system according to Example 13, wherein the vehicle control module is configured to track the at least one protruding area by identifying locations where detected objects in the multiple frames repeatedly disappear during movement of the vehicle.

[0119] Example 15. The vehicle camera lens occlusion detection system of Example 13, wherein the at least one vehicle motion signal comprises at least one of a steering wheel rotation signal and a wheel speed sensor signal.

[0120] Example 16. The vehicle camera lens occlusion detection system of Example 12, wherein initiating automatic lens cleaning comprises at least one of dispensing a cleaning solution onto the lens of the vehicle camera or moving a wiper across the surface of the lens.

[0121] Example 17. A vehicle camera lens occlusion detection system according to Example 12, wherein the initiation of the camera lens occlusion signal includes alerting at least one vehicle movement control system of the lens occlusion of the vehicle camera to inhibit the generation of at least one of automatic vehicle braking or an audible warning based on input received from the vehicle camera.

[0122] Example 18. A vehicle camera lens occlusion detection system according to Example 12, wherein the vehicle camera lens includes a rear reversing camera.

[0123] Example 19. A vehicle camera lens occlusion detection system, comprising:

[0124] a vehicle camera mounted on the vehicle, the vehicle camera comprising a lens and configured to capture an image; and

[0125] a vehicle control module in communication with the vehicle camera, the vehicle control module being configured to:

[0126] receiving an image from a camera on the vehicle;

[0127] extracting a texture sub-region of the image;

[0128] discretizing the image blur level according to the pixel texture richness of the texture sub-region;

[0129] extracting a color spectrum sub-region of the image;

[0130] generating a camera lens dirtiness map according to the pixel texture richness of the texture sub-region and the color spectrum sub-region of the image; and

[0131] At least one of automatic lens cleaning or a camera lens blocking signal is initiated according to the camera lens dirtiness map.

[0132] Example 20. A vehicle camera lens occlusion detection system according to Example 19, wherein:

[0133] The initiating automatic lens cleaning comprises at least one of dispensing a cleaning solution onto a lens of the vehicle camera or moving a wiper across a surface of the lens; and

[0134] The initiating a camera lens block signal includes alerting at least one vehicle motion control system of a lens blockage of a vehicle camera to inhibit generation of at least one of automatic vehicle braking or an audible alarm based on input received from the vehicle camera.

[0135] The foregoing description is essentially illustrative only, and there is no intention to limit the present disclosure, its application or use. The broad teachings of the present disclosure can be implemented in various forms. Therefore, although the present disclosure includes specific examples, the true scope of the present disclosure should not be limited thereto, because after studying the drawings, the specification and the appended claims, other modifications can be made apparent. It should be understood that one or more steps in the method can also be performed in different orders (or simultaneously) without changing the principles of the present disclosure. In addition, although each of the embodiments is described above as having certain features, any one or more of those features described with reference to any embodiment of the present invention can also be implemented in the features of any of the other embodiments and / or combined with the features of any of the other embodiments, even if the combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and the permutation of one or more embodiments to each other remains within the scope of the present disclosure.

[0136] Various terms are used to describe the spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.), including "connected," "engaged," "coupled," "adjacent," "next to," "on top," "above," "below," and "disposed." Unless explicitly described as "direct," when describing the relationship between a first and a second element in the above disclosure, the relationship can be a direct relationship in which there are no other intermediate elements between the first and second elements, but can also be an indirect relationship in which there are one or more intermediate elements (spatially or functionally) between the first and second elements. As used herein, the phrase at least one of A, B, and C should be interpreted as meaning a logical (A or B or C) using a non-exclusive logical "or," and should not be interpreted as meaning "at least one of A, at least one of B, and at least one of C."

[0137] In the drawings, the direction of the arrow, as indicated by the arrow, generally represents the flow of information (e.g., data or instructions) of interest to the illustration. For example, when component A and component B exchange various information, but the information transmitted from component A to component B is relevant to the illustration, the arrow may be marked as pointing from component A to component B. The unidirectional arrow does not imply that no other information is transmitted from component B to component A. In addition, for information sent from component A to component B, component B may also send a request for the information or a confirmation of receipt of the information to component A.

[0138] In this application, including the following definitions, the term "module" or the term "controller" may be replaced with the term "circuit". The term "module" may refer to, be part of, or include: an application specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the functionality; or a combination of some or all of the above, such as in a system on a chip.

[0139] The module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functions of any given module of the present disclosure may be distributed in multiple modules connected via the interface circuits. For example, multiple modules may allow load balancing. In other examples, a server (also referred to as a remote or cloud) module may perform some functions on behalf of a client module.

[0140] The term "code" as used above may include software, firmware and / or microcode, and may refer to a program, a routine, a function, a class, a data structure and / or an object. The term "shared processor circuit" covers a single processor circuit that executes some or all of the code from multiple modules. The term "group processor circuit" covers a processor circuit that executes some or all of the code from one or more modules in combination with an additional processor circuit. References to multi-processor circuits cover multi-processor circuits on discrete dies, multi-processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term "shared memory circuit" covers a single memory circuit that stores some or all of the code from multiple modules. The term "group memory circuit" covers a memory circuit that stores some or all of the code from one or more modules in combination with additional memory.

[0141] The term "memory circuit" is a subset of the term "computer-readable medium". The term "computer-readable medium" as used herein does not encompass transient electrical or electromagnetic signals propagated through a medium (such as on a carrier wave). Therefore, the term "computer-readable medium" may be considered to be both tangible and non-transitory. Non-limiting examples of non-transitory, tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tapes or hard drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0142] The apparatus and methods described in this application may be implemented in part or in whole by a special purpose computer created by configuring a general purpose computer to perform one or more specific functions contained in a computer program. The above-mentioned function blocks, flow chart components and other elements are used as software specifications, which can be converted into a computer program by routine work of a skilled technician or programmer.

[0143] The computer program includes processor executable instructions stored on at least one non-transitory tangible computer readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0144] A computer program may include: a descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated by a compiler from a source code; (iv) source code executed by an interpreter; (v) source code compiled and executed by a just-in-time compiler, etc. For example, a computer program may be used from languages ​​including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Fortran, Perl, Pascal, Curl, OCaml, HTML5 (Hypertext Markup Language, Revision 5), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Visual Lua, MATLAB, SIMULINK, and The syntax of the language is used to write source code.

Claims

1. A method for processing vehicle camera lens occlusion, the method comprising: Obtaining images from a vehicle camera; generating a first camera lens dirtiness map based on pixel texture richness and color spectrum levels extracted from the image; Obtaining a plurality of frames of sequential feed images from the vehicle camera; generating a second camera lens dirtiness map based on at least one salient region extracted from the plurality of frames; combining the first camera lens dirt map and the second camera lens dirt map to generate a combined camera lens dirt map; as well as At least one of automatic lens cleaning or a camera lens occlusion signal is initiated based on the combined camera lens dirtiness map.

2. The method according to claim 1, wherein: The initiating automatic lens cleaning includes at least one of dispensing a cleaning solution onto a lens of the vehicle camera or moving a wiper across a surface of the lens.

3. The method according to claim 1, wherein: The initiating camera lens block signal includes alerting at least one vehicle motion control system of the lens blockage of the vehicle camera to inhibit generation of at least one of automatic vehicle braking or an audible warning based on input received from the vehicle camera.

4. The method according to claim 1, further comprising: extracting a texture sub-region of the image; discretizing the image blur level according to the pixel texture richness of the texture sub-region; as well as extracting a chromatographic sub-region of the image, Wherein, the first camera lens dirtiness map is generated by combining the image blur level and the color spectrum sub-area.

5. The method according to claim 1, further comprising: extracting and tracking at least one salient region in the plurality of frames; as well as detecting at least one blocked area of ​​the vehicle camera according to the at least one protruding area, The second camera lens dirtiness map is generated according to the at least one occlusion area.

6. The method according to claim 5, further comprising: obtaining at least one vehicle motion signal; as well as generating a timeline of pixel statistics in the plurality of frames based on the at least one vehicle motion signal, wherein the at least one salient region is tracked based on a timeline of pixel statistics in the plurality of frames, and Wherein, at least one occlusion area is detected according to a time line of pixel statistics in the plurality of frames.

7. The method according to claim 5, wherein: Tracking at least one salient region includes identifying locations in the plurality of frames where detected objects repeatedly disappear during vehicle motion.

8. The method according to claim 6, wherein: The at least one vehicle motion signal includes at least one of a steering wheel rotation signal and a wheel speed sensor signal.

9. A vehicle camera lens occlusion detection system, comprising: a vehicle camera mounted on the vehicle, the vehicle camera comprising a lens, and the vehicle camera being configured to capture an image; as well as a vehicle control module in communication with the vehicle camera, the vehicle control module being configured to: receiving a plurality of frames of sequential feed images from the vehicle camera; extracting and tracking at least one salient region in the plurality of frames; detecting at least one blocked area of ​​the vehicle camera based on the at least one protruding area; generating a camera lens dirtiness map according to the at least one occluded area; as well as At least one of automatic lens cleaning or a camera lens blocking signal is initiated according to the camera lens dirtiness map.

10. A vehicle camera lens occlusion detection system, comprising: a vehicle camera mounted on the vehicle, the vehicle camera comprising a lens, and the vehicle camera being configured to capture an image; as well as a vehicle control module in communication with the vehicle camera, the vehicle control module being configured to: receiving an image from a camera on the vehicle; extracting a texture sub-region of the image; discretizing the image blur level according to the pixel texture richness of the texture sub-region; extracting a color spectrum sub-region of the image; generating a camera lens dirtiness map according to the pixel texture richness of the texture sub-region and the color spectrum sub-region of the image; as well as At least one of automatic lens cleaning or a camera lens blocking signal is initiated according to the camera lens dirtiness map.

Images