How to warn about dirt on car lenses

Abstract

A vehicle lens dirt warning method that provides faster and more accurate identification of dirt areas. [Solution] A method for warning dirt on an in-vehicle lens, comprising the steps of: acquiring a series of images of the shooting range by an in-vehicle lens when the vehicle is in a reversing state; forming a specific region by a processing module by acquiring at least three corner points in each of the Nth frame images of the series based on bounding box information, wherein the specific region does not cover the four corners of the Nth frame image and the area of ​​the specific region is smaller than the area of ​​the Nth frame image; performing dirt image feature detection on the Nth frame image of the series according to an image recognition model by the processing module to acquire dirt information including the dirt region; and if the processing module determines, based on the dirt information, that the ratio of the dirt area overlapping the specific region of the dirt region to the area of ​​the specific region exceeds a specific ratio, it generates a dirt warning.

Description

Technical Field

[0004] ,

[0001] The present invention relates to a method for warning of dirt on an in-vehicle lens. More specifically, the present invention relates to a method for warning of dirt on an in-vehicle lens that extracts an environmental image in a reverse state using the in-vehicle lens, immediately determines if the lens is dirty, warns the user, and achieves the purpose of improving driving safety.

Background Art

[0002] In recent years, due to the miniaturization of various imaging lenses and sensor elements, a large number of applications based on images and various environmental information have spread. Currently, applying imaging lenses to driving records or driving safety has become almost standard equipment for automobiles, motorcycles, and bicycles. In the case of automobiles, the currently commonly used electronic rearview mirror provides environmental detection of the rear of the automobile during reverse using an imaging lens with a pedestrian recognition function mounted on the rear of the automobile, and transmits the coordinates and relative speed of pedestrians behind the vehicle. This is used in combination with Autonomous Emergency Braking (AEB) to automatically brake the vehicle in a dangerous situation, effectively reducing the occurrence of traffic accidents.

[0003] However, since these imaging lenses are mainly exposed outside the vehicle, they may become dirty when driving under extreme road conditions. For example, when water droplets, mud, ice and snow on the road surface splash onto the surface of the camera and the camera becomes dirty, the gradient of the image decreases. This affects pedestrian recognition, cannot reflect actual road information, causes the control unit to perform incorrect analysis, reduces the safety of vehicle reverse, and greatly affects the safe driving of the automobile.

[0004] The dirt recognition means of conventional cameras has poor accuracy. Although it can be judged only by a single-frame picture, it is easily affected by factors such as light changes and background interference, has extremely poor fault tolerance, and low recognition accuracy. In addition, due to the distortion caused by shooting at the edge of the camera lens, problems of false detection and detection omission are likely to occur.

Summary of the Invention

[0005] This invention provides a method for warning of dirt on automotive lenses. Its purpose is to avoid false detections due to edge distortion by dividing the entire image into specific areas. Because the area that needs to be processed for recognition is small, the dirt warning method for automotive lenses of this invention can provide faster and more accurate determination of the dirt area. [Means for solving the problem]

[0006] The present invention provides a method for warning of dirt on an in-vehicle lens, applicable to a vehicle comprising an in-vehicle lens, a processing module, and a warning joule, the method comprising: when the vehicle is in reverse, acquiring at least one sequence of images of the shooting range with the in-vehicle lens; the processing module acquiring at least three corner points in the Nth frame of the at least one sequence of images based on bounding box information, and forming a specific region in the Nth frame of the image, wherein the specific region does not cover the four corners of the Nth frame of the image, and its area is smaller than the area of ​​the Nth frame of the image; the processing module performing dirt image feature detection on the Nth frame of the at least one sequence of images according to an image recognition model to acquire dirt information including the dirt region; and the processing module generating a dirt warning if it determines, based on the dirt information, that the ratio of the dirt area overlapping the specific region of the dirt region to the area of ​​the specific region exceeds a specific ratio.

[0007] In one embodiment of the present invention, the vehicle described above further comprises at least one memory cell for storing the boundary box information and the image recognition model, the at least one memory cell further includes a boundary box database and an image recognition model database.

[0008] In one embodiment of the present invention, the above-mentioned on-board lens is a retracted lens that is provided on the centerline of the vehicle and located at the rear of the vehicle.

[0009] In one embodiment of the present invention, the specific region of the Nth frame image described above is a first pedestrian recognition region located in the image coordinate system, and is used to detect whether or not a pedestrian is present at a specific position in the second pedestrian recognition region located in the world coordinate system after coordinate transformation when the pedestrian is in a backward state.

[0010] In one embodiment of the present invention, the processing module described above acquires at least three corner points in the Nth frame of at least one sequence of images based on bounding box information, and the step of forming a specific region in the Nth frame of images further includes selecting the at least three corner points as reference points based on the bounding box information, and defining the closed region enclosed by the lines connecting the at least three corner points as the specific region.

[0011] In one embodiment of the present invention, the specific region of the Nth frame image described above is a first pedestrian recognition region located in the image coordinate system. After coordinate transformation, the first pedestrian recognition region corresponds to a second pedestrian recognition region located in the world coordinate system. In the world coordinate system, where the position of the on-board lens is the origin of the X and Y axis coordinates, the second pedestrian recognition region is located within the region from -400 cm to 400 cm in the X-axis direction of the on-board lens, and within the region from 0 cm to 900 cm in the Y-axis direction of the on-board lens.

[0012] In one embodiment of the present invention, the area of ​​the specified region described above is 1 / 2 to 2 / 3 of the area of ​​the Nth frame image.

[0013] In one embodiment of the present invention, the above-described image recognition model is a semantic segmentation model, which is used to detect dirt image features and acquire dirt information, and the dirt information further includes the dirt located at the coordinate position of the specific region and its size.

[0014] In one embodiment of the present invention, the semantic segmentation model described above is a fully convolutional network model (FCN model), a U-net model, or an E-net model (efficient neural network model).

[0015] In one embodiment of the present invention, the specific ratio mentioned above is one of 10%, 15%, 20%, and 30%.

[0016] The effects of the present invention are as follows: By dividing the entire image acquired by the lens into specific areas, the drawbacks of distortion and low resolution of the screen captured at the edge of the lens are avoided. Furthermore, by using an AI recognition model to accurately recognize whether the area of ​​dirt in the specific area exceeds a certain ratio, and issuing a warning, the time required to judge dirt using the entire screen as in the conventional method is effectively reduced, recognition accuracy is improved, and false detections and missed detections are further reduced. [Brief explanation of the drawing]

[0017] [Figure 1] This is a step flowchart of the vehicle lens dirt warning method according to the present invention. [Figure 2] This is a schematic diagram of the contaminated area in the vehicle lens contaminated warning method according to the present invention. [Figure 3] This is a schematic diagram of the first pedestrian recognition area of ​​the vehicle lens dirt warning method according to the present invention. [Figure 4]It is a schematic diagram of the second pedestrian recognition area of the in-vehicle lens dirt warning method according to the present invention. [Figure 5] It is a schematic diagram of the calibration method of the second pedestrian recognition area of the in-vehicle lens dirt warning method according to the present invention.

Embodiments for Carrying Out the Invention

[0018] In order to make the above-mentioned features and advantages of the present invention clearer and easier to understand, the following examples will be given and described in detail with reference to the accompanying drawings.

[0019] FIG. 1 is a flowchart of the in-vehicle lens dirt warning method according to the present invention. The flowchart is as follows.

[0020] Step S110: When the vehicle is in a reverse state, at least one continuous image of the shooting range is acquired by the in-vehicle lens.

[0021] Step S120: The processing module acquires at least three corner points respectively in the Nth frame video of the at least one continuous image based on the bounding box information, forms a specific area in the Nth frame video, and the specific area does not cover the four corners of the Nth frame video and its area is smaller than the area of the Nth frame video.

[0022] Step S130: The processing module performs dirt video feature detection on the Nth frame video of the at least one continuous image according to the video recognition model to obtain dirt information including the dirt area.

[0023] Step S140: When the processing module determines that the ratio of the dirt area overlapping the specific area to the area of the specific area exceeds a specific ratio based on the dirt information, a dirt warning is generated.

[0024] In this embodiment, the vehicle further includes at least one memory cell for storing the boundary box information and the video recognition model. These at least one memory cell further includes a boundary box database and a video recognition model database.

[0025] In this embodiment, the in-vehicle lens is a reverse lens provided on the center line of the vehicle and located behind the vehicle.

[0026] In this embodiment, the specific area of the Nth frame video is the first pedestrian recognition area located in the video coordinate system, and is used to detect whether there is a pedestrian at a specific position in the second pedestrian recognition area located in the world coordinate system after coordinate transformation when in the reverse state.

[0027] In this embodiment, based on the boundary box information, at least three corner points are selected as reference points, and a closed area surrounded by a line connecting the at least three corner points is used as the specific area. The specific area may be a circular area, an elliptical area, a triangular area, a quadrilateral area or a polygonal area, but is not limited thereto.

[0028] In this embodiment, the specific area of the Nth frame video is the first pedestrian recognition area located in the video coordinate system. After coordinate transformation, the first pedestrian recognition area corresponds to the second pedestrian recognition area located in the world coordinate system. In the world coordinate system with the position of the in-vehicle lens as the origin of the X-axis and Y-axis coordinates, the second pedestrian recognition area is located within the area from -400 cm to 400 cm in the X-axis direction of the in-vehicle lens and within the area from 0 cm to 900 cm in the Y-axis direction of the in-vehicle lens. The second pedestrian recognition area is located in a quadrilateral area in the world coordinate system. Preferably, the quadrilateral area is 9m×8m (length×width).

[0029] In this embodiment, the area of the specific area is from 1 / 2 to 2 / 3 of the area of the Nth frame video.

[0030] In this embodiment, the image recognition model is a semantic segmentation model, used to detect dirt features in the image and acquire dirt information. This dirt information further includes the location and size of the dirt in the specific region. The semantic segmentation model acquires multiple semantic segmentation areas for the Nth frame of each of the continuous images, and performs dirt feature detection.

[0031] In this embodiment, the semantic segmentation model is a fully convolutional network model (FCN model), a U-net model, or an E-net model (efficient neural network model).

[0032] In this embodiment, the specific ratio is one of 10%, 15%, 20%, and 30%. If it is determined that the soiled area is equal to or greater than the specific ratio of the total area of ​​the specific region (for example, the specific ratio is 30%), the warning unit will present a warning based on the soiling, either as a lamp signal or an audible warning.

[0033] Figure 2 is a schematic diagram of a dirty area in the automotive lens dirt warning method according to the present invention. As shown in Figure 2, the processing module obtains at least three corner points in the Nth frame video 1 of at least one sequence of images based on bounding box information, forms a specific region 11 in the Nth frame video 1, the specific region 11 does not cover the four corners of the Nth frame video 1, its area is smaller than the area of ​​the Nth frame video 1, and dirt video feature detection is performed on the Nth frame video 1 of at least one sequence of images according to an image recognition model to obtain dirt information including the dirty area 12. Based on the dirt information, the processing module determines that the ratio of the dirt area of ​​the dirty area 12 overlapping the specific region 11 to the area of ​​the specific region 11 exceeds a specific ratio (for example, the specific ratio is 30%), and generates a dirt warning.

[0034] In this embodiment, at least three corner points are selected as reference points based on the bounding box information, and the closed region enclosed by the lines connecting these at least three corner points is defined as the specific region 11. The specific region 11 may be, but is not limited to, a circular region, an elliptical region, a triangular region, a quadrilateral region, or a polygonal region.

[0035] Refer to Figures 3 and 4 simultaneously. Figure 3 is a schematic diagram of the first pedestrian recognition area of ​​the vehicle lens dirt warning method according to the present invention. Figure 4 is a schematic diagram of the second pedestrian recognition area of ​​the vehicle lens dirt warning method according to the present invention.

[0036] In this embodiment, the specific region 11 is a first pedestrian recognition region 21 located in the video coordinate system, and is used to detect whether or not a pedestrian is present at a specific position in the second pedestrian recognition region 22 located in the world coordinate system after coordinate transformation when in a backward state.

[0037] Automatic Emergency Braking (AEB) is currently being promoted by new car assessment programs in various countries. For example, the rearview test method specified in the European New Car Assessment Programme (Euro-NCAP) requires that the vehicle be able to recognize a stationary adult and child behind it at reverse speeds of 4 km / h and 8 km / h. It also requires that the vehicle be able to recognize an adult and child moving at a speed of 5 km / h behind it at reverse speeds of 4 km / h and 8 km / h.

[0038] In this embodiment, as shown in Figures 2, 3, and 4, the specific region 11 is a first pedestrian recognition region 21 located in the video coordinate system. After coordinate alignment by the calibration method and coordinate transformation, the first pedestrian recognition region 21 corresponds to a second pedestrian recognition region 22 located in the world coordinate system. In the world coordinate system, where the position of the on-board lens is the origin of the X and Y axis coordinates, the second pedestrian recognition region 22 is located within the region from -400 cm to 400 cm in the X-axis direction of the on-board lens, and within the region from 0 cm to 900 cm in the Y-axis direction of the on-board lens.

[0039] As shown in Figure 4, when the backward speed is 8 km / h and the pedestrian's movement speed is 5 km / h, the second pedestrian recognition region 22 in the world coordinate system is estimated to be a rectangular region of 9 m × 8 m (length × width).

[0040] The specified region 11 may be, but is not limited to, a circular region, an elliptical region, a triangular region, a quadrilateral region, or a polygonal region.

[0041] Figure 5 is a schematic diagram of the calibration method for the second pedestrian recognition area of ​​the vehicle lens dirt warning method according to the present invention.

[0042] In this embodiment, according to the calibration method for the second pedestrian recognition area specified in the European New Car Assessment Programme (Euro-NCAP), for example, when the reverse speed is 8 km / h, only a child test is required, and in this case, 50% of the rear of car 3 becomes the collision point. When pedestrian 2 starts walking from the side of car 3 at a distance of 4 m from point L, the speed is 5 km / h, and F = 1.5 m is the acceleration distance.

[0043] Let t1 be the time it takes for pedestrian 2 to travel through the acceleration section F = 1.5 m, a be the acceleration, and t be the time it takes to reach point L.

[0044]

number

[0045]

number

[0046]

number

[0047] It takes pedestrian 2 3.96 seconds to walk to point L.

[0048] JPEG2026120065000005.jpg24145

[0049]

number

[0050] JPEG2026120065000007.jpg26145

[0051] JPEG2026120065000008.jpg33145

[0052] If the AEB response time is 0.5s, then when pedestrian 2 walks to point L, the distance traveled by car 3 will be... The filename is JPEG2026120065000009.jpg11145.

[0053] Therefore, for pedestrian 2 to walk to point L without being hit by car 3, it must start backing up from a position at least 9m away from point L. According to this estimation, the area of ​​at least the second pedestrian recognition region 22 is defined as length × width = 9m × 8m.

[0054] As described above, the present invention avoids the drawbacks of distortion and low resolution of the image captured at the edge of the lens by dividing the entire image acquired by the lens into a specific area. Furthermore, by using an AI recognition model to accurately recognize whether the area of ​​dirt in this specific area exceeds a specific ratio, and issuing a warning, the present invention effectively reduces the time required to judge dirt using the entire screen as in the conventional method, improves recognition accuracy, and further improves false detections and missed detections.

[0055] Although the present invention has been disclosed as described above by the embodiments, the embodiments are not intended to limit the present invention. Equivalent substitutions of modifications and improvements made by those skilled in the art will still be within the scope of the patent protection of the present invention without departing from the spirit and scope of the invention. [Explanation of Symbols]

[0056] S110~S140 Step Flow 1. The Nth frame of the video. 11 Specific areas 12 Dirt Area 21 First pedestrian recognition area 22 Second pedestrian recognition area 2 Pedestrians 3. Automobile

Claims

1. A method for warning of dirt on an in-vehicle lens, applicable to a vehicle comprising an in-vehicle lens, a processing module, and a warning joule, The steps include: when the vehicle is in reverse, acquiring at least one continuous image of the shooting range with the on-board lens; The processing module obtains at least three corner points in the Nth frame of at least one sequence of images based on bounding box information, and forms a specific region in the Nth frame, wherein the specific region does not cover the four corners of the Nth frame and its area is smaller than the area of ​​the Nth frame. The processing module performs dirt image feature detection on the Nth frame of at least one sequence of images according to an image recognition model to obtain dirt information including the dirt region. The processing module includes the step of generating a dirt warning if it determines, based on the dirt information, that the ratio of the dirt area overlapping a specific area with the dirt area to the area of ​​the specific area exceeds a specific ratio. How to warn about dirt on a car's lens.

2. The vehicle further comprises at least one memory cell for storing the boundary box information and the image recognition model, wherein at least one of these memory cells further includes a boundary box database and an image recognition model database, the vehicle lens dirt warning method according to claim 1.

3. The vehicle-mounted lens is a retracted lens located at the rear of the vehicle and provided on the centerline of the vehicle, as described in claim 1, for warning of dirt on a vehicle-mounted lens.

4. The vehicle lens dirt warning method according to claim 1, wherein the specific region of the Nth frame image is a first pedestrian recognition region located in the image coordinate system, and is used to detect whether or not a pedestrian is present at a specific position in the second pedestrian recognition region located in the world coordinate system after coordinate transformation when the vehicle is in a reversing state.

5. The method for warning of dirt on an in-vehicle lens according to claim 1, wherein the processing module obtains at least three corner points in the Nth frame of at least one sequence of images based on bounding box information, and the step of forming a specific region in the Nth frame further includes the step of selecting the at least three corner points as reference points based on the bounding box information, and defining the closed region enclosed by the lines connecting the at least three corner points as the specific region.

6. The method for warning of dirt on an in-vehicle lens according to claim 1, wherein the specific region of the Nth frame image is a first pedestrian recognition region located in the image coordinate system, and after coordinate transformation, the first pedestrian recognition region corresponds to a second pedestrian recognition region located in the world coordinate system, and in the world coordinate system with the position of the in-vehicle lens as the origin of the X and Y axis coordinates, the second pedestrian recognition region is located within the region from -400 cm to 400 cm in the X-axis direction of the in-vehicle lens and within the region from 0 cm to 900 cm in the Y-axis direction of the in-vehicle lens.

7. The method for warning of dirt on an in-vehicle lens according to claim 1, wherein the area of ​​the specific region is 1 / 2 to 2 / 3 of the area of ​​the Nth frame image.

8. The method for warning of dirt on an in-vehicle lens according to claim 1, wherein the image recognition model is a semantic segmentation model, and is used to detect dirt image features and acquire dirt information, and the dirt information further includes the dirt located at the coordinate position and size of the specific region.

9. The method for warning of dirt on an in-vehicle lens according to claim 8, wherein the semantic segmentation model is a Fully Convolutional Networks model (FCN model), a U-net model, or an Enet model (efficient neural network model).

10. The method for warning of dirt on an in-vehicle lens according to claim 1, wherein the specific ratio is one of 10%, 15%, 20%, and 30%.

Images