Video processing system and video processing method thereof

Abstract

The invention relates to the technical field of image processing, in particular to a video processing system and a video processing method thereof. The video processing method comprises the steps that a to-be-processed video is acquired, the to-be-processed video comprises at least one to-be-processed image group, and the to-be-processed image group comprises a frame of to-be-processed sample image and at least one frame of to-be-processed following image; respectively analyzing and judging the to-be-processed sample image and the to-be-processed following image to obtain processing modes of the to-be-processed sample image and the to-be-processed following image; the processing modes comprise a demisting mode, a rain removing mode, a rain and fog removing mode and a normal processing mode; and processing the to-be-processed sample image and the to-be-processed following image based on the processing modes of the to-be-processed sample image and the to-be-processed following image, and generating and outputting a display video. According to the data processing method, defogging and rain removal intelligent processing of the video can be realized, and the calculation amount can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and more particularly, to a video processing system and a video processing method thereof. Background Art

[0002] Since the information captured by a high-definition camera, an electronic rearview mirror can provide a wider field of view than a traditional rearview mirror, and can eliminate the blind spots brought by the traditional rearview mirror, especially in the lateral and rear directions. Moreover, with the development of electronic rearview mirror regulations, its application will become more and more common.

[0003] In rainy or foggy environments, the camera system of the electronic rearview mirror will have problems such as low contrast, blurriness, overall image color being grayish-white, and color shift in the captured images due to low visibility of the scene, which will greatly affect the normal and stable operation of related recognition systems.

[0004] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0005] The purpose of the present disclosure is to overcome the above-mentioned deficiencies of the prior art, and to provide a video processing system and a video processing method thereof, which can realize intelligent defogging and de-raining processing of videos and reduce the amount of calculation.

[0006] According to one aspect of the present disclosure, there is provided a video processing method for an electronic rearview mirror, including:

[0007] Obtain a video to be processed, where the video to be processed includes at least one group of images to be processed, and the group of images to be processed includes a sample image to be processed and at least one following image to be processed;

[0008] Analyze and judge the sample image to be processed and the following image to be processed respectively to obtain the processing modes of the sample image to be processed and the following image to be processed; the processing modes include a defogging mode, a de-raining mode, a de-rain-and-fog mode, and a normal processing mode;

[0009] Based on the processing modes of the sample image to be processed and the following image to be processed, process the sample image to be processed and the following image to be processed, and generate and output a display video.

[0010] In an embodiment of the present disclosure, analyzing and judging the sample image to be processed and the following image to be processed respectively to obtain the processing modes of the sample image to be processed and the following image to be processed includes the following steps:

[0011] S100. Obtain the light transmittance of the to-be-processed sample image, and determine whether the to-be-processed sample image is a foggy image based on the light transmittance; obtain the sample rain line density of the to-be-processed sample image, and determine whether the to-be-processed sample image is a rainy image based on the sample rain line density.

[0012] S200. Obtain the image features of the to-be-processed sample image based on step S100.

[0013] S300. Obtain the processing mode of the to-be-processed sample image based on the image features.

[0014] S400. Obtain the processing mode of the to-be-processed following image in the same manner as the to-be-processed sample image.

[0015] In an embodiment of the present disclosure, obtaining the light transmittance of the to-be-processed sample image and determining whether the to-be-processed sample image is a foggy image based on the light transmittance includes:

[0016] Obtain the light transmittance of the to-be-processed sample image, and compare the light transmittance with a light transmittance threshold; if the light transmittance is greater than the light transmittance threshold, it indicates that the to-be-processed sample image is a non-foggy image, and conversely, if the light transmittance is not greater than the light transmittance threshold, it indicates that the to-be-processed sample image is a foggy image.

[0017] In an embodiment of the present disclosure, obtaining the sample rain line density of the to-be-processed sample image and determining whether the to-be-processed sample image is a rainy image based on the sample rain line density includes:

[0018] Obtain the sample rain line density of the to-be-processed sample image, and compare the sample rain line density with a sample rain line density threshold; if the sample rain line density is not less than the sample rain line density threshold, it indicates that the to-be-processed sample image is a rainy image, and conversely, if the sample rain line density is less than the sample rain line density threshold, it indicates that the to-be-processed sample image is a non-rainy image.

[0019] In an embodiment of the present disclosure, obtaining the rain line density of the to-be-processed sample image includes:

[0020] Successively perform gray processing, histogram equalization processing, and binarization processing on the to-be-processed sample image to generate a sample intermediate image; the sample intermediate image has multiple high-brightness stripes.

[0021] Calibrate the high-brightness stripes that meet the sample rain line size threshold as rain lines, and obtain the sample rain line density.

[0022] In an embodiment of the present disclosure, obtaining the rain line density of the to-be-processed following image includes:

[0023] After performing grayscale processing, histogram equalization processing, and binarization processing on the to-be-processed following image in sequence, a following intermediate image is generated; the following intermediate image has multiple highlighted stripes;

[0024] Calibrate the highlighted stripes that meet the following rain line size threshold as rain lines, and obtain the following rain line density; within the same group of to-be-processed images, the following rain line size threshold is the same as the sample rain line size threshold.

[0025] In an embodiment of the present disclosure, obtaining the rain line density of the to-be-processed sample image includes:

[0026] After performing grayscale processing, histogram equalization processing, and binarization processing on the to-be-processed sample image in sequence, a sample intermediate image is generated; the sample intermediate image has multiple highlighted stripes;

[0027] Calibrate the highlighted stripes that meet the sample rain line size threshold as the first sample stripes, record the inclination angles of each of the first sample stripes, calibrate the first sample stripes whose inclination angles meet the sample rain line inclination angle as rain lines, and obtain the sample rain line density;

[0028] Obtaining the rain line density of the to-be-processed following image includes:

[0029] After performing grayscale processing, histogram equalization processing, and binarization processing on the to-be-processed following image in sequence, a following intermediate image is generated; the following intermediate image has multiple highlighted stripes;

[0030] Calibrate the highlighted stripes that meet the following rain line size threshold as the first following stripes, record the inclination angles of each of the first following stripes, calibrate the first following stripes whose inclination angles meet the following rain line inclination angle as rain lines, and obtain the following rain line density, wherein, within the same group of to-be-processed images, the following rain line size threshold is the same as the sample rain line size threshold, and the following rain line inclination angle is the same as the sample rain line inclination angle.

[0031] In an embodiment of the present disclosure, obtaining the rain line density of the to-be-processed sample image includes:

[0032] After performing grayscale processing, histogram equalization processing, and binarization processing on the to-be-processed sample image in sequence, a sample intermediate image is generated; the sample intermediate image has multiple highlighted stripes;

[0033] Calibrate the highlighted stripes that meet the sample rain line size threshold as the first sample stripes, record the inclination angles of each of the first sample stripes, calibrate the first sample stripes with inclination angles that meet the sample rain line inclination angle as the second sample stripes, and after removal, calibrate the second sample stripes that increase the image contrast as rain lines, and obtain the sample rain line density;

[0034] Obtaining the rain line density of the to-be-processed following image includes:

[0035] After sequentially performing gray processing, histogram equalization processing, and binarization processing on the to-be-processed following image, generate a following intermediate image; the following intermediate image has a plurality of highlighted stripes;

[0036] Calibrate the highlighted stripes that meet the following rain line size threshold as the first following stripes, record the inclination angles of each of the first following stripes, calibrate the first following stripes with inclination angles that meet the following rain line inclination angle as the second following stripes, and after removal, calibrate the second following stripes that increase the image contrast as rain lines, and obtain the following rain line density, wherein, within the same to-be-processed image group, the following rain line size threshold is the same as the sample rain line size threshold, and the following rain line inclination angle is the same as the sample rain line inclination angle.

[0037] In an embodiment of the present disclosure, in the rain removal mode and the rain and fog removal mode, the rain lines are removed and pixel filling is used for rain removal processing.

[0038] In an embodiment of the present disclosure, the display video includes at least one display image group, and the display image group includes one frame of display sample image and at least one frame of display following image; the display sample image is formed after processing the to-be-processed sample image, and the display following image is formed after processing the to-be-processed following image;

[0039] Obtaining the transmittance of the to-be-processed sample image includes:

[0040] Based on the fog image model, combine the dark channel of the to-be-processed sample image to obtain the sample global atmospheric light constant; obtain the transmittance of the to-be-processed sample image based on the expression of the sample global atmospheric light constant and the fog image model.

[0041] In an embodiment of the present disclosure, obtaining the transmittance of the following image includes:

[0042] Based on the fog image model and the sample global atmospheric light constant, obtain the transmittance of the following image.

[0043] In an embodiment of the present disclosure, in the defogging mode and the rain and fog removal mode, defogging processing is performed based on the dark channel prior algorithm.

[0044] In an embodiment of the present disclosure, the processing mode is selected by direct designation by the user.

[0045] According to another aspect of the present disclosure, there is provided a video processing system for an electronic rearview mirror, including:

[0046] A video acquisition module configured to output a video to be processed;

[0047] A video processing module electrically connected to the video acquisition module; the video processing module is configured to receive the video to be processed and execute the above video processing method to output a display video;

[0048] A video display module electrically connected to the video processing module; the video display module is configured to receive and display the display video.

[0049] In an embodiment of the present disclosure, the video processing system further includes a user selection module;

[0050] The user selection module is electrically connected to the video processing module; the user selection module is configured to designate and select the processing mode.

[0051] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0053] Figure 1 It is a schematic structural diagram of a video processing system in an embodiment of the present disclosure.

[0054] Figure 2 It is a schematic flow diagram of a video processing module in an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. Like reference numerals in the figures denote like or similar structures, and thus their detailed description will be omitted. Further, the drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale.

[0056] The terms "a", "an", "the", "said", and "at least one" are used to denote the presence of one or more elements / components / etc.; the terms "comprising" and "having" are used to mean an open inclusion and mean that there may be additional elements / components / etc. in addition to the listed elements / components / etc.; the terms "first", "second", "third", etc. are used only as labels and are not a limitation on the quantity of their objects.

[0057] In this application, unless otherwise clearly specified and defined, the term "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral one; it can be directly connected or indirectly connected through an intermediate medium.

[0058] Because of the information captured by a high-definition camera, an electronic rearview mirror can provide a wider field of view than a traditional rearview mirror, can eliminate the blind spots brought by the traditional rearview mirror, especially in the lateral and rear directions, and with the development of electronic rearview mirror regulations, its application will become more and more common.

[0059] In a rainy or foggy environment, the camera system of the electronic rearview mirror will have problems such as low contrast, blurring, the overall color of the image being grayish-white, and color deviation in the captured imaging due to the low visibility of the scene, which will greatly affect the normal and stable operation of the relevant recognition system.

[0060] To solve the above problems, the present disclosure provides a video processing system for use on an in-vehicle electronic rearview mirror. Among them, referring to Figure 1 , the video processing system includes a video acquisition module, a video processing module, and a video display module. The video acquisition module is electrically connected to the video processing module, and the video processing module is electrically connected to the video display module.

[0061] Optionally, the video acquisition module is configured to acquire the video to be processed on the electronic rearview mirror and send the video to be processed to the video processing module. In one example, the video acquisition module includes a camera for acquiring the required field of view of the vehicle and a reading unit for reading the image data acquired by the camera. In this example, the camera acquires the video to be processed, and the reading unit reads the video to be processed acquired by the camera and sends it to the video processing module.

[0062] To improve the image clarity under extreme conditions, the camera in the present disclosure may adopt a camera with the ability to collect images at high frequencies. In other words, the camera in the present disclosure is a high-speed camera (a high-speed camera can capture images at an extremely high frame rate (hundreds to thousands of frames per second or even higher)), which can facilitate real-time information collection. The acquisition frequency of the camera can be set and adjusted according to requirements, and the present disclosure does not make any limitations. Of course, the acquisition frequency of the camera needs to meet the requirements of normal driving. Of course, the present disclosure can also adopt other methods to collect the video to be processed.

[0063] Optionally, the video processing module is configured to receive the video to be processed and output a display video after performing corresponding processing on the video to be processed. Specifically, referring to Figure 2 , the video processing module may include an analysis unit and multiple processing units (in this example, the processing units include a first processing unit, a second processing unit, a third processing unit, and a fourth processing unit). Among them, the analysis unit is used to receive the video to be processed, analyze and judge the processing modes of multiple frames of images in the video to be processed, and select different processing units for image processing based on different processing modes, and then output a display video. In the present disclosure, the video to be processed includes multiple frames of images to be processed, and the images to be processed may be foggy images, or rainy images, or rainy and foggy images, or images without rain and fog. When the image to be processed is a rainy image, the first processing unit is used for rain removal processing. When the image to be processed is a foggy image, the second processing unit is used for fog removal processing. When the image to be processed is a rainy and foggy image, the third processing unit is used for fog and rain removal processing. When the image to be processed is an image without rain and fog (normal mode), the fourth processing unit directly outputs it.

[0064] In an implementation manner of the present disclosure, the video display module is configured to receive and display the display video for the user to view.

[0065] In an implementation manner of the present disclosure, the video processing system further includes a user selection module; the user selection module is electrically connected to the video processing module; the user selection module is configured to be able to directly specify and select a processing mode, so that after the video processing module receives the video to be processed, it processes it using the specified processing mode and outputs a display video.

[0066] In the present disclosure, the user selection module is provided so that the user can, when performing fog and rain removal according to the system determination, also perform fog and rain removal processing according to their own needs, with stronger applicability.

[0067] The present disclosure also provides a video processing method, which includes the following steps:

[0068] SA00. The video acquisition module acquires the video to be processed of the electronic rearview mirror and sends it to the video processing module;

[0069] SB00. The video processing module receives the video to be processed sent by the video acquisition module. The video to be processed includes at least one image group to be processed. The image group to be processed is multiple frames of images acquired within a time period T. The image group to be processed includes one frame of sample image to be processed and at least one frame of following image to be processed.

[0070] In step SB00, the first frame of the image group to be processed is selected as the sample image to be processed; the following image to be processed is the remaining frames of the image group to be processed. In other words, the following image to be processed is the image between two adjacent sample images to be processed;

[0071] SC00. Analyze and judge the sample image to be processed and the following image to be processed respectively to obtain the processing modes of the sample image to be processed and the following image to be processed; the processing modes include defogging mode, de-raining mode, de-rain-and-fog mode, and normal processing mode;

[0072] S100. Obtain the transmittance of the sample image to be processed, and judge whether the sample image to be processed is a fog image based on the transmittance; obtain the sample rain line density of the sample image to be processed, and judge whether the sample image to be processed is a rain image based on the sample rain line density;

[0073] S110. Judge whether the sample image to be processed is a fog image;

[0074] In the RGB color mode, each pixel is composed of three channels (a channel refers to the corresponding component used to represent color in each pixel), namely the red channel (R), the green channel (G), and the blue channel (B). The value range of each channel is 0 - 255, where 0 represents no light, and 255 represents the maximum light brightness. By adjusting the brightness of each channel, different colors can be obtained. For example, if the brightness of the red channel is 255 and the brightness of the green and blue channels is 0, then red is obtained; if the brightness of the red channel is 255 and the brightness of the green channel is 255; and the brightness of the blue channel is 0, then yellow is obtained. By adjusting the brightness of each channel in the RGB color mode, more than 16 million different colors can be obtained. This method is very intuitive because it corresponds to the way of human visual perception. The human eye is most sensitive to the three primary colors of red, green, and blue, so the RGB color mode is very accurate in representing the colors of natural scenes.

[0075] Under normal circumstances, due to the presence of fog in almost all natural images, there are usually some pixel values (color level values) in one of the RGB channels of any pixel that are very low and close to 0. In the presence of fog, even in the red region, blue region, and green region of the image, the dark channel region is much greater than 0, making it appear grayish-white. Moreover, the more severe the fog, the more grayish it becomes, making the image blurred.

[0076] The foggy image model can be expressed by the following formula:

[0077] I(Z) = J(Z)F(Z) + A(1 - F(Z))

[0078] Where, I(Z) represents the color level value of the image with fog at pixel Z, J(Z) represents the color level value of the defogged image at pixel Z, F(Z) represents the light transmittance at pixel Z, and A is the global atmospheric light constant (A is independent of the spatial coordinates, so in this disclosure, it is assumed to be a global constant).

[0079] The global atmospheric light constant A can be obtained based on the formula I(Z) = J(Z)F(Z) + A(1 - F(Z)), and then the light transmittance of the image with fog can be obtained using the global atmospheric light constant A; after obtaining the light transmittance and A, the defogged image J(Z) can be deduced in reverse.

[0080] Based on this concept, in this disclosure, the light transmittance is obtained in the following manner:

[0081] S1101. Obtain the light transmittance of the sample image to be processed; in this disclosure, the light transmittance of the sample image to be processed is used to characterize the fog concentration in the sample image to be processed. In other words, the image light transmittance reflects the fog concentration in the sample image to be processed.

[0082] S11011. Obtain the dark channel of the sample image to be processed (solve the dark channel according to the sample image to be processed);

[0083] S11012. Select the N brightest pixel points in the dark channel; in this example, the N pixel points can be the top N pixel points with the largest color level values in the dark channel.

[0084] In one example, N = 0.001 * Z, where Z is the number of pixel points in the sample image to be processed. Of course, in other examples, N can be selected according to actual needs.

[0085] For the brightest pixel points in the dark channel, their light transmittance is almost 0. Therefore, for any pixel point X (1 ≤ X ≤ N) among the N pixel points, the formula

[0086] I Y (X) = J Y (X)FY (X) + A Y (1 - F Y (X)) where F Y (X) = 0 (transmittance is 0), at this time, I Y (X) = A Y . It can be understood that the value of I Y (X) is the estimated value A of the sample global atmospheric light based on pixel point X Y . Among them, I Y (X) represents the color level value of the pixel point at X in the sample image to be processed, J Y (X) is the color level value of the pixel point at X in the sample image to be processed after defogging, F Y (X) represents the transmittance of the sample image to be processed based on pixel point X, A Y is the estimated value of the sample global atmospheric light (based on pixel point X). In the present disclosure, the upper left corner of the sample image to be processed can be taken as the origin. Of course, in other examples, other regions can also be selected as the origin

[0087] To further ensure the accuracy of the calculation, the estimated values of the sample global atmospheric light based on N pixel points are calculated respectively, and the average value of the N estimated values of the global atmospheric light is taken as the sample global atmospheric light constant A of the sample image to be processed Y .

[0088] S11013. The brightness of the dark channel of the image itself should tend to 0. Therefore, for any one of the N pixel points, for example, for the Xth pixel point, J Y (X)F Y (X) = 0, then I Y (X) = A Y (1 - F Y (X)), substituting the color level value I Y (X) of the sample image to be processed at pixel point X, and the global atmospheric light constant A Y into the formula I Y (X) = A Y (1 - F Y (X)), the transmittance F Y (X) (based on pixel point X) of the sample image to be processed is obtained

[0089] To further ensure the accuracy of the calculation, the transmittances based on N pixel points are calculated respectively, and the average value of the transmittances of the N pixel points is taken as the transmittance F of the sample image to be processed Y .

[0090] S1102. Determine whether the sample image to be processed is a foggy image based on the transmittance of the sample image to be processed

[0091] Compare the light transmittance F of the sample image to be processed Y with the light transmittance threshold FYT. If the light transmittance F Y ≤ the light transmittance threshold FYT, it means that the sample image to be processed is a foggy image. Conversely, if the light transmittance F > the light transmittance threshold FYT, it means that the sample image to be processed is a non-foggy image.

[0092] S120. Determine whether the sample image to be processed is a rainy image;

[0093] Because usually raindrops have a strong reflection effect on light, the pixel gray value of raindrops is significantly larger than that of the background pixels. Therefore, the image can be processed by layering, and the high-frequency bright lines can be screened to determine the raindrops.

[0094] S1201. Obtain the sample raindrop density of the sample image to be processed; the sample raindrop density is used to characterize the raindrop distribution in the sample image to be processed. In other words, the sample raindrop density reflects the raindrop distribution in the sample image to be processed.

[0095] In the first implementation manner of the present disclosure, the sample raindrop density is obtained in the following manner:

[0096] S12011. Convert the sample image to be processed into a grayscale image to form a first image;

[0097] S12012. Perform histogram equalization processing on the first image (make the high-brightness points brighter and the background dark points darker) to form a second image;

[0098] S12013. Perform local binarization processing on the second image to generate a sample intermediate image. In this example, the high-brightness stripes are regarded as 1 value and the background layer is regarded as 0 value to distinguish the high-brightness stripes from the background layer;

[0099] S12014. Through connected component selection, calibrate the high-brightness stripes that meet the sample raindrop size threshold as raindrops (calibrate the high-brightness stripes that do not meet the sample raindrop size threshold as non-raindrops, or do not calibrate the high-brightness stripes that do not meet the sample raindrop size threshold).

[0100] Among them, in the present disclosure, setting the sample rain line size threshold includes a width range threshold WY (it can be understood that WY is a range. For example, WY can be (5, 10), and this (5, 10) can be understood as a range where 5 ≤ width ≤ 10; for another example, WY can be (7.3, 18.2), and this (7.3, 18.2) can be understood as a range where 7.3 ≤ width ≤ 18.2) and a length range threshold LY (it can be understood that LY is a range. For example, LY can be (3, 20), and this (3, 20) can be understood as a range where 3 ≤ length ≤ 20; for another example, LY can be (4.4, 30.6), and this (4.4, 30.6) can be understood as a range where 4.4 ≤ length ≤ 30.6). In the present disclosure, the units of the width value and the length value can be adjusted according to the actual situation.

[0101] If the width W of the highlighted stripe is within the width range threshold WY and the length L is within the length range threshold LY, then the highlighted stripe is calibrated as a rain line. Conversely, if at least one of the width W and the length L of the highlighted stripe is not within the corresponding width range threshold WY or the corresponding length range threshold LY, it means that the highlighted stripe is not a rain line and is not calibrated (it can be understood that in the following three cases: the width W of the highlighted stripe is not within the width range threshold WY and the length L is within the length range threshold LY; the width W of the highlighted stripe is within the width range threshold WY and the length L is not within the length range threshold LY; the width W of the highlighted stripe is not within the width range threshold WY and the length L is not within the length range threshold LY, all indicate that the highlighted stripe is not a rain line and is not calibrated, or is calibrated as a non-rain line).

[0102] S12015. Summarize the number of calibrated rain lines and define this number of rain lines as the sample rain line density NLY of the to-be-processed sample image.

[0103] In the second implementation manner of the present disclosure, the sample rain line density is obtained through the following method:

[0104] S12011. Convert the to-be-processed sample image into a grayscale image to form a first image;

[0105] S12012. Perform histogram equalization processing on the first image (to make the high-brightness points brighter and the background dark points darker) to form a second image;

[0106] S12013. Perform local binarization processing on the second image to generate a sample intermediate image. In this example, the highlighted stripe is regarded as a value of 1 and the background layer is regarded as a value of 0 to distinguish the highlighted stripe from the background layer;

[0107] S12014. Through connected component selection, the highlighted stripes that meet the sample rain line size threshold are marked as the first sample stripes (the highlighted stripes that do not meet the sample rain line size threshold are marked as non-rain lines, or the highlighted stripes that do not meet the sample rain line size threshold are not marked).

[0108] Among them, in the present disclosure, setting the sample rain line size threshold includes a width range threshold WY (it can be understood that WY is a range. For example, WY can be (5, 10), and this (5, 10) can be understood as a range of 5 ≤ width ≤ 10; for another example, WY can be (7.3, 18.2), and this (7.3, 18.2) can be understood as a range of 7.3 ≤ width ≤ 18.2) and a length range threshold LY (it can be understood that LY is a range. For example, LY can be (3, 20), and this (3, 20) can be understood as a range of 3 ≤ length ≤ 20; for another example, LY can be (4.4, 30.6), and this (4.4, 30.6) can be understood as a range of 4.4 ≤ length ≤ 30.6). In the present disclosure, the units of the width value and the length value can be adjusted according to the actual situation.

[0109] If the width W of the highlighted stripe is within the width range threshold WY and the length L is within the length range threshold LY, then the highlighted stripe is marked as the first sample stripe. On the contrary, if at least one of the width W and the length L of the highlighted stripe is not within the corresponding width range threshold WY or the corresponding length range threshold LY, it means that the highlighted stripe is not a rain line and is not marked or marked as a non-rain line (it can be understood that in the following three cases: the width W of the highlighted stripe is not within the width range threshold WY and the length L is within the length range threshold LY; the width W of the highlighted stripe is within the width range threshold WY and the length L is not within the length range threshold LY; the width W of the highlighted stripe is not within the width range threshold WY and the length L is not within the length range threshold LY, all mean that the highlighted stripe is not a rain line and is not marked, or marked as a non-rain line).

[0110] S12015. Record the inclination angle θ of each first sample stripe q (θ q represents the inclination angle of the q-th first sample stripe). Since the rain line distribution is usually parallel. Therefore, in the present disclosure, the inclination angle that appears the most times is marked as the inclination angle θ of the rain line Y , and the first sample stripes with the inclination angle within θ Y ±θ Y (the inclination angle of the sample rain line) range are marked as rain lines (in the present disclosure, a certain measurement error is allowed for the inclination angle. Therefore, the inclination angle is within θ Y ±θ YFor the range limit), the remaining first sample stripes are not calibrated or calibrated as non-rain lines according to the above.

[0111] S12016. Summarize the number of calibrated rain lines and define this number of rain lines as the sample rain line density NLY of the sample image to be processed.

[0112] Based on the first implementation manner of the present disclosure, the second implementation manner further limits the characteristics of rain lines, increases the accuracy of rain line confirmation, and prevents misjudgment.

[0113] In the third implementation manner of the present disclosure, the sample rain line density is obtained by the following method:

[0114] S12011. Convert the sample image to be processed into a grayscale image to form a first image;

[0115] S12012. Perform histogram equalization processing on the first image (make the high-brightness points brighter and the background dark points darker) to form a second image;

[0116] S12013. Perform local binarization processing on the second image to generate a sample intermediate image. In this example, the high-brightness stripes are regarded as 1 value and the background layer is regarded as 0 value to distinguish the high-brightness stripes from the background layer;

[0117] S12014. Through connected component selection, calibrate the high-brightness stripes that meet the sample rain line size threshold as the first sample stripes (calibrate the high-brightness stripes that do not meet the sample rain line size threshold as non-rain lines, or do not calibrate the high-brightness stripes that do not meet the sample rain line size threshold).

[0118] Among them, in the present disclosure, the sample rain line size threshold is set to include a width range threshold WY (it can be understood that WY is a range. For example, WY can be (5, 10), and this (5, 10) can be understood as a range of 5 ≤ width ≤ 10; for another example, WY can be (7.3, 18.2), and this (7.3, 18.2) can be understood as a range of 7.3 ≤ width ≤ 18.2) and a length range threshold LY (it can be understood that LY is a range. For example, LY can be (3, 20), and this (3, 20) can be understood as a range of 3 ≤ length ≤ 20; for another example, LY can be (4.4, 30.6), and this (4.4, 30.6) can be understood as a range of 4.4 ≤ length ≤ 30.6). In the present disclosure, the units of the width value and the length value can be adjusted according to the actual situation.

[0119] If the width W of the highlighted stripe is within the width range threshold WY and the length L is within the length range threshold LY, then the highlighted stripe is calibrated as the first sample stripe. Conversely, if at least one of the width W and the length L of the highlighted stripe is not within the corresponding width range threshold WY or the corresponding length range threshold LY, it means that the highlighted stripe is not a rain line and is not calibrated or is calibrated as a non-rain line (it can be understood that in the following three cases: the width W of the highlighted stripe is not within the width range threshold WY and the length L is within the length range threshold LY; the width W of the highlighted stripe is within the width range threshold WY and the length L is not within the length range threshold LY; the width W of the highlighted stripe is not within the width range threshold WY and the length L is not within the length range threshold LY, all indicate that the highlighted stripe is not a rain line, is not calibrated, or is calibrated as a non-rain line).

[0120] S12015. Record the tilt angle θ of each first sample stripe q (θ q represents the tilt angle of the q-th first sample stripe). Since the distribution of rain lines is usually parallel. Therefore, the present disclosure calibrates the tilt angle that appears most frequently as the tilt angle θ of the rain line Y , and calibrates the first sample stripes with tilt angles within θ Y ±θ Y as the second sample stripes (in the present disclosure, a certain measurement error is allowed for the tilt angle. Therefore, the tilt angle is restricted within the range of θ Y ±θ Y ), and the remaining first sample stripes are not calibrated or are calibrated as non-rain lines according to the above.

[0121] S12016. Since the rain streaks are basically blurred and out of focus, the appearance of rain streaks will cause the image contrast to decrease. After removing the rain streaks, the image contrast will increase. Other interfering lines are generally the outlines of various objects in the image. After removing the object outlines, the image contrast will deteriorate. Therefore, in the present disclosure, each of the second sample streaks can be removed in sequence, and after removing each of the second sample streaks, the change in the contrast of the sample intermediate image can be judged respectively. If the contrast increases, the second sample streak is calibrated as a rain streak. On the contrary, if the contrast decreases, the second sample streak is calibrated as a non-rain streak as described above, or no calibration is performed. (In the present disclosure, the contrast can be compared after each of the second sample streaks is removed. Of course, it is also possible to use this method to confirm whether some of the second sample streaks are rain streaks. For example, when there are two sets of second sample streaks with the same number but different inclination angles (in other words, each of the second sample streaks is divided into two groups, and the two groups have the same number. The inclination angle of one group of the to-be-processed streaks satisfies θ1±θ1, and the inclination angle of the other group of the to-be-processed streaks satisfies θ2±θ2), one can be arbitrarily selected from each of the two groups of second sample streaks to determine the change in contrast, so as to determine which group of second sample streaks is a rain streak).

[0122] S12017. Summarize the calibrated number of rain streaks, and define this number of rain streaks as the sample rain streak density NLY of the to-be-processed sample image.

[0123] S1202. Based on the sample rain streak density NLY, judge whether the to-be-processed sample image is a rain image;

[0124] Compare the sample rain streak density NLY with the sample rain streak density threshold NLYY. If the sample rain streak density NLY is not less than the sample rain streak density threshold NLYY, it means that the to-be-processed sample image is a rain image. On the contrary, if the sample rain streak density NLY is less than the sample rain streak density threshold NLYY, it means that the to-be-processed sample image is a non-rain image.

[0125] S200. Based on the image features obtained in step S100 for the to-be-processed sample image;

[0126] Among them, when the to-be-processed sample image is a fog image and a non-rain image, the image feature of the to-be-processed sample image is defined as a fog image; when the to-be-processed sample image is a fog image and a rain image, the image feature of the to-be-processed sample image is defined as a rain and fog image; when the to-be-processed sample image is a non-fog image and a rain image, the image feature of the to-be-processed sample image is defined as a rain image; when the to-be-processed sample image is a non-fog image and a non-rain image, the image feature of the to-be-processed sample image is defined as a non-rain and fog image.

[0127] S300. Obtain the processing mode of the sample image to be processed based on the image features. If the sample image to be processed is a rain and fog image, the processing mode is the rain and fog removal mode; if the sample image to be processed is a fog image, the processing mode is the fog removal mode; if the sample image to be processed is a rain image, the processing mode is the rain removal mode; if the sample image to be processed is a non-rain and fog image, the processing mode is the normal processing mode.

[0128] S400. Obtain the processing mode of the following image to be processed in the same way as the sample image to be processed.

[0129] Within a certain period of time, the change of each frame of continuously collected images is extremely small. Therefore, based on the fog image, within the same tiny time period (for example, within the period T (t1 to tn, several seconds)), the foggy weather will not suddenly change, and the global atmospheric light constant will not change. Therefore, in the present disclosure, within the time period t1 to tn, for each frame of all the videos to be processed collected by the video acquisition module within the same period (the same group of images to be processed), the global atmospheric light constant calculated from the frame of image (the sample image to be processed) collected at time t1 can be used to process all the videos to be processed collected within the time period t1 to tn. In this way, it is not necessary to calculate the global atmospheric light constant for each frame of image first and then perform fog removal processing, reducing the amount of data processing and greatly saving the data processing time.

[0130] In other words, in the present disclosure, during the process of obtaining the processing mode of the following image to be processed, when judging whether the following image to be processed is a fog image, the global atmospheric light constant A of the following image is not calculated. g The sample global atmospheric light constant A Y is directly assigned to the following global atmospheric light constant A g , that is, the following global atmospheric light constant A g is the same as the sample global atmospheric light constant A Y .

[0131] Obtaining the transmittance of the following image includes:

[0132] Based on formula I g (Y) = J g (Y)F g (Y) + A g (1 - F g (Y)), the brightness of the dark channel of the image itself should tend to 0. Therefore, for any one of the N pixel points, for example, for the Y-th pixel point, J g (Y)F g (Y) = 0, I g (Y) = A g (1 - F g(Y)), calculate the transmittance of the follow-up image to be processed; among them, the follow-up global atmospheric light constant A g is the same as the sample global atmospheric light constant A Y ; I g (Y) represents the color level value of the follow-up image to be processed at pixel Y, and J g (Y) represents the color level value of the display follow-up image at pixel Y, and F g (Y) represents the transmittance of the follow-up image to be processed at pixel Y, and obtain the transmittance F of the follow-up image to be processed g (Y) (based on pixel point Y);

[0133] To further ensure the accuracy of the calculation, calculate the transmittance based on N pixel points respectively, and obtain the average value of the transmittances of the N pixel points as the transmittance F of the sample image to be processed g .

[0134] Obtain the transmittance F of the sample image to be processed g , and then judge whether it is a foggy image.

[0135] Similar to the foggy day image, for the rainy day image, within the same small time period from t1 to tn (for example, within a few seconds), the climate will not change suddenly, and the image rain stripe feature value caused by rain will not change. Therefore, within the time period of t1~tn, all the image information collected by the video acquisition module can be processed by the image rain line feature (sample rain line size threshold, tilt angle within θ Y ±θ Y ) of the frame image (sample image to be processed) collected in the t1 time period, so that the amount of data processing is greatly reduced and the data processing time is saved.

[0136] In the first implementation manner, obtaining the rain line density of the follow-up image to be processed includes:

[0137] After sequentially performing gray-scale processing, histogram equalization processing, and binarization processing on the follow-up image to be processed, generate a follow-up intermediate image; the follow-up intermediate image has multiple high-brightness stripes;

[0138] Calibrate the high-brightness stripes that meet the follow-up rain line size threshold as rain lines, and obtain the follow-up rain line density; within the same group of images to be processed, the follow-up rain line size threshold is the same as the sample rain line size threshold.

[0139] In the second implementation manner, obtaining the rain line density of the follow-up image to be processed includes:

[0140] After sequentially performing gray-scale processing, histogram equalization processing, and binarization processing on the follow-up image to be processed, generate a follow-up intermediate image; the follow-up intermediate image has multiple high-brightness stripes;

[0141] Calibrate the highlighted stripes that meet the following rain line size threshold as the first following stripes, record the inclination angles of each first following stripe, calibrate the first following stripes with inclination angles that meet the following rain line inclination angle as rain lines, and obtain the following rain line density. Among them, within the same group of images to be processed, the following rain line size threshold is the same as the sample rain line size threshold, and the following rain line inclination angle is the same as the sample rain line inclination angle.

[0142] In the third implementation manner, obtaining the rain line density of the image to be processed for following includes:

[0143] After sequentially performing gray processing, histogram equalization processing, and binarization processing on the image to be processed for following, generate an intermediate following image; the intermediate following image has multiple highlighted stripes;

[0144] Calibrate the highlighted stripes that meet the following rain line size threshold as the first following stripes, record the inclination angles of each first following stripe, calibrate the first following stripes with inclination angles that meet the following rain line inclination angle as the second following stripes, and after removal, calibrate the second following stripes that increase the image contrast as rain lines, and obtain the following rain line density. Among them, within the same group of images to be processed, the following rain line size threshold is the same as the sample rain line size threshold, and the following rain line inclination angle is the same as the sample rain line inclination angle.

[0145] After obtaining the following rain line density, use the same method to confirm the processing mode of the image to be processed for following.

[0146] SD00. Process the image to be processed for the sample and the image to be processed for following based on the processing mode of the image to be processed for the sample and the processing mode of the image to be processed for following;

[0147] SD001. When it is a foggy image, use the following method for defogging processing;

[0148] Perform defogging processing based on the dark channel prior algorithm.

[0149] Based on the intelligent judgment of foggy images and rainy images in the present disclosure, taking the image to be processed for the sample as an example, the transmittance F of the image to be processed for the sample Y (The transmittance of each pixel point in the image to be processed for the sample all uses the transmittance F of the image to be processed for the sample Y ), the global atmospheric light constant A Y , and the color level values of each pixel point in the image to be processed for the sample are substituted into the formula I Y (X) = J Y (X)F Y (X) + A Y (1 - F Y (X)), calculate the color level value J of each pixel point after defogging processingY (X), the displayed sample image after defogging can be obtained.

[0150] Specifically, if the total number of pixel points in the sample image to be processed is Z, then for the Mth pixel point, formula I Y (M) = J Y (M)F Y (M) + A Y (1 - F Y (M)), where 1 ≤ M ≤ Z, F Y (M) is the transmittance F of the sample image to be processed Y , substituting the color level value I Y (M) of the Mth pixel point in the sample image to be processed and substituting the global atmospheric light constant A Y into it, the color level value J Y (M) after defogging of the Mth pixel point is obtained;

[0151] By using the above method, the color level values after defogging of Z pixel points are calculated respectively, and the sample image to be output after defogging is obtained and output.

[0152] In the present disclosure, when judging the foggy image and performing defogging processing, the three channels of the image are separated and its dark channel is obtained. Based on the formula of the foggy image model and the characteristics of the dark channel, the transmittance and atmospheric light constant of each pixel point are calculated. Finally, through the transmittance and atmospheric light constant, reverse derivation is carried out to directly obtain the sample image to be output after defogging.

[0153] SD002. When it is a rainy image, the following method is used for rain removal processing;

[0154] Based on the sample image to be processed, all rain lines are removed, and pixel filling is performed in the corresponding rain line area, and the sample image to be output after rain removal is obtained.

[0155] SD003. Based on the processing mode, after processing each frame of the sample image to be processed, the video is output and displayed frame by frame in real time.

[0156] When the next group of images to be processed is collected, the above steps are repeated.

[0157] In an implementation manner of the present disclosure, after step SA00 and before step SB00, preprocessing of the video to be processed is further included. In this example, the preprocessing process includes but is not limited to filtering and noise reduction, ROI (Region of Interest, region of interest, abbreviated as ROI) target area interception, color correction, etc. In the present disclosure, preprocessing the video to be processed can improve the image quality, unify the image format, reduce the data volume, and highlight key information, etc., which is beneficial to subsequent judgment of the image state and improves the accuracy of judgment.

[0158] In an embodiment of the present disclosure, there is no sequence requirement for determining whether the sample image to be processed is a rainy image and determining whether the sample image to be processed is a foggy image. In other words, determining whether the sample image to be processed is a rainy image and determining whether the sample image to be processed is a foggy image can be performed simultaneously, or can be performed successively (for example, it can first be determined whether the sample image to be processed is a foggy image, and then whether the sample image to be processed is a rainy image).

[0159] In an embodiment of the present disclosure, after the user specifies the processing mode, the sample image to be processed and the following image to be processed are directly processed.

[0160] The video processing method and video processing system in the present disclosure can process foggy images and rainy images, and have strong applicability. Moreover, the video processing method in the present disclosure can intelligently determine the weather state of the video collected by the video acquisition module, eliminating the need for manual judgment and selection, thus achieving intelligence. Additionally, based on user selection, video processing can be selectively performed, further enhancing adaptability. Furthermore, in the present disclosure, a segmented approach is adopted. Within a time period, for foggy images, the global atmospheric light constant calculated from the first frame image (the sample image to be processed) is used, and for rainy images, the rain line size threshold and tilt angle threshold set based on the first frame image (the sample image to be processed) are used for calculation, which can greatly reduce the amount of data processing.

[0161] It should be noted that although the steps of the video processing method in the present disclosure are described in a specific order in the drawings, this does not require or imply that these steps must be executed in that specific order, or that all the steps shown must be executed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc.

[0162] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the appended claims.

Claims

1. A video processing method for an electronic rearview mirror, characterized in that: include: Acquire a video to be processed, wherein the video to be processed includes at least one image group to be processed, and the image group to be processed includes a frame of a sample image to be processed and at least one frame of a follow-up image to be processed; Analyze and judge the sample image to be processed and the following image to be processed respectively, and obtain processing modes of the sample image to be processed and the following image to be processed; the processing modes include a defogging mode, a rain removal mode, a rain and fog removal mode, and a normal processing mode; Based on the processing modes of the sample image to be processed and the following image to be processed, the sample image to be processed and the following image to be processed are processed to generate and output a display video.

2. The video processing method according to claim 1, characterized in that: Analyzing and judging the sample image to be processed and the follow-up image to be processed respectively to obtain the processing modes of the sample image to be processed and the follow-up image to be processed comprises the following steps: S100, obtaining the transmittance of the sample image to be processed, and judging whether the sample image to be processed is a fog image based on the transmittance; obtaining the sample rain line density of the sample image to be processed, and judging whether the sample image to be processed is a rain image based on the sample rain line density; S200, obtaining image features of the sample image to be processed based on step S100; S300, obtaining a processing mode of the sample image to be processed based on the image feature; S400: Obtaining a processing mode of the to-be-processed follow-up image in the same manner as the to-be-processed sample image.

3. The video processing method according to claim 2, characterized in that: Acquiring the transmittance of the sample image to be processed, and judging whether the sample image to be processed is a fog image based on the transmittance, including: The transmittance of the sample image to be processed is obtained, and the transmittance is compared with a transmittance threshold; if the transmittance is greater than the transmittance threshold, it indicates that the sample image to be processed is a non-fog image; otherwise, if the transmittance is not greater than the transmittance threshold, it indicates that the sample image to be processed is a foggy image.

4. The video processing method according to claim 2, characterized in that: Acquiring a sample rain line density of a sample image to be processed, and judging whether the sample image to be processed is a rain image based on the sample rain line density, including: The sample rain line density of the sample image to be processed is obtained, and the sample rain line density is compared with a sample rain line density threshold; if the sample rain line density is not less than the sample rain line density threshold, it indicates that the sample image to be processed is a rain image; otherwise, if the sample rain line density is less than the sample rain line density threshold, it indicates that the sample image to be processed is a non-rain image.

5. The video processing method according to claim 2, characterized in that: Obtaining the sample rain line density of the sample image to be processed, comprising: After the sample image to be processed is subjected to grayscale processing, histogram equalization processing and binarization processing in sequence, a sample intermediate image is generated; the sample intermediate image has a plurality of highlight stripes; The highlighted stripes that meet the sample rain line size threshold are marked as rain lines, and the sample rain line density is obtained.

6. The video processing method according to claim 5, characterized in that: Obtaining the following rain line density of the following image to be processed, comprising: After the to-be-processed following image is subjected to grayscale processing, histogram equalization processing and binarization processing in sequence, a following intermediate image is generated; the following intermediate image has a plurality of highlight stripes; The highlighted stripes satisfying the following rain line size threshold are marked as rain lines, and the following rain line density is obtained; within the same to-be-processed image group, the following rain line size threshold is the same as the sample rain line size threshold.

7. The video processing method according to claim 2, characterized in that: Obtaining the sample rain line density of the sample image to be processed, comprising: After the sample image to be processed is subjected to grayscale processing, histogram equalization processing and binarization processing in sequence, a sample intermediate image is generated; the sample intermediate image has a plurality of highlight stripes; Marking the highlighted stripes that meet the sample rain line size threshold as first sample stripes, recording the inclination angle of each of the first sample stripes, marking the first sample stripes whose inclination angles meet the sample rain line inclination angles as rain lines, and obtaining the sample rain line density; Obtaining the following rain line density of the following image to be processed, comprising: After the to-be-processed following image is subjected to grayscale processing, histogram equalization processing and binarization processing in sequence, a following intermediate image is generated; the following intermediate image has a plurality of highlight stripes; The highlighted stripes satisfying the following rain line size threshold are calibrated as first following stripes, the inclination angles of the first following stripes are recorded, the first following stripes whose inclination angles satisfy the following rain line inclination angles are calibrated as rain lines, and the following rain line density is obtained, wherein, in the same group of images to be processed, the following rain line size threshold is the same as the sample rain line size threshold, and the following rain line inclination angle is the same as the sample rain line inclination angle.

8. The video processing method according to claim 6, characterized in that: Obtaining the sample rain line density of the sample image to be processed, comprising: After the sample image to be processed is subjected to grayscale processing, histogram equalization processing and binarization processing in sequence, a sample intermediate image is generated; the sample intermediate image has a plurality of highlight stripes; Marking the highlighted stripes that meet the sample rain line size threshold as first sample stripes, recording the inclination angle of each of the first sample stripes, marking the first sample stripes whose inclination angles meet the sample rain line inclination angles as second sample stripes, marking the second sample stripes that increase the image contrast after being removed as rain lines, and obtaining the sample rain line density; Obtaining the following rain line density of the following image to be processed, comprising: After the to-be-processed following image is subjected to grayscale processing, histogram equalization processing and binarization processing in sequence, a following intermediate image is generated; the following intermediate image has a plurality of highlight stripes; The highlighted stripes satisfying the following rain line size threshold are calibrated as first following stripes, the inclination angle of each of the first following stripes is recorded, the first following stripes whose inclination angles satisfy the following rain line inclination angle are calibrated as second following stripes, the second following stripes whose image contrast is increased after being removed are calibrated as rain lines, and the following rain line density is obtained, wherein, in the same group of images to be processed, the following rain line size threshold is the same as the sample rain line size threshold, and the following rain line inclination angle is the same as the sample rain line inclination angle.

9. The video processing method according to any one of claims 5 to 8, characterized in that: In the rain removal mode and the rain and fog removal mode, rain removal processing is performed by removing rain lines and performing pixel filling.

10. The video processing method according to claim 2, characterized in that: The display video includes at least one display image group, and the display image group includes a display sample image frame and at least one display follow-up image frame; the display sample image is formed after the sample image to be processed is processed, and the display follow-up image is formed after the follow-up image to be processed is processed; Obtaining the light transmittance of the sample image to be processed, comprising: Based on the fog image model and in combination with the dark channel of the sample image to be processed, a global atmospheric light constant of the sample is obtained; The transmittance of the sample image to be processed is obtained based on the expression of the sample global atmospheric light constant and the fog image model.

11. The video processing method according to claim 10, characterized in that: Obtaining the transmittance of the following image, comprising: Based on the fog image model and the sample global atmospheric light constant, the transmittance of the follow-up image is obtained.

12. The video processing method according to claim 11, characterized in that: In the defogging mode and the rain and fog removal mode, defogging processing is performed based on a dark channel priori algorithm.

13. The video processing method according to claim 1, characterized in that: The processing mode is selected by direct designation by the user.

14. A video processing system for an electronic rearview mirror, characterized in that: include: A video acquisition module, configured to output a video to be processed; A video processing module, electrically connected to the video acquisition module; The video processing module is configured to receive the video to be processed, and execute the video processing method in any one of claims 1 to 13, and output a display video; The video display module is electrically connected to the video processing module; the video display module is configured to receive and display the display video.

15. The video processing system according to claim 14, characterized in that: The video processing system also includes a user selection module; The user selection module is electrically connected to the video processing module; the user selection module is configured to specify and select the processing mode.

Images