A camera occlusion detection algorithm for a vehicle during driving combines spatial and temporal features

Abstract

The application provides a camera shielding detection algorithm for a vehicle during driving, which combines space and time domain features, and comprises an image acquisition step, a frame judgment step, a first frame judgment step, a change judgment step, a current frame judgment step and an updating step, and the camera shielding is judged for a region of interest, so that the false detection caused by the environment is reduced; meanwhile, the local statistical information is used to replace the global information, so that the local shielding of the camera can be better detected, and meanwhile, the combination of the image features on the space domain and the motion features on the time domain can effectively improve the accuracy of the shielding detection during driving, reduce the false judgment, and the camera is calculated in real time, so that the detection accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly to a camera occlusion detection algorithm for vehicles during driving combining spatial and temporal features. BACKGROUND

[0002] With the development of computer vision and pattern recognition, visual algorithms are widely used in the field of automatic driving, playing a crucial role in the safe driving of vehicles. However, due to the influence of environmental information such as weather and road conditions, the camera may be occluded by various objects such as fallen leaves, water droplets, and fog, resulting in a smaller or even disappearing field of view of the camera. At this time, all visual algorithms based on the camera will fail, posing a huge safety hazard to vehicle driving. Therefore, real-time detection of camera occlusion is one of the keys to the normal operation of visual algorithms and a guarantee for the safe driving of vehicles.

[0003] Existing camera occlusion detection algorithms mainly fall into two categories: 1). data-driven detection algorithms; 2). model-driven detection algorithms. The former trains a neural network through a large amount of data to achieve occlusion detection. The latter models the rules in the data through various mathematical models to achieve occlusion detection. The method with the aid of neural networks has high precision and can be applied to various scenarios. However, this method requires the construction of a large amount of data and has certain requirements for the operation device. Therefore, on the edge device with limited resources, many solutions focus on studying model-driven detection algorithms. However, there are still defects in today's model-driven algorithms: algorithms that rely on image brightness cannot be used normally in scenarios such as tunnels; methods that use image sharpness have low precision in rainy weather; methods based on local binary patterns and other features have high precision but high computational complexity. SUMMARY

[0004] In view of the deficiencies in the prior art, the present application aims to provide a camera occlusion detection algorithm for vehicles during driving combining spatial and temporal features, which overcomes the above-mentioned defects in the prior art.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] A camera occlusion detection algorithm for vehicles during driving combining spatial and temporal features, characterized in that it comprises

[0007] An image acquisition step: acquiring an image frame captured by a camera, cropping and extracting a region of interest, and cutting the region of interest into a plurality of image blocks to enter a frame judgment step;

[0008] A frame judgment step: judging whether the image frame is the first frame, if it is the first frame, entering a first frame judgment step, if it is not the first frame, entering a change judgment step;

[0009] The first frame judging step: if the current frame is the first frame after the camera is turned on, three non-continuous frames in the subsequent time are obtained as key image frames, three-frame difference method is performed on the three groups of key image frames, the first calculation formula is used to calculate the cumulative motion information of each image block in the interest region, and the updating step is entered. When the cumulative motion information is not less than the time information threshold, it is judged as occlusion.

[0010] The change judging step: if the camera has been turned on for a period of time, the picture comparison information of all frames is obtained, and the current frame judging step is entered. When the image acquisition information meets the second preset condition, it is judged as a dramatic change, and the current frame judging step is entered. When the image acquisition information is not less than the motion information threshold, it is judged as no dramatic change, and the updating step is entered.

[0011] The current frame judging step: for the current frame with dramatic change, three non-continuous frames in the subsequent time are obtained as key image frames, three-frame difference method is performed on the three groups of key image frames, the first calculation formula is used to calculate the cumulative motion information of each image block in the interest region, and the updating step is entered. When the cumulative motion information is not less than the time information threshold, it is judged as occlusion.

[0012] The updating step: update the current frame queue, and return to the image acquisition step.

[0013] As preferred, the first calculation formula is:

[0014]

[0015]

[0016] Wherein, m is the number of image blocks;

[0017] is the binary image of the image block;

[0018] is the image mean value of

[0019] S is the cumulative motion information;

[0020] T s is the set motion information threshold.

[0021] As preferred, the change judging step includes the following steps:

[0022] The statistical sub-step: according to the characteristics of each image block, the statistical information of the entire interest region is obtained by using the second calculation formula, that is, the overall information of the image obtained by the camera in the past period of time during the vehicle driving, and the judging sub-step is entered. ​

[0023] The judging sub-step: calculating the average statistical information of all frames in the frame queue by using a third calculation formula, and calculating the mean square error of the statistical information of the current frame and the average statistical information by using a fourth calculation formula, further comprising an error threshold, when the error is greater than the error threshold, it is judged that the dramatic change occurs, and enters the current frame judging step, if the error is less than the threshold, it is judged that the dramatic change does not occur, and enters the updating step.

[0024] As a preferred, the second calculation formula is:

[0025]

[0026]

[0027] Wherein, is the sharpness of the i-th image block of the region of interest corresponding to the current frame;

[0028] is the standard deviation of the i-th image block of the region of interest corresponding to the current frame;

[0029] w i and v i are the weights of the sharpness and the standard deviation, respectively;

[0030] C q is the weighted sharpness of the current frame;

[0031] C s is the weighted standard deviation of the current frame.

[0032] As a preferred, the third calculation formula is:

[0033]

[0034]

[0035] Wherein, n is a time parameter;

[0036] Cq j is the weighted sharpness of the region of interest corresponding to the j-th key frame in the past;

[0037] Cs j is the standard deviation of the region of interest corresponding to the j-th key frame in the past;

[0038] M q is the average sharpness of the region of interest;

[0039] M s is the average standard deviation of the region of interest.

[0040] As a preferred, the fourth calculation formula is:

[0041]

[0042] Wherein mse is the error between the current frame and the previous frame.

[0043] As preferred, the current frame in the current frame judging step is the initial frame, and the images of the 30th frame, the 150th frame and the 300th frame are recorded as the key image frames.

[0044] As preferred, the first frame in the first frame judging step is the initial frame, and the images of the 1st frame, the 30th frame and the 150th frame are recorded as the key image frames.

[0045] As preferred, the image captured by the camera in the image obtaining step is I, and the width and height of the image after the size adjustment are I and I respectively. w and I h The region of interest I is intercepted, and I = I[0:I, I / 3:I]. roi , I roi / 3:I w ] is obtained. h h

[0046] As preferred, the updating step comprises the following steps: after the current frame judging step, judging whether the number of frames in the frame queue is greater than the maximum storage number, if the frame queue is full, deleting the frame at the head of the queue, and adding the current frame to the tail.

[0047] The present application has the following advantages: the present application judges whether the camera is blocked according to the region of interest, reduces the false detection caused by the environment, uses the local statistical information instead of the global information, can better detect the local blocking of the camera, uses the combination of the image features on the video space and the motion features on the time domain, can effectively improve the accuracy of the blocking detection during the driving, reduces the false judgment, and performs the real-time calculation on the camera, and improves the detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the principle diagram of the camera blocking detection algorithm of the present application.

[0049] Figure 2 is the flow chart of the camera blocking detection algorithm of the present application. DETAILED DESCRIPTION

[0050] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all the other embodiments obtained by the ordinary skilled in the art without any creative work are within the protection scope of the present application.​​

[0051] It should be understood that when a component is referred to as being "on" another component that it can be directly on the other component or intervening components can also be present. When a component is referred to as being "connected" to another component that it can be directly connected to the other component or intervening components can also be present. When a component is referred to as being "disposed" on another component that it can be directly disposed on the other component or intervening components can also be present. The terms "vertical", "horizontal", "left", "right", and similar terms as used herein are for purposes of description only.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0053] The embodiments of the present application will be further described in conjunction with the drawings:

[0054] Image acquisition step: input the image frame I collected by the camera, and let the width and height of the image after adjustment be I w and I h , the interested region I roi is intercepted, and the interested region I roi =I[0:I w ,I h / 3:I h ]; define the coordinates of the upper left corner of the image frame as (0, 0), the width of the interested region I w , and the height of the interested region I , that is, the region of I h / 3-I h is intercepted as the interested region, through the interception of the interested region, the error caused by the environment is reduced; the image frame is cut and equally divided to obtain nine image blocks, and they are denoted as and enter the frame judgment step, the interested region is cut to depict the changes caused by local occlusion;

[0055] Frame judgment step: judge whether the image frame is the first frame, if it is the first frame, enter the first frame judgment step, if it is not the first frame, enter the change judgment step

[0056] First frame judgment steps: For vehicles in the initial starting phase, the camera acquires the first frame after startup and records three discontinuous frames over several seconds as key image frames, namely the 1st, 120th, and 150th frames after startup. During the initial starting phase, the vehicle's speed is relatively slow. Acquiring the 1st and 120th frames allows for a significant change in the scene captured by the camera, thus preventing false alarms due to slow speed. The regions of interest for each of the three key image frames are then calculated. Binary segmentation was performed on three regions of interest using the three-frame difference method, and binary maps of nine image patches were obtained. (i = 0, 1, ... 8). The cumulative motion information S of each image block within the region of interest is obtained using the first calculation formula, where... for The average value of the image. The accumulated motion information S is compared with a set threshold T. block In contrast, if the value is not less than the threshold, the camera is considered to be occluded; otherwise, it is not occluded, and the update process begins.

[0057] First calculation formula:

[0058]

[0059]

[0060] Change judgment step: If the current camera has been turned on for a period of time, the image is compared with all frames to determine whether there is a drastic change in the image; if the image acquisition information meets the second preset condition, it is judged as a drastic change, and the current frame judgment step is entered; if the image acquisition information is not less than the motion information threshold, it is judged as no drastic change has occurred, and the update step is entered.

[0061] Statistical sub-step: Based on the features of each image patch, and using the second calculation formula, obtain the statistical information of the entire region of interest, that is, the overall information of the image acquired by the camera over a period of time during the vehicle's journey, and proceed to the judgment sub-step.

[0062] For vehicles in the driving phase, calculate the current frame I. roi Each Sharpness and standard deviation The weighted sharpness C calculated using the second formula q and standard deviation C s , to be used as the global feature of the current frame. Where, w i and v i The weights for sharpness and standard deviation are respectively set to w in this scheme. i =v i =1 (i=0,1,…8).

[0063] The second calculation formula is:

[0064]

[0065]

[0066] The judgment sub-step is to calculate the average statistical information of all frames in the frame queue by using a third calculation formula, and to calculate the mean square error of the statistical information of the current frame and the average statistical information by using a fourth calculation formula, and further comprising an error threshold, when the error is greater than the error threshold, it is judged that a drastic change has occurred, and enters the current frame judgment step, if the error is less than the threshold, it is judged that no drastic change has occurred, and enters the updating step;

[0067] Frame queue I ROI Store the corresponding interest region of the past n image frames Cq j and Cs j , j = 0, 1,..., n-1, and in the present scheme, n = 10. The average sharpness Mq and the average standard deviation Ms of I ROI are calculated by the third calculation formula. Mq and Ms reflect the overall information of the images acquired by the camera in the past period of time during the vehicle driving;

[0068] The third calculation formula is:

[0069]

[0070]

[0071] The fourth calculation formula is used to calculate the error mse between the current frame and the previous frame. The size of mse can be used to judge whether the current camera state has changed.

[0072] The fourth calculation formula is:

[0073]

[0074] Further comprising a threshold T mse , compare mse with the threshold T mse , if mse is greater than the threshold T mse . It is considered that the current camera state may have undergone a drastic change.

[0075] The current frame judgment step: for the current frame that has undergone a drastic change, set it as the initial frame, and record the subsequent three frames that are not continuous within a few seconds as key image frames, i.e. the images of the 30th frame, the 150th frame and the 300th frame. In the process of vehicle driving, the interest regions of the above three frames within a long time are Ir ′ oi0 I r ′ oi1 I r ′ oi2 The three regions of interest are segmented into binary values ​​using the three-frame difference method, and the binary images in the nine image blocks are obtained. The first calculation formula can be used to determine whether the current camera is occluded.

[0076] Update steps: After the current frame is judged, check frame queue I. ROI Has the number of inner frames reached the maximum storage limit? If I ROI If the inner frame is full, delete the frame at the head of the queue, add the current frame to the tail, and re-enter the image acquisition step.

[0077] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A camera occlusion detection algorithm for in-vehicle driving, combining spatial and temporal features, characterized in that, Comprising The image acquisition step: acquiring the image frame collected by the camera, cropping and extracting the region of interest, and cutting the region of interest to obtain a plurality of image blocks, and entering the frame judgment step; The frame judgment step: judging whether the image frame is the first frame, if it is the first frame, entering the first frame judgment step, if it is not the first frame, entering the change judgment step; The first frame judgment step: if the current is the first frame after the camera is turned on, three non-continuous frames in the subsequent time are obtained as key image frames, three frame difference method is performed on the three key image frames, the first calculation formula is used to calculate the cumulative motion information of each image block in the region of interest, and the update step is entered, if the number of image blocks with cumulative motion information less than the motion information threshold is not less than the set number threshold, it is judged as occlusion; The change judgment step: if the camera has been turned on for a period of time, the picture comparison information of all frames is obtained, and the current frame judgment step is entered, when the image acquisition information meets the second preset condition, it is judged as a dramatic change, and then the current frame judgment step is entered, when the image acquisition information is not less than the motion information threshold, it is judged as no dramatic change, and then the update step is entered; The current frame judgment step: for the current frame with dramatic change, three non-continuous frames in the subsequent time are obtained as key image frames, three frame difference method is performed on the three key image frames, the first calculation formula is used to calculate the cumulative motion information of each image block in the region of interest, and the update step is entered, when the time information threshold is not less than the time information threshold, it is judged as occlusion; The update step: updating the current frame queue, and returning to the image acquisition step.

2. The camera occlusion detection algorithm for vehicles in motion combining spatial and temporal features according to claim 1, characterized in that, The first calculation formula is: Wherein, m is the number of image blocks; a binary image of the image block; For each image, the mean value of the image is computed as follows: The mean value of the image is computed as follows: S is the cumulative motion information; T s is a set threshold value for motion information.

3. The camera occlusion detection algorithm for vehicles in motion combining spatial and temporal features according to claim 1, characterized in that, The change judgment step includes the following steps: The statistical sub-step: according to the characteristics of each image block, and using the second calculation formula to obtain the statistical information of the entire region of interest, that is, the overall information of the image obtained by the camera in the past period of time during the vehicle driving, and entering the judgment sub-step; The judgment sub-step: using the third calculation formula to calculate the average statistical information of all frames in the frame queue, and using the fourth calculation formula to calculate the mean square error of the statistical information of the current frame and the average statistical information, and including an error threshold, when the error is greater than the error threshold, it is judged that the dramatic change occurs, and then the current frame judgment step is entered, if the error is less than the threshold, it is judged that no dramatic change occurs, and then the update step is entered.

4. The camera occlusion detection algorithm for vehicles in motion combining spatial and temporal features according to claim 3, characterized in that, The second calculation formula is: wherein, is the sharpness of the i-th image block of the region of interest corresponding to the current frame; Standard deviation of the i-th image block of the region of interest corresponding to the current frame; w i and v i are the weights of sharpness and standard deviation, respectively; C q Weighted sharpness for the current frame; C s Weighted standard deviation for the current frame.

5. The camera occlusion detection algorithm for in-vehicle driving according to claim 4, wherein, The third calculation formula is: Wherein, n is a time parameter; Cq j is the weighted sharpness of the jth key frame for the region of interest. Cs j is the weighted standard deviation of the interest region corresponding to the jth key frame in the past. M q is the average sharpness of the region of interest; M s The average standard deviation for the region of interest.

6. The camera occlusion detection algorithm for in-vehicle driving according to claim 5, wherein, The fourth calculation formula is: Wherein, mse is the error between the current frame and the previous frame.

7. The camera occlusion detection algorithm for in-vehicle driving according to claim 1, wherein, In the current frame judgment step, the current frame is the initial frame, and the 30th, 150th and 300th frames of images are recorded as key image frames.

8. The camera occlusion detection algorithm for in-vehicle driving according to claim 1, wherein, In the first frame judgment step, the first frame is the initial frame, and the 1st, 30th and 150th frames of images are recorded as key image frames.

9. The camera occlusion detection algorithm for in-vehicle driving according to claim 1, wherein, The image captured by the camera in the image acquisition step is I, and the width and height of the image after size adjustment are I w and I h , and the interested region I roi is intercepted, the interested region I roi = I[0:I w , I h / 3:I h ].

10. The camera occlusion detection algorithm for in-vehicle driving according to claim 1, wherein, The update step includes whether the number of frames in the frame queue is greater than the maximum storage number after the current frame judgment is completed, if the frame queue is full, the frame at the head of the queue is deleted, and the current frame is added to the tail.

Images