An algorithm for detecting highway litter based on image block division with low frame rate

Abstract

This application relates to the field of video analysis technology and discloses an algorithm for detecting highway litter based on image block segmentation, which successively includes: video frame extraction, automatic road area recognition, road segmentation, regional block expansion, expanded block grayscale image calculation, suspected moving target extraction, and litter discrimination. In this application, the detected picture area is segmented, and then each block area is processed independently. If there are obvious pixel changes in the expanded area corresponding to this block area, foreground detection is performed on this block area, and then the foreground targets obtained from each block area are merged to obtain the entire foreground image, and subsequent processing of litter target verification is performed on the foreground image. Due to the obvious spatio-temporal imbalance in highway traffic flow, there will be obvious scenes without vehicle flow continuously at different positions and different time periods in the monitoring area. By means of segmentation, the huge computational cost caused by detecting the entire picture can be avoided, and the computational amount can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the technical field of video analysis, and in particular to an algorithm for detecting highway spilled objects at a low frame rate based on image segmentation. Background Art

[0002] Highway spill identification mainly refers to a series of cognitive and analytical processes, including identifying various abnormal objects that appear on the highway, judging their sources, properties, and possible impacts on traffic.

[0003] For example, the spilled object detection method, device, electronic device and storage medium disclosed in the Chinese patent application number CN202110125460.7, the spilled object detection method includes: obtaining an original image of a road; segmenting the original image to obtain a segmentation map of the original image, wherein the segmentation map contains semantic information of multiple non-spilled objects; synthesizing the segmentation map according to a preset generative adversarial network to obtain a composite image of the road; comparing the original image and the composite image to obtain a difference feature map between the original image and the composite image; determining whether there is spilled objects on the road according to the difference feature map.

[0004] This technology obtains a synthetic image of the road by synthesizing the segmented images. In essence, it is based on foreground detection of the entire image or road area, and there is a situation where the image area to be processed is large. In addition, in the current technology, the foreground detection algorithm is calculated based on each pixel, which has a large amount of calculation and the related algorithm has high requirements on machine performance. Summary of the invention

[0005] The purpose of this application is to provide an algorithm for detecting highway spills at a low frame rate based on image segmentation, so as to solve the problem that the existing system proposed in the background technology is not effective in heating when used in winter due to the upper limit of underground temperature, and external equipment is still needed to adjust the indoor heating temperature.

[0006] To achieve the above purpose, the present application discloses the following technical solution: an algorithm for detecting highway spills at a low frame rate based on image segmentation, which comprises the following steps in sequence:

[0007] Video frame extraction: Get full-frame images in high-frame rate videos and extract one image every F frames;

[0008] Automatically identify road areas: Identify the extracted images to obtain road areas;

[0009] Road block division: Divide the obtained road area into N*N blocks;

[0010] Area block expansion: Expand the edge of each block outward by size s to cover the scattered objects falling on the boundary, and obtain the extend-block block;

[0011] Grayscale image calculation of extended blocks: Calculate the grayscale image of each extend-block. When the number of points in the grayscale images corresponding to two adjacent pictures exceeds M, perform GMM background modeling on the consecutive K frames of the extend-block corresponding to the later extracted picture of the two adjacent pictures, and extract the foreground target.

[0012] Extracting suspected moving targets: extracting suspicious targets from the foreground targets, tracking the suspicious targets, and extracting stationary targets;

[0013] Identification of scattered objects: The stationary target is traced back L frames for target matching to obtain its position in the L frame, and it is determined whether there is a change between its position in the L frame and the position in the current frame. If there is a change, it means that the motion state of the stationary target has changed from moving to stationary, and it is a scattered object, so it is reported. Otherwise, it is judged as a false alarm and the stationary target is discarded.

[0014] Preferably, the step of acquiring a full-frame image in a high frame rate video specifically includes:

[0015] Full-frame images of high-frame rate videos are obtained from road cameras, and one image is extracted every F frames.

[0016] Preferably, the step of identifying the extracted image to obtain the road area specifically includes:

[0017] The images extracted from the video frames are identified by a deep learning image semantic segmentation model to obtain ROI regions {[x1, y1], [x2, y2], [x3, y3]...[xn, yn]} where target detection is required, where xn and yn represent the coordinate positions of the center of each region in the image.

[0018] Preferably, the step of dividing the obtained road area into N*N blocks specifically includes:

[0019] Draw a polygon grayscale image grep according to the obtained ROI area;

[0020] A preprocessed image is obtained by performing a bitwise AND operation on the extracted image and the polygon grayscale image;

[0021] The preprocessed image is divided into N×N blocks.

[0022] Preferably, the area block expansion specifically includes:

[0023] The size of each block is [x, y, w, h], where x is the horizontal coordinate of the block in the image, y is the vertical coordinate of the block in the image, w is the length of the block, and h is the width of the block;

[0024] Expand the edge of the block by size s to obtain extend-block blocks. The area of ​​each extend-block block is [x, y, w+2s, h+2s].

[0025] Preferably, the calculation of the extended block grayscale image specifically includes:

[0026] Calculate the grayscale image of each extend-block block in frame t, recorded as , where i and j are the corresponding rows and columns of the extend-block in the N×N blocks respectively;

[0027] like , then perform GMM background modeling on the (i, j)th block of the t frame and the K consecutive frames to extract the foreground target.

[0028] Preferably, extracting the suspected moving target specifically includes:

[0029] The extracted foreground target is tracked using a target tracking algorithm. Based on the IOU of the current frame and the historical frame, the suspected target in different frames is associated and the motion trajectory of the suspected target is analyzed. The IOU calculation formula is:

[0030]

[0031] Among them, A is the current block and B is the historical frame.

[0032] Preferably, the scattered object identification specifically includes:

[0033] Set a stationary IOU threshold Ths, and compare the calculated IOU value corresponding to each target foreground with the stationary IOU threshold Ths. When the IOU value is greater than the stationary IOU threshold Ths, the suspected target is judged to be stationary and defined as the stationary target. The target motion trajectory of the stationary target is traced back to determine whether the state of the stationary target is from moving to stationary and meets the movement characteristics of scattered objects. If so, report the scattered objects. Otherwise, it is judged as a false alarm and the stationary target is discarded.

[0034] Preferably, the target motion trajectory backtracking specifically includes:

[0035] A corner detection algorithm is used to extract feature values ​​of the stationary target to form a feature vector representing target feature point information, and a target matching template is used to match the historical frame image to achieve target registration;

[0036] Set the window offset , get the grayscale change expression of the pixel before and after the moving window, and define the corner response value to quickly find the corner point; the grayscale change expression of the pixel is ,in, is the pixel coordinate corresponding to the window, is a window function, for The gray value of the pixel at the coordinate, for Gray value of the pixel at the coordinate;

[0037] The extracted feature values ​​are classified using a clustering algorithm, and the boundary points of stationary targets are removed using a Gaussian filter method. The images to be matched are overlapped and placed in a coordinate system. The slopes of the straight lines between the extracted feature points are first calculated. The same or similar feature points are correctly matched, and then the Hausdorff distance is used for accurate registration.

[0038] The similarity between two image blocks is determined using the absolute error sum algorithm, which is: , where, in the search graph S, take (i, j) as the upper left corner, take a subgraph of size M×m, and calculate its similarity with the template, It is the search graph S The pixel value of the point, is the pixel value of the matching image at coordinate (s, t);

[0039] Set a sliding window to cover the template image in sequence to obtain all the pixels in the window coverage area, and use the window to cover the image to be matched to obtain the pixels; subtract the pixels in the template coverage area from the pixels in the coverage area to be matched to obtain the absolute value sum of the grayscale differences of all the pixels; move the window on the image to be matched, and repeat the calculation of the absolute value sum, and end when the set search range is exceeded;

[0040] Find the window with the smallest SAD value within the search range as the target that best matches the template, obtain the coordinate point U of the matching target, and obtain the position sequence of the stationary target from the tLth frame to the tth frame before it stops. .

[0041] Preferably, the determining whether the state of the stationary target is from moving to stationary and meets the motion characteristics of the thrown object specifically includes:

[0042] The position change of the stationary target from the tL frame to the t frame is determined. If there is a position change, it means that the state of the stationary target is from moving to stationary and meets the motion characteristics of the scattered object.

[0043] Technical effect: The algorithm of the present application for detecting spilled objects on highways at a low frame rate based on image segmentation divides the detected picture area into blocks, and then processes each area independently. If there is an obvious pixel change in the extended area corresponding to the block area, foreground detection is performed on this area, and then the foreground targets obtained in each area are merged to obtain the entire foreground image, and then the foreground image is subsequently processed to verify the target of the spilled objects. Due to the obvious spatiotemporal imbalance of highway traffic, there will be scenes with obvious continuous lack of traffic at different locations and different time periods in the monitoring area. The huge computational cost of detecting the entire picture can be avoided by segmenting the image, effectively reducing the amount of calculation. Therefore, the present application can effectively improve the speed of the detection algorithm, making the algorithm have better landing value. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 A flowchart of an algorithm for detecting highway spilled objects at a low frame rate based on image segmentation provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0047] In this article, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprising..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0048] In the management of highways, the identification of scattered objects is not only conducive to improving the road environment, but also to improving the driving safety of vehicles on the road. Therefore, accurate identification of scattered objects has far-reaching significance for the supervision of driving behavior, the improvement of vehicle driving safety, and the maintenance of highways. In this regard, this embodiment discloses how Figure 1 The algorithm shown is a low frame rate algorithm for detecting spilled objects on highways based on image segmentation, which includes video frame extraction, automatic recognition of road areas, road segmentation, area block expansion, grayscale image calculation of expanded blocks, extraction of suspected moving targets and spilled object identification.

[0049] Below, the specific content of each step is explained in detail.

[0050] Video frame extraction: Get full-frame images in high-frame rate videos and extract one image every F frames.

[0051] Among them, the full-frame image is obtained from the high-frame rate video of the road camera.

[0052] Automatically identify road areas: Identify the extracted images to obtain road areas. Specifically include:

[0053] The images extracted from the video frames are identified by a deep learning image semantic segmentation model to obtain ROI regions {[x1, y1], [x2, y2], [x3, y3]...[xn, yn]} where target detection is required, where xn and yn represent the coordinate positions of the center of each region in the image.

[0054] Road block division: Divide the obtained road area into N*N blocks. Specifically including:

[0055] Draw a polygon grayscale image grep according to the obtained ROI area;

[0056] A preprocessed image is obtained by performing a bitwise AND operation on the extracted image and the polygon grayscale image;

[0057] The preprocessed image is divided into N×N blocks.

[0058] Block extension: Expand the edge of each block by a size of s to cover the scattered objects on the boundary, and obtain the extend-block block. Specifically, it includes:

[0059] The size of each block is [x, y, w, h], where x is the horizontal coordinate of the block in the image, y is the vertical coordinate of the block in the image, w is the length of the block, and h is the width of the block;

[0060] Expand the edge of the block outward by size s to obtain extend-block blocks. The area of ​​each extend-block block is [x, y, w+2s, h+2s]. The purpose of each extend-block block is to completely cover the small scattered objects that fall on the boundary of the original block.

[0061] Grayscale image calculation of extended blocks: Calculate the grayscale image of each extend-block. When the number of points in the grayscale images corresponding to two adjacent pictures exceeds M, perform GMM background modeling on the consecutive K frames of the extend-block corresponding to the later extracted picture of the two adjacent pictures, and extract the foreground target. Specifically include:

[0062] Calculate the grayscale image of each extend-block block in frame t, recorded as , where i and j are the corresponding rows and columns of the extend-block in the N×N blocks respectively;

[0063] like , then perform GMM background modeling on the (i, j)th block of the t frame and the K consecutive frames to extract the foreground target.

[0064] Extract suspected moving targets: extract suspicious targets from the foreground targets, track the suspicious targets, and extract stationary targets. Specifically include:

[0065] The extracted foreground target is tracked using a target tracking algorithm. Based on the IOU of the current frame and the historical frame, the suspected target in different frames is associated and the motion trajectory of the suspected target is analyzed. The IOU calculation formula is:

[0066]

[0067] Among them, A is the current block and B is the historical frame.

[0068] Identification of scattered objects: Go back L frames to match the static object, obtain its position in the L frame, and determine whether there is a change between its position in the L frame and the current frame. If there is a change, it means that the motion state of the static object has changed from motion to stillness, and it is a scattered object, so it is reported. Otherwise, it is judged as a false alarm and the static object is discarded. Specifically, it includes:

[0069] Set a stationary IOU threshold Ths, and compare the calculated IOU value corresponding to each target foreground with the stationary IOU threshold Ths. When the IOU value is greater than the stationary IOU threshold Ths, the suspected target is judged to be stationary and defined as the stationary target. The target motion trajectory of the stationary target is traced back to determine whether the state of the stationary target is from moving to stationary and meets the movement characteristics of scattered objects. If so, report the scattered objects. Otherwise, it is judged as a false alarm and the stationary target is discarded.

[0070] In the identification of scattered objects, the process of tracing back the target motion trajectory specifically includes:

[0071] A corner detection algorithm is used to extract feature values ​​of the stationary target to form a feature vector representing target feature point information, and a target matching template is used to match the historical frame image to achieve target registration;

[0072] Set the window offset , get the grayscale change expression of the pixel before and after the moving window, and define the corner response value to quickly find the corner point; the grayscale change expression of the pixel is ,in, is the pixel coordinate corresponding to the window, is a window function, for The gray value of the pixel at the coordinate, for Gray value of the pixel at the coordinate;

[0073] The extracted feature values ​​are classified using a clustering algorithm, and the boundary points of stationary targets are removed using a Gaussian filter method. The images to be matched are overlapped and placed in a coordinate system. The slopes of the straight lines between the extracted feature points are first calculated. The same or similar feature points are correctly matched, and then the Hausdorff distance is used for accurate registration.

[0074] The similarity between two image blocks is determined using the absolute error sum algorithm, which is: , where, in the search graph S, take (i, j) as the upper left corner, take a subgraph of size M×m, and calculate its similarity with the template, It is the search graph S The pixel value of the point, is the pixel value of the matching image at coordinate (s, t);

[0075] Set a sliding window to cover the template image in sequence to obtain all the pixels in the window coverage area, and use the window to cover the image to be matched to obtain the pixels; subtract the pixels in the template coverage area from the pixels in the coverage area to be matched to obtain the absolute value sum of the grayscale differences of all the pixels; move the window on the image to be matched, and repeat the calculation of the absolute value sum, and end when the set search range is exceeded;

[0076] Find the window with the smallest SAD value within the search range as the target that best matches the template, obtain the coordinate point U of the matching target, and obtain the position sequence of the stationary target from the tLth frame to the tth frame before it stops. .

[0077] By acquiring the above position sequence, it is determined whether the state of the stationary target is from moving to stationary and meets the motion characteristics of the scattered object, specifically including:

[0078] The position change of the stationary target from the tL frame to the t frame is determined. If there is a position change, it means that the state of the stationary target is from moving to stationary and meets the motion characteristics of the scattered object.

[0079] Based on the above, the algorithm of low frame rate detection of spilled objects on highways based on image segmentation in this embodiment divides the detected picture area into blocks, and then processes each area independently. If there is an obvious pixel change in the extended area corresponding to the block area, foreground detection is performed on this area, and then the foreground targets obtained in each area are merged to obtain the entire foreground image, and then the foreground image is subsequently processed for target verification of the spilled objects. Due to the obvious spatiotemporal imbalance of highway traffic, there will be scenes with obvious continuous lack of traffic at different locations and different time periods in the monitoring area. The huge computational cost of detecting the entire picture can be avoided by segmenting, effectively reducing the amount of calculation. It has a good promoting effect on reducing the cost of machine application and improving the response efficiency of the detection algorithm, and has higher landing value.

[0080] Finally, it should be noted that the above are only preferred embodiments of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An algorithm for detecting highway spills at a low frame rate based on image segmentation, characterized in that: The following steps are included in sequence: Video frame extraction: Get full-frame images in high-frame rate videos and extract one image every F frames; Automatically identify road areas: Identify the extracted images to obtain road areas; Road block division: Divide the obtained road area into N*N blocks; Area block expansion: Expand the edge of each block outward by size s to cover the scattered objects falling on the boundary, and obtain the extend-block block; Grayscale image calculation of extended blocks: Calculate the grayscale image of each extend-block. When the number of points in the grayscale images corresponding to two adjacent pictures exceeds M, perform GMM background modeling on the consecutive K frames of the extend-block corresponding to the later extracted picture of the two adjacent pictures, and extract the foreground target. Extracting suspected moving targets: extracting suspicious targets from the foreground targets, tracking the suspicious targets, and extracting stationary targets; Dispersed object identification: The stationary target is traced back L frames to perform target matching, and its position in the L frame is obtained, and it is determined whether there is a change between its position in the L frame and the position in the current frame. If there is a change, it means that the motion state of the stationary target has changed from moving to stationary, and it is a scattered object, and it is reported. Otherwise, it is determined to be a false alarm and the stationary target is discarded; The extended block grayscale image calculation specifically includes: Calculate the grayscale image of each extend-block block in frame t, recorded as , where i and j are the corresponding rows and columns of the extend-block in the N×N blocks respectively; like , then perform GMM background modeling on the (i, j)th block of the t frame in K consecutive frames to extract the foreground target; The extracting of the suspected moving target specifically includes: The extracted foreground target is tracked using a target tracking algorithm. Based on the IOU of the current frame and the historical frame, the suspected target in different frames is associated and the motion trajectory of the suspected target is analyzed. The IOU calculation formula is: Among them, A is the current block and B is the historical frame; The scattered object identification specifically includes: Set a static IOU threshold Ths, compare the calculated IOU value corresponding to each target foreground with the static IOU threshold Ths, and when the IOU value is greater than the static IOU threshold Ths, determine that the suspected target is static and define it as the static target, backtrack the target motion trajectory of the static target, and determine whether the state of the static target is from motion to static and meets the motion characteristics of the scattered object. If yes, report the scattered object, otherwise it is determined as a false alarm and the static target is discarded; The target motion trajectory backtracking specifically includes: A corner detection algorithm is used to extract feature values ​​of the stationary target to form a feature vector representing target feature point information, and a target matching template is used to match the historical frame image to achieve target registration; Set the window offset , get the grayscale change expression of the pixel before and after the moving window, and define the corner response value to quickly find the corner point; the grayscale change expression of the pixel is ,in, is the pixel coordinate corresponding to the window, is a window function, for The gray value of the pixel at the coordinate, for Gray value of the pixel at the coordinate; The extracted feature values ​​are classified using a clustering algorithm, and the boundary points of stationary targets are removed using a Gaussian filter method. The images to be matched are overlapped and placed in a coordinate system. The slopes of the straight lines between the extracted feature points are first calculated. The same or similar feature points are correctly matched, and then the Hausdorff distance is used for accurate registration. The similarity between two image blocks is determined using the absolute error sum algorithm, which is: , where, in the search graph S, take (i, j) as the upper left corner, take a subgraph of size M×m, and calculate its similarity with the template, It is the search graph S The pixel value of the point, is the pixel value of the matching image at coordinate (s, t); Set a sliding window to cover the template image in sequence to obtain all the pixels in the window coverage area, and use the window to cover the image to be matched to obtain the pixels; subtract the pixels in the template coverage area from the pixels in the coverage area to be matched to obtain the absolute value sum of the grayscale differences of all the pixels; move the window on the image to be matched, and repeat the calculation of the absolute value sum, and end when the set search range is exceeded; Find the window with the smallest SAD value within the search range as the target that best matches the template, obtain the coordinate point U of the matching target, and obtain the position sequence of the stationary target from the tLth frame to the tth frame before it stops. .

2. The algorithm for detecting highway spills based on low frame rate image segmentation according to claim 1 is characterized in that: The method of obtaining a full-frame image in a high frame rate video specifically includes: Full-frame images of high-frame rate videos are obtained from road cameras, and one image is extracted every F frames.

3. The algorithm for detecting highway spills based on low frame rate image segmentation according to claim 1 is characterized in that: The step of identifying the extracted image to obtain the road area specifically includes: The images extracted from the video frames are identified by a deep learning image semantic segmentation model to obtain ROI regions {[x1, y1], [x2, y2], [x3, y3]...[xn, yn]} where target detection is required, where xn and yn represent the coordinate positions of the center of each region in the image.

4. The algorithm for detecting highway spills based on low frame rate image segmentation according to claim 3 is characterized in that: The step of dividing the obtained road area into N*N blocks specifically includes: Draw a polygon grayscale image grep according to the obtained ROI area; A preprocessed image is obtained by performing a bitwise AND operation on the extracted image and the polygon grayscale image; The preprocessed image is divided into N×N blocks.

5. The algorithm for detecting highway spills based on low frame rate based on image segmentation according to claim 4 is characterized in that: The area block expansion specifically includes: The size of each block is [x, y, w, h], where x is the horizontal coordinate of the block in the image, y is the vertical coordinate of the block in the image, w is the length of the block, and h is the width of the block; Expand the edge of the block by size s to obtain extend-block blocks. The area of ​​each extend-block block is [x, y, w+2s, h+2s].

6. The algorithm for detecting highway spillage based on low frame rate image segmentation according to claim 1 is characterized in that: The step of judging whether the state of the stationary target is from moving to stationary and meets the motion characteristics of the scattered object specifically includes: The position change of the stationary target from the tL frame to the t frame is determined. If there is a position change, it means that the state of the stationary target is from moving to stationary and meets the motion characteristics of the scattered object.

Images