Road facility shielding detection, background modeling and anomaly detection method based on monitoring camera
Through example segmentation model and background modeling technology, the problems of road facilities occlusion detection and abnormal identification are solved, and automated and intelligent monitoring of road facilities are realized, and operation and maintenance efficiency and traffic safety are improved.
Patent Information
- Application Number
- CN202410983328.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-22
AI Technical Summary
The existing technology is difficult to timely detect and deal with abnormal situations such as safety guardrails, lane lines, stop lines, pedestrian crossings, road signs, etc., resulting in traffic safety hazards and operation and maintenance Inefficient.
An example segmentation model is used to extract road facilities and traffic participants information from the monitoring camera video frame, and the minimum external rectangle is used to identify whether the road facilities are blocked, and a background model is constructed for abnormal detection to realize automated and intelligent monitoring of road facilities.
Real-time monitoring and abnormal identification of road facilities have been realized, which significantly reduces personnel and time costs and improves the level of informatization and digital construction of urban roads and highways.
Smart Images

Figure CN118918544A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of traffic monitoring and road facility monitoring, and in particular to a road facility occlusion detection, background modeling and anomaly detection method based on a monitoring camera. Background Art
[0002] With the rapid development of economic and social construction, the mileage of urban roads and the number of cars have gradually increased. With the rapid development of information and digital construction, intelligent transportation systems and transportation information technology have played an increasingly important role in traffic management. From large cities to county towns, important intersections and sections of roads are basically all covered by key transportation facilities such as traffic signals, traffic lights, checkpoints, electric police camera capture facilities, radar detectors, induction screens, and traffic signs. The information and digital construction of urban roads and highways has achieved remarkable results.
[0003] The detection methods of existing operation and maintenance systems basically only support facilities with network communication capabilities, such as signal machines, induction screens, camera capture facilities, radars, etc., and mainly rely on the network communication capabilities provided by the facilities themselves to determine whether they have failed, thereby reminding operation and maintenance personnel to conduct manual investigation and processing.
[0004] However, road facilities such as safety guardrails, lane lines, stop lines, crosswalks, road signs, signal poles, speed limit signs, traffic signs, etc. that do not have network communication capabilities may also be damaged, missing, misplaced, and other abnormal conditions due to various internal and external factors such as traffic accidents, item quality, and weather, which can affect the driver's judgment and may cause traffic accidents in severe cases.
[0005] At present, the damage, missing, position displacement and other abnormal conditions of such road infrastructure can only be reported by citizens or regularly inspected by operation and maintenance personnel. This method will cause a large number of personnel consumption, and more importantly, the efficiency is too low and the delay is extremely high. It can be seen that the depth and breadth of the integration of advanced information technology and transportation are still insufficient, and the model and standards that can be replicated and promoted on a large scale have not yet been formed. The problem of focusing on construction and neglecting operation and maintenance cannot be ignored. Summary of the invention
[0006] In order to overcome the problem in the above-mentioned prior art that it is difficult to timely detect abnormalities such as occlusion, damage, missingness, and position displacement of road infrastructure, the present invention proposes a road facility occlusion detection method, which can timely detect the problem of road facilities being obstructed by traffic participants.
[0007] The present invention proposes a road facility occlusion detection method. First, the identification object information is extracted from the video frame collected by the road monitoring camera through the pre-trained instance segmentation model. The identification objects include road facilities and traffic participants. The identification object information includes: the type, shape, size, color, and coordinates of all pixels contained in the identification object; then, the directory index information of the road facilities is constructed and stored in the data set D1, and the directory index information of the traffic participants is constructed and stored in the data set D2, and the following steps are performed to determine whether the road facilities are occluded by the traffic participants:
[0008] SA1, for the road facilities in the data set D1, extract their boundary information in the original image, construct the boundary pixel point set Po of the road facilities, and associate it with the directory index information;
[0009] SA2, for data sets D1 and D2, obtain the minimum bounding rectangle of each identified object and associate it with the directory index information;
[0010] SA3, traverse the data set D1, and select road facilities that are not blocked by traffic participants based on the overlap of the minimum circumscribed rectangle.
[0011] Preferably, in SA3, for a single road facility P, the steps of determining whether it is blocked by a traffic participant are as follows:
[0012] SA31, extracting traffic participant C from data set D2 and comparing it with road facility P, and determining whether there is a risk of overlap between the minimum bounding rectangle corresponding to road facility P and the minimum bounding rectangle of traffic participant C; if not, executing step SA34; if yes, executing step SA32;
[0013] SA32, let the overlapping area of the minimum circumscribed rectangle of the road facility P and the minimum circumscribed rectangle of the traffic participant C be recorded as the PC rectangle, and count the coordinate pixel points in the PC rectangle;
[0014] SA33, determine whether there is a coordinate pixel point in the PC rectangle that exists both in the road facility P and in the traffic participant C; if yes, determine that P and C overlap, that is, the road facility P is blocked by the traffic participant C, and add a valid state flag "False" to the road facility P; if no, execute step S34;
[0015] SA34. Determine whether the data set D2 has been traversed; if not, update the traffic participant C and return to step SA31; if yes, determine that the road facility P is not blocked by the traffic participant, and add a valid state flag "True" to the road facility P.
[0016] Preferably, the method for determining whether the minimum bounding rectangles of the two identified objects P1 and P2 have an overlapping risk is:
[0017] Let the lower left corner of the minimum circumscribed rectangle of P1 be (x1 min ,y1 min ), the upper right corner is (x1 max ,y1 max ); Let the lower left corner point of the minimum circumscribed rectangle of the identified object P2 be (x2 min ,y2 min ), the upper right corner is (x2 max ,y2 max );
[0018] If x1 max <x2 min And y1 max <y2 min , or x2 max <x1 min And y2 max <y1 min , then it is judged that P1 and P2 do not overlap;
[0019] If x2 min ≤x1 max And y2 min ≤y1 max , or x1 min ≤x2 max And y1 min ≤y2 max , then it is judged that there is an overlapping risk between P1 and P2.
[0020] Preferably, after SA3, the method further includes: judging one by one whether the road facilities marked as “True” in the data set D1 are blocked by the road facilities; for the road facilities Q marked as “True” in the data set D1, the judging method is as follows:
[0021] SA41, extracting road facility P from data set D1 and comparing it with road facility Q, and determining whether there is a risk of overlap between the minimum bounding rectangle corresponding to road facility Q and the minimum bounding rectangle of road facility P; if not, executing step SA45; if yes, executing step SA42;
[0022] SA42, let the overlapping area of the minimum circumscribed rectangle of the road facility Q and the minimum circumscribed rectangle of the road facility P be recorded as the QP rectangle, and count the coordinate pixel points in the QP rectangle;
[0023] SA43, traverse the coordinate pixel points in the QP rectangle, count and select the number of coordinate pixel points in the QP rectangle that exist in both the road facility Q and the road facility P, record it as S QP’ ;
[0024] SA44. Let the number of pixels contained in the road facility Q be S Q, calculate the proportionality factor N Q =S QP’ / S Q ; Determine whether N is satisfied Q >N, where N is the set proportionality coefficient; if yes, the overlapping area between the road facility Q and the road facility P is associated with the road facility Q as the auxiliary information, and then the next road facility Q with the valid status marked as "True" is executed; if no, step SA45 is executed;
[0025] SA45. Determine whether the data set D1 has been traversed; if not, update the road facility P and return to step SA41; if yes, determine whether the road facility Q is not blocked by other road facilities.
[0026] Preferably, the data sets D1 and D2 use lists to store directory index information of the identified objects, and the directory index information includes a network video stream address, a timestamp, a video frame number, and a self-incrementing serial number of the identified object in the original image.
[0027] The present invention proposes a road background modeling method based on a surveillance camera, comprising the following steps:
[0028] S1, sampling the real-time network video stream information of the road monitoring camera over a continuous period, and obtaining the number of picture frames, recorded as Sum;
[0029] S2. Extract the identification object information of each picture frame through the instance segmentation model, and construct the data sets D1 and D2 corresponding to each picture frame; use the road facility occlusion detection method to process the identification object information of each picture frame, and determine the identification object information, auxiliary information and valid state identification of the road infrastructure contained in each picture frame; the auxiliary information is the boundary pixel set Po and the overlapping area between the road facilities;
[0030] S3. Track the same road facilities in different picture frames according to the generation time of the picture frames, and make the road facilities with the valid status identification "True" associated in each picture frame as automatic modeling objects, and the road facilities other than the automatic modeling objects as supplementary modeling objects; model each automatic modeling object and the supplementary modeling object to obtain a background model of the road facilities;
[0031] S4, assigning a pixel value of a region of the background model corresponding to the road infrastructure in each picture frame to 0, then gray-processing the picture frame as a candidate background image, then clustering the candidate background images, obtaining all the candidate background images in the largest cluster, performing pixel fitting, and obtaining a final fitting image as the final background image;
[0032] S5. Use the SURF corner feature algorithm to detect and extract M corner feature descriptors in the background image, and delete the corner feature descriptors whose minimum distance from the road infrastructure in the background image is less than the set distance threshold; sort the remaining corner feature descriptors in descending order of confidence, and the obtained ordered sequence of corner feature descriptors is the background model of the surveillance camera.
[0033] Preferably, the modeling steps of a single automatic modeling object include the following steps:
[0034] S31. Extract the recognition object information and affiliated information corresponding to the automatic modeling object in the picture frame as the associated information of the picture frame, cluster the associated information of each picture frame, and obtain the number T of picture frames included in the largest cluster;
[0035] S32. When T≥Sum×t, then fit the associated information of the picture frames in the largest cluster to obtain the background model of the automatic modeling object, including the contour curve, color and affiliated information; t is the set proportionality coefficient.
[0036] Preferably, the modeling method for the supplementary modeling object and the road facilities with T<Sum×t in S32 is:
[0037] Sample the real-time network video stream information of the road surveillance camera in another selected time period until the automatic modeling object satisfies T≥Sum×t, and then use the road background modeling method based on the surveillance camera for modeling;
[0038] Or perform manual modeling.
[0039] A method for detecting and identifying anomalies in road facilities based on a surveillance camera proposed by the present invention includes the following steps:
[0040] After the road facilities are completed, use the road background modeling method based on the surveillance camera to obtain the background model of the road facilities and the background model of the surveillance camera as the reference frame background model;
[0041] During the monitoring process, obtain the network video stream of the surveillance camera, and use the road background modeling method based on the surveillance camera to obtain the background model of the road facilities and the background model of the surveillance camera as the comparison frame background model;
[0042] Calculate the rotation angle and origin offset vector of the comparison frame background model of the surveillance camera relative to the reference frame background model; according to the rotation angle and origin offset vector, adjust the comparison frame background model of each road facility;
[0043] For each road facility, compare the adjusted comparison frame background model with the reference frame background model, calculate the similarity between the two, and judge whether the road facility is abnormal according to the similarity.
[0044] Preferably, calculating the similarity between the comparison frame background model and the reference frame comparison model of the road facilities with the valid state marked as False in the comparison frame includes the following steps:
[0045] St61, comparing the comparison frame background model with the reference frame comparison model, screening the missing pixels of the comparison frame background model relative to the reference frame comparison model, adding the screened pixels to the comparison frame background model, and assigning values to the pixels added to the comparison frame background model according to the reference frame background model;
[0046] St62. Use a similarity algorithm to calculate the similarity between the comparison frame background model and the reference frame background model.
[0047] The advantages of the present invention are:
[0048] 1. The road facility occlusion detection method proposed in the present invention uses an instance segmentation module to extract road facilities and traffic participants from the video frames of the monitoring video stream, and identifies whether the road facilities are blocked by traffic participants based on the minimum circumscribed rectangle, thereby realizing real-time monitoring of abnormal problems such as damage, missing, and position displacement of road facilities, and realizing timely identification, discovery, and maintenance of road facilities.
[0049] 2. The present invention further monitors the problem of mutual obstruction of road facilities, and realizes all-round automated and intelligent monitoring of road infrastructure that do not have network communication capabilities, such as safety guardrails, lane lines, stop lines, crosswalks, road signs, signal poles, speed limit signs, and traffic signs, thereby significantly reducing personnel costs and time costs, and improving the level of informatization and digital construction of urban roads and highways.
[0050] 3. The road background modeling method based on surveillance cameras proposed in the present invention classifies road facilities in combination with the effective sign status, and then models the two categories separately, and models the surveillance cameras, providing a reliable comparison basis for subsequent road facility abnormality judgment.
[0051] 4. The road facility anomaly detection and identification method based on surveillance cameras proposed in the present invention fully considers the occlusion effect between the target road infrastructure and traffic participants, as well as other road infrastructure, selects more complete and accurate road facility target feature information for feature comparison, and determines the abnormal state. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 It is a flow chart of a road facility occlusion detection method;
[0053] Figure 2 A flowchart of the steps for determining whether a single road facility P is blocked by a traffic participant;
[0054] Figure 3 A flowchart of the steps of determining whether a road facility Q marked as “True” in a data set D1 is blocked by other road facilities;
[0055] Figure 4 The present invention is a flow chart of a method for detecting and identifying anomalies of road facilities based on surveillance cameras. DETAILED DESCRIPTION
[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0057] Instance Segmentation Models and Datasets
[0058] In this embodiment, an instance segmentation model is proposed, which is built based on a neural network and trained using a machine learning method. The instance segmentation model can be, but is not limited to, a deep learning neural network model such as VGG16, FCN, DeconvNet, SegNet, DeepLab, Mask-RCNN, etc., which is used to identify specific objects and perform instance segmentation on them.
[0059] The input of the instance segmentation model is the original image captured by the road monitoring camera, and the output of the instance segmentation model is the identification object information; the identification objects include road facilities and traffic participants.
[0060] Traffic participants are dynamic objects, including motor vehicles, pedestrians, non-motor vehicles, etc. Road facilities are static objects, including road traffic signs, road traffic markings, traffic lights, signal light poles, safety guardrails, speed limit signs, etc.
[0061] Specifically, road traffic signs include warning signs, prohibition signs, instruction signs, guide signs, tourist area signs, road construction safety signs, and auxiliary signs. Road traffic markings include prohibition markings, instruction markings, and warning markings. They are traffic control facilities that are directly painted on the road surface or laid out into lines and symbols with concrete prefabricated blocks, etc., and are coordinated with road traffic signs. Road traffic markings include continuous lines, discontinuous lines, arrow indicator lines, etc., and are mostly painted in white or yellow, such as lane lines, stop lines, crosswalk lines, and tidal lane lines.
[0062] The identification object information includes: the type, shape, size, color, and coordinates of all pixels contained in the identification object.
[0063] To facilitate information storage, in this implementation, a list is constructed to store the identification object information extracted by the instance segmentation model. The list is named using identifiers that correspond one-to-one to the identification objects. The identifiers are used for target indexing to enable fast, efficient and accurate query matching of the required identification objects, and then extract relevant data for application.
[0064] The naming rules of the identifier can be multi-segment. For example, the first segment uses the network video stream address, so that all image information from the same network video stream information can be assigned the same first index information. The second segment uses information such as timestamps and video frames to assign second index information to different images from the same network video stream information according to time sequence feature information. The third segment uses a self-incrementing serial number to assign third index information to different targets detected and segmented from the same image information according to position distribution sequence feature information. In this way, by combining the first index information, the second index information, and the third index information to construct a complete directory index information, the network video stream information, image information, and target attribute information to which the target to be determined belongs can be determined.
[0065] In this embodiment, the identified objects extracted by the instance segmentation model are divided into two categories: road facilities and traffic participants. The directory index information of the road facilities is stored in the data set D1, and the directory index information of the traffic participants is stored in the data set D2; thereby, the data sets D1 and D2 corresponding to each original image are obtained, so as to judge whether the identified objects in a single original image overlap based on the data sets.
[0066] A road facility occlusion detection method proposed in this embodiment preliminarily evaluates whether the road facilities are normal or not by judging the target overlap. Figure 1 ,The target overlap judgment method includes the following steps.
[0067] SA1. For the road facilities in the data set D1, extract their boundary information in the original image, construct the boundary pixel point set Po of the road facilities, and associate it with the directory index information.
[0068] In specific implementation, the data set D1 is traversed, and the horizontal and vertical coordinates of each pixel point on the boundary of the identified object corresponding to each directory index information are queried in turn. Then, starting from the upper left corner point, the boundary pixel points of the identified object are added to the boundary pixel point set Po in a clockwise direction. The boundary pixel point set Po is associated as the auxiliary information of the directory index information.
[0069] In specific implementation, the boundary coordinate extreme value of the road facility can also be calculated in combination with the auxiliary information, that is, for each boundary pixel point set Po, the minimum horizontal coordinate value Pox of the pixel point is extracted. min , the maximum value of the horizontal axis Pox maxAnd the minimum value of the vertical coordinate Poy min , maximum value of vertical coordinate Poy max
[0070] SA2. For data sets D1 and D2, obtain the minimum bounding rectangle of each recognized object and associate it with the directory index information.
[0071] The method for obtaining the minimum enclosing rectangle of the identified object is as follows: first, the pixel coordinate set of the identified object in the original image is obtained by combining the directory index information, and then the minimum horizontal coordinate value x of the identified object is obtained. min , the maximum value of the horizontal axis x max And the minimum value of the ordinate y min , maximum value of ordinate y max ; Then the lower left corner point of the minimum circumscribed matrix of the identified object is (x min ,y min ), the upper right corner point of the minimum circumscribed matrix of the identified object is (x max ,y max ).
[0072] SA3, traverse the data set D1, and select road facilities that are not blocked by traffic participants; Figure 2 ,For a single road facility P, the steps to determine whether it is blocked by traffic participants are as follows.
[0073] SA31, extracting traffic participants C from data set D2 and comparing them with road facilities P, and determining whether there is a risk of overlap between the minimum bounding rectangle corresponding to road facility P and the minimum bounding rectangle of traffic participants C;
[0074] Specifically, let the lower left corner point of the minimum circumscribed rectangle of the road facility P in the data set D1 be (px min ,py min ), the upper right corner is (px max ,py max ); Let the lower left corner point of the minimum bounding rectangle of the traffic participant C in the data set D2 be (cx min ,cy min ), the upper right corner is (cx max ,cy max );
[0075] If px max <cx min And py max <cy min , or cx max <px min Andcy max <py min, it is determined that the road facility P and the traffic participant C do not overlap, and then step SA34 is executed;
[0076] If cx min ≤px max Andcy min ≤py max , or px min ≤cx max And py min ≤cy max , it is determined that there is an overlap risk between the road facility P and the traffic participant C, and then step SA32 is executed;
[0077] SA32: Let the overlapping area of the minimum circumscribed rectangle of the road facility P and the minimum circumscribed rectangle of the traffic participant C be denoted as the PC rectangle, and the lower left corner point of the PC rectangle be denoted as (pcx min ,pcy min ), the upper right corner is recorded as (pcx max ,pcy max );
[0078] When cx min ≤px max Andcy min ≤py max , then pcx min =cx min , pcy min =cy min ,pcx max =px max , pcy max =py max ;
[0079] When px min ≤cx max And py min ≤cy max , then pcx min =px min , pcy min =py min ,pcx max =cx max , pcy max =cy max ;
[0080] SA33, determine whether there is a coordinate pixel point in the PC rectangle that exists both in the road facility P and in the traffic participant C; if yes, determine that P and C overlap, that is, the road facility P is blocked by the traffic participant C, and add a valid state flag "False" to the road facility P; if no, execute step S34;
[0081] SA34. Determine whether the data set D2 has been traversed; if not, update the traffic participant C and return to step SA31; if yes, determine that the road facility P is not blocked by the traffic participant, and add a valid state flag "True" to the road facility P.
[0082] In specific implementation, the target index directories in data sets D1 and D2 can be sorted according to the coordinate pixel positions. For example, the target index directories in the data sets can be sorted in the order of the vertical coordinate from small to large and the horizontal coordinate from small to large. Then, when traversing data set D1 in SA3, the road facility P is selected in turn according to the sorting of the target index directories; when selecting road facility P, the traffic participants in data set D2 are selected in turn according to the sorting of the target index directory as C to compare the coordinates with the road facility P.
[0083] It is worth noting that in the process of traversing the data set D2, once a traffic participant overlapping with the road facility P is found, the road facility P is marked as being obscured by the traffic participant, that is, the road facility P is added with a valid state flag "False"; then the search for the data set D2 is abandoned, and the data set D1 is returned to update the road facility P, and then the data set D2 is searched again to determine whether the new road facility P is obscured by the traffic participant.
[0084] SA4, for the road facilities marked as “True” in the data set D1, filter the parts that are not blocked by the remaining road facilities in the data set D1;
[0085] In this step, the road facilities marked as “True” in the data set D1 are judged one by one to see whether they are blocked by other road facilities in the data set D1.
[0086] Reference Figure 3 , the steps to determine whether the road facility Q marked as “True” in the data set D1 is blocked by other road facilities are as follows:
[0087] SA41, extracting road facility P from data set D1 and comparing it with road facility Q, and determining whether there is a risk of overlap between the minimum bounding rectangle corresponding to road facility Q and the minimum bounding rectangle corresponding to road facility P;
[0088] Let the lower left corner of the minimum circumscribed rectangle of the road facility Q be (Qx min ,Qy min ), the upper right corner is (Qx max ,Qy max ); Let the lower left corner point of the minimum circumscribed rectangle of the road facility P in the data set D1 be (Px min ,Py min ), the upper right corner is (Px max ,Py max );
[0089] If Qx max <Px min And Qy max <Py min , or Px max <Qx min And Py max <Qy min , it is determined that the two do not overlap, and then step SA45 is executed;
[0090] If Px min ≤Qx max And Py min ≤Qy max , or Qx min ≤Px max And Qy min ≤Py max , it is determined that there is an overlap risk between the two, and then step SA42 is executed;
[0091] SA42: Let the overlapping area of the minimum circumscribed rectangle of road facility Q and the minimum circumscribed rectangle of road facility P be denoted as QP rectangle, and the lower left corner point of QP rectangle be denoted as (QPx min ,QP min ), the upper right corner is denoted as (QPx max ,QP max );
[0092] When Px min ≤Qx max And Py min ≤Qy max , then QPx min =Px min ,QPy min =Py min , QPx max =Qx max ,QPy max =Qy max ;
[0093] When Qx min ≤Px max And Qy min ≤Py max , then QPx min =Qx min ,QPy min =Qy min , QPx max =Px max ,QPy max =Py max ;
[0094] SA43, traverse the coordinate pixel points in the QP rectangle, count and select the number of coordinate pixel points in the QP rectangle that exist in both the road facility Q and the road facility P, record it as S QP’ ;
[0095] SA44. Let the number of pixels contained in the road facility Q be S Q , calculate the proportionality factor N Q =S QP’ / S Q ; Determine whether N is satisfied Q >N, where N is the set proportionality coefficient; if yes, the overlapping area between the road facility Q and the road facility P is associated with the road facility Q as the auxiliary information, and then the next road facility Q with the valid status marked as "True" is executed; if no, step SA45 is executed;
[0096] In specific implementation, the number of pixels contained in the road facility P can also be recorded as S P , calculate the proportionality factor N P =S QP’ / S P ; If N is satisfied P > N and the valid status mark of road facility P is "True", then the overlapping area of road facility Q and road facility P is associated with road facility P as auxiliary information, and the subsequent execution of road facility Q can directly skip the area that satisfies N. P >N and the road facility P with valid status flag “True”;
[0097] N represents the threshold value of the ratio of the set overlapping area to the original image area, and can be specifically taken in the interval [0.01, 0.5], for example, 0.05, 0.1, etc.
[0098] SA45. Determine whether the data set D1 has been traversed; if not, update the road facility P and return to step SA41; if yes, determine whether the road facility Q is not blocked by other road facilities.
[0099] A road background modeling method based on surveillance camera
[0100] The road background modeling method based on surveillance cameras proposed in this embodiment is used to obtain the background model of road facilities and surveillance cameras. The method includes the following steps:
[0101] S1. Sample the real-time network video stream information of the road monitoring camera over a continuous period of time, and obtain the number of picture frames, which is recorded as Sum.
[0102] In specific implementation, the real-time network video stream information can be set to be sampled every other frame, for example, one frame is extracted every 2, 5 or 10 frames. Obviously, sampling every other frame can make the adjacent target image information have a relatively sufficient time span, so that the foreground information such as dynamic traffic participants disappears quickly, making the background modeling more efficient and accurate. Sum should be greater than 100 and not greater than 1000. On the one hand, it avoids the incomplete and inaccurate background modeling caused by the small amount of picture frame data, and on the other hand, it avoids the low modeling efficiency caused by too much data.
[0103] S2. Extract the identification object information of each picture frame through the instance segmentation model, and construct the data sets D1 and D2 corresponding to each picture frame; use the target overlap judgment method recorded in the above SA1-SA4 to process the identification object information of each picture frame, and determine the identification object information, ancillary information and valid status identification of the road infrastructure contained in each picture frame; the identification object information includes: the type, shape, size, color of the identification object, and the coordinates of all the pixels contained therein, and the ancillary information is the overlapping area between the boundary pixel point set Po and the road facilities.
[0104] S3. Track the same road facilities in different picture frames according to the generation time of the picture frames, and divide the road infrastructure into two categories. One category of road facilities is automatic modeling objects. The valid state identifiers associated with the automatic modeling objects in each picture frame are all "True". Road facilities other than the automatic modeling objects are supplementary modeling objects. Model each automatic modeling object and supplementary modeling object to obtain a background model of the road facilities.
[0105] The modeling steps for a single automatically modeled object include the following steps:
[0106] S31. Extract the recognition object information, ancillary information and overlapping area information corresponding to the automatically modeled object in the picture frame as the associated information of the picture frame, cluster the associated information of each picture frame, and obtain the number T of picture frames contained in the largest cluster; the ancillary information includes the contour point set of the road facilities and the overlapping area with other road facilities; the overlapping area information is the overlapping area of the road facilities and traffic participants.
[0107] S32. When T≥Sum×t, the associated information of the picture frames in the largest cluster is fitted to obtain the background model of the automatically modeled object, including the contour curve, color, ancillary information and overlapping area information, and the shape and size of the automatically modeled object are determined according to the contour curve; t is the set scale coefficient; specifically, the value can be taken in the interval [0.4,0.85].
[0108] Specifically, the color of the background model is obtained by pixel fitting of the picture frame, and the contour, auxiliary information and overlapping area information of the background model are obtained by contour fitting.
[0109] It should be noted that if the background models of all road facilities in the picture frames are to be obtained, for the supplementary modeling objects and the road facilities with T < Sum×t in S32, the real-time network video stream information of the road monitoring cameras can be sampled at another time period until the automatic modeling object satisfies T≥Sum×t, or the management personnel can be informed to conduct manual modeling.
[0110] S4. Assign the regional pixel values of the background models of the corresponding road infrastructure in each picture frame to 0, then perform gray-scale processing on the picture frame as an alternative background image, and then cluster the alternative background image. Fit the pixels of all alternative background images in the largest cluster to obtain the final fitted image as the final background image;
[0111] S5. Use the SURF corner feature algorithm to detect and extract M corner feature descriptors in the background image, and delete the corner feature descriptors whose minimum distance from the road infrastructure in the background image is less than the set distance threshold; sort the remaining corner feature descriptors in descending order of confidence, and the ordered sequence of corner feature descriptors is the background model of the monitoring camera.
[0112] M is a set value, and M is greater than or equal to 10.
[0113] Refer to Figure 4 , a method for detecting and identifying road facility anomalies based on a monitoring camera proposed by this facility method, includes the following steps:
[0114] St1. After the road facilities are completed, use the above-mentioned road background modeling method based on the monitoring camera to obtain the background models of the road facilities and the monitoring camera as the reference frame background models;
[0115] It should be noted that for the supplementary modeling objects and the road facilities with T < Sum×t in S32, the real-time network video stream information of the road monitoring cameras can be sampled at another time period until the automatic modeling object satisfies T≥Sum×t, or the management personnel can be informed to conduct manual modeling to obtain the reference frame background models of all road facilities.
[0116] St2. During the monitoring process, the network video stream of the monitoring camera is obtained, and the picture frames are extracted from it as comparison frames. The identification object information, auxiliary information and overlapping area information corresponding to the road facilities in the comparison frames are extracted as the association information of the comparison frames. The association information of each comparison frame is clustered to obtain the number of comparison frames T contained in the largest cluster; the association information of the comparison frames in the largest cluster is fitted to obtain the background model of the road facilities as the comparison frame background model of the road facilities, which includes contour curves, colors, auxiliary information and overlapping area information, and the shape and size of the road facilities are determined according to the contour curves; t is the set proportional coefficient; specifically, it can take values in the interval [0.4,0.85].
[0117] St3, assigning the regional pixel value of the corresponding contrast frame background model in each contrast frame to 0, then gray-processing the contrast frame as the second candidate background image, and then clustering the second candidate background image to obtain all the second candidate background images in the largest cluster for pixel fitting, and obtaining the final fitting image as the final contrast frame background image;
[0118] St4. Use the SURF corner feature algorithm to detect and extract M corner feature descriptors in the comparison frame background image, and delete the corner feature descriptors whose minimum distance to the road foundation in the comparison frame background image is less than the set distance threshold; sort the remaining corner feature descriptors from large to small according to the confidence level, and obtain the ordered sequence of corner feature descriptors, which is the comparison frame background model of the surveillance camera; M is a set value, and M is greater than or equal to 10.
[0119] St5, calculating the rotation angle and origin offset vector of the contrast frame background model of the surveillance camera relative to the reference frame background model; adjusting the contrast frame background model of each road facility according to the rotation angle and the origin offset vector;
[0120] St6. For each road facility, the adjusted comparison frame background model is compared with the reference frame background model, the similarity between the two is calculated, and the road facility is judged to be abnormal based on the similarity. Specifically, if the similarity reaches the set safety threshold, it means that the road facility is normal; otherwise, the road facility is judged to be abnormal and the abnormality is reported.
[0121] In this embodiment, in step St6, the road facilities in the comparison frame are divided into two categories. For the road facilities whose valid state is marked as True in the comparison frame, the similarity between the comparison frame background model and the reference frame background model is directly calculated using a similarity algorithm; for the road facilities whose valid state is marked as False in the comparison frame, the similarity between the comparison frame background model and the reference frame background model is calculated using the following steps.
[0122] St61, comparing the comparison frame background model with the reference frame comparison model, screening the missing pixels of the comparison frame background model relative to the reference frame comparison model, adding the screened pixels to the comparison frame background model, and assigning values to the pixels added to the comparison frame background model according to the reference frame background model;
[0123] St62. Use a similarity algorithm to calculate the similarity between the comparison frame background model and the reference frame background model.
[0124] Of course, it is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any reference numerals in the claims should not be regarded as limiting the claims involved.
[0125] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.
[0126] The techniques, shapes, and structural parts not described in detail in the present invention are all well-known techniques.
Claims
1. A road facility occlusion detection method, characterized in that: First, the pre-trained instance segmentation model is used to extract the identification object information from the video frames collected by the road monitoring camera. The identification objects include road facilities and traffic participants. The identification object information includes: the type, shape, size, color, and coordinates of all pixels contained in the identification object; then the directory index information of the road facilities is constructed and stored in the data set D1, and the directory index information of the traffic participants is constructed and stored in the data set D2. The following steps are used to determine whether the road facilities are blocked by traffic participants: SA1, for the road facilities in the data set D1, extract their boundary information in the original image, construct the boundary pixel point set Po of the road facilities, and associate it with the directory index information; SA2, for data sets D1 and D2, obtain the minimum bounding rectangle of each identified object and associate it with the directory index information; SA3, traverse the data set D1, and select road facilities that are not blocked by traffic participants based on the overlap of the minimum circumscribed rectangle.
2. The road facility occlusion detection method according to claim 1, characterized in that: In SA3, for a single road facility P, the steps to determine whether it is blocked by traffic participants are as follows: SA31, extracting traffic participant C from data set D2 and comparing it with road facility P, and determining whether there is a risk of overlap between the minimum bounding rectangle corresponding to road facility P and the minimum bounding rectangle of traffic participant C; if not, executing step SA34; if yes, executing step SA32; SA32, let the overlapping area of the minimum circumscribed rectangle of the road facility P and the minimum circumscribed rectangle of the traffic participant C be recorded as the PC rectangle, and count the coordinate pixel points in the PC rectangle; SA33, determine whether there is a coordinate pixel point in the PC rectangle that exists both in the road facility P and in the traffic participant C; if so, determine that P and C overlap, that is, the road facility P is blocked by the traffic participant C, and add a valid state flag "False" to the road facility P; If not, execute step S34; SA34, determine whether the data set D2 has been traversed; if not, update the traffic participant C and return to step SA31; if yes, determine that the road facility P is not blocked by the traffic participant, and add a valid state flag "True" to the road facility P.
3. The road facility occlusion detection method according to claim 2, characterized in that: The method for determining whether the minimum bounding rectangles of two identified objects P1 and P2 have overlapping risks is as follows: Let the lower left corner of the minimum circumscribed rectangle of P1 be (x1 min ,y1 min ), the upper right corner is (x1 max ,y1 max ); Let the lower left corner point of the minimum circumscribed rectangle of the identified object P2 be (x2 min ,y2 min ), the upper right corner is (x2 max ,y2 max ); If x1 max <x2 min And y1 max <y2 min , or x2 max <x1 min And y2 max <y1 min , then it is judged that P1 and P2 do not overlap; If x2 min ≤x1 max And y2 min ≤y1 max , or x1 min ≤x2 max And y1 min ≤y2 max , then it is judged that there is an overlapping risk between P1 and P2.
4. The road facility occlusion detection method according to claim 3, characterized in that: After SA3, it also includes: judging one by one whether the road facilities marked as "True" in the data set D1 are blocked by the road facilities; for the road facilities Q marked as "True" in the data set D1, the judgment method is as follows: SA41, extracting road facility P from data set D1 and comparing it with road facility Q, and determining whether there is a risk of overlap between the minimum bounding rectangle corresponding to road facility Q and the minimum bounding rectangle of road facility P; if not, executing step SA45; if yes, executing step SA42; SA42, let the overlapping area of the minimum circumscribed rectangle of the road facility Q and the minimum circumscribed rectangle of the road facility P be recorded as the QP rectangle, and count the coordinate pixel points in the QP rectangle; SA43, traverse the coordinate pixel points in the QP rectangle, count and select the number of coordinate pixel points in the QP rectangle that exist in both the road facility Q and the road facility P, record it as S QP’ ; SA44. Let the number of pixels contained in the road facility Q be S Q , calculate the proportionality factor N Q =S QP’ / S Q ; Determine whether N is satisfied Q >N, where N is the set proportionality coefficient; if yes, the overlapping area between the road facility Q and the road facility P is associated with the road facility Q as the auxiliary information, and then the next road facility Q with the valid status marked as "True" is executed; if no, step SA45 is executed; SA45. Determine whether the dataset D1 has been traversed completely; if not, update the road facility P and return to step SA41; if so, determine whether the road facility Q is blocked by other road facilities.
5. The road facility occlusion detection method according to claim 1, characterized in that: In the dataset D1 and the dataset D2, the directory index information of the recognized objects is stored in a list. The directory index information includes the network video stream address, timestamp, video frame number, and the auto-increment serial number of the recognized object in the original image.
6. A road background modeling method based on a surveillance camera using the road facility occlusion detection method as claimed in claim 4, characterized in that: It includes the following steps: S1. Sample the real-time network video stream information of the road monitoring camera over a continuous time period, and record the number of picture frames obtained as Sum. S2. Extract the recognized object information of each picture frame through an instance segmentation model, and construct the corresponding datasets D1 and D2 for each picture frame; use the road facility occlusion detection method described in claim 4 to process the recognized object information of each picture frame, and determine the recognized object information, accessory information, and valid status flag of the road infrastructure included in each picture frame; the accessory information is the set of boundary pixel points Po and the overlapping area between road facilities. S3. Track the same road facility in different picture frames according to the generation time of the picture frames, and let the road facilities with the valid status flag "True" associated in each picture frame be the automatic modeling objects, and the road facilities other than the automatic modeling objects be the supplementary modeling objects; model each automatic modeling object and supplementary modeling object to obtain the background model of the road facility. S4. Assign the pixel values of the area of the background model corresponding to the road infrastructure in each picture frame to 0, then perform grayscale processing on the picture frame as the alternative background image, and then perform clustering on the alternative background image, and perform pixel fitting on all alternative background images in the largest cluster to obtain the final fitting image as the final background image. S5. Use the SURF corner feature algorithm to detect and extract M corner feature descriptors in the background image, and delete the corner feature descriptors whose minimum distance from the road infrastructure in the background image is less than the set distance threshold; sort the remaining corner feature descriptors in descending order of confidence, and the ordered sequence of corner feature descriptors is the background model of the monitoring camera.
7. The road background modeling method based on surveillance cameras as claimed in claim 6, characterized in that: The modeling steps of a single automatic modeling object include the following steps: S31. Extract the recognized object information and accessory information corresponding to the automatic modeling object in the picture frame as the associated information of the picture frame, and perform clustering on the associated information of each picture frame to obtain the number T of picture frames included in the largest clustering cluster. S32. When T ≥ Sum × t, fit the associated information of the picture frames in the largest clustering cluster to obtain the background model of the automatic modeling object, including the contour curve, color, and accessory information; t is the set proportionality coefficient.
8. The road background modeling method based on surveillance cameras as claimed in claim 7, characterized in that: The modeling method for supplementary modeling objects and road facilities where T < Sum × t in S32 is as follows: Sample the real-time network video stream information of the road monitoring camera in another selected time period until the automatic modeling object satisfies T ≥ Sum × t, and then use the road background modeling method based on the monitoring camera described in claim 6 for modeling. Or perform manual modeling.
9. A method for detecting and identifying abnormalities of road facilities using the road background modeling method based on surveillance cameras as claimed in claim 6, 7 or 8, characterized in that: It includes the following steps: After the road facilities are completed, the road background modeling method based on the surveillance camera as described in claim 6, 7 or 8 is used to obtain the background model of the road facilities and the background model of the surveillance camera as the reference frame background model; During the monitoring process, a network video stream of a monitoring camera is obtained, and a road background modeling method based on a monitoring camera as described in claim 6, 7 or 8 is used to obtain a background model of road facilities and a background model of the monitoring camera as a comparison frame background model; Calculate the rotation angle and origin offset vector of the contrast frame background model of the surveillance camera relative to the reference frame background model; adjust the contrast frame background model of each road facility according to the rotation angle and the origin offset vector; For each road facility, the adjusted comparison frame background model is compared with the reference frame background model, the similarity between the two is calculated, and whether the road facility is abnormal is determined based on the similarity.
10. The method for detecting and identifying abnormalities of road facilities based on surveillance cameras according to claim 9, characterized in that: Calculating the similarity between the comparison frame background model and the reference frame comparison model of the road facilities with the valid state marked as False in the comparison frame includes the following steps: St61, comparing the comparison frame background model with the reference frame comparison model, screening the missing pixels of the comparison frame background model relative to the reference frame comparison model, adding the screened pixels to the comparison frame background model, and assigning values to the pixels added to the comparison frame background model according to the reference frame background model; St62. Use a similarity algorithm to calculate the similarity between the comparison frame background model and the reference frame background model.
Citation Information
Patent Citations
Method for detecting shielding of road region monitoring video by leaves
CN108805042A
A bus lane occupation detection method and device
CN109711264A
Obstacle detection method and device
CN110222605A
Object detection device and object detection method
US20200097756A1
