Vehicle object throwing monitoring method and monitoring system

Abstract

The invention discloses a vehicle object throwing monitoring method, which comprises the steps of detecting object targets in a monitoring area and tracking each object target in the monitoring area, the object targets including a human body target, a vehicle target and a suspected remnant target; performing real-time analysis on the state of the monitoring area based on the tracking result; when the state is that a pedestrian or a vehicle enters and exits the monitoring area, whether the suspected remnant target is a real remnant is judged according to a tracking result; whether the behavior of throwing the remnants by the vehicle occurs or not is analyzed according to the real remnants; and when the vehicle throws the remnants, judging the vehicle and the remnants and detecting that the remnants are picked up by people. The invention further provides a monitoring system applying the vehicle object throwing monitoring method.

Description

Technical Field

[0001] The present application relates to the technical field of digital image processing, and more specifically, to a vehicle-thrown object monitoring method and monitoring system. Background Art

[0002] When vehicles pass through some customs ports, the inspection unit needs to inspect the goods of the passing vehicles and collect taxes on the relevant goods. Some customs vehicles may throw goods from the vehicle onto the road, and have pre-arranged personnel move the goods thrown on the road away, so that these vehicles can avoid inspection and tax payment. In order to prevent the above-mentioned behavior of vehicles, the inspection unit arranges staff to be on duty on the customs road to manually monitor the vehicle throwing behavior, but this method not only increases labor costs, but also requires long-term manual duty, which causes inconvenience. Summary of the invention

[0003] In view of the existing technology, the technical problem solved by the present application is to provide a vehicle object throwing monitoring method and monitoring system for automatically detecting thrown objects from vehicles on the road based on image processing. The vehicle object throwing monitoring method and monitoring system can effectively and conveniently realize unattended vehicle object throwing monitoring.

[0004] In order to solve the above technical problems, in a first aspect, the present application provides a vehicle thrown object monitoring method, which comprises:

[0005] Detecting objects in the monitoring area and tracking each object in the monitoring area, wherein the object includes a human target, a vehicle target, and a suspected object target;

[0006] Real-time analysis of the status of the monitoring area based on the tracking results;

[0007] When there are pedestrians or vehicles entering or leaving the monitoring area, determine whether the suspected leftover object is a real leftover object based on the tracking results;

[0008] Analyze the real remains to see if there is any behavior of throwing the remains from the vehicle; and,

[0009] When a vehicle throws away objects, the vehicle and object identification and the person picking up the objects are detected.

[0010] In a possible implementation, detecting an object target in a monitoring area and tracking each object target in the monitoring area includes:

[0011] A target tracking set is established for each object target in each frame image of the monitoring area, and the target tracking set includes the target tracking set Vehicle of the vehicle target u tracked to the i-th frame i,u , the target tracking set Abandon of the suspected abandoned target v tracked to the i-th framei,v and the target tracking set Person of the pedestrian target w tracked to the i-th frame i,w ;

[0012] Through the deep learning model, all objects in each frame are detected and each object is assigned a life value;

[0013] Match and track the objects of the same category in two adjacent frames: if there is an object m in the previous frame n that matches an object k in the next frame n+1, then update the object k in the target tracking set Subset n,m The target tracking set Subset corresponding to the object k in the next frame image n+1 is formed n+1,k ; If there is an object m in the previous frame image n that is not successfully matched with any object in the next frame image n+1, then the life value of the object m is updated to life=life-1 and Subset n+1,m =Subset n,m If there is an object k in the next frame image n+1 that is not successfully matched with any object in the previous frame image n, then the object k is a new object and is added to the Subset n+1,k ; Where Subset∈{Vehicle,Abandon,Person};

[0014] In each frame tracking process, it is determined whether the target tracking set corresponding to each object target is deleted according to the life value life of the object target.

[0015] In a possible implementation, judging whether the target tracking set corresponding to each object target is to be deleted according to the life value life of the object target includes:

[0016] When the life value of an object target is 0, the target tracking set corresponding to the object target is marked as a target set to be deleted;

[0017] Record the number of image frames L to which the target belongs when it is marked as the target set to be deleted. If the target with a life value of 0 fails to match successfully between the L+1th to L+Mth frames, delete the target set to be deleted.

[0018] The number of image frames L to which the target object belongs when it is marked as a target set to be deleted is recorded. When the target object with a life value of 0 successfully matches in any frame from L+1 to L+M, the target set to be deleted is restored to the state before marking and the life value is restored to the initial value.

[0019] In a possible implementation, judging whether a suspected relic target is a real relic according to the tracking result is: judging whether each suspected relic target detected at each interval T1 is a real relic, including:

[0020] Obtain the time t1 when each suspected relic target first appears;

[0021] If the time difference between the first appearance time t1 and the current time is greater than the time threshold T2, the tracking trajectory, size and background of the monitored area of ​​the suspected relic target are comprehensively considered to determine whether the suspected relic target is a real relic.

[0022] In a possible implementation, judging whether the suspected relic target is a real relic based on the tracking trajectory, size and background of the monitoring area of ​​the suspected relic target includes:

[0023] Get the target tracking set Abandon corresponding to the suspected abandoned target Y in the current frame image j j,Y ;

[0024] Root calculation target tracking set Abandon j,Y The standard deviation D of the center point coordinates of all target boxes in std ;

[0025] Calculate target tracking set Abandon j,Y The standard deviation S of the sizes of all target boxes in std ;

[0026] Calculate the time t2 that the suspected remains target stays in the monitoring area in the current image frame;

[0027] Training a binary classification recognition model for judging whether the background area of ​​the monitoring area is a real relic by taking into account the background area of ​​the monitoring area: obtaining image sets of the monitoring area at different times, intercepting multiple image blocks about the background area in each image set to form a first data set, collecting a second data set of images about the suspected relic target to be monitored, and inputting the first data set and the second data set into the binary classification model for training to obtain a binary classification recognition model;

[0028] When the following conditions (1) to (3) are met, the suspected relic target is determined to be a real relic;

[0029] (I) Standard deviation D std Less than the threshold D1 and standard deviation S std Less than threshold D2;

[0030] (II) T2<t2<T3, T3 is the set time threshold;

[0031] (III) Abandon the target tracking setj,Y In any N1 target input binary classification recognition model, there are at least N2 true recognition results in the binary classification recognition model, where N2≤N1.

[0032] In a possible implementation, the occurrence of a vehicle throwing away a leftover object is determined as follows: when a new real leftover object target appears in the current frame image j relative to the previous frame image, then a vehicle throwing away a leftover object behavior occurs.

[0033] In a possible implementation, when the current frame image j shows a behavior of a vehicle throwing away a leftover object, a vehicle-object determination is performed, including:

[0034] The target tracking set Abandon corresponding to the newly added suspected abandoned target TrueAban when obtaining the current frame image j j,TrueAban ;

[0035] Get the target tracking set of all vehicle targets in the current frame image j;

[0036] Calculate target tracking set Abandon j,TrueAban The target tracking set Vehicle for each target box center point and any vehicle target u j,u The Euclidean distance of the center point of each target frame in the image is used to calculate the minimum value of all Euclidean distance values ​​as the distance D from the vehicle target u to the true suspected relic target TrueAban. u,TrueAban ;

[0037] At all distances D u,TrueAban The vehicle target corresponding to the minimum value is the vehicle Throw that throws the real suspected relic target TrueAban;

[0038] The target tracking set corresponding to the real suspected abandoned target TrueAban and the vehicle throwing the real suspected abandoned target TrueAban is stored in the background database in the form of combination Z.

[0039] In a possible implementation, the vehicle-object determination further includes:

[0040] Using the bayonet camera on the exit side of the monitoring area to photograph the vehicles coming out of the monitoring area;

[0041] Identify license plates of vehicles photographed by bayonet cameras;

[0042] The vehicle that throws the real suspected relic target TrueAban is matched with the vehicle photographed by the bayonet camera for multi-dimensional features. When the match is successful, the vehicle that throws the real suspected relic target TrueAban is added to the combination Z.

[0043] In a possible implementation, when a vehicle throws away a leftover object in the current frame image j, detection of a person picking up the leftover object is performed, including:

[0044] When the current frame image j is obtained, the target tracking set Abandon corresponding to the real suspected abandoned target TrueAban j,TrueAban ;

[0045] Get the target tracking set of all pedestrian targets in the current frame image j;

[0046] The target frame corresponding to the real suspected relic target TrueAban and the target frame corresponding to the pedestrian target w in the same frame image are matched for overlap;

[0047] If there are at least T8 successful matches, the pedestrian target w picks up the true suspected abandoned object target TrueAban and adds the pedestrian target w to the combination Z.

[0048] In a second aspect, the present application provides a vehicle object throwing monitoring system, which includes a processor, a memory and a camera; the camera is used to capture a video image of a monitoring area; the memory is used to store an executable code; the processor is used to execute the executable code stored in the memory to perform the steps of the vehicle object throwing monitoring method.

[0049] The beneficial effects of the vehicle object throwing monitoring method and monitoring system provided by the embodiments of the present application are: detecting and tracking pedestrians, vehicles and suspected objects of interest in the monitoring area, analyzing the status of the monitoring area in real time based on the tracking results, and analyzing whether the objects of interest are genuine if pedestrians or vehicles enter or leave the monitoring area. For suspected objects of interest actually thrown by vehicles, vehicle and object determination and human-picked object detection are performed, thereby achieving the determination of people, vehicles and objects. The vehicle object throwing monitoring method and monitoring system realize the function of unattended automatic object throwing detection by means of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 This is a flow chart of the steps of the vehicle object throwing monitoring method according to an embodiment of the present application.

[0052] Figure 2 This is a schematic diagram of a large truck throwing away leftovers in an embodiment of the present application.

[0053] Figure 3 This is a schematic diagram of the result of a pedestrian picking up leftovers in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0056] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0057] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0058] The vehicle object throwing monitoring method and monitoring system of the present application are now described in detail with reference to the accompanying drawings.

[0059] Please refer to Figure 1 The vehicle object throwing monitoring method provided in the embodiment of the present application includes the following steps S100 to S500:

[0060] Step S100: Detecting objects in the monitoring area and tracking each object in the monitoring area, wherein the object includes a human target, a vehicle target, and a suspected object target;

[0061] Step S200: Perform real-time analysis on the status of the monitoring area based on the tracking results; when the status indicates that a pedestrian or vehicle enters or leaves the monitoring area, execute step S300, otherwise return to execute step S100;

[0062] Step S300: removing interference targets that are not thrown by vehicles from the suspected leftover targets according to the tracking results to obtain the real leftover targets;

[0063] Step S400: Analyze whether there is any behavior of throwing away the leftovers by the vehicle according to the real leftovers;

[0064] Step S500: When a vehicle throws away objects, the vehicle objects are bound and a person picking up the objects is detected.

[0065] In the above-mentioned vehicle-thrown object monitoring method, pedestrians, vehicles and suspected abandoned objects that require attention in the monitoring area are detected and tracked, and the status of the monitoring area is analyzed in real time based on the tracking results. When pedestrians or vehicles enter or leave the monitoring area, the authenticity of the suspected abandoned objects is analyzed. For the abandoned objects that are determined to be truly thrown out by the vehicle, vehicle-object judgment and human-picked abandoned objects detection are performed, thereby realizing the binding of people, vehicles and objects.

[0066] It is worth noting that cameras are deployed around the monitoring area, and the cameras are used to continuously shoot videos of the monitoring area. The above steps S100 to S500 are performed for each frame of the video. In each frame, it is necessary to determine whether there are real relics, whether there is a behavior of vehicles throwing relics, and when it is determined that there is a behavior of vehicles throwing relics, vehicle binding and human picking up relics detection are performed. Among them, the aforementioned real relics can be understood as items thrown from the vehicle while the vehicle is driving.

[0067] In one embodiment, for step S100, detecting objects in a monitoring area and tracking each object in the monitoring area includes the following steps S110 to S140.

[0068] Step S110: Establish a target tracking set for each object target in each frame image of the monitoring area, and the target tracking set includes the target tracking set Vehicle of the vehicle target u tracked to the i-th frame i,u , the target tracking set Abandon of the suspected abandoned target v tracked to the i-th frame i,v and the target tracking set Person of the pedestrian target w tracked to the i-th frame i,w .

[0069] Step S120: Detect all object targets in each frame image through the deep learning model, and assign a life value life to each object target.

[0070] Wherein, the deep learning model can be but not limited to the yolox model, specifically, the pedestrian data, vehicle data and legacy data are collected respectively to train the yolox model, and the yolox model that can identify pedestrians, the yolox model that can identify vehicles and the yolox model that can identify suspected legacy targets are obtained respectively. It is worth noting that what the embodiment of the present application really needs to pay attention to and monitor is the real legacy left behind by throwing from the vehicle, wherein, the vehicle throwing is not limited to the various goods that need to be checked and taxed by the inspection unit such as packages and luggage bags, and the yolox model that can identify suspected legacy targets is trained, and the articles that need to be checked that appear in real life and are specified by the inspection unit are collected to carry out model training, and the collected articles are as diverse as possible. However, since there may be objects that are relatively close to the shapes such as packages and luggage bags in the outdoor background area, the aforementioned relatively close objects can be understood as a part of the interference target in step S300, so it may be possible to: the legacy detected by the yolox model is not entirely the articles thrown by the vehicle during driving that the present application really needs to pay attention to and monitor. Therefore, in step S100, the remnants detected by the yolox model are temporarily referred to as suspected remnant targets, and are marked as true remnant targets only after the operation of the subsequent step S300.

[0071] It can be understood that when the deep learning model is used to detect object targets in each frame of image, the category and target frame information of each object target can be obtained. The target frame information includes the coordinates of the center point of the target frame, the height and width of the target frame, and the coordinates of the upper left corner point of the target frame. The coordinates of the center point of the target frame correspond to the coordinate position of the object target in the image, and the categories are pedestrians, vehicles, and suspected relics.

[0072] Step S130: Match and track the objects of the same category in two adjacent frames: if an object m in the previous frame n successfully matches an object k in the next frame n+1, then update the object k in the target tracking set Subset n,m The target tracking set Subset corresponding to the object k in the next frame image n+1 is formed n+1,k ; If there is an object m in the previous frame image n that is not successfully matched with any object in the next frame image n+1, then the life value of the object m is updated to life=life-1 and Subset n+1,m =Subset n,m ; If there is an object k in the next frame image n+1 that is not successfully matched with any object in the previous frame image n, then the object k is a new object and the Subset is formed by the object k n+1,k ; where Subset∈{Vehicle,Abandon,Person}.

[0073] It is worth noting that when two adjacent frames of images are tracked and matched, targets of the same category are matched, which can reduce unnecessary matching processes. n represents any frame image, n+1 is the next frame image of image n, and m is any object target in any frame image n. k is any object target in the next frame image n+1, and n can be a digital label, for example, 1, 2, 3... represent the 1st, 2nd, 3rd... frame images respectively. m and k can be identification symbols composed of numbers and letters of object targets, for example: P1, P2, P3... represent the 1st, 2nd, 3rd... pedestrian targets respectively, V1, V2, V3... represent the 1st, 2nd, 3rd... vehicle targets respectively, and A1, A2, A3... represent the 1st, 2nd, 3rd... suspected relics targets respectively. It can be understood that the same identification symbol represents the same object target. Subset n,m Subset represents the target tracking set corresponding to the object m in the nth frame image, n+1,m 、Subset n+1,k And so on, they are the target tracking set corresponding to the object target m in the n+1th frame image and the target tracking set corresponding to the object target k in the n+1th frame image.

[0074] Taking the object target of pedestrian category as an example, the updating process of the target tracking set corresponding to each target in the matching tracking process is explained:

[0075] When a pedestrian target P1 is detected in the first frame image of the monitoring area, a target tracking set Subset needs to be established for the pedestrian target P1. 1,P1 , and assign an initial life value to the pedestrian target P1, for example, life = 5, Subset 1,P1 It stores the information of the target box box1 of the pedestrian target P1 in the first frame, the target category, and the frame from which box1 comes. For the second frame image, continue to perform target detection to obtain all pedestrian targets in the second frame image, match each pedestrian target in the first frame image with each pedestrian target, and assume that a pedestrian target C is detected in the second frame image; if there is a pedestrian target P1 and a pedestrian target C that match successfully, then the identification symbol of the pedestrian target C in the second frame image is also P1, which means that the pedestrian target P1 and the pedestrian target C are the same pedestrian target. As mentioned above, a target tracking set is established for each object target in each frame image. For the second frame, when the pedestrian target P1 is tracked to the second frame, the corresponding target tracking set is Subset 2,P1 , Subset 2,P1The target box information box1 of the pedestrian target P1 in the first frame image, the target box box2 information in the second frame image, the frame number of box1, the frame number of box2, and the category of the pedestrian target P1 are stored in the target box box2 information in the second frame image, and the target box box2 information in the second frame image is the target box information of the pedestrian target C detected in the second frame image; if the pedestrian target P1 in the first frame does not successfully match any pedestrian target in the second frame image, then the corresponding target tracking set when the pedestrian target P1 is tracked to the second frame is Subset 2,P1 =Subset 1,P1 , which means: in the second frame, the pedestrian target P1 is tracked and recognized once, then the life value of the pedestrian target P1 is updated to life-1, at this time life-1=4, the set Subset 2,P1 Relative to the set Subset 1,P1 No new elements are updated; if the pedestrian target C detected from the second frame image does not match any pedestrian target in the first frame image, then the pedestrian target C is a new pedestrian target in the monitoring area. If there are 5 pedestrian targets in the first frame, then the pedestrian target C can be marked as P6. Then the target tracking set corresponding to the pedestrian target C when tracking to the second frame is Subset 2,P6 , Subset 2,P6 The target frame information of pedestrian target C in the second frame is stored, and the pedestrian target P6 is assigned an initial life value. In this way, pedestrian target detection and matching tracking are continuously performed in the third frame and subsequent frames, and the target tracking set of the same pedestrian target is updated in the same way as the initial frame and the second frame image, and a new target tracking set is established for the newly appeared pedestrian target. The above content takes pedestrian targets as an example, and the same is true for vehicle targets and suspected relics targets, which will not be repeated here.

[0076] In a specific embodiment, for two adjacent frames of images, the matching method for two targets of the same category is: the intersection-over-union (IOU) of the target frames of the two targets is calculated. If the value of the IOU is greater than a set threshold, it indicates that the match is successful, otherwise the match is unsuccessful. The threshold value set here can be 0.5.

[0077] Step S140: In each frame tracking process, judging whether the target tracking set corresponding to each target object is to be deleted according to the life value life of the target object, specifically:

[0078] When the life value of an object target is 0, the target tracking set corresponding to the object target is marked as a target set to be deleted;

[0079] Record the number of image frames L to which the target belongs when it is marked as the target set to be deleted. If the target with a life value of 0 fails to match successfully between the L+1th to L+Mth frames, delete the target set to be deleted.

[0080] The number of image frames L to which the target object belongs when it is marked as a target set to be deleted is recorded. When the target object with a life value of 0 successfully matches in any frame from L+1 to L+M, the target set to be deleted is restored to the state before marking and the life value is restored to the initial value.

[0081] It is worth noting that, on the one hand, the life value of an object target reflects whether the object target is successfully matched during the matching and tracking process. If the object target maintains the initially set life value during the matching and tracking process, it indicates that the object target is successfully matched and tracked. If the life value of the object target decreases by one during the matching and tracking process, it indicates that the object target fails to match and track once. As long as the life value of the object target is not 0, the target tracking set of the object target still maintains its original state. On the other hand, when the target tracking set of the object target is marked as a target set to be deleted, the target tracking set of the object target still maintains a normal use state; and even if the life value life becomes 0, the object target suddenly matches successfully again in the subsequent M frames, the target set to be deleted is restored to the state before marking and the life value is restored to the initial value. After the aforementioned matching tracking fails once, the target-related collection is not cleared immediately, because there may be occlusions between different targets in the monitoring area or other factors that cause accidental tracking failure of the target. At this time, the target has not really disappeared in the monitoring area, and the life value planning also provides each object with a fault-tolerant mechanism for occasional tracking failures caused by other influencing factors. Only when the target appears life+M times in a row will it be determined that the target has really disappeared in the monitoring area and then cleared.

[0082] It is also worth mentioning that when the life value life just becomes 0, the target tracking set of the object target is still maintained in normal use. Only when the life value life reaches 0 and continues to be 0 for M frames, the target tracking set of the object target will be cleared. This is because it indicates that the tracking of the object target has failed for a long time. At this time, there is no need to continue to maintain the target tracking set of the object target, so as to avoid continuing to match and track the object target, which will cause unnecessary calculations and waste time, and unnecessary sets occupying memory.

[0083] In a specific embodiment, the initial value of the life value life is 5, and the value of M can be 5, which means that when there are ten consecutive matching failures, the corresponding target tracking set will be cleared.

[0084] In one embodiment, for step S200, the status of the monitoring area is analyzed in real time based on the tracking result:

[0085] In the previous frame image and the current frame image, the target tracking set corresponding to the pedestrian target and the target tracking set corresponding to the vehicle target are both empty, and the current state of the monitoring area is marked as 0, which means that there is no car and no person in the monitoring area, so there is no need to analyze the vehicle-thrown objects in the current frame;

[0086] In the current frame image, the target tracking set corresponding to the pedestrian target and the target tracking set corresponding to the vehicle target are both empty. In the previous frame image, the target tracking set corresponding to the pedestrian target or the target tracking set corresponding to the vehicle target is not empty, then the current state of the monitoring area is marked as 1, indicating that there are pedestrians or vehicles passing through the monitoring area;

[0087] In the current frame image, if the target tracking set corresponding to the pedestrian target or the target tracking set corresponding to the vehicle target is not empty, the current state of the monitoring area is marked as 2, indicating that there is a person or a vehicle in the monitoring area.

[0088] Understandably, status 1 and 2 indicate that there are people or vehicles entering or exiting the monitoring area, and only then are the operations of steps S300 to S500 performed. When the status is 0 and there are no vehicles or people entering or exiting the monitoring area, there is no need to perform a real debris analysis, a vehicle throwing debris behavior analysis, a vehicle-object judgment, or a person picking up debris detection, so as to avoid unnecessary algorithm running time and unnecessary waste of memory resources.

[0089] In one embodiment, for step S300, determining whether each suspected relic target is a real relic is determining whether each suspected relic target is a real relic at each interval T1, including:

[0090] Obtain the time t1 when each suspected relic target first appears;

[0091] If the time difference between the current time and the first appearance time t1 is greater than the time threshold T2, the tracking trajectory, size and background of the monitored area of ​​the suspected relic target are comprehensively considered to determine whether the suspected relic target is a real relic.

[0092] It can be understood that in addition to storing the information of each target frame of the object target and the frame from which each target frame comes, the time corresponding to the frame can also be recorded in the target tracking set corresponding to each object target. Then, the time when each suspected relic target first appears can also be known. The time when each suspected relic target first appears is the time when the target frame of the relic is first detected.

[0093] Further, assuming that the current frame image is image j, the tracking trajectory, size and background of the monitored area of ​​the suspected relic target are comprehensively considered to determine whether the suspected relic target is a real relic, including the following steps S210 to S260.

[0094] Step S310: Obtain the target tracking set Abandon corresponding to the suspected abandoned object Y in the current frame image j j,Y .

[0095] Step S320: Root calculation target tracking set Abandon j,Y The standard deviation D of the center point coordinates of all target boxes in std .

[0096] Among them, the standard deviation D is calculated std The process is:

[0097] Calculate target tracking set Abandon j,Y The mean of the horizontal coordinates of all target boxes in

[0098] Calculate target tracking set Abandon j,Y The mean of the ordinates of all target boxes in

[0099] Standard Deviation D std for:

[0100] In the above formula, M represents the current frame j set Abandon j,Y The total number of all target boxes about the suspected relic target Y in x N Indicates the horizontal coordinate of the center point of the Nth target box, y N represents the ordinate of the center point of the Nth target box, and Σ represents the sum.

[0101] Step S330: Calculate target tracking set Abandon j,Y The standard deviation S of the sizes of all target boxes in std .

[0102] Among them, the standard deviation S is calculated std The process is:

[0103] Calculate target tracking set Abandon j,Y The mean width of all target boxes in

[0104] Calculate target tracking set Abandon j,Y The mean height of all target boxes in

[0105] Standard Deviation std for:

[0106] In the above formula, M represents the current frame j set Abandon j,Y The total number of all target boxes about the suspected relic target Y, w N Indicates the width of the Nth target box, y N Indicates the height of the Nth target box.

[0107] Step S340: Calculate the time t2 that the suspected remains target stays in the monitoring area in the current image frame:

[0108] Time t2 = the time corresponding to the current frame - the time when the suspected relic target Y first appears.

[0109] Step S350: training a binary classification recognition model for judging whether the background area of ​​the monitoring area is a real relic: obtaining image sets of the monitoring area at different times, intercepting multiple image blocks about the background area in each image set to form a first data set, collecting a second data set of images of suspected relic targets to be monitored, and inputting the first data set and the second data set into a binary classification model for training to obtain a binary classification recognition model. The binary classification recognition model is based on the efficientnet-b1 network model. It can be understood that the output result of the binary classification recognition model is either true or false, wherein the image blocks of the background area constituting the first data set can be images of fallen leaves and garbage on the ground that appear outdoors.

[0110] Step S360: When the following conditions (1) to (3) are met, the suspected relic target is determined to be a real relic;

[0111] (I) Standard deviation D std Less than the threshold D1 and standard deviation S std Less than threshold D2;

[0112] (II) T2<t2<T3, T3 is the set time threshold;

[0113] (III) Abandon the target tracking set j,Y In any N1 target input binary classification recognition model, there are at least N2 true recognition results in the binary classification recognition model, where N2≤N1.

[0114] In the above step S300, it is determined whether the remains obtained by target detection through the deep learning model in step S100 are real remains in each interval T1, that is, the remains detected and matched and tracked in step S100 are only tentative remains, and whether they are the remains thrown by the vehicle as recognized by the embodiment of the present application needs to be further determined. Since the camera is constantly collecting video of the monitored area, the determination of whether the remains obtained by target detection in step S100 are real remains is also carried out over time.

[0115] In the above step S300, firstly, the judgment condition of the standard deviation can exclude the false detection caused by the target detection and target matching tracking process. Secondly, since the real suspected relic target concerned by the embodiment of the present application is: the target thrown by the vehicle that should not exist in the monitoring area at the beginning, in order to exclude the false detection caused by environmental background elements such as pits, shadows and external light on the ground that may be similar to the appearance of the relic, the above-mentioned residence time t2 is designed; specifically, for the relic that has been in the monitoring area for a long time with a residence time t2 greater than T3, it can be identified as a background element or an object that stays in the background for a long time. For the relic that appears too short with a residence time t2 less than T2, it can be identified as a temporary error, such as a relic that temporarily flies into the monitoring area and is close to the appearance of the relic, and then flies out of the monitoring area. Furthermore, the outdoor scene is more complex, and the binary classification recognition model excludes other targets that are similar to the remains that appear outdoors. For example, the pits in some backgrounds or the interference targets that are not thrown by vehicles may also meet the above conditions (I) and (II). At this time, condition (III) can be used to exclude them. Understandably, when the judgment of step S200 is performed, the target of the initial detection of the remains in the middle of step S100 can be considered as the suspected remains target detected by step S100 as a suspicious suspected remains target, and its authenticity is subject to further judgment of step S200.

[0116] In one embodiment, for step S400, it is determined that a vehicle throwing away a relic occurs: if a new real relic target appears in the current frame image j relative to the previous frame image, then a vehicle throwing away a relic occurs.

[0117] It is worth noting that the determination of the vehicle throwing away the remains in step S400 is based on the presence of the real remains target in step S300. Figure 2The large truck shown, when the object is thrown from this type of vehicle, the thrown object is the aforementioned so-called relic. The object thrown is easily blocked by the large vehicle at the beginning. If the vehicle is used as a reference, the parabolic trajectory near the vehicle is detected to detect whether the parabolic behavior occurs in the monitoring area. It is obviously not easy to succeed, because the object that may be thrown may be blocked by the vehicle for a period of time, and will only be revealed as the vehicle moves, and the parabolic trajectory of the object is difficult to detect. In view of this, the embodiment of the present application is based on a real scene, and the scheme conceived is: compare the real relic target in the previous frame image and the current frame image. If after comparison, a real relic target that has not appeared before is added to the current frame image, it indicates that there is a vehicle throwing relic behavior in the current frame image. It is understandable that if multiple real relic targets are added in the current frame compared to the previous frame, then there are multiple vehicles throwing relic behaviors. As for whether the suspected relic target thrown is from the same vehicle or different vehicles, it will be further determined later, and its determination process is realized by the vehicle object judgment in step S500.

[0118] In one embodiment of the application, for step S500, when the current frame image j shows a vehicle throwing away a leftover object, a vehicle-object determination is performed, including:

[0119] Step S511: Obtain the target tracking set Abandon corresponding to the real abandoned target TrueAban newly added in the current frame image j j,TrueAban ;

[0120] Step S512: Acquire a target tracking set of all vehicle targets in the current frame image j;

[0121] Step S513: Calculate target tracking set Abandon j,TrueAban The target tracking set Vehicle for each target box center point and any vehicle target u j,u The Euclidean distance of the center point of each target frame in the image is used to calculate the minimum value of all Euclidean distances as the distance D from the vehicle target u to the true legacy target TrueAban. u,TrueAban ;

[0122] Step S514: Take all distances D u,TrueAban The vehicle target corresponding to the minimum value is the vehicle that throws the true relic target TrueAban.

[0123] Step S515: The target tracking set corresponding to the real abandoned object target TrueAban and the vehicle that threw the real abandoned object target TrueAban is stored in the background database in the form of combination Z.

[0124] Furthermore, when determining whether a vehicle is an object, the following steps are also included:

[0125] Using the bayonet camera on the exit side of the monitoring area to photograph the vehicles coming out of the monitoring area;

[0126] Identify the license plate of every vehicle photographed by the bayonet camera;

[0127] The vehicle that threw the real relic target TrueAban is matched with the vehicle photographed by the bayonet camera for multi-dimensional features, and the vehicle that threw the real relic target TrueAban is added to the combination Z when the match is successful.

[0128] Among them, the oncoming vehicle refers to the vehicle driving towards the bayonet camera, that is, the bayonet camera can capture the front of the oncoming vehicle, and the license plate is recognized by any existing license plate recognition method. Multidimensional feature matching is: the formula for calculating the similarity sim between multidimensional features is: sim=α*F1+β*F2+θ*F3, wherein F1, F2 and F3 are shape feature similarity, color feature similarity, and deep learning feature similarity respectively, and F1 is the similarity between the shape feature of a vehicle throwing the real relic target TrueAban and the shape feature of a vehicle photographed by the bayonet camera, F2 is the similarity between the color feature of a vehicle throwing the real relic target TrueAban and the color feature of a vehicle photographed by the bayonet camera, F3 is the similarity between the deep learning feature of a vehicle throwing the real relic target TrueAban and the deep learning feature of a vehicle photographed by the bayonet camera, wherein the solution of the similarity of F1, F2 and F3 can adopt but is not limited to cosine similarity, and the license plate information of the real relic target TrueAban can be obtained through the above step S500.

[0129] In an embodiment of the application, for step S500, when the current frame image j shows a vehicle throwing away a leftover object, a person picking up the leftover object detection is performed, including:

[0130] When the current frame image j is obtained, the target tracking set Abandon corresponding to the true abandoned target TrueAban j,TrueAban ;

[0131] Get the target tracking set of all pedestrian targets in the current frame image j;

[0132] Match the target frame corresponding to the true legacy target TrueAban and the target frame corresponding to the pedestrian target w in the same frame image;

[0133] If there are at least N3 successful matches, the pedestrian target w picks up the true relic target TrueAban and adds the pedestrian target w to the combination Z.

[0134] The overlap matching is as follows: the intersection-over-union ratio of the target frame corresponding to the real legacy target TrueAban and the target frame corresponding to the pedestrian target w is calculated. When the intersection-over-union ratio is greater than the set threshold, the overlap matching is successful, otherwise the matching is unsuccessful; the threshold here can be set to 0.5. Figure 3 Schematic diagram of the results of picking up leftovers for people.

[0135] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for monitoring objects thrown from a vehicle, characterized in that: include: Detecting objects in the monitoring area and tracking each object in the monitoring area, wherein the object includes a human target, a vehicle target, and a suspected object target; Real-time analysis of the status of the monitoring area based on the tracking results; When there are pedestrians or vehicles entering or leaving the monitoring area, the interference targets that are not thrown by vehicles are removed from the suspected leftover targets according to the tracking results to obtain the real leftover targets; Analyze the real remains to see if there is any behavior of throwing the remains; as well as When a vehicle throws away objects, the vehicle and object identification and the person picking up the objects are detected.

2. The vehicle object throwing monitoring method according to claim 1, characterized in that: Detecting objects in the monitoring area and tracking each object in the monitoring area, including: A target tracking set is established for each object target in each frame image of the monitoring area, and the target tracking set includes the target tracking set Vehicle of the vehicle target u tracked to the i-th frame i,u , the target tracking set Abandon of the suspected abandoned target v tracked to the i-th frame i,v and the target tracking set Person of the pedestrian target w tracked to the i-th frame i,w ; Through the deep learning model, all objects in each frame are detected and each object is assigned a life value; Match and track the objects of the same category in two adjacent frames: if there is an object m in the previous frame n that matches an object k in the next frame n+1, then update the object k in the target tracking set Subset n,m The target tracking set Subset corresponding to the object k in the next frame image n+1 is formed n+1,k ; If there is an object m in the previous frame image n that is not successfully matched with any object in the next frame image n+1, then the life value of the object m is updated to life=life-1 and Subset n+1,m =Subset n,m If there is an object k in the next frame image n+1 that is not successfully matched with any object in the previous frame image n, then the object k is a new object and is added to the Subset n+1,k ; Where Subset∈{Vehicle,Abandon,Person}; In each frame tracking process, it is determined whether the target tracking set corresponding to each object target is deleted according to the life value life of the object target.

3. The vehicle object throwing monitoring method according to claim 2, characterized in that: According to the life value of each object target, it is determined whether the target tracking set corresponding to the object target is deleted, including: When the life value of an object target is 0, the target tracking set corresponding to the object target is marked as a target set to be deleted; Record the number of image frames L to which the target belongs when it is marked as the target set to be deleted. If the target with a life value of 0 fails to match successfully between the L+1th to L+Mth frames, delete the target set to be deleted. The number of image frames L to which the target object belongs when it is marked as a target set to be deleted is recorded. When the target object with a life value of 0 successfully matches in any frame from L+1 to L+M, the target set to be deleted is restored to the state before marking and the life value is restored to the initial value.

4. The vehicle-throwing monitoring method according to claim 2, characterized in that: The suspected relic target removes the interference targets that are not thrown by the vehicle from the suspected relic target according to the tracking results, and obtains the real relic target: whether each suspected relic target detected at each interval T1 is a real relic, including: Obtain the time t1 when each suspected relic target first appears; If the time difference between the first appearance time t1 and the current time is greater than the time threshold T2, the tracking trajectory, size and background of the monitored area of ​​the suspected relic target are comprehensively considered to determine whether the suspected relic target is a real relic.

5. The vehicle-throwing object monitoring method according to claim 4, characterized in that: The suspected relic target is judged as a real relic based on its tracking trajectory, size and the background of the monitoring area, including: Get the target tracking set Abandon corresponding to the suspected abandoned target Y in the current frame image j j,Y ; Root calculation target tracking set Abandon j,Y The standard deviation D of the center point coordinates of all target boxes in std ; Calculate target tracking set Abandon j,Y The standard deviation S of the sizes of all target boxes in std ; Calculate the time t2 that the suspected remains target stays in the monitoring area in the current image frame; Training a binary classification recognition model for judging whether the background area of ​​the monitoring area is a real relic by taking into account the background area of ​​the monitoring area: obtaining image sets of the monitoring area at different times, intercepting multiple image blocks about the background area in each image set to form a first data set, collecting a second data set of images about the suspected relic target to be monitored, and inputting the first data set and the second data set into the binary classification model for training to obtain a binary classification recognition model; When the following conditions (I) to (III) are met, the suspected relic target is determined to be a real relic; (I) Standard deviation D std Less than the threshold D1 and standard deviation S std Less than threshold D2; (II) T2<t2<T3, T3 is the set time threshold; (III) Abandon the target tracking set j,Y In any N1 target input binary classification recognition model, there are at least N2 true recognition results in the binary classification recognition model, where N2≤N1.

6. The vehicle-throwing object monitoring method according to claim 1, characterized in that: Determine whether a vehicle has thrown away objects: if a new real object appears in the current frame image j relative to the previous frame image, then a vehicle has thrown away objects.

7. The vehicle-throwing object monitoring method according to claim 6, characterized in that: When the current frame image j shows a vehicle throwing away objects, the vehicle-object judgment is performed, including: Get the target tracking set Abandon corresponding to the newly added true abandoned target TrueAban when the current frame image j is obtained j,TrueAban ; Get the target tracking set of all vehicle targets in the current frame image j; Calculate target tracking set Abandon j,TrueAban The target tracking set Vehicle for each target box center point and any vehicle target u j,u The Euclidean distance of the center point of each target frame in the image is used to calculate the minimum value of all Euclidean distances as the distance D from the vehicle target u to the true legacy target TrueAban. u,TrueAban ; At all distances D u,TrueAban The vehicle target corresponding to the minimum value is the suspected vehicle target that threw the real relic target TrueAban; The target tracking set corresponding to the real abandoned object target TrueAban and the vehicle throwing the real abandoned object target TrueAban is stored in the background database in the form of combination Z.

8. The vehicle-throwing object monitoring method according to claim 7, characterized in that: The vehicle and property judgment also includes: Using the bayonet camera on the exit side of the monitoring area to photograph the vehicles coming out of the monitoring area; Identify license plates of vehicles photographed by bayonet cameras; The vehicle that threw the real relic target TrueAban is matched with the vehicle photographed by the bayonet camera for multi-dimensional features, and the vehicle that threw the real relic target TrueAban is added to the combination Z when the match is successful.

9. The vehicle-throwing object monitoring method according to claim 7, characterized in that: When the current frame image j shows a vehicle throwing away objects, detection of a person picking up objects is performed, including: When the current frame image j is obtained, the target tracking set Abandon corresponding to the true abandoned target TrueAban j,TrueAban ; Get the target tracking set of all pedestrian targets in the current frame image j; Match the target frame corresponding to the true legacy target TrueAban and the target frame corresponding to the pedestrian target w in the same frame image; If there are at least N 38 If the match is successful, the pedestrian target w picks up the true relic target TrueAban and adds the pedestrian target w to the combination Z.

10. A vehicle object throwing monitoring system, characterized in that: It includes a processor, a memory and a camera; the camera is used to capture video images of the monitored area; the memory is used to store executable codes; The processor is used to execute the executable code stored in the memory to perform the steps of the vehicle object throwing monitoring method as described in any one of claims 1 to 9.

Images