A method and apparatus for detecting a litter

Abstract

The application discloses a method and device for detecting a thrown object, which are used for detecting the thrown object through a motion detection model and a deep learning thrown object target detection model, and improving the accuracy of the detection of the thrown object. The method provided by the application comprises the following steps: acquiring a collection image obtained for a target shooting scene, and determining a region of interest in the collection image; performing motion target detection on the region of interest through a motion detection model, and obtaining at least one motion target image; performing thrown object detection on the region of interest and the at least one motion target image through a deep learning thrown object target detection model, and determining a thrown object detection result corresponding to the collection image.

Description

Technical Field

[0001] This application relates to the field of intelligent transportation, and in particular to a method and apparatus for detecting spilled materials. Background Technology

[0002] Traffic debris detection is a key and challenging area in the field of intelligent transportation, as it not only makes roads dirty and messy but can also endanger drivers' lives in severe cases. In recent years, incidents of personal safety being threatened by traffic debris have occurred frequently, prompting traffic management bureaus across the country to place increasing emphasis on controlling this behavior. Traffic debris can be detected using bullet cameras and PTZ cameras, typically employing target detection methods. However, due to the diverse types of debris and numerous environmental interferences, the recall and accuracy rates of detection are relatively poor. Summary of the Invention

[0003] This application provides a method and apparatus for detecting projectiles, which uses a motion detection model and a deep learning projectile target detection model to detect projectiles and improve the accuracy of projectile detection.

[0004] This application provides a method for detecting debris, the method comprising:

[0005] Acquire images of the target shooting scene and determine the region of interest in the acquired images;

[0006] The region of interest is used to detect moving targets using a motion detection model to obtain at least one image of a moving target.

[0007] By using a deep learning object detection model to detect objects in the region of interest and the image of at least one moving target, the object detection result corresponding to the acquired image is determined.

[0008] This method acquires images of a target shooting scene and determines the region of interest (ROI) within the acquired images. A motion detection model is used to detect moving targets within the ROI, resulting in at least one moving target image. A deep learning object detection model is then used to detect objects within the ROI and the at least one moving target image, determining the object detection result corresponding to the acquired image. This method utilizes both the motion detection model and the deep learning object detection model to improve the accuracy of object detection.

[0009] In some embodiments, the step of detecting moving targets in the region of interest using a motion detection model to obtain at least one moving target image includes:

[0010] Based on the region of interest, the contour of each moving target is determined, and a motion detection box is drawn on the contour to obtain the motion detection box of each moving target; wherein, the image in the motion detection box is the image of the moving target;

[0011] Based on the preset motion detection bounding box for non-scattered objects, the motion detection bounding box of each moving target is filtered to obtain at least one moving target image.

[0012] This method enables the drawing of motion detection bounding boxes for moving targets using a motion detection model, while filtering out motion detection bounding boxes for non-projectile objects, thereby improving detection accuracy.

[0013] In some embodiments, filtering the motion detection frame of each moving target based on a preset non-scattered object motion detection frame to obtain at least one moving target image includes:

[0014] For each motion detection frame, a cross-over and union ratio (CUP) is performed with a preset non-projectile motion detection frame. When the CUP is greater than a preset threshold, the motion detection frame is filtered out.

[0015] For each remaining motion detection box, the motion detection box is expanded based on the width and height of the original acquired image, and the image of the moving target is extracted from the expanded motion detection box to obtain the moving target image.

[0016] This method filters motion detection boxes and extracts images from the remaining motion detection boxes, thereby improving the accuracy of subsequent detection.

[0017] In some embodiments, the step of performing object detection on the region of interest and the at least one moving target image using a deep learning object detection model, and determining the object detection result corresponding to the acquired image, includes:

[0018] By performing object detection on the region of interest, it can be determined whether an image of an object is present in the region of interest;

[0019] If so, the at least one moving target image is filtered based on the object image, and the remaining moving target image and the object image are used to obtain the object detection result corresponding to the acquired image;

[0020] Otherwise, the object detection result corresponding to the acquired image is obtained using the at least one moving target image.

[0021] This method enables the filtering of moving target images based on the detection results of regions of interest by a deep learning projectile target detection model.

[0022] In some embodiments, the method further includes the step of pre-training the deep learning projectile target detection model as follows:

[0023] Calculate the total loss function value of a preset number of sample images; wherein, the total loss function value is the sum of the preset category loss function value and the contrastive encoding prior loss function value;

[0024] Based on the total loss function value, adjust the weights of each layer of the neural network in the deep learning projectile target detection model.

[0025] This method improves the accuracy of deep learning-based projectile detection models for detecting and classifying projectiles by increasing the contrastive encoding prior loss function to narrow the distance between similar projectiles and widen the distance between different projectiles.

[0026] In some embodiments, the method further includes:

[0027] Based on the object detection results corresponding to the captured images of the target shooting scene obtained in multiple consecutive frames, it is determined whether to issue an object warning message.

[0028] This method enables the determination of whether to issue a warning about spilled materials, thereby achieving a rapid response to the warning information.

[0029] In some embodiments, determining whether to issue a warning message regarding littering based on the littering detection results corresponding to the captured images obtained from multiple consecutive frames of the target shooting scene includes:

[0030] When it is determined that the same object image exists in the object detection results of multiple consecutive frames of images captured for the target shooting scene, it is determined whether there is an image in the preset database used to store non-object images.

[0031] If so, a warning message for the same object being thrown will be issued when none of the images in the database match.

[0032] Otherwise, issue a warning message about the spilled material directly.

[0033] This method automatically matches the same scattered object detected multiple times with non-scattered objects in the database to filter out false alarms, thereby improving the accuracy and recall of detection.

[0034] Another embodiment of this application provides a projectile detection device, which includes a memory and a processor, wherein the memory is used to store program instructions, and the processor is used to call the program instructions stored in the memory and execute any of the methods described above according to the obtained program.

[0035] Furthermore, according to embodiments, for example, a computer program product for a computer is provided, which includes software code portions that, when the product is run on the computer, perform the steps of the methods defined above. The computer program product may include a computer-readable medium on which the software code portions are stored. Furthermore, the computer program product may be directly loaded into the computer's internal memory and / or sent via a network through at least one of an upload process, a download process, and a push process.

[0036] Another embodiment of this application provides a computer-readable storage medium storing computer-executable instructions for causing the computer to perform any of the methods described above. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic flowchart illustrating a method for detecting projectiles provided in an embodiment of this application.

[0039] Figure 2 An image acquisition method provided in an embodiment of this application;

[0040] Figure 3 This application provides a motion detection model with a detection effect diagram after background modeling is completed.

[0041] Figure 4 This application provides a schematic diagram illustrating a specific process for obtaining a moving target image through a motion detection model in an embodiment of the present application.

[0042] Figure 5 A schematic diagram of a motion detection box drawn using a motion detection model, provided as an embodiment of this application;

[0043] Figure 6 This is a schematic diagram of an expanded motion detection frame provided in an embodiment of this application;

[0044] Figure 7 A schematic diagram of a moving target image provided in an embodiment of this application;

[0045] Figure 8 This is a schematic diagram illustrating a process for detecting projectiles using a deep learning-based projectile target detection model, provided as an embodiment of this application.

[0046] Figure 9 This is a schematic diagram of the structure of a projectile detection system provided in an embodiment of this application;

[0047] Figure 10 A flowchart illustrating a rapid response to false alarms provided in an embodiment of this application;

[0048] Figure 11 This is a schematic diagram of a warning information for scattered materials provided in an embodiment of this application;

[0049] Figure 12 This is a schematic diagram of the overall process of a method for detecting spilled material provided in an embodiment of this application;

[0050] Figure 13 This is a schematic diagram of the structure of a debris detection device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0052] This application provides a method and apparatus for detecting projectiles, which uses a motion detection model and a deep learning projectile target detection model to detect projectiles and improve the accuracy of projectile detection.

[0053] The method and apparatus are based on the same concept of the application. Since the methods and apparatus solve problems in similar ways, the implementation of the apparatus and methods can refer to each other, and the repeated parts will not be described again.

[0054] The terms "first," "second," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0055] The following examples and embodiments are to be understood as illustrative only. While this specification may refer to "a," "an," or "some" examples or embodiments in several places, this does not mean that every such reference relates to the same example or embodiment, nor does it mean that the feature applies only to a single example or embodiment. Individual features of different embodiments may also be combined to provide other embodiments. Furthermore, terms such as "comprising" and "including" should be understood not to limit the described embodiments to consisting only of those features mentioned; such examples and embodiments may also include features, structures, units, modules, etc., not specifically mentioned.

[0056] The various embodiments of this application will now be described in detail with reference to the accompanying drawings. It should be noted that the order in which the embodiments are presented in this application represents only a chronological order and does not represent the superiority or inferiority of the technical solutions provided by the embodiments.

[0057] It should be noted that the technical solution provided in this application embodiment is illustrated by taking the detection of littered objects in continuous image and video frames on a highway obtained by a checkpoint camera as an example, but it is not limited thereto.

[0058] The following are explanations of some of the terms that appear in the text:

[0059] 1. In the embodiments of this application, the term "region of interest" refers to the area to be processed in machine vision and image processing, which is delineated from the image being processed in the form of a box, circle, ellipse, irregular polygon, etc.

[0060] 2. In the embodiments of this application, the term "small sample" refers to a sample with a sample size of less than or equal to 30.

[0061] 3. In the embodiments of this application, the term "IOU" stands for Intersection over Union, which is the overlap ratio between the predicted border and the actual border, i.e., the ratio of their intersection to their union.

[0062] 4. In the embodiments of this application, the term "Ghost region" refers to the area that was originally covered by the object but now moves. When the background difference detection algorithm starts to detect the object as moving, the area becomes the Ghost region. Alternatively, the Ghost region is also introduced when the object that was originally moving becomes stationary.

[0063] 5. In the embodiments of this application, the term "bitstream" refers to the data flow rate of a video image after encoding and compression per unit time, also known as bit rate.

[0064] 6. In the embodiments of this application, the term "motion detection box" refers to the rectangular box drawn on the contour of each moving target found by the motion detection model in the image.

[0065] The predicted bounding box is the box calculated and output by the deep learning projectile target detection model based on the anchor box, predicting the smallest outer bounding box of the target object;

[0066] Anchor boxes are boxes clustered based on the location of objects in a dataset, used to predict target objects;

[0067] The ground truth bounding boxes (also known as annotation boxes) are the locations of manually annotated target objects, stored in the annotation file, and used to match them with the predicted boxes to remove unusable predicted boxes.

[0068] By acquiring continuous image and video frames on highways using checkpoint cameras, configuring regions of interest (ROIs) based on these frames, and combining traditional machine learning and deep learning algorithms to detect targets within these ROIs, the detection and classification of rare, small-sample types of projectiles remains a challenge. This is due to the wide range of projectile categories, large size differences, limited training data for some categories, and different pose characteristics at different angles.

[0069] To address the aforementioned issues, this application proposes a method for detecting projectiles, which can be applied to applications such as checkpoint cameras. For small samples of uncommon projectiles, especially small targets, the method can accurately detect and classify their categories, enhancing the recall rate for small sample targets. Simultaneously, it can effectively filter out falsely detected targets, thereby improving the detection accuracy.

[0070] Since some of the scattered objects have small target pixels, they are difficult to detect and are not easily distinguishable from the background when they land on the road. The best time to detect them is when they are scattered. Therefore, the scattered object detection method proposed in this application not only uses a deep learning target detection algorithm (the algorithm used by the deep learning scattered object target detection model) to detect scattered objects, but also uses a motion detection algorithm (the algorithm used by the motion detection model) to extract small images of potential scattered object targets, thereby reducing the false negative rate of scattered objects.

[0071] See Figure 1 The present application provides a method for detecting spilled material, which includes the following steps:

[0072] Step S11: Acquire images of the target shooting scene;

[0073] Step S12: Determine the region of interest in the captured image based on the region of interest configured by the user;

[0074] Step S13: Detect moving targets in the region of interest in the image obtained in step S12 using a motion detection model to obtain a moving target image;

[0075] In this step, before detecting moving targets, the motion detection model needs to perform background modeling using the acquired images (e.g., 8 frames; the motion detection model caches each detected frame for reference in the next detection). Only after the background modeling is complete can moving target detection be performed. Figure 2 The captured image shown has moving pedestrians within the rectangular frame, for example... Figure 3 As shown, this is the result of background modeling. Figure 2 The identified moving targets. In other words, for the first 8 frames of images, the motion detection model could not detect any moving targets;

[0076] In this step, the motion detection model, for example, uses a motion detection algorithm to detect motion in the region of interest;

[0077] Step S14: Use a deep learning object detection model to detect objects in the region of interest obtained in step S12 and the moving target image obtained in step S13. When objects are detected, the category and feature image of the objects are output through the user interface (the feature image is the object marked with a detection box on the original acquired image). When no objects are detected, the original acquired image is output through the user interface.

[0078] In this step, the deep learning projectile target detection model, for example, uses a deep learning target detection algorithm to detect projectiles in the region of interest and moving target images;

[0079] Step S15: When the preset conditions are met, determine whether to issue a warning message for the spilled material;

[0080] Specifically, for example, when multiple consecutive frames of images detect litter, the feature image of the litter is matched with the feature images of non-litter in a preset database using cosine similarity. If the similarity exceeds a preset threshold (e.g., 0.8), no litter warning is issued; if the similarity is less than or equal to the preset threshold, a litter warning is issued, that is, the feature image of the litter corresponding to the most recent frame is output to the user interface, and the litter and its category are marked on the feature image.

[0081] See Figure 4 The specific process for obtaining the moving target image in step S13 above includes:

[0082] Step S131: Acquire images of the target shooting scene;

[0083] Step S132: Based on the acquired image obtained in step S131, determine the region of interest in the image according to the region of interest configured by the user;

[0084] Step S133: Based on the region of interest, determine the contour of each moving target and draw a motion detection box on the contour to obtain the motion detection box of each moving target, wherein the image in the motion detection box is the image of the moving target;

[0085] In this step, for example, the ViBe algorithm is used to obtain the motion detection box of each moving target from the region of interest. The ViBe algorithm model establishes a sample set, which stores all pixels and saves the past pixel values ​​of each pixel and the pixel values ​​of its neighbors. By comparing the current pixel value with the historical pixel values ​​in the sample set, it is determined whether the pixel corresponding to the current pixel value belongs to the background. If the current pixel value is close to the historical value in the sample set (for example, the pixel value differs within a preset range), then the pixel corresponding to the current pixel value is determined to belong to the background. The ViBe algorithm has better detection performance than other common algorithms and consumes less hardware memory.

[0086] Step S134: Filter the motion detection frame obtained in step S133 according to the preset motion detection frame for non-scattered objects;

[0087] In this step, each motion detection box obtained in step S133 is matched with the preset motion detection boxes for motor vehicles, pedestrians, and non-motor vehicles using IOU. When the IOU is greater than the preset threshold, the motion detection box is filtered out; otherwise, the motion detection box is retained.

[0088] Step S135: For each remaining motion detection box, expand the motion detection box based on the width and height of the original acquired image, and extract the image of the moving target from the expanded motion detection box to obtain the moving target image;

[0089] In this step, to avoid situations where moving targets are not completely covered by the motion detection frame, affecting the accuracy of subsequent object detection, the motion detection frame is expanded, for example... Figure 5 As shown, taking the center point of the motion detection box of the thrown object as the center, the motion detection box is expanded based on the width and height of the corresponding original acquired image (e.g., twice the width and height) (that is, the motion detection box is enlarged while the size of the image within the motion detection box remains unchanged). For example... Figure 6 As shown, the image of the moving target within the expanded motion detection bounding box is extracted, for example... Figure 7 As shown, the image of the moving target is obtained.

[0090] See Figure 8 The specific process of step S14 above, which uses a deep learning projectile target detection model to detect projectiles in the region of interest and moving target images, includes:

[0091] Step S141: Detect objects in the region of interest using a deep learning object detection model. If objects are detected, proceed to step S142; otherwise, proceed directly to step S143.

[0092] Step S142: Filter each moving target image based on the detected projectile images;

[0093] In this step, the cross-union ratio (CUI) of each moving target image and the object image is performed. When the CUI is greater than a preset threshold, the moving target image is filtered out to avoid repeatedly detecting objects in the same image.

[0094] Step S143: Perform object detection on the remaining moving target images. When objects are detected, integrate them with the detection results of step S141 and output the category and feature image of the objects through the user interface.

[0095] In this step, if debris is detected, the detection result is integrated with the detection result of step S141. That is, all debris is marked with a rectangle in the region of interest in the original acquired image.

[0096] See Figure 9 The present application provides a projectile detection system, which includes, for example, a deep learning target detection module 100 and a motion detection module 200.

[0097] The deep learning object detection module 100 is used to detect projectiles in the region of interest in the image (for example, using the deep learning projectile object detection model mentioned above).

[0098] The motion detection module 200 (for example, using the motion detection model described above) is used to detect moving targets in the region of interest in the image, obtain a moving target image, and send it to the deep learning target detection module 100.

[0099] The deep learning target detection module 100 is also used to receive moving target images sent by the motion detection module 200 and to perform object detection on the moving target images.

[0100] In some embodiments, the motion detection module 200 includes, for example, a first motion detection module 201 and a second motion detection module 202;

[0101] The first motion detection module 201 is used to determine the contour of each moving target based on the region of interest in the image, and draw a motion detection box on the contour to obtain the motion detection box of each moving target, wherein the image in the motion detection box is the image of the moving target.

[0102] The first motion detection module 201 identifies all moving targets from the region of interest in the image, ensuring that no thrown targets are missed during subsequent object detection, thereby improving detection accuracy.

[0103] Since the first motion detection module 201 can only identify moving targets, but not all moving targets are thrown objects, such as pedestrians, motor vehicles, and non-motorized vehicles, further screening of moving targets is required to filter out these moving targets. Therefore, a second motion detection module 202 is added to the motion detection module 200 to filter out motion detection boxes with known moving target categories (e.g., motion detection boxes for motor vehicles, pedestrians, and non-motorized vehicles) from the motion detection boxes determined by the first motion detection module 201. In some embodiments, for each remaining motion detection box, the motion detection box can be further expanded based on the width and height of the original acquired image, and the image of the moving target can be extracted from the expanded motion detection box to obtain the motion target image, which is then sent to the aforementioned deep learning target detection module 100.

[0104] In some embodiments, the projectile detection system further includes a registration database module 300 for storing feature images of non-projectiles. When the deep learning target detection module 100 determines that the same projectile image exists in the projectile detection results corresponding to multiple consecutive frames (which can be set) of captured images of a target shooting scene, it outputs the feature image of the projectile to the registration database module 300. The registration database module 300 performs cosine similarity matching between the feature image of the projectile and the feature images of non-projectiles stored in the database. If the similarity is less than or equal to a preset threshold (e.g., 0.8), the deep learning target detection module 100 issues a projectile warning; otherwise, it does not issue a projectile warning. If the similarity is greater than the preset threshold, the original captured image is stored in a non-projectile sample set. This non-projectile sample set is used to train the deep learning projectile target detection model. The trained deep learning projectile target detection model is used for projectile detection, thereby further improving the detection accuracy.

[0105] In some embodiments, the first motion detection module 201 described above extracts the moving target image from the acquired image, for example, using the ViBe model. The ViBe model mainly includes three aspects: the working principle of the ViBe model, the initialization method of the background model, and the update strategy of the background model.

[0106] I. How the ViBe model works:

[0107] The ViBe model pre-establishes a sample set containing all pixels, saving the past pixel values ​​of each pixel and the pixel values ​​of its neighbors. By comparing the current pixel value with the historical pixel values ​​in the sample set, it determines whether the pixel corresponding to the current pixel value belongs to the background. If the current pixel value is close to the historical value in the sample set, it is determined that the pixel corresponding to the current pixel value belongs to the background.

[0108] Let V(x,y) be the pixel value at coordinates (x,y) in the current image frame; M(x,y) = {V 1 (x,y),...V n (x,y) is the background sample set with a sample set size of n; R is a range of allowable fluctuations. In the ViBe model used in this application embodiment, it is set to 20. For example, the historical pixel value at coordinate (1,1) is 200. Assuming the current frame pixel value is 230, the difference between this pixel value and its historical pixel value is 30, which is greater than R. This pixel value is regarded as a potential foreground target, and otherwise as a background target.

[0109] In some embodiments, the number of neighborhood sample points n in the sample set is selected as 50, which can ensure the memory consumption of the debris detection system and increase the system's adaptability to complex scenarios.

[0110] The difference between the 50 neighboring points of a pixel value and the historical pixel value in the sample set M(x,y) is compared with R. If the absolute value of the 50 differences is less than the number of R, that is, the number of potential background target pixels N>th (th represents the preset threshold, for example, 43), then the pixel is similar to the historical pixel and can be judged as a background pixel. If the difference is greater than R or less than the number of -R N1, that is, the number of potential foreground target pixels N1>T (T=n-th), then the pixel is judged as a foreground pixel.

[0111] II. Initialization of the background model:

[0112] Initialization is the process of building the background model. The ViBe model primarily uses a single frame of image to initialize the background model. For a given pixel, considering the spatial distribution characteristics of neighboring pixels having similar pixel values, the pixel values ​​of the neighboring pixels are randomly selected as the model sample values ​​for that pixel. Background model initialization is essentially the process of filling the sample set of pixels. In some embodiments, the background model is initialized by selecting, for example, the first 8 frames of the image to be processed, and selecting 50 neighboring pixels around each pixel as the sample set. This not only reduces the computational load but also preserves the spatiotemporal information of the image.

[0113] III. Background Model Update Strategy:

[0114] Background model updates allow the background model to adapt to continuous changes in the background, such as changes in lighting or background image. In some embodiments, an update strategy, such as a foreground point counting method, is used to count pixels. If a pixel is detected as a foreground point for N consecutive frames (e.g., 20), it is updated to a background point. Each background point detected by the ViBe model has... The probability of updating the sample value of that pixel ( There is a probability that the pixel will be re-evaluated to confirm whether it is no longer a background pixel. The probability is used to update the sample values ​​of the neighboring points of a pixel, where l represents the time sampling factor (considering the computational time, it is not calculated for every pixel in every frame of the image, but rather a probability is used to determine whether a pixel and its neighbors have changed from a background point to a foreground point). In this model, its value is set to 16. Simultaneously, for a detected foreground point, if it is detected as a foreground point for 20 consecutive frames, it is updated to a background point, and the same applies. The probability is used to update its sample value. In other words, if a foreground object has been stationary in the picture for 20 consecutive frames, it can be regarded as part of the background and changed from foreground to background.

[0115] This update strategy leverages the spatial propagation characteristics of pixel values, effectively avoiding the generation of ghost regions. Furthermore, by randomly selecting a value from the sample set to be replaced, the smoothness of the sample values ​​is ensured. Since the update of sample values ​​in this ViBe model is time-independent, this update strategy is quite reasonable.

[0116] In some embodiments, the deep learning object detection module 100 described above, for example, employs a single-stage object detection algorithm, taking YOLOv5s as an example. In the first stage, a large number of labeled common category objects (e.g., cardboard boxes, cartons, bags) samples are used to pre-train the detection model, resulting in a detection model with initial parameters. In the second stage, small samples of uncommon category objects (e.g., beverage bottles, shopping bags) are used to fine-tune the detection model with initial parameters. Unlike two-stage object detection algorithms that freeze convolutional layers and only fine-tune parameters in fully connected layers, in order to ensure that each layer of the YOLOv5s network learns the category features of small samples, in some embodiments, the fine-tuning of the detection model is performed based on the entire YOLOv5s network. However, the imbalance between the basic category and the small sample training data ratio still significantly affects the detection results of small samples, making it impossible to accurately classify some easily confused or small targets with few training samples. To solve this problem, in some embodiments, the relevant parameters of the detection model are adjusted during the second stage of training.

[0117] Because of the limited amount of small sample data, the confidence level of pre-trained models (i.e., detection models with initial parameters) when detecting small sample targets is often not very high, and many predicted boxes will be directly suppressed by the NMS (non-maximum suppression) operation. During the second stage of training the pre-trained model, a confidence score threshold (e.g., 0.2) is set. Predicted boxes with confidence scores greater than or equal to this threshold inferred by the pre-trained model are retained, thereby increasing the number of small sample predicted boxes and preventing some predicted boxes with low confidence scores from being filtered out.

[0118] However, increasing the number of predicted boxes introduces more predicted boxes that do not contain target semantic information, making it difficult for the pre-trained model to converge and reducing accuracy. In some embodiments, when performing the second stage of training on the pre-trained model, an IOU threshold (e.g., 0.6) is set. When the IOU between the predicted box and the labeled box (manually labeled coordinate boxes) is greater than or equal to this threshold, the predicted box is retained, thereby ensuring the usability of the predicted box. In other words, the predicted box can only be kept for training if the intersection-union ratio between the predicted box and the labeled box reaches 60%.

[0119] To increase the similarity between similar targets and to make the feature vectors of different targets closer together, thereby improving the accuracy of the pre-trained model in detecting and classifying projectiles, in some embodiments, a CPE (Contrastive Proposal Encoding) loss is added to the original loss function of the pre-trained model during the second stage of training. The formula for calculating the CPE loss is as follows:

[0120]

[0121] in, This represents the CPE loss of the current predicted bounding box for a batch (i.e., multiple image samples). This indicates that items with the same label y in a batch i The number of predicted boxes, y i The label (also called the identifier) ​​of the prediction box, y j Z represents the label of the annotation box. i Z represents the feature vector of the target image in the current prediction bounding box. j Z represents the feature vector of the target image of the predicted bounding box with the same label as the current predicted bounding box in a batch; k This represents the feature vector of the target image of all predicted bounding boxes except the current predicted bounding box; This represents the cosine similarity between predicted bounding boxes with the same label in a batch. τ represents a preset hyperparameter, which is a preset constant, for example, with a value of 0.2.

[0122]

[0123] Among them, L CPE This represents the sum of CPE losses for all predicted boxes in a batch, where N is the number of predicted boxes in a batch. This represents the confidence score when the Intersection over Union (IOU) between the predicted bounding box and the labeled bounding box is greater than a preset threshold. For example, for the existing Yolov5s, the loss function calculation formula after adding CPE loss in this embodiment is as follows:

[0124]

[0125] Where λ represents the weight hyperparameters controlling various loss functions (e.g., λ1 = λ2 = λ3 = 1, λ4 = 0.7), L box L represents the error loss between the predicted bounding box and the labeled bounding box. obj L represents the confidence loss for calculating the predicted bounding box. cls This represents the loss calculated to determine whether the predicted bounding box category matches the labeled bounding box category. During the second stage of training the pre-trained model, the calculated total loss L can be used as a reference. total To judge the training effect of the model, the smaller the total loss, the better the effect. In this embodiment, the calculation of CPE loss is added, which can shorten the feature vector distance between prediction boxes of the same type and widen the feature vector distance between prediction boxes of different types. This makes it less likely for small sample targets to be interfered with by common large sample targets. This loss function is applicable to all single-stage and two-stage deep learning object detection network models.

[0126] The detection method proposed in this application, which primarily employs deep learning for object detection and supplements it with machine learning, can improve detection robustness and enhance recall for small samples. However, in practical applications, some false alarms (i.e., inaccurate detection) may still occur. Therefore, in some embodiments, the aforementioned registration database module 300 is used to achieve rapid response processing for false alarms. See [link to relevant documentation]. Figure 10 The specific steps include:

[0127] Step S21: Acquire the captured image of the target shooting scene, and determine the region of interest in the image based on the region of interest configured by the user;

[0128] Step S22: The motion detection module 200 performs motion target detection on the region of interest in the image to obtain a moving target image;

[0129] Step S23: The deep learning target detection module 100 performs object detection on the region of interest and moving target image in the image. When object is detected, it is determined whether the preset conditions are met.

[0130] It should be noted that the preset conditions may include multiple conditions. For example, when the first preset condition is met, it can be further determined whether the second preset condition is met, and a warning message for scattered objects is issued only when multiple conditions are met. For example, for any image of scattered objects, if the image of scattered objects exists in multiple consecutive frames, it is considered that the preset conditions are met; or, for any image of scattered objects, if the image of scattered objects exists in multiple consecutive frames, and the similarity between the image of scattered objects and a preset non-scattered object image is less than or equal to a preset threshold, it is considered that the preset conditions are met.

[0131] The preset non-scattered object image is stored in the registration base library module 300.

[0132] Step S24: When the preset conditions are met, the deep learning target detection module 100 issues a warning message for the scattered object.

[0133] In some embodiments, a warning message about littering can be issued via a user interface. For example, a warning message about littering issued via a user interface may be such as... Figure 11 As shown, this allows staff to utilize Figure 11 The warning information regarding the spilled material is manually analyzed to determine... Figure 11 To determine whether the marked object is indeed a thrown object, we can confirm whether the warning message about the thrown object was a false alarm.

[0134] Step S25: If, for example, a user instruction is received through the user interface that the currently issued warning information about scattered objects is a false alarm, then it is determined that the currently falsely detected scattered object is actually a non-scattered object, and the image of the non-scattered object is stored in the registration base module 300, that is, the registration base module 300 is updated in real time to further improve the accuracy of subsequent false alarm judgments.

[0135] If the user confirms that the current warning message about debris is correct, then it is finally determined that debris has been detected.

[0136] In some embodiments, when the amount of non-spraying object data stored in the aforementioned registration database module 300 is sufficient, for example, exceeding a preset threshold (the specific value can be determined according to actual needs), during subsequent detection, before issuing a spraying object warning message when the reporting conditions are triggered, a cosine similarity match is performed with the feature vector of the non-spraying object feature image stored in the database. If the similarity exceeds the preset threshold (e.g., 0.8), no spraying object warning message is issued. That is, the aforementioned spraying object warning message is issued only when multiple preset conditions are met. In other words, the registration database module 300 can replace manual judgment of warning events triggered by the same or similar targets, thus eliminating the step of manually confirming whether the current warning is a false alarm on the user interface.

[0137] In some embodiments, the original images corresponding to the non-sprayed objects detected by false alarms can also be stored in the registration base module 300. Based on the saved original image frame number of the non-sprayed objects, the false alarm data can be effectively located, collected, and included in the training set, thereby enabling iterative updates to the deep learning target detection algorithm, optimizing the detection accuracy and recall rate, and improving robustness.

[0138] In summary, see Figure 12 This application provides a method for detecting spilled material, comprising:

[0139] Step S31: Acquire images of the target shooting scene and determine the region of interest in the acquired images;

[0140] In this step, the region of interest in the captured image is determined based on the region of interest configured by the user.

[0141] Step S32: Detect moving targets in the region of interest using a motion detection model to obtain at least one moving target image;

[0142] For example, the motion detection module 200 described above can be used to detect moving targets in the region of interest of the image.

[0143] Step S33: Use a deep learning object detection model to detect objects in the region of interest and the image of at least one moving target, and determine the object detection result corresponding to the acquired image;

[0144] In this step, for example, the deep learning target detection module 100 described above is used to detect projectiles in the region of interest and the moving target image;

[0145] Step S33 enables the detection of projectiles from the original acquired image and the moving target image extracted by the motion detection model, thereby improving the accuracy of projectile detection.

[0146] To improve detection accuracy, in some embodiments, the step of detecting moving targets in the region of interest using a motion detection model to obtain at least one moving target image includes:

[0147] Based on the region of interest, the contour of each moving target is determined, and a motion detection box is drawn on the contour to obtain the motion detection box of each moving target (for example, by the first motion detection module 201 described above); wherein, the image in the motion detection box is the image of the moving target;

[0148] Based on the preset motion detection frame for non-scattered objects, the motion detection frame of each moving target is filtered (for example, by the second motion detection module 202 mentioned above, filtering out the motion detection frames of motor vehicles, pedestrians, and non-motor vehicles) to obtain at least one moving target image.

[0149] To improve the accuracy of subsequent detection, in some embodiments, filtering the motion detection frame of each moving target based on a preset non-projectile motion detection frame to obtain at least one moving target image includes:

[0150] For each motion detection frame, a cross-union ratio (CUNR) is performed with a preset non-projectile motion detection frame. When the CUNR is greater than a preset threshold (e.g., 0.8), the motion detection frame is filtered out.

[0151] For each remaining motion detection box, the motion detection box is expanded based on the width and height of the original acquired image (e.g., twice the width and height of the image), and the image of the moving target is extracted from the expanded motion detection box to obtain the moving target image.

[0152] To avoid repeated detection of the same image, in some embodiments, the step of using a deep learning projectile target detection model to perform projectile detection on the region of interest and the at least one moving target image, and determining the projectile detection result corresponding to the acquired image, includes:

[0153] By performing object detection on the region of interest, it can be determined whether an image of an object is present in the region of interest;

[0154] If so, the at least one moving target image is filtered according to the image of the scattered object (for example, the cross-union ratio of the image of the scattered object and the moving target image is performed, and the moving target image is filtered out when the cross-union ratio is greater than a preset threshold), and the remaining moving target image and the image of the scattered object are used to obtain the scattered object detection result corresponding to the acquired image.

[0155] Otherwise, the object detection result corresponding to the acquired image is obtained using the at least one moving target image.

[0156] To improve the accuracy of deep learning projectile target detection models in detecting and classifying projectiles, in some embodiments, the method further includes the following step of pre-training the deep learning projectile target detection model:

[0157] Calculate the total loss function value of a preset number of sample images; wherein, the total loss function value is the sum of the preset category loss function value and the contrastive encoding prior loss function value;

[0158] Based on the total loss function value, adjust the weights of each layer of the neural network in the deep learning projectile target detection model;

[0159] The preset quantity is, for example, the batch size mentioned above; the preset category loss function value is, for example, the original loss function of the detection model; and the contrastive encoding prior loss function value is, for example, the CPE loss.

[0160] To improve the accuracy of detection, in some embodiments, the method further includes:

[0161] Based on the object detection results corresponding to the captured images obtained from multiple consecutive frames of the target shooting scene, it is determined whether to issue an object warning message.

[0162] For example, a warning message can be issued when the same object is detected in multiple consecutive frames of images, or when the same object is detected in multiple consecutive frames of images and the object does not match the non-objects stored in a preset database.

[0163] The warning information for the debris is, for example, outputting a feature image of the debris corresponding to the most recent frame to the user interface, on which the debris and its category are marked.

[0164] To automatically filter out false alarms from non-projectile sources, in some embodiments, determining whether to issue a projectile warning based on the projectile detection results corresponding to the acquired images obtained from multiple consecutive frames of the target shooting scene includes:

[0165] When it is determined that the same object image exists in the object detection results of multiple consecutive frames of images captured for the target shooting scene, it is determined whether there is an image in the preset database used to store non-object images.

[0166] If so, when the same image of the object being thrown does not match any of the images in the database (by performing an intersection-union comparison between the image of the object being thrown and the images in the database, and when the intersection-union comparison is less than or equal to a preset threshold, it indicates that the object being thrown does not match any of the images in the database), an object throwing warning message is issued.

[0167] Otherwise, issue a warning message about the spilled material directly.

[0168] The following describes the device or apparatus provided in the embodiments of this application, and the explanations or examples of the same or corresponding technical features as those described in the above methods will not be repeated hereafter.

[0169] See Figure 13 This application provides an embodiment of a debris detection device, comprising:

[0170] Processor 600 is used to read the program from memory 620 and execute the following procedures:

[0171] Acquire images of the target shooting scene and determine the region of interest in the acquired images;

[0172] The region of interest is used to detect moving targets using a motion detection model to obtain at least one image of a moving target.

[0173] By using a deep learning object detection model to detect objects in the region of interest and the image of at least one moving target, the object detection result corresponding to the acquired image is determined.

[0174] In some embodiments, the step of detecting moving targets in the region of interest using a motion detection model to obtain at least one moving target image includes:

[0175] Based on the region of interest, the contour of each moving target is determined, and a motion detection box is drawn on the contour to obtain the motion detection box of each moving target; wherein, the image in the motion detection box is the image of the moving target;

[0176] Based on the preset motion detection bounding box for non-scattered objects, the motion detection bounding box of each moving target is filtered to obtain at least one moving target image.

[0177] In some embodiments, filtering the motion detection frame of each moving target based on a preset non-scattered object motion detection frame to obtain at least one moving target image includes:

[0178] For each motion detection frame, a cross-over and union ratio (CUP) is performed with a preset non-projectile motion detection frame. When the CUP is greater than a preset threshold, the motion detection frame is filtered out.

[0179] For each remaining motion detection box, the motion detection box is expanded based on the width and height of the original acquired image, and the image of the moving target is extracted from the expanded motion detection box to obtain the moving target image.

[0180] In some embodiments, the step of performing object detection on the region of interest and the at least one moving target image using a deep learning object detection model, and determining the object detection result corresponding to the acquired image, includes:

[0181] By performing object detection on the region of interest, it can be determined whether an image of an object is present in the region of interest;

[0182] If so, the at least one moving target image is filtered based on the object image, and the remaining moving target image and the object image are used to obtain the object detection result corresponding to the acquired image;

[0183] Otherwise, the object detection result corresponding to the acquired image is obtained using the at least one moving target image.

[0184] In some embodiments, the processor 600 is further configured to read a program from the memory 620 and execute the following steps for pre-training the deep learning projectile target detection model:

[0185] Calculate the total loss function value of a preset number of sample images; wherein, the total loss function value is the sum of the preset category loss function value and the contrastive encoding prior loss function value;

[0186] Based on the total loss function value, adjust the weights of each layer of the neural network in the deep learning projectile target detection model.

[0187] In some embodiments, the processor 600 is further configured to read a program from the memory 620 and execute it:

[0188] Based on the object detection results corresponding to the captured images of the target shooting scene obtained in multiple consecutive frames, it is determined whether to issue an object warning message.

[0189] In some embodiments, determining whether to issue a warning message regarding littering based on the littering detection results corresponding to the captured images obtained from multiple consecutive frames of the target shooting scene includes:

[0190] When it is determined that the same object image exists in the object detection results of multiple consecutive frames of images captured for the target shooting scene, it is determined whether there is an image in the preset database used to store non-object images.

[0191] If so, a warning message for the same object being thrown will be issued when none of the images in the database match.

[0192] Otherwise, issue a warning message about the spilled material directly.

[0193] In some embodiments, the debris detection device provided in this application further includes a transceiver 610 for receiving and sending data under the control of a processor 600.

[0194] Among them, Figure 13In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 600) and memory (memory 620). The bus architecture can also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 610 can be multiple elements, including a transmitter and a receiver, providing a unit for communicating with various other devices over a transmission medium.

[0195] In some embodiments, the debris detection device provided in this application further includes a user interface 630. The user interface 630 may be an interface that can connect to external or internal devices, including but not limited to keypads, displays, speakers, microphones, joysticks, etc.

[0196] The processor 600 is responsible for managing the bus architecture and general processing, while the memory 620 can store the data used by the processor 600 when performing operations.

[0197] In some embodiments, the processor 600 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a CPLD (Complex Programmable Logic Device).

[0198] This application provides a computing device, which may specifically be a desktop computer, portable computer, smartphone, tablet computer, personal digital assistant (PDA), etc. The computing device may include a central processing unit (CPU), memory, input / output devices, etc. Input devices may include a keyboard, mouse, touchscreen, etc., and output devices may include display devices, such as a liquid crystal display (LCD) or a cathode ray tube (CRT).

[0199] The memory may include read-only memory (ROM) and random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In the embodiments of this application, the memory may be used to store the program of any of the methods provided in the embodiments of this application.

[0200] The processor executes any of the methods described in the embodiments of this application according to the program instructions stored in the memory.

[0201] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the methods described in the above embodiments. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0202] This application provides a computer-readable storage medium for storing computer program instructions used in the apparatus provided in the above-described embodiments, including a program for performing any of the methods provided in the above-described embodiments. The computer-readable storage medium may be a non-transitory computer-readable medium.

[0203] The computer-readable storage medium can be any available medium or data storage device that a computer can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0204] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0205] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0206] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0207] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0208] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for detecting spilled material, characterized in that, The method includes: Acquire images of the target shooting scene and determine the region of interest in the acquired images; The region of interest is used to detect moving targets using a motion detection model to obtain at least one image of a moving target. By using a deep learning object detection model to detect objects in the region of interest and the image of at least one moving target, the object detection result corresponding to the acquired image is determined. The step of detecting moving targets in the region of interest using a motion detection model to obtain at least one moving target image includes: Based on the region of interest, the contour of each moving target is determined, and a motion detection box is drawn on the contour to obtain the motion detection box of each moving target; wherein, the image in the motion detection box is the image of the moving target; Based on the preset motion detection frame for non-scattered objects, the motion detection frame of each moving target is filtered to obtain at least one moving target image; The step of using a deep learning-based object detection model to detect objects in the region of interest and the at least one moving target image, and determining the object detection result corresponding to the acquired image, includes: By performing object detection on the region of interest, it can be determined whether an image of an object is present in the region of interest; If so, the at least one moving target image is filtered based on the object image, and the remaining moving target image and the object image are used to obtain the object detection result corresponding to the acquired image; Otherwise, the object detection result corresponding to the acquired image is obtained using the at least one moving target image.

2. The method according to claim 1, characterized in that, The step of filtering the motion detection box of each moving target according to the preset non-scattered object motion detection box to obtain at least one moving target image includes: For each motion detection frame, a cross-over and union ratio (CUP) is performed with a preset non-projectile motion detection frame. When the CUP is greater than a preset threshold, the motion detection frame is filtered out. For each remaining motion detection box, the motion detection box is expanded based on the width and height of the original acquired image, and the image of the moving target is extracted from the expanded motion detection box to obtain the moving target image.

3. The method according to claim 1, characterized in that, The method further includes the following steps for pre-training the deep learning projectile target detection model: Calculate the total loss function value of a preset number of sample images; wherein, the total loss function value is the sum of the preset category loss function value and the contrastive encoding prior loss function value; Based on the total loss function value, adjust the weights of each layer of the neural network in the deep learning projectile target detection model.

4. The method according to claim 1, characterized in that, The method further includes: Based on the object detection results corresponding to the captured images of the target shooting scene obtained in multiple consecutive frames, it is determined whether to issue an object warning message.

5. The method according to claim 4, characterized in that, The determination of whether to issue a warning message regarding littering based on the litter detection results corresponding to the captured images obtained from multiple consecutive frames of the target shooting scene includes: When it is determined that the same object image exists in the object detection results of multiple consecutive frames of images captured for the target shooting scene, it is determined whether there is an image in the preset database used to store non-object images. If so, a warning message for the same object being thrown will be issued when none of the images in the database match. Otherwise, issue a warning message about the spilled material directly.

6. A device for detecting spilled material, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method according to any one of claims 1 to 5.

7. A computer program product for use in a computer, characterized in that, Includes a software code portion that, when the product is run on the computer, is used to perform the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing the computer to perform the method according to any one of claims 1 to 5.

Images