Method, device, electronic device and storage medium for detecting spilled objects

Abstract

The present application relates to a method, device, electronic device and storage medium for detecting spilled objects, wherein the method for detecting spilled objects comprises: detecting a moving target in an image to be detected, obtaining first position information of the moving target, and establishing a first mask image based on the first position information; detecting a suspected spilled object in the image to be detected, obtaining second position information of the suspected spilled object; and determining whether the suspected spilled object is a target spilled object based on the first mask image and the second position information. Through the present application, the problem that the traditional method for detecting spilled objects is prone to false detection is solved, and the technical effect of reducing the probability of false detection and improving detection accuracy is achieved.

Description

Technical Field

[0001] The present application relates to the field of image detection technology, and in particular to a method, device, electronic device and storage medium for detecting spilled objects. Background Art

[0002] In the process of continuous urban development and construction, there are a large number of muck trucks and concrete mixer trucks transporting construction sand and cement materials to construction sites, or transporting soil and construction waste from construction sites. Due to overloading, bumps and other reasons, muck trucks often cause sand and gravel spillage and dripping during transportation. This seriously affects the cleanliness of urban roads, increases the workload of sanitation workers, and road spillage poses a major safety hazard to subsequent vehicles.

[0003] The traditional method of detecting spilled objects simply uses image detection algorithms to determine whether there are spilled objects in the image, which is prone to false detection and has low accuracy.

[0004] With regard to the technical problems in related technologies such as the traditional spilled object detection method being prone to false detection and having low accuracy, no effective solution has been proposed so far. Summary of the invention

[0005] In this embodiment, a spilled object detection method, device, electronic device and storage medium are provided to solve the problem that the traditional spilled object detection method in the related art is prone to false detection and has low accuracy.

[0006] In a first aspect, a method for detecting spilled objects is provided in this embodiment, comprising:

[0007] Detecting a moving target in an image to be detected, obtaining first position information of the moving target, and establishing a first mask image based on the first position information;

[0008] Detecting a suspected spilled object in the image to be detected, and obtaining second position information of the suspected spilled object;

[0009] Determine whether the suspected spilled object is a target spilled object based on the first mask image and the second position information.

[0010] In one embodiment, a moving target detection frame obtained based on the first position information is marked on the first mask map, and determining whether the suspected spilled object is a target spilled object based on the first mask map and the second position information includes: determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; if the suspected spilled object is outside the moving target detection frame, determining it to be a target spilled object.

[0011] In one embodiment, a moving target detection frame obtained based on the first position information is marked on the first mask map, and determining whether the suspected spilled object is a target spilled object based on the first mask map and the second position information also includes: determining a first moving target based on the type of the moving target, the first moving target including a dump truck; determining a motion trajectory of the first moving target in multiple frames of the image to be detected based on a target tracking algorithm; determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; determining whether the suspected spilled object is located within the range of the motion trajectory based on the second position information; if the suspected spilled object is outside the moving target detection frame and within the range of the motion trajectory, it is determined to be a target spilled object, and the target spilled object is associated with the first moving target.

[0012] In one embodiment, a moving target detection frame obtained based on the first position information is marked on the first mask map, and determining whether the suspected spilled object is a target spilled object based on the first mask map and the second position information further includes: acquiring a reference image, wherein the reference image is a historical image of the image to be detected; determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; determining a first similarity between the image to be detected and the reference image; if the suspected spilled object is outside the moving target detection frame and the first similarity is less than a first preset threshold, determining that the suspected spilled object is a target spilled object.

[0013] In one embodiment, the acquiring of the reference image comprises: determining a background area image based on the first mask image of a preset number of frames and the second position information; and determining a reference image based on the background area image.

[0014] In one embodiment, a moving target detection frame obtained based on the first position information is marked on the first mask map, and determining whether the suspected spilled object is a target spilled object based on the first mask map and the second position information further includes: obtaining historical position information of historical spilled objects; determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; determining an intersection-and-union ratio of an image area corresponding to the second position information and an image area corresponding to the historical position information; if the suspected spilled object is outside the moving target detection frame and the intersection-and-union ratio is less than a second preset threshold, determining that the suspected spilled object is a target spilled object.

[0015] In one embodiment, a moving target detection frame obtained based on the first position information is marked on the first mask image, and the determination of whether the suspected spilled object is the target spilled object based on the first mask image and the second position information also includes: obtaining a target spilled object detection result of the image to be detected for a preset frame number; determining the first frame number in which the target spilled object exists in the preset frame number based on the target spilled object detection result; and if the ratio of the first frame number to the preset frame number is greater than a second preset threshold, an alarm is issued.

[0016] In a second aspect, a spilled object detection device is provided in this embodiment, comprising:

[0017] A first detection module, used for detecting a moving target in an image to be detected, obtaining first position information of the moving target, and establishing a first mask image based on the first position information;

[0018] A second detection module is used to detect the suspected spilled object in the image to be detected and obtain second position information of the suspected spilled object;

[0019] A target determination module is used to determine whether the suspected spilled object is a target spilled object based on the first mask image and the second position information.

[0020] In a third aspect, an electronic device is provided in this embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for detecting spilled objects described in the first aspect when executing the computer program.

[0021] In a fourth aspect, in this embodiment, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for detecting spilled objects described in the first aspect is implemented.

[0022] Compared with the related art, the method, device, electronic device and storage medium for detecting spilled objects provided in the present embodiment detect a moving target in an image to be detected to obtain first position information of the moving target, and establish a first mask map based on the first position information; detect suspected spilled objects in the image to be detected to obtain second position information of the suspected spilled objects; determine whether the suspected spilled object is a target spilled object based on the first mask map and the second position information, comprehensively consider the position information of the moving target and the suspected spilled object, and then judge whether the suspected spilled object is a target spilled object, thereby reducing the probability of false detection and improving detection accuracy.

[0023] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] Figure 1 is a hardware structure block diagram of a terminal according to a method for detecting spilled objects in an embodiment of the present application;

[0026] Figure 2 is a flow chart of a method for detecting spilled objects according to an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of a spilled object detection method according to another embodiment of the present application;

[0028] Figure 4 It is a structural block diagram of a spilled object detection device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0030] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0031] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 is a hardware structure block diagram of a terminal according to the method for detecting spilled objects in an embodiment of the present application. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0032] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for detecting spilled objects in the present embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0034] With the development of computer vision technology, the detection technology of scattered objects is also constantly improving. An existing method for detecting scattered objects is to use the image after removing the moving object as the historical background image, determine the change area in the current background image based on the historical background image, and then input the image of the change area into a binary classification model to determine whether there is a scattered object in the image. The so-called binary classification model is a recognition model that can only identify whether the image contains scattered objects or does not contain scattered objects after the image to be detected is input into the scattered object detection model. The disadvantage of this method is that changes in targets such as road rain and snow, light and shadow, water stains, and fallen leaves will frequently generate change areas, which will not only increase the amount of calculation, but also the morphology of these targets is similar to that of scattered objects. Directly inputting them into the binary classification model for scattered object detection will produce a large number of false detections. Another method for detecting scattered objects is to establish a Gaussian mixture model for each pixel in the image, use the Gaussian mixture model to separate the moving target from the background, and then use the binary classification model to detect scattered objects in the moving target image. Compared with the previously proposed method of determining the changed area by comparison, this method can directly separate the moving target from the image through the Gaussian mixture model. However, the actual road scenes are complex and diverse. In such scenes, directly using the binary classification model for spilled object detection will still result in false detection. At the same time, since most of the spilled objects become stationary targets after dripping, this method will treat the stationary targets as background, resulting in missed detection of spilled objects. Therefore, how to improve the detection accuracy of spilled objects is still an important issue that needs to be solved.

[0035] In this embodiment, a method for detecting spilled objects is provided. Figure 2 is a flow chart of a method for detecting spilled objects according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0036] Step S201: Detect a moving target in an image to be detected, obtain first position information of the moving target, and establish a first mask image based on the first position information.

[0037] Specifically, the image to be detected in this embodiment is obtained by a camera placed at a traffic road checkpoint and a camera installed on an electronic police. A moving target refers to a common moving target object in a traffic road scene, such as a pedestrian, a motor vehicle, a non-motor vehicle, etc. By detecting the moving target in the image to be detected, the position information of the moving target in the image is obtained and used as the first position information. The type position information of the moving target can be obtained through a target recognition detection algorithm or a pre-trained deep learning model. The mask image is a mask bitmap, which is a monochrome bitmap. In this embodiment, the first mask image is a binary image of the same size as the input image. When constructing the first mask image, the values ​​of all target corresponding areas on the image are filled with 1, and the values ​​of the remaining positions are filled with 0.

[0038] Step S202: Detect the suspected spilled object in the image to be detected, and obtain second position information of the suspected spilled object.

[0039] Specifically, the type of suspected spilled objects and the second location information can be determined by a pre-trained deep learning model. Spilled objects refer to items that are spilled and dripped onto the road during vehicle transportation, such as mud, sand, sewage stains, building materials, construction waste, etc. Suspected spilled objects refer to objects that are initially determined to be spilled objects. This is because in actual scenarios, there are situations where the deep learning model mistakenly detects the vehicle's own dirt as spilled objects. Therefore, in this embodiment, only the preliminary recognition results are used as suspected spilled objects for further detection.

[0040] Step S203: determining whether the suspected spilled object is a target spilled object based on the first mask image and the second position information.

[0041] Specifically, based on the positional relationship between the moving target and the suspected spilled object in the first mask image, it is further determined whether the suspected spilled object is the target spilled object. For example, the judgment is made based on whether the distance from the center of the moving target in the first mask image to the center of the suspected spilled object exceeds a preset distance. Another example is: the judgment is made based on the positional relationship between the center of the suspected spilled object and the moving target detection frame. In one embodiment, the judgment can also be made based on the intersection and union ratio of the moving target detection frame and the suspected spilled object detection frame.

[0042] Through the above steps, the present application obtains the position information of the moving target and the suspected spilled object, and judges whether there is a spilled object in the image to be detected according to the positional relationship between the moving target and the suspected spilled object. On the basis of the traditional use of a binary classification model to directly identify objects, according to the characteristics of the spilled object dripping from the vehicle in the actual scene, the targets in the image are divided into moving targets and suspected spilled objects. Then, according to the positional relationship between the two targets, the suspected spilled objects are further screened and identified, and the situation where the vehicle's own dirt is mistakenly detected as a spilled object is filtered out, and the target spilled object is finally determined. The technical problem of easy false detection and low accuracy in traditional detection schemes is solved, and the detection accuracy is improved.

[0043] Figure 3 is a schematic diagram of a method for detecting spilled objects according to another embodiment of the present application, such as Figure 3 As shown in the figure, after the camera's dump truck spillage and dripping detection function is turned on, the image in the surveillance video is obtained, and after image preprocessing processes such as filtering and denoising, the image is input into the first deep learning model and the second deep learning model for target detection respectively. The image preprocessing process can improve the clarity and contrast of the image, thereby improving the accuracy of target recognition.

[0044] The first deep learning model is used to detect common moving target objects such as pedestrians, motor vehicles, and non-motor vehicles in traffic road scenes. For motor vehicle targets, the first deep learning model will further distinguish the types of motor vehicles, such as cars, vans, buses, muck trucks, concrete mixer trucks, engineering vehicles, etc. The output results of the first deep learning model include the rectangular frame coordinates of the moving target and the confidence of the target category. The first deep learning model is trained based on the first sample image and the annotated labels of the first sample image. The first sample image is: traffic road scene image. The annotated labels are: pedestrians, various types of motor vehicles, and non-motor vehicles. Based on the first deep learning model, common moving target objects in road scenes can be quickly and accurately identified. And the first deep learning model can distinguish the types of motor vehicles, which is conducive to the subsequent association of different types of vehicles with scattered objects.

[0045] The second deep learning model is used to detect spilled objects in road scenes, as well as negative sample categories with similar morphological appearance to the spilled objects. The output results of the second deep learning model include the target detection frame coordinates and target category confidence of the suspected spilled objects, and the second deep learning model is trained based on the second sample image and the annotated labels of the second sample image. The second sample image is: a traffic road scene image. The annotated labels are: multiple types of spilled objects and multiple types of negative samples. The types of spilled objects include: soil, gravel, sewage stains, building materials, construction waste, etc. The negative sample types include: sunlight, lights, stagnant water, water stains, rain and snow, fallen leaves, lane lines, damaged road surfaces, car exhaust, smoke, ordinary garbage, etc. The second deep learning model of this embodiment is trained together with road spilled objects and multiple common negative sample categories. Compared with the recognition model that only uses spilled objects for training, the second deep learning model of this embodiment can greatly improve the accuracy and recall rate of identifying spilled objects.

[0046] In this embodiment, the network structures of the first deep learning model and the second deep learning model for target detection are not specifically limited. The network structures can be flexibly configured in combination with project requirements and hardware conditions, and appropriate network structures can be selected, such as FasterRCNN series, YOLO series, SSD series, CenterNet series, DETR series, etc.

[0047] The existing method for identifying spilled objects uses a binary classification model to train a single spilled object model. However, in real traffic road scenarios, due to the interference of multiple complex factors such as time, weather, lighting, and surrounding environment, multiple negative sample categories are very similar to the spilled object category in morphology and appearance, making it difficult for a single-category model to distinguish them, and there are a large number of false detections. Therefore, a multi-classification model is trained using the spilled object category and multiple negative sample categories, and the powerful learning ability of the neural network allows the model to adaptively learn and distinguish different categories. Compared with a single-class model that only uses spilled objects for training, the accuracy and recall rate of spilled object recognition have been greatly improved, achieving the technical effect of accurately identifying spilled objects.

[0048] In one embodiment, a confidence threshold is pre-configured for the deep learning model. When the confidence of the detection target of the first deep learning model exceeds the first confidence threshold, a first background mask map, i.e., a first mask map, is constructed based on the rectangular coordinate frames of all moving targets that exceed the first confidence threshold. The first mask map is a binary image of the same size as the input image. When constructing the first mask map, the values ​​within the coordinate frames of all moving targets on the map are filled with 1, and the values ​​of the remaining positions are filled with 0. The first mask map represents the position and range of all moving targets in the current input image frame. By setting the first confidence threshold, the initial filtering of the detection target is achieved, and the first mask map is constructed based on the rectangular coordinate frames of the moving targets with higher credibility, which is conducive to reducing the amount of calculation and improving the efficiency of scattered object recognition. Further, by the initial filtering of the detection target, the erroneous detection of scattered objects detected by pedestrians, motor vehicles, non-motor vehicles and other targets themselves is removed, and the scattered object recognition of the detection target with lower credibility is avoided, which can reduce the probability of false detection at the source.

[0049] In one embodiment, a moving target detection frame obtained based on the first position information is marked on the first mask map, and determining whether the suspected spilled object is a target spilled object based on the first mask map and the second position information includes: determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; if the suspected spilled object is outside the moving target detection frame, determining it to be a target spilled object.

[0050] Specifically, when the confidence of the detection target of the second deep learning model exceeds the corresponding second confidence threshold, for all targets predicted to be of the type of spilled objects, the first mask map is used to determine whether they are within the range of the moving target. The calculation method is: according to the coordinates of the detection frame of the suspected spilled object, the coordinates of the center point of the detection frame of the suspected spilled object are calculated, and the coordinates of the center point are mapped to the first mask map. If the value on the first mask map is 1, it indicates that the suspected spilled object is within the range of the moving target, and it is the dirt of the moving target itself, not a spilled object, and this part of the suspected spilled object detection target should be filtered and ignored. If the value on the first mask map is 0, it indicates that the suspected spilled object is located in the background of the input image, and it can be determined as a target spilled object, and subsequent analysis is performed on it. Through the pre-established first mask map, the complex position relationship calculation process is converted into a code value comparison process between the center point of the suspected spilled object and the center point of the moving target detection frame. It greatly reduces the amount of calculation and improves the efficiency of spilled object recognition.

[0051] In one embodiment, a moving target detection frame obtained based on the first position information is marked on the first mask map, and determining whether the suspected spilled object is a target spilled object based on the first mask map and the second position information also includes: determining a first moving target based on the type of the moving target, the first moving target including a dump truck; determining a motion trajectory of the first moving target in multiple frames of the image to be detected based on a target tracking algorithm; determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; determining whether the suspected spilled object is located within the range of the motion trajectory based on the second position information; if the suspected spilled object is outside the moving target detection frame and within the range of the motion trajectory, it is determined to be a target spilled object, and the target spilled object is associated with the first moving target.

[0052] Specifically, in this implementation, a dump truck is used as the first moving target for spilled object identification. In an actual detection scenario, the first moving target may also be a concrete mixer truck or other vehicle type that may generate spilled objects. All dump trucks that exceed the first confidence threshold are tracked using a target tracking algorithm, and the trajectory point information of all dump truck targets is saved and recorded. The trajectory point information is used to characterize the motion trajectory of the dump truck in multiple frames of images to be detected. The trajectory point information is: the position coordinates of the moving target detection box of the dump truck on each input image frame.

[0053] The coordinates of the moving target detection frame of the muck truck in multiple frames are used to construct the connected area and generate the motion trajectory mask of the muck truck in the view of the surveillance camera. The construction process of the motion trajectory mask is similar to that of the first mask. The positions corresponding to the moving target detection frames of the muck truck in multiple frames on the motion trajectory mask are filled with 1, and the remaining positions are filled with 0.

[0054] Then, according to the coordinates of the suspected spilled object detection frame, the coordinates of the center point of the suspected spilled object detection frame are calculated, and the coordinates of the center point are mapped to the motion trajectory mask map of the muck truck.

[0055] If the value on the motion trajectory mask is 1, it means that the suspected spilled object is within the motion trajectory of the muck truck and can be determined as the target spilled object. The suspected spilled object is associated with the muck truck, so that the muck truck corresponding to the suspected spilled object can be found.

[0056] If the value on the motion trajectory mask is 0, it means that the suspected spilled object is not within the motion trajectory of the current muck truck. If all muck trucks are traversed and the suspected spilled object is not within their motion trajectory, it means that the suspected spilled object is a false detection, and its false detection information is recorded.

[0057] It should be emphasized that in the embodiments of the present application, the present application does not make any specific limitations on the algorithm for target tracking. It can be flexibly set in combination with project requirements and hardware conditions to select a suitable method, such as the traditional meanshift algorithm, target tracking based on Kalman filtering, target tracking based on particle filtering, contour-based tracking, centroid tracking algorithm, target tracking based on motion target modeling, or FlowTrack based on deep learning, etc.

[0058] Through the above steps, this embodiment completes the judgment of whether the suspected spilled objects are within the motion trajectory of the dump truck by calculating the trajectory point information of the dump truck target and constructing the motion trajectory mask map. The technical effect of associating the spilled objects with the dump truck and locating and tracking the vehicle that generated the spilled objects is achieved. Based on the association results, the relevant departments can get rid of the dilemma of only being able to passively clean up the roads, actively find the vehicles that caused the spillage and educate and punish them, and solve the road spillage problem from the root.

[0059] In one embodiment, a moving target detection frame obtained based on the first position information is marked on the first mask map, and determining whether the suspected spilled object is a target spilled object based on the first mask map and the second position information further includes: acquiring a reference image, wherein the reference image is a historical image of the image to be detected; determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; determining a first similarity between the image to be detected and the reference image; if the suspected spilled object is outside the moving target detection frame and the first similarity is less than a first preset threshold, determining that the suspected spilled object is a target spilled object.

[0060] Specifically, the image of the area corresponding to the suspected spilled object and the historical image of the image to be detected in the area are input into the third deep learning model to calculate the similarity between the two. The third deep learning model is a twin network, which receives two input images at the same time, makes the two images enter two different network branches to extract features, extracts feature vectors from the two sub-networks, performs similarity calculation, and outputs the first similarity of the two images.

[0061] A first similarity threshold, i.e., a first preset threshold, is pre-configured, and its specific value can be adjusted according to the specific situation, for example, it is set to 0.7. This is not specifically limited in the present embodiment. When the first similarity of the two images is less than or equal to the first preset threshold, the suspected spilled object is determined as the target spilled object, and it is considered that there is a new spilled object in the picture frame currently taken by the surveillance camera, and the first warning information recording the spilled object is saved. The first warning information includes the first detection time of the spilled object, the detection rectangular frame position coordinates, the detection confidence, the first similarity, and the cumulative number of detected frames.

[0062] If the first similarity between the two images is higher than a first preset threshold, it is considered that the suspected spilled object is an object already spilled on the road surface, or is a misdetected spilled object, and its misdetection information is recorded.

[0063] Through the above steps, the present application introduces historical data as support in the process of determining the target spilled objects, and compares the historical images with the images of the corresponding areas of the current suspected spilled objects through the third deep learning model, i.e., the twin network model, to determine whether the currently detected suspected spilled objects are new spilled objects, and only file a case for new spilled objects. In addition, the present application can correct the possible false detection of the second deep learning model, i.e., further identify and judge the situation in which the second deep learning model identifies ordinary road surfaces as target spilled objects. It can accurately determine whether the detected spilled objects are new spilled objects, and avoid false alarms of spilled objects.

[0064] In one embodiment, the acquiring of the reference image comprises: determining a background area image based on the first mask image of a preset number of frames and the second position information; and determining a reference image based on the background area image.

[0065] Specifically, on the basis of the first mask map, the rectangular coordinate frame area corresponding to all the spilled objects recorded as non-false detections is filled with 1, thereby obtaining a second background mask map. Points with a value of 1 on the second background mask map indicate that there is a target to be analyzed at this position and it does not belong to the background image. According to all points with a position of 0 on the second background mask map, the pixel value is copied from the input image frame, and the image frame number of the point is recorded and updated, thereby generating a second background map, that is, a background area image. For the spilled objects within the movement trajectory of the muck truck, according to its spilled object detection frame, the image is cut out on the background area image to obtain a historical image of the spilled objects in the image, that is, a reference image. The similarity between the image of the area corresponding to the current spilled object and the reference image is obtained. If the first similarity of the two images is higher than the first preset threshold, it is considered that the spilled object is an existing spilled object on the road surface, or a falsely detected spilled object, and its false detection information is recorded. After the current frame is analyzed, the false detection information is updated to the background area image.

[0066] In one embodiment, the image frame number of the point on the background area image that has not been copied to the pixel value is filled with -1, indicating that the background of the area has not been established. The background area image always exists in the spilled object detection function and is continuously updated with the input of the image frame. Each point can be set to be updated after a certain period of time, and the interval time can be 10 minutes. The specific value can be adjusted according to the actual application. By setting the update interval time, it is possible to avoid frequent updates of the background area image that consume a lot of time.

[0067] It should be noted that in order to ensure that the background area image is not frequently blocked by moving targets and spilled objects, the background is not established. In the first N frames after the spilled object detection function is turned on, for example, the first 30 frames, no spilled object alarm is performed, and it is only used to build the background area image. At this time, all detected spilled objects are ignored and considered to be existing spilled objects, and no alarm is triggered. This is conducive to improving the stability of background establishment and avoiding false alarms when the spilled object detection function is turned on.

[0068] If a spilled object eventually triggers a spilled object alarm, the image range corresponding to the spilled object needs to be updated synchronously to the background area image, indicating that the area has triggered a spilled object alarm, avoiding subsequent repeated alarms for the spilled object.

[0069] In one embodiment, a moving target detection frame obtained based on the first position information is marked on the first mask map, and determining whether the suspected spilled object is a target spilled object based on the first mask map and the second position information further includes: obtaining historical position information of historical spilled objects; determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; determining an intersection-and-union ratio of an image area corresponding to the second position information and an image area corresponding to the historical position information; if the suspected spilled object is outside the moving target detection frame and the intersection-and-union ratio is less than a second preset threshold, determining that the suspected spilled object is a target spilled object.

[0070] Specifically, the dump truck spillage and dripping detection function continues to detect subsequent image frames of the surveillance camera. If new spilled objects continue to be detected in subsequent image frames, and the overlap rate with the rectangular frame of the spilled object in the saved first warning information exceeds a threshold, for example, the threshold is set to 0.6, it is considered to be the same spilled object, and the number of detection frames of the first warning information of the spilled object is accumulated, and the rectangular frame coordinates, detection confidence, and first similarity are updated at the same time. The overlap rate of the newly created rectangular frame of the spilled object and the saved spilled object is calculated according to the following intersection-and-union ratio formula:

[0071]

[0072] In the formula, Area1 and Area2 represent the areas of two rectangular boxes respectively. The numerator represents the intersection of the areas of the two rectangular boxes, that is, the overlapping area of ​​the two rectangular boxes. The denominator is the union of the areas of the two rectangular boxes, that is, Area1+Area2-Area1∩Area2.

[0073] In one embodiment, a moving target detection frame obtained based on the first position information is marked on the first mask image, and the determination of whether the suspected spilled object is the target spilled object based on the first mask image and the second position information also includes: obtaining a target spilled object detection result of the image to be detected for a preset frame number; determining the first frame number in which the target spilled object exists in the preset frame number based on the target spilled object detection result; and if the ratio of the first frame number to the preset frame number is greater than a second preset threshold, an alarm is issued.

[0074] Since changes in light and shadow, water stains, or other debris on the road surface in a short period of time are usually misdetected as spilled objects, or some inconspicuous spilled objects disappear naturally after staying for a period of time, the relevant departments are more concerned about the spilled objects that stay on the road surface for a long time and cause continuous pollution. Therefore, for the spilled objects in the first warning state, if after a period of time, for example, after an interval of 30 minutes, the cumulative proportion of detected frames in the first warning information of the spilled object exceeds the first warning threshold, for example, the first warning threshold is set to 0.8, then it is finally determined to be a new spilled object that has stayed for a long time. The cumulative number of detected frames is the first frame number, and the first warning threshold is the second preset threshold. The spilled object and the dump truck associated with the spilled object will be alarmed for not reporting new spilled objects. The cumulative proportion of detected frames is calculated according to the following formula:

[0075]

[0076] If the cumulative number of detected frames in the first warning information of the spilled object does not exceed the first warning threshold, it is considered that the spilled object is a false detection or has disappeared naturally in a short period of time. After the current frame is analyzed, the area of ​​the spilled object that has been alarmed and the area of ​​the spilled object that does not meet the alarm will be updated to the background area image to avoid subsequent repeated alarms for the spilled object.

[0077] The first warning information accumulated over a long period of time and in multiple frames is used to ultimately determine whether the detected spilled objects are real ones, and to eliminate false detections of targets such as light and shadow changes, water stains, and other debris on the road in a short period of time, as well as spilled objects that disappear naturally in a short period of time, thereby improving the reliability and accuracy of the spilled object alarm.

[0078] Through the above steps, the spilled object detection method of the present application, by comprehensively utilizing multiple technical means such as the second deep learning model, the first mask image, the third deep learning model, the background area image, and the first warning information accumulated over a long period of time, has greatly improved the accuracy of the alarm time for spillage and dripping of the muck truck. In addition, by utilizing the location information of the spilled object and the movement trajectory information of the muck truck, the association between the spilled object and the muck truck where the spillage occurred is achieved, which is helpful for assisting law enforcement departments to quickly find the muck truck that caused the accident.

[0079] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0080] In this embodiment, a spilled object detection device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit" and the like used below can implement a combination of software and / or hardware of predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0081] Figure 4 is a structural block diagram of a spilled object detection device according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:

[0082] A first detection module 10 is used to detect a moving target in an image to be detected, obtain first position information of the moving target, and establish a first mask image based on the first position information;

[0083] The second detection module 20 is used to detect the suspected spilled object in the image to be detected and obtain the second position information of the suspected spilled object;

[0084] The target determination module 30 is used to determine whether the suspected spilled object is a target spilled object based on the first mask image and the second position information.

[0085] The target determination module 30 is further used to determine whether the suspected spilled object is located within the moving target detection frame based on the second position information; if the suspected spilled object is located outside the moving target detection frame, it is determined to be a target spilled object.

[0086] The target determination module 30 is also used to determine a first moving target based on the type of the moving target, the first moving target including a dump truck; determine a motion trajectory of the first moving target in multiple frames of the image to be detected based on a target tracking algorithm; determine whether the suspected spilled object is located within the moving target detection frame based on the second position information; determine whether the suspected spilled object is located within the range of the motion trajectory based on the second position information; if the suspected spilled object is outside the moving target detection frame and within the range of the motion trajectory, it is determined to be a target spilled object, and the target spilled object is associated with the first moving target.

[0087] The target determination module 30 is also used to obtain a reference image, which is a historical image of the image to be detected; determine whether the suspected spilled object is located within the moving target detection frame based on the second position information; determine a first similarity between the image to be detected and the reference image; if the suspected spilled object is located outside the moving target detection frame and the first similarity is less than a first preset threshold, determine that the suspected spilled object is a target spilled object.

[0088] The target determination module 30 is further configured to determine a background area image based on the first mask image of a preset number of frames and the second position information; and determine a reference image based on the background area image.

[0089] The target determination module 30 is also used to obtain historical position information of historical spilled objects; determine whether the suspected spilled object is located within the moving target detection frame based on the second position information; determine the intersection and union ratio of the image area corresponding to the second position information and the image area corresponding to the historical position information; if the suspected spilled object is located outside the moving target detection frame and the intersection and union ratio is less than a second preset threshold, then determine that the suspected spilled object is a target spilled object.

[0090] The target determination module 30 is also used to obtain the target scattered object detection result of the image to be detected of a preset frame number; determine the first frame number in the preset frame number where the target scattered object exists based on the target scattered object detection result; if the ratio of the first frame number to the preset frame number is greater than a second preset threshold, an alarm is issued.

[0091] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0092] In this embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0093] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0094] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0095] S1, detecting a moving target in an image to be detected, obtaining first position information of the moving target, and establishing a first mask image based on the first position information.

[0096] S2, detecting a suspected spilled object in the image to be detected, and obtaining second position information of the suspected spilled object.

[0097] S3: Determine whether the suspected spilled object is a target spilled object based on the first mask image and the second position information.

[0098] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0099] In addition, in combination with the spilled object detection method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any of the spilled object detection methods in the above embodiments is implemented.

[0100] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0101] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0102] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0103] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A method for detecting spilled objects, characterized in that: include: Detecting a moving target in an image to be detected, obtaining first position information of the moving target, and establishing a first mask image based on the first position information; The first mask image is a binary image of the same size as the input image; Detecting a suspected spilled object in the image to be detected, and obtaining second position information of the suspected spilled object; Determining whether the suspected spilled object is a target spilled object based on the first mask image and the second position information; The determining whether the suspected spilled object is a target spilled object based on the first mask image and the second position information includes: It is determined whether the suspected spilled object is the target spilled object based on the positional relationship between the moving target in the first mask image and the second position information of the suspected spilled object.

2. The method for detecting spilled objects according to claim 1, characterized in that: The first mask image is marked with a moving target detection frame obtained based on the first position information, and the determining whether the suspected spilled object is a target spilled object based on the first mask image and the second position information includes: Determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; If the suspected spilled object is outside the moving target detection frame, it is determined to be the target spilled object.

3. The method for detecting spilled objects according to claim 1, characterized in that: The first mask image is marked with a moving target detection frame obtained based on the first position information, and the determining whether the suspected spilled object is a target spilled object based on the first mask image and the second position information further includes: Determine a first moving target based on the type of the moving target, wherein the first moving target includes a muck truck; Determine the motion trajectory of the first moving target in the multiple frames of the to-be-detected images based on a target tracking algorithm; Determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; Determining whether the suspected spilled object is within the range of the motion trajectory based on the second position information; If the suspected spilled object is outside the moving target detection frame and within the range of the motion trajectory, it is determined to be a target spilled object, and the target spilled object is associated with the first moving target.

4. The method for detecting spilled objects according to claim 1, characterized in that: The first mask image is marked with a moving target detection frame obtained based on the first position information, and the determining whether the suspected spilled object is a target spilled object based on the first mask image and the second position information further includes: Acquire a reference image, where the reference image is a historical image of the image to be detected; Determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; Determining a first similarity between the image to be detected and the reference image; If the suspected spilled object is outside the moving target detection frame and the first similarity is less than a first preset threshold, the suspected spilled object is determined to be a target spilled object.

5. The method for detecting spilled objects according to claim 4, characterized in that: The obtaining of the reference image comprises: Determine a background area image based on the first mask image of a preset number of frames and the second position information; A reference image is determined based on the background area image.

6. The method for detecting spilled objects according to claim 1, characterized in that: The first mask image is marked with a moving target detection frame obtained based on the first position information, and the determining whether the suspected spilled object is a target spilled object based on the first mask image and the second position information further includes: Get the historical location information of historical spills; Determining whether the suspected spilled object is located within the moving target detection frame based on the second position information; Determine an intersection-and-union ratio of an image area corresponding to the second position information and an image area corresponding to the historical position information; If the suspected scattered object is outside the moving target detection frame and the intersection-over-union ratio is less than a second preset threshold, the suspected scattered object is determined to be a target scattered object.

7. The method for detecting spilled objects according to claim 1, characterized in that: The first mask image is marked with a moving target detection frame obtained based on the first position information, and the method further includes: determining whether the suspected spilled object is a target spilled object based on the first mask image and the second position information: Obtaining target spilled object detection results of the to-be-detected images of a preset number of frames; Determine the first frame number in which the target scattered object exists in the preset frame number based on the target scattered object detection result; If the ratio of the first frame number to the preset frame number is greater than a second preset threshold, an alarm is generated.

8. A spilled object detection device, characterized in that: include: A first detection module, used for detecting a moving target in an image to be detected, obtaining first position information of the moving target, and establishing a first mask image based on the first position information; The first mask image is a binary image of the same size as the input image; A second detection module is used to detect the suspected spilled object in the image to be detected and obtain second position information of the suspected spilled object; a target determination module, configured to determine whether the suspected spilled object is a target spilled object based on the first mask image and the second position information; The target determination module is further used to determine whether the suspected spilled object is the target spilled object based on the positional relationship between the moving target in the first mask image and the second position information of the suspected spilled object.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the spilled object detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the spilled object detection method according to any one of claims 1 to 7 are implemented.

Images