Methods, devices, electronic equipment, storage media and products for identifying spilled debris

Abstract

This disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for identifying litter, relating to the field of image processing technology, and particularly to the field of intelligent transportation technology. One embodiment of the litter identification method involves: determining a foreground target in the image to be identified; filtering the foreground target to obtain a suspected target; identifying traffic elements and road surface elements on the suspected target; and determining that the suspected target is litter if it is neither a traffic element nor a road surface element, thereby effectively improving the accuracy of litter identification.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, in particular to the technical field of intelligent transportation, and more particularly to a litter identification method and device, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] Road congestion, traffic accidents, and environmental pollution are common problems in the field of road transportation. In particular, traffic accidents caused by litter on highways are one of the important reasons for current traffic accidents.

[0003] Road litter identification plays a very important role in the field of traffic safety. If the litter on the highway is not handled in a timely manner, it can easily cause serious traffic accidents. Therefore, it is necessary to monitor this event, report and handle the litter in a timely manner, and restore normal operation of the road. SUMMARY

[0004] The present disclosure provides a litter identification method and device, an electronic device, a computer readable storage medium, and a computer program product, which effectively improves the accuracy of litter identification.

[0005] According to an aspect of the present disclosure, a litter identification method is provided, which includes:

[0006] determining a foreground target in a to-be-identified image; filtering the foreground target to obtain a suspected target; performing traffic element identification and road surface element identification on the suspected target; and in response to the suspected target not being a traffic element and not being a road surface element, determining that the suspected target is litter.

[0007] According to another aspect of the present disclosure, a litter identification device is provided, which includes:

[0008] a foreground target determination module configured to determine a foreground target in a to-be-identified image; a filtering module configured to filter the foreground target to obtain a suspected target; an identification module configured to perform traffic element identification and road surface element identification on the suspected target; and a litter determination module configured to determine that the suspected target is litter in response to the suspected target not being a traffic element and not being a road surface element.

[0009] According to still another aspect of the present disclosure, an electronic device is provided, which includes at least one processor, and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the aforementioned litter identification method.

[0010] According to still another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the aforementioned litter identification method.

[0011] According to still another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the aforementioned litter identification method.

[0012] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are used to better understand the present scheme, and do not constitute a limitation on the present disclosure. Among them:

[0014] Figure 1 is an exemplary system architecture diagram to which an embodiment of the present application can be applied;

[0015] Figure 2 is a flowchart of an embodiment of the litter identification method according to the present application;

[0016] Figure 3 is a schematic diagram of a foreground target according to an embodiment of the present application;

[0017] Figure 4 is a flowchart of still another embodiment of the litter identification method according to the present application;

[0018] Figures 5a-5c is a schematic diagram of an image to be identified, a foreground image and a foreground target according to an embodiment of the present application;

[0019] Figure 6 is a flowchart of still another embodiment of the litter identification method according to the present application;

[0020] Figures 7a-7b is a schematic diagram of a bounding box region and an extended target according to an embodiment of the present application;

[0021] Figure 8 is a schematic diagram of feature extraction and feature splicing according to an embodiment of the present application;

[0022] Figure 9 is a flowchart of an application scenario of the litter identification method of the present application;

[0023] Figure 10 is a structural schematic diagram of an embodiment of the litter identification device according to the present application;

[0024] Figure 11 is a block diagram of an electronic device for implementing the litter identification method of the embodiments of the present application. DETAILED DESCRIPTION

[0025] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, in which various specific details are set forth to assist in a thorough understanding of the embodiments of the present disclosure. It will be understood by those of ordinary skill in the art that various changes in the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, the description is made in the order of the accompanying drawings for clarity and conciseness.

[0026] Figure 1 An exemplary system architecture 100 to which embodiments of the litter identification method or litter identification apparatus of the present application can be applied is shown.

[0027] As shown in FIG. 1, the system architecture 100 can include a camera 101, a network 102, and a server 103. The network 102 is a medium to provide a communication link between the camera 101 and the server 103. The network 102 can include various connection types, such as wired, wireless communication links, etc. Figure 1 A user can use the camera 101 to interact with the server 103 through the network 102 to receive or send messages, etc.

[0028] The camera 101 refers to a camera used for vehicle and road monitoring in general. It can be an electronic police that takes a snapshot of a vehicle violating a law (e.g., crossing a solid line, driving against the direction, occupying a non-motor vehicle lane, not driving according to a guide sign, running a red light, etc.) at an intersection. It can also be a toll camera installed at some key sections of expressways, provincial roads, and national roads to take a snapshot of a vehicle driving at a speed exceeding a limit. The camera 101 can also be a camera for taking a snapshot of a vehicle parked in violation of a law, a traffic monitoring camera, a sky net monitoring camera, a flow monitoring camera, etc.

[0029] The server 103 can be a server providing various services, such as a background analysis server that provides analysis of road images collected by the camera 101. The background analysis server can perform analysis on the received road images, etc., and output a result of the processing (e.g., whether there is litter on the road).

[0030]

[0031] ​It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or as a single software or software module. This is not specifically limited here.

[0032] It should be noted that the litter identification method provided in the embodiments of the present application is generally executed by the server 103, and correspondingly, the litter identification apparatus is generally arranged in the server 103.

[0033] It should be understood that Figure 1 The number of cameras, networks and servers in the above is only illustrative. According to the needs of implementation, there can be any number of cameras, networks and servers.

[0034] With reference to Figure 2 , a flow 200 of one embodiment of the litter identification method according to the present application is shown. The litter identification method comprises the following steps:

[0035] Step 201, determining a foreground target in a to-be-identified image.

[0036] This step aims to determine a foreground target in a to-be-identified image by the execution subject of the litter identification method (for example, the server 103 shown in Figure 1 ). The to-be-identified image is a road video image collected by a camera, which can be a static image or a dynamic image, and the present application does not limit this; the way the camera collects the road video image can include one or more of any combination of timing collection, real-time collection, panoramic collection, and slice collection. Figure 1 In addition, it should be understood that the foreground target refers to a person or a scene in front of the subject or close to the lens, for example, a pedestrian, a vehicle, a road, a lane line, a guardrail, a barrier, a plant, a cone barrel, a road sign, soil, a soil slope or litter, etc.

[0037] For example, the foreground target is a vehicle 301, litter 302, a lane line 303, a tree 304, a road sign 305, 306, etc. Figure 3

[0038] It should be noted that the foreground target can be determined based on manual annotation or extracted by background modeling, and the present application does not limit this.

[0039] Step 202, filtering the foreground target to obtain a suspected target.

[0040] ​It should be understood that the foreground target in step 201 contains a large number of non-spraying objects, so this step aims to filter the foreground target to obtain a suspected target, that is, only the suspected target is identified as a spraying object, thereby improving the efficiency and accuracy of spraying object identification.

[0041] In actual application, the foreground target can be filtered according to actual conditions. For example, the foreground target with an area less than an area threshold is removed from the foreground target to obtain a suspected target. Through this method, the foreground target with a small area that does not affect the road condition can be filtered out. For example, the foreground target with a number of continuous image frames less than a frame threshold is removed from the foreground target to obtain a suspected target. Through this method, the foreground target that self-drifts (i.e., does not stay on the road surface) and does not affect the road condition can be filtered out. For example, the foreground target with a coincidence rate greater than a coincidence threshold with a tree, a vehicle, etc. can also be removed from the foreground target to obtain a suspected target. Through this method, the foreground target in a static state such as a green belt, a street lamp, and a parked vehicle can be filtered out.

[0042] Step 203, traffic element identification and road surface element identification are performed on the suspected target.

[0043] It should be understood that step 202 only performs preliminary filtering on the foreground target, so after obtaining the suspected target, in order to further improve the accuracy of spraying object identification, this step aims to perform traffic element identification and road surface element identification on the suspected target. Among them:

[0044] Traffic element identification is used to determine whether the suspected target is a traffic element, wherein the traffic element includes a pedestrian, a vehicle, and an easily confused object. If the suspected target is identified as a traffic element, it is determined that the suspected target is not a spraying object. For example, referring to Figure 3 Traffic element identification is performed on the suspected target 301, and the identification result is a vehicle. The vehicle belongs to a traffic element, so the suspected target 301 is not a spraying object.

[0045] Road surface element identification is used to determine whether the suspected target is a road surface element, wherein the road surface element includes a road surface and an easily confused object. If the suspected target is identified as a road surface element, it is determined that the suspected target is not a spraying object. For example, referring to Figure 3 Road surface element identification is performed on the suspected target 303, and the identification result is a lane line. It should be understood that the lane line belongs to a road surface, so the suspected target 303 is not a spraying object.

[0046] It should be noted that the easily confused object in the embodiment is an object that is easily missed or misdetected in actual application, and the easily confused object includes at least one of rain and snow, fixed traffic facilities, and trees and plants, wherein the rain and snow can be rainwater or accumulated snow at a position such as a road surface or a vehicle; the fixed traffic facilities can be a rainwater grate, a guardrail, a separation belt, a conical bucket, or a road sign; and the trees and plants are green belts or trees arranged on a roadside. The embodiment increases the identification and judgment of the easily confused object in the process of traffic element identification and road surface element identification, so that the missed detection and misdetected problems caused by the easily confused object can be reduced, and the accuracy of the litter identification can be improved.

[0047] Step 204, in response to the suspected target not being a traffic element and not being a road surface element, determining that the suspected target is a litter.

[0048] Based on the identification result of the traffic element identification and the road surface element identification of the suspected target in step 203, the step aims to determine the suspected target as a litter on the basis that the suspected target is not a traffic element and not a road surface element. For example, referring to Figure 3 The traffic element identification is performed on the suspected target 302, it is determined that the suspected target 302 is not a traffic element, and then the road surface element identification is performed on the suspected target 302, it is determined that the suspected target 302 is not a road surface element, and thus it is determined that the suspected target 302 is a litter.

[0049] The method provided by the above embodiments of the application can identify a suspected target as a litter when the suspected target is neither a traffic element nor a road surface element, thereby effectively improving the accuracy of the litter identification. In addition, the rain and snow, the trees and plants, and the fixed traffic facilities are used as the easily confused objects in the traffic element identification and the road surface element identification, so that the misidentification in a complex situation can be filtered, the outdoor factors such as weather and brightness are less likely to affect the litter identification, and the efficiency and the accuracy of the road litter identification are further improved.

[0050] Further referring to Figure 4 , another embodiment of a litter identification method is shown. The litter identification method includes the following steps:

[0051] Step 401, separating a foreground image from a to-be-identified image.

[0052] In the embodiment, as shown in Figure 5a , the to-be-identified image is obtained by Figure 1The image consists of a road video image captured by a camera; the foreground image is obtained by extracting the foreground from the image to be identified. This foreground extraction can be achieved using background modeling algorithms, such as Gaussian mixture modeling, which separates the background and foreground of the image to be identified, thus obtaining the foreground image and reducing interference from objects in the background for litter identification. For example, from... Figure 5a The foreground image separated from the image to be identified is shown below. Figure 5b As shown, background that does not belong to the road area is removed from the image, thereby extracting the road area. It should be noted that the background modeling algorithm is existing technology and will not be described in detail here.

[0053] Step 402: Identify the foreground image to obtain the foreground target.

[0054] In this embodiment, foreground image recognition is performed, that is, objects in the foreground image are extracted, and the extracted objects are used as foreground targets. For example, for Figure 5b The foreground image shown is used for identification, and the resulting foreground target is as follows: Figure 5c As shown. Clearly, compared to the previous embodiments, this embodiment, based on the foreground image, can remove objects located in the background of the original image to be identified, such as... Figure 3 Road signs such as 305 and 306 in the road network have been improved, thereby increasing the efficiency and accuracy of debris identification.

[0055] Step 403: Perform cross-union tracking on the foreground target to obtain the cross-union ratio.

[0056] It should be understood that road monitoring area images are used to reflect scene changes at a certain location, and can include static and dynamic objects in the scene. In some embodiments, static objects can be objects that remain in a fixed position for a long time. For example, static objects can include one or any combination of trees, buildings, streetlights, traffic signs, guardrails, display boards, parked vehicles, etc.; dynamic objects can be objects whose positions change continuously within a certain time frequency. For example, one or any combination of moving vehicles, walking people, moving animals, objects carried by the wind (such as fallen leaves, empty cardboard boxes), etc.

[0057] This step aims to perform cross-union (CUI) tracking on foreground targets in the form of dynamic objects. CUI tracking measures the relative size of the overlap between two bounding boxes. For example, it can obtain the ratio of the intersection of the bounding boxes of a foreground target in two adjacent frames. The higher the CUI ratio, the more the object's motion state tends to be static between frames. In other words, based on CUI tracking, it is possible to obtain foreground targets that change from a moving state to a static state. Since these foreground targets conform to the characteristics of the spill process, they can be identified as potential targets.

[0058] Step 404, remove the foreground target with the intersection-over-union value less than the preset ratio threshold value in the foreground target to obtain a suspected target.

[0059] This step aims to remove the foreground target with the intersection-over-union value less than the preset ratio threshold value in the foreground target to obtain a suspected target. In this embodiment, the preset ratio threshold value is, for example, 90%, that is, when the intersection-over-union value is less than 90%, it can be determined that the moving object continues to move and thus leaves the monitoring area image, for example, a vehicle driving, a tree leaf drifting by itself, litter, etc. Since such foreground targets do not affect the road conditions, in order to improve the efficiency and accuracy of litter identification, the foreground target with the intersection-over-union value less than the preset ratio threshold value in the foreground target should be removed to obtain a suspected target.

[0060] In some optional manners of this embodiment, a background modeling algorithm can also be used to detect the moving foreground, and the intersection-over-union is used for tracking. If the foreground target continues to match the result of the previous motion detection for more than a preset number threshold value, it is considered that the foreground target stops and does not move, and it is considered that the foreground target is a suspected target.

[0061] Step 405, performing traffic element identification on the suspected target based on a pre-trained traffic model.

[0062] In this embodiment, the suspected target can be identified based on the trained traffic model. For example, the traffic model can be a binary classification model, and the traffic elements are used as positive samples, and the litter and the road surface are used as negative samples to train the machine learning model, and the trained traffic model can be obtained.

[0063] Further, the suspected target is input into the trained traffic model, and an output result about the suspected target can be obtained. When the output result is 1, it indicates that the suspected target is a traffic element such as a pedestrian, a vehicle, a rain grate, rain or snow, or a tree; and when the output result is 0, it indicates that the suspected target is a road surface or litter.

[0064] Step 406, performing road surface element identification on the suspected target based on a pre-trained road surface model in response to the suspected target not being a traffic element.

[0065] Based on step 405, when the output result of the traffic model is 0, it indicates that the suspected target is a road surface or a litter, and thus, in order to achieve accurate identification of the litter, this step aims to, in response to the suspected target not being a traffic element, perform road surface element identification on the suspected target based on a pre-trained road surface model. Similar to the traffic model, the road surface model can be a binary classification model, and the road surface element is taken as a positive sample and the litter is taken as a negative sample to train the machine learning model, so as to obtain the trained road surface model. It should be noted that the road surface element can also be taken as a positive sample, and the litter, the pedestrian and the vehicle are taken as negative samples to train the machine learning model, so as to obtain the trained road surface model, which is not limited in the present application.

[0066] Further, the suspected target is input into the trained road surface model, and the output result of the suspected target can be obtained. When the output result is 1, it indicates that the suspected target is a road surface, a rainwater grate, rain and snow, a tree, etc. When the output result is 0, it indicates that the suspected target is a litter.

[0067] Step 407, in response to the suspected target not being a traffic element and not being a road surface element, it is determined that the suspected target is a litter.

[0068] In this embodiment, when the output results of the road surface model and the traffic model are both 1, it indicates that the suspected target is neither a road surface element nor a traffic element, and at this time, it is determined that the suspected target is a litter. In addition, the description of step 407 can also refer to step 204, which will not be repeated here.

[0069] The method provided by the above embodiments of the present application can obtain more accurate foreground targets by extracting foreground images, thereby improving the accuracy of litter identification. On the other hand, by using the method combining traditional images and machine learning, by mining the characteristics of road litter and non-litter, for the difficult cases such as rain, snow and trees, based on the double filtering of the road surface model and the traffic model, multiple identifications of the litter are realized. Compared with the existing technology which only performs one identification, the false negatives or false positives of the litter can be greatly reduced, and through experimental comparison, it is found that the accuracy of litter identification in the embodiments of the present application is improved by nearly 50% compared with the existing technology.

[0070] Further reference is made to Figure 6 which shows another embodiment of a litter identification method flow 600. The litter identification method flow 600 includes the following steps:

[0071] Step 601, determining a foreground target in a to-be-identified image.

[0072] In this embodiment, the description of step 601 refers to step 201 or steps 401-402, which will not be repeated here.

[0073] Step 602: Filter the foreground targets to obtain suspected targets.

[0074] In this embodiment, the description of step 602 is the same as that of step 202 or steps 403-404, and will not be repeated here.

[0075] Step 603: Determine the annotation box area corresponding to the suspected target.

[0076] In this embodiment, with Figure 3 Taking the suspected target corresponding to label 303 as an example, the area marked with a box is the same as the area containing the rectangle corresponding to label 303. (e.g.) Figure 7a (As shown)

[0077] Step 604: Expand the suspected target based on the marked area to obtain the expanded target.

[0078] In this embodiment, after determining the bounding box region, the suspected target can be expanded based on the bounding box region to obtain an expanded target. For example, taking the bounding box region as the center, the bounding box region can be expanded along the four sides or in a certain direction (up, down, left, right) to obtain an expanded target larger than the image size of the bounding box region. Still using... Figure 3 Taking the suspected target corresponding to the code number 303 as an example, the expanded target is as follows: Figure 7b As shown.

[0079] Step 605: Input the suspected target and extended target into the road surface model to identify road surface elements.

[0080] In this embodiment, suspected targets and extended targets are input into the road surface model for road surface element identification. For example... Figure 8 As shown, the road surface model includes a feature extraction network and a classification network. Specifically, after inputting the suspected target and the extended target into the road surface model, firstly, the feature extraction network extracts features from the suspected target and the extended target respectively, obtaining two local features, and then concatenates the two local features to obtain a concatenated feature. Secondly, the concatenated feature is input into the classification network to obtain the output result. Compared with the previous embodiment, this embodiment inputs the suspected target and the extended target into the road surface model and concatenates feature maps of different scales in the depth direction. Based on the concatenated feature, road surface element recognition can be performed, which can improve the accuracy of road surface element recognition.

[0081] Step 606: In response to the suspected target not being a road element, input the suspected target and extended target into the traffic model to perform traffic element identification.

[0082] In this embodiment, when the suspected target is not a road surface element, the suspected target and the extended target are input into the traffic model for traffic element recognition. It should be noted that the traffic model of this embodiment is consistent with the network structure of the road surface model, and it also includes a feature extraction network and a classification network, that is, when performing traffic element recognition, feature extraction and splicing of the suspected target and the extended target can also be realized, and the specific description is referred to the foregoing examples, thereby improving the accuracy of traffic element recognition.

[0083] Step 607, in response to the suspected target not being a traffic element and not being a road surface element, determining that the suspected target is a litter.

[0084] In this embodiment, the description of step 607 is referred to step 204 and step 407, which will not be repeated here.

[0085] Step 608, in response to determining that the suspected target is a litter, determining the litter level of the litter based on a pre-trained litter level model, wherein the litter level is positively correlated with the traffic risk degree.

[0086] In this embodiment, in order to realize more comprehensive litter monitoring, after determining that the suspected target is a litter, the litter level of the litter is determined based on the pre-trained litter level model.

[0087] In this embodiment, the litter level is positively correlated with the traffic risk degree, for example, the litter level includes serious danger, general danger and no danger; wherein the serious danger is used to represent that the litter will cause a traffic accident; the general danger is used to represent that the vehicle can pass safely, but the litter will cause a certain panic and discomfort to the driver; and the no danger is used to represent that the litter will not affect the road conditions, such as paper, leaves, etc.

[0088] It should be noted that the litter level model of this embodiment is consistent with the network structure of the traffic model, and it also includes a feature extraction network and a classification network, that is, when performing road surface element recognition, feature extraction and splicing of the suspected target and the extended target can also be realized, and the specific description is referred to the foregoing examples, thereby improving the accuracy of litter level recognition. Specifically, when the output result is 0, it means that the litter affects the traffic safety and needs to be warned and handled; when the output result is 1, it means that the litter does not affect the traffic safety but causes panic to the driver; and when the output result is 2, it means that the litter does not affect the traffic safety and can be safely driven through.

[0089] It should be noted that, in this embodiment, after obtaining the image to be identified, traffic elements such as pedestrians, vehicles or cones can be removed from the image based on the traffic model. Then, a series of steps such as background modeling, foreground extraction, and determination of foreground targets and suspected targets in steps 601-608 are performed on the processed image to be identified. This method can improve the identification efficiency of litter.

[0090] The method provided in the above embodiments of this application, on the one hand, can improve the accuracy of debris identification by expanding the suspected target, that is, by identifying traffic elements and road elements based on feature maps of different scales; on the other hand, after determining that the suspected target is debris, the method can realize early warning and handling of potential road risks by determining the level of debris, thereby effectively improving the safety of vehicle driving.

[0091] See also Figure 9 , Figure 9 This is a schematic flowchart illustrating an application scenario of the debris identification method according to this embodiment. In practical applications, a camera at a certain location on the road captures an image to be identified. After processing the image by foreground extraction, filtering, and expansion, a suspected target is obtained. The suspected target is input into a traffic model. If the suspected target is a traffic element, it is determined that the suspected target is not debris, and the debris identification process ends. If the suspected target is not a traffic element, it is input into a road surface model for road surface element identification. If the suspected target is identified as a road surface element, it is determined that the suspected target is not debris, and the debris identification ends. If the suspected target is not a traffic element, it is determined that the suspected target is debris, and the debris level is output based on the debris level model.

[0092] Further reference Figure 10 As an implementation of the litter identification method shown in the above figures, this disclosure provides an embodiment of a litter identification device 1000, which is compatible with... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0093] like Figure 10 As shown, the litter identification device 1000 includes:

[0094] The foreground target determination module 1001 is configured to determine foreground targets in the image to be recognized;

[0095] The filtering module 1002 is configured to filter foreground targets to obtain suspected targets;

[0096] The identification module 1003 is configured to identify traffic elements and road surface elements of suspected targets;

[0097] The debris determination module 1004 is configured to determine the suspected target as debris in response to the suspected target not being a traffic element or a road surface element.

[0098] In some alternative embodiments of this example, traffic elements include pedestrians, vehicles, and easily confused objects; road surface elements include road surface and easily confused objects.

[0099] In some alternative embodiments of this embodiment, the confounding factors include at least one of rain and snow, fixed transportation facilities, and tree vegetation.

[0100] In some alternative embodiments of this example, the identification module is further configured as follows:

[0101] Based on a pre-trained traffic model, traffic element identification is performed on suspected targets;

[0102] In response to the fact that the suspected target is not a traffic element, road element identification is performed on the suspected target based on the pre-trained road surface model.

[0103] In some alternative embodiments of this example, the identification module is further configured as follows:

[0104] Based on a pre-trained road surface model, road surface elements are identified for suspected targets.

[0105] In response to the fact that the suspected target is not a road element, traffic element identification is performed on the suspected target based on a pre-trained traffic model.

[0106] In some optional embodiments of this example, the following are also included:

[0107] The annotation box region determination module is configured to determine the annotation box region corresponding to the suspected target;

[0108] The extension module is configured to expand the suspected target based on the labeled area to obtain the expanded target; and

[0109] The identification module is further configured to: input suspected targets and extended targets into the traffic model for traffic element identification; and input suspected targets and extended targets into the road surface model for road surface element identification.

[0110] In some alternative embodiments of this example, wherein,

[0111] The traffic model is obtained by training a machine learning model using traffic elements as positive samples and debris and road surface as negative samples.

[0112] The road surface model is obtained by training a machine learning model using road surface elements as positive samples and debris, pedestrians, and vehicles as negative samples.

[0113] In some optional embodiments of this example, the following are also included:

[0114] The litter level determination module is configured to determine the litter level based on a pre-trained litter level model in response to identifying a suspected target as litter. The litter level is positively correlated with the degree of traffic risk.

[0115] In some alternative embodiments of this example, the foreground target determination module is further configured to:

[0116] Separate the foreground image from the image to be identified;

[0117] The foreground image is identified to obtain the foreground target.

[0118] In some alternative embodiments of this example, the filtering module is further configured as follows:

[0119] The intersection-union ratio (IU) is used to track the foreground targets and obtain the IU ratio.

[0120] Remove foreground targets whose intersection-union ratio is less than a preset threshold to obtain suspected targets.

[0121] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0122] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product, wherein:

[0123] An electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the litter identification method of the foregoing embodiments.

[0124] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the litter identification method of the foregoing embodiments.

[0125] A computer program product includes a computer program that, when executed by a processor, implements the litter recognition method of the foregoing embodiments.

[0126] Figure 11A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0127] like Figure 11 As shown, device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1102 or a computer program loaded from storage unit 1108 into random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. Input / output (I / O) interface 1105 is also connected to bus 1104.

[0128] Multiple components in device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0129] The computing unit 1101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as the litter identification method. For example, in some embodiments, the litter identification method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by the computing unit 1101, one or more steps of the litter identification method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the litter identification method by any other suitable means (e.g., by means of firmware).

[0130] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0131] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0132] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0133] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0134] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0135] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0136] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A litter identification method, comprising: determining a foreground object in a to-be-identified image; filtering the foreground object to obtain a suspected object; determining a bounding box region corresponding to the suspected object; extending the suspected object according to the bounding box region to obtain an extended object; inputting the suspected object and the extended object into a traffic model to perform traffic element identification, wherein the traffic element includes at least one of a pedestrian, a vehicle, and a confusing object, and the confusing object includes rain, snow, a fixed traffic facility, and a tree; in response to the suspected object not being a traffic element, inputting the suspected object and the extended object into a road surface model to perform road surface element identification, wherein the road surface element includes a road surface and a confusing object; in response to the suspected object not being a traffic element or a road surface element, determining that the suspected object is a litter.

2. The method of claim 1, wherein: the traffic model is obtained by training a machine learning model with traffic elements as positive samples and litters and road surfaces as negative samples; the road surface model is obtained by training a machine learning model with road surface elements as positive samples and litters as negative samples.

3. The method of claim 1 or 2, further comprising: in response to determining that the suspected object is a litter, determining a litter level of the litter based on a pre-trained litter level model, wherein the litter level is positively correlated with a traffic risk level. The determination of the foreground object in the to-be-identified image comprises:

4. The method of claim 1 or 2, wherein, separating a foreground image from the to-be-identified image; identifying the foreground image to obtain the foreground object. The filtering of the foreground object to obtain the suspected object comprises:

5. The method of claim 1 or 2, wherein, performing intersection over union tracking on the foreground object to obtain an intersection over union ratio value; removing foreground objects with an intersection over union ratio value less than a preset ratio threshold from the foreground object to obtain the suspected object.

6. A litter identification apparatus, comprising: a foreground object determination module configured to determine a foreground object in a to-be-identified image; a filtering module configured to filter the foreground object to obtain a suspected object; a bounding box region determination module configured to determine a bounding box region corresponding to the suspected object; an extension module configured to extend the suspected object according to the bounding box region to obtain an extended object; an identification module configured to: input the suspected object and the extended object into a traffic model to perform traffic element identification, wherein the traffic element includes at least one of a pedestrian, a vehicle, and a confusing object, and the confusing object includes rain, snow, a fixed traffic facility, and a tree; and in response to the suspected object not being a traffic element, input the suspected object and the extended object into a road surface model to perform road surface element identification, wherein the road surface element includes a road surface and a confusing object; a litter determination module configured to determine that the suspected object is a litter in response to the suspected object not being a traffic element or a road surface element.

7. The apparatus of claim 6, wherein: ​ The traffic model is obtained by training a machine learning model with traffic elements as positive samples, litter and road surface as negative samples, and obtaining a trained traffic model; The road surface model is obtained by training a machine learning model with road surface elements as positive samples and litter as negative samples, and obtaining a trained road surface model.

8. The apparatus of claim 6 or 7, further comprising: a litter level determination module configured to, in response to determining that the suspected target is litter, determine a litter level of the litter based on a pre-trained litter level model, wherein the litter level is positively correlated with the degree of traffic risk.

9. The apparatus of claim 6 or 7, wherein, The foreground target determination module is further configured to: separate a foreground image from the image to be recognized; identify the foreground image to obtain a foreground target.

10. The apparatus of claim 6 or 7, wherein, The filtering module is further configured to: perform intersection over union tracking on the foreground target to obtain an intersection over union ratio; remove the foreground target from the foreground target whose intersection over union ratio is less than a preset ratio threshold to obtain a suspected target.

11. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-5.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-5.

Images