Method and apparatus for training a litter detection model, and electronic device

Abstract

The embodiment of the present disclosure discloses a method and device for training a litter detection model and an electronic device. The method comprises: generating a training sample data set of diverse litters based on pre-collected road image data and litter image data; training a preset neural network model based on the training sample data set, the neural network model comprising a cascaded detection segmentation model and a noise reduction model, to obtain a litter detection model for predicting litter probability. This technical solution not only improves the richness and diversity of litter samples, but also improves the training efficiency of the litter detection model, the detection recall rate and accuracy of the litter, and the automation of litter detection.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of visual image, in particular to a method and device for training a model for detecting a litter and an electronic device. BACKGROUND

[0002] With the development and progress of society, there are more and more vehicles on the road. Litter on the road is a dangerous item that affects driving safety, especially on highways, national roads, provincial roads and other roads where vehicles are allowed to drive at high speed. The road maintenance department usually removes litter through manual inspection by maintenance vehicles. In the process of manual inspection, since litter is a low-frequency disease, the process of finding litter is the most time-consuming, so the algorithm model is needed to realize automatic identification of litter.

[0003] However, since litter is a low-frequency disease and the types of litter are rich, it is difficult to find sufficient and rich litter samples in real life to train the algorithm model, resulting in low accuracy of using the model to identify litter. Therefore, for automatic identification (finding) of litter, it is urgent to provide a new scheme with high identification accuracy. SUMMARY

[0004] The present disclosure provides a method and device for training a model for detecting litter and an electronic device.

[0005] In a first aspect, the present disclosure provides a method for training a model for detecting litter.

[0006] Specifically, the method for training a model for detecting litter includes:

[0007] Based on pre-collected road image data and litter image data, a diverse training sample data set of litter is generated, wherein the training sample data set of litter includes training sample images, detection region identification results of the litter corresponding to the training sample images, litter contour region segmentation results, and probability values of the training sample images being litter images, and the training sample images include training positive sample images and training negative sample images.

[0008] Based on the training sample data set, a preset neural network model is trained, the neural network model includes a cascaded detection segmentation model and a noise reduction model, and a litter detection model for predicting litter probability is obtained.

[0009] In an implementation manner of the present disclosure, the method further includes generating a training positive sample image, including:

[0010] acquire a first road surface collection image, perform semantic segmentation on the first road surface collection image to obtain a drivable area in the first road surface collection image;

[0011] Based on the pre-acquired sample of the scattering object, a diverse sample of the scattering object is obtained through random variation.

[0012] The sample of the scattering object is synthesized with the drivable area in the first road surface collection image to generate the training positive sample image.

[0013] In an implementation manner of the present disclosure, the diverse sample of the scattering object is obtained through random variation based on the pre-acquired sample of the scattering object, including:

[0014] The pre-acquired sample of the scattering object is subjected to multi-class data enhancement processing to obtain a diverse sample of the scattering object corresponding to the sample of the scattering object.

[0015] In an implementation manner of the present disclosure, the pre-set neural network model is trained based on the training sample data set to obtain a scattering object detection model for predicting a scattering object probability, including:

[0016] The training sample image, the detection region recognition result of the scattering object corresponding to the training sample image, and the scattering object contour region segmentation result are taken as the training input data of the detection segmentation model, and the detection segmentation model is trained;

[0017] The training sample image is input into the trained detection segmentation model to obtain the detection region detection result of the scattering object and the scattering object contour region detection result output by the detection segmentation model;

[0018] The annotation result of the detection region detection result of the scattering object output by the detection segmentation model and the annotation result of the scattering object contour region detection result are acquired, and the annotation result includes: a scattering object detection positive sample and a scattering object detection negative sample, a scattering object segmentation positive sample and a scattering object segmentation negative sample;

[0019] The scattering object detection positive sample, the scattering object detection negative sample, the scattering object segmentation positive sample, and the scattering object segmentation negative sample, and the probability value of the training sample image as a scattering object image are taken as the training input data of the noise reduction model, and the noise reduction model is trained;

[0020] The detection segmentation model and the noise reduction model are cascaded to obtain a scattering object detection model for predicting a scattering object probability.

[0021] In an implementation manner of the present disclosure, it further includes:

[0022] The road surface collected image used when the scattering object detection model is used for scattering object detection and the corresponding scattering object detection result are added to the scattering object training sample data set as new training data, and the scattering object detection model is retrained.

[0023] In a second aspect, the embodiments of the present disclosure provide a scattering object detection method.

[0024] Specifically, the scattering object detection method comprises:

[0025] obtaining a road surface collected image to be recognized;

[0026] inputting the road surface collected image to be recognized into the scattering object detection model trained by the training method, to obtain a probability value of the road surface collected image being a scattering object image, and determining that the road surface collected image includes a scattering object when the probability value of the road surface collected image being a scattering object image is greater than or equal to a preset probability threshold.

[0027] In an implementation manner of the present disclosure, when the probability value of the road surface collected image being a scattering object image is greater than or equal to the preset probability threshold, the scattering object detection model further outputs a detection result of a detection region where a scattering object corresponding to the road surface collected image is located and a detection result of a scattering object contour region.

[0028] In a third aspect, the embodiments of the present disclosure provide a scattering object detection model training device.

[0029] Specifically, the scattering object detection model training device comprises:

[0030] a generation module configured to generate a training sample data set of scattering objects with diversity based on pre-collected road surface image data and scattering object image data, wherein the training sample data set of scattering objects comprises training sample images, a detection region recognition result of a scattering object corresponding to the training sample images, a scattering object contour region segmentation result, and a probability value of the training sample images being scattering object images, and the training sample images comprise training positive sample images and training negative sample images;

[0031] a training module configured to train a preset neural network model based on the training sample data set, the neural network model comprising a cascaded detection segmentation model and a noise reduction model, to obtain a scattering object detection model for predicting a scattering object probability.

[0032] In a fourth aspect, the embodiments of the present disclosure provide a scattering object detection device.

[0033] Specifically, the scattering object detection device comprises:

[0034] An acquisition module is configured to acquire a road surface collection image to be recognized;

[0035] A detection module is configured to input the road surface collection image to be recognized into the litter detection model trained by the training device to obtain a probability value of the road surface collection image being a litter image, and determine that the road surface collection image includes litter when the probability value of the road surface collection image being a litter image is greater than or equal to a preset probability threshold.

[0036] In a fifth aspect, an electronic device is provided, including a memory and at least one processor, wherein the memory is configured to store one or more computer instructions, and the one or more computer instructions are executed by the at least one processor to implement the method steps of the litter detection model training method and / or the litter detection method.

[0037] In a sixth aspect, a computer-readable storage medium is provided, configured to store computer instructions for a litter detection device, and the computer instructions are related to the litter detection model training method and / or the litter detection method.

[0038] In a seventh aspect, a computer program product is provided, including computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the method steps of the litter detection model training method and / or the litter detection method.

[0039] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0040] The technical solutions provided by the embodiments of the present disclosure can include the following beneficial effects:

[0041] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0042] Other features, objects and advantages of the present disclosure will become more apparent from the following detailed description of the non-limiting embodiments, taken in conjunction with the accompanying drawings. In the drawings:

[0043] Figure 1A flowchart showing a method of training a litter detection model according to an embodiment of the present disclosure is shown.

[0044] Figure 2 A flowchart showing a litter detection method according to an embodiment of the present disclosure is shown.

[0045] Figure 3 A flowchart showing a litter detection method according to an embodiment of the present disclosure is shown.

[0046] Figure 4 A structural block diagram of a litter detection model training device according to an embodiment of the present disclosure is shown.

[0047] Figure 5 A structural block diagram of a litter detection device according to an embodiment of the present disclosure is shown.

[0048] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0049] Figure 7 A structural schematic diagram of a computer system suitable for implementing a litter detection model training method and / or a litter detection method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so as to be easily implemented by those skilled in the art. Also, portions unrelated to the description of the exemplary embodiments are omitted in the accompanying drawings for the sake of clarity.

[0051] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate that there are features, numbers, steps, actions, components, parts or combinations thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, actions, components, parts or combinations thereof exist or are added.

[0052] It should also be noted that the embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0053] The technical solution provided by the embodiments of the present disclosure provides a litter training sample generation and litter detection model training method, which can not only generate diverse litter training samples, but also achieve efficient and accurate detection of litter by virtue of the joint of the detection segmentation model and the noise reduction model. This technical solution not only improves the richness and diversity of litter samples, but also improves the training efficiency of the litter detection model, improves the detection recall rate and accuracy of the litter, and improves the degree of automation of litter detection.

[0054] Figure 1 A flow chart of a spillage detection model training method according to an embodiment of the present disclosure is shown in FIG. 1. As shown in FIG. 1, the spillage detection model training method includes the following steps S101-S102: Figure 1

[0055] In step S101, a set of training sample data of diverse spillages is generated based on pre-collected road surface image data and spillage image data, wherein the set of training sample data of spillages includes training sample images, detection region recognition results of spillages corresponding to the training sample images, spillage contour region segmentation results, and probability values that the training sample images are spillage images, and the training sample images include training positive sample images and training negative sample images.

[0056] In step S102, a preset neural network model is trained based on the set of training sample data, the neural network model includes a cascaded detection segmentation model and a noise reduction model, and a spillage detection model for predicting spillage probability is obtained.

[0057] As mentioned above, with the development and progress of society, there are more and more vehicles on the road, and spillages on the road belong to dangerous goods that affect driving safety, especially on roads such as expressways, national highways, and provincial highways that allow vehicles to drive at high speed. The influence of spillages on driving safety is greater. The road maintenance department usually removes spillages through artificial inspection by maintenance vehicles. In the process of artificial inspection, since spillages are low-frequency diseases, the process of discovering spillages is the most time-consuming, and therefore automatic identification of spillages is achieved through the algorithm model. However, since spillages are low-frequency diseases and the types of spillages are diverse, it is difficult to find sufficient and diverse spillage samples in real life to train the algorithm model, resulting in a low accuracy rate of using the model to identify spillages. Therefore, there is an urgent need to provide a new solution with a high identification accuracy rate for automatic identification (discovery) of spillages.

[0058] In view of the above defects, in this embodiment, a method for generating training samples of diverse spillages and training a spillage detection model is provided, which not only generates diverse training samples of spillages, but also achieves efficient and accurate detection of spillages by virtue of the joint of the detection segmentation model and the noise reduction model. This technical solution not only improves the richness and diversity of spillage samples, but also improves the training efficiency of the spillage detection model, improves the recall rate and accuracy of spillage detection, and improves the degree of automation of spillage detection.

[0059] ​In an embodiment of the present disclosure, the method can be applied to a computer, a computing device, an electronic device, a server, a service cluster, or the like for training the litter detection model.

[0060] In an embodiment of the present disclosure, the set of training sample data refers to a set of training sample data with diversity, litter, and for training the litter detection model. The set of training sample data includes training sample images, detection region identification results of litters corresponding to the training sample images, litter contour region segmentation results, and probability values of the training sample images being litter images.

[0061] In an embodiment of the present disclosure, the training sample image refers to data as one of the input data for training the litter detection model. The training sample image includes training positive sample images and training negative sample images. The training positive sample image refers to a training sample image containing litter, and the training negative sample image refers to a training sample image not containing litter.

[0062] In an embodiment of the present disclosure, the detection region identification result of the litter corresponding to the training sample image refers to an identification result of identifying a detection region of the litter in the training sample image based on the training sample image. The detection region of the litter can be set according to actual application requirements, and the present disclosure does not make a special limitation thereon. For example, the detection region of the litter can be set as a minimum rectangular region capable of completely surrounding the litter.

[0063] In an embodiment of the present disclosure, the litter contour region segmentation result corresponding to the training sample image refers to a segmentation result of identifying a contour region of the litter in the training sample image based on the training sample image. The contour region of the litter refers to a region surrounded by an outer contour of the litter.

[0064] In addition, as mentioned above, the training positive sample image contains litter, and the training negative sample image does not contain litter. Therefore, for the training positive sample image, the detection region identification result of the litter and the litter contour region segmentation result are results of existing detection region and contour region of the litter, and for the training negative sample image, the detection region identification result of the litter and the litter contour region segmentation result are notification results of being empty or not being able to detect the detection region and the contour region of the litter.

[0065] In an embodiment of the present disclosure, the neural network model refers to a pre-set model that can be used as an initial form of the litter detection model. The neural network model includes a cascaded detection segmentation model and a noise reduction model. The detection segmentation model is used to detect the detection region of the litter and the contour region of the litter. The noise reduction model is used to reduce the detection result of the final litter to improve the accuracy of litter detection.

[0066] In the above embodiment, when training the litter detection model, first, based on the pre-acquired road image data and litter image data, a diverse litter training sample data set is generated, including training sample images, detection region identification results of litters corresponding to the training sample images, contour region segmentation results of litters, and probability values of the training sample images being litter images. Then, based on the training sample data set, a pre-set neural network model including a cascaded detection segmentation model and a noise reduction model is trained, and finally a litter detection model capable of predicting litter probability is obtained. The learning and training of the above litter detection model can be realized by using the learning and training methods mastered by those skilled in the art, and the specific learning and training implementation method of the litter detection is not particularly limited in the present disclosure.

[0067] In an embodiment of the present disclosure, the method can further include generating a training positive sample image. The step of generating a training positive sample image can include the following steps:

[0068] Obtaining a first road surface acquisition image, performing semantic segmentation on the first road surface acquisition image to obtain a road drivable region in the first road surface acquisition image;

[0069] Based on the pre-acquired litter samples, diverse litter samples are obtained by random variation;

[0070] The litter samples and the road drivable region in the first road surface acquisition image are synthesized to generate the training positive sample image.

[0071] In order to obtain a sufficient number of training positive sample images containing litter of various types, in this embodiment, the training positive sample images are generated by sample synthesis. Specifically:

[0072] First, a first road surface acquisition image is obtained. The first road surface acquisition image refers to an acquired road surface image containing one or more road surface elements. The road surface elements can be, for example, drivable road surface, non-drivable road surface, sidewalk, median strip, fence, traffic light, vehicle on the road, zebra crossing, lane line, traffic sign, etc.

[0073] Then semantic segmentation is performed on the first road surface collected image. With the aid of semantic segmentation, semantic classification results corresponding to the road surface elements of all pixels contained in the first road surface collected image can be achieved, such as which road surface element category a certain pixel in the first road surface collected image belongs to. Then, by determining all pixels of the category of drivable road surface, the drivable road surface area in the first road surface collected image can be obtained.

[0074] Then, based on the pre-acquired litter sample, a diverse litter sample is obtained through random variation, wherein the pre-acquired litter sample refers to a pre-acquired sample image containing only litter.

[0075] Then, the litter sample is synthesized with the drivable road surface area in the first road surface collected image, and the training positive sample image is generated. The litter sample can be randomly pasted in different positions in the drivable road surface area of the first road surface collected image in a copy-paste manner.

[0076] In this embodiment, the step of obtaining the litter sample can include the following steps:

[0077] A second road surface collected image is obtained, and the litter area in the second road surface collected image is labeled. The labeled litter area is taken as a first litter sample.

[0078] From the object sample database, object samples that can become litter are screened to obtain a second litter sample.

[0079] The first litter sample and the second litter sample are combined to obtain the litter sample.

[0080] In this embodiment, the litter sample can be obtained in various ways, for example, the litter sample can be obtained based on a collection data approach and a classified data approach. For the collection data approach, a second road surface collection image is first obtained, wherein the second road surface collection image refers to a collected road surface image containing litter, then the region where the litter is located in the second road surface collection image is labeled, and the region where the litter is located obtained by labeling is taken as a first litter sample, wherein the region where the litter is located can be a detection region where the litter is located or a region surrounded by the outer contour of the litter. For the classified data approach, object samples that can become litter are first screened from an object sample database to be taken as second litter samples, wherein the object refers to an object with a certain shape, such as a cup, a book, a mobile phone, a medicine bottle, a carton, an automobile part, a stone, a computer, a household appliance, etc., the object sample database refers to a conventional database for classified objects but not established for litter, that is, the object sample images stored in the object sample database are images of various objects that can be seen in daily life, which include objects that can become litter and objects that are less likely to become litter, in addition, the object sample images stored in the object sample database can be object sample images collected indoors or outdoors. By classifying the object samples in the object sample database, object samples that can become litter can be obtained, for example, the cup, the book, the mobile phone, the medicine bottle, the carton, the automobile part, the stone, etc. in the above object examples are objects that can be thrown and become litter. Finally, the first litter sample and the second litter sample obtained are combined to be the final litter sample.

[0081] In order to further enrich the number and types of litter samples and improve the completeness of the litter training data set and the litter detection model training, in an embodiment of the present disclosure, the obtained litter sample can also be changed randomly to obtain diverse litter samples. The random change can be, for example, random scale transformation, random flipping, adding random noise, random change of RGB channel order, random proportion change, etc. By processing the litter sample based on the random processing method, litter samples after various changes corresponding to the random processing method can be obtained, thereby achieving the purpose of enhancing the diversity of litter samples, effectively improving the completeness of the litter training data set and the litter detection model training, and improving the robustness of the litter detection model training.

[0082] In an embodiment of the present disclosure, the step S102, i.e., training the preset neural network model based on the set of training sample data to obtain a litter detection model for predicting litter probability, can include the following steps:

[0083] training the detection segmentation model by taking the training sample image, the detection region recognition result of the litter corresponding to the training sample image, and the litter contour region segmentation result as training input data of the detection segmentation model;

[0084] inputting the training sample image into the trained detection segmentation model to obtain the detection region detection result of the litter and the litter contour region detection result output by the detection segmentation model;

[0085] obtaining the labeling result of the detection region detection result of the litter output by the detection segmentation model and the labeling result of the litter contour region detection result, wherein the labeling result includes litter detection positive samples and litter detection negative samples, litter segmentation positive samples and litter segmentation negative samples;

[0086] training the noise reduction model by taking the litter detection positive samples, the litter detection negative samples, the litter segmentation positive samples, the litter segmentation negative samples, and the probability value of the training sample image being a litter image as training input data of the noise reduction model;

[0087] concatenating the detection segmentation model and the noise reduction model to obtain the litter detection model for predicting litter probability.

[0088] As mentioned above, the litter detection model includes the detection segmentation model and the noise reduction model, and the detection segmentation model and the noise reduction model are in a concatenated relationship. Therefore, when the litter detection model for predicting litter probability is finally obtained, the detection segmentation model and the noise reduction model need to be concatenated and trained. Specifically:

[0089] First, the training sample image, the detection region recognition result of the litter corresponding to the training sample image, and the litter contour region segmentation result obtained in advance are taken as training input data of the detection segmentation model to train the detection segmentation model, and a trained detection segmentation model is obtained.

[0090] Then, the training sample image is input into the trained detection segmentation model to obtain the detection region detection result of the litter and the litter contour region detection result output by the detection segmentation model. At this time, the training sample image input into the trained detection segmentation model can be the same as or different from the training sample image taken as the training input data of the detection segmentation model.

[0091] Then, the detection region detection result of the detection region where the litter is located output by the detection segmentation model is labeled, and the detection result of the detection region where the litter is located is labeled as a litter detection positive sample, and the detection result of the detection region where the litter is not located is labeled as a litter detection negative sample; similarly, the litter contour region detection result is also labeled, and the detection result of the litter contour region is labeled as a litter segmentation positive sample, and the detection result of the litter contour region is labeled as a litter segmentation negative sample; in this way, two sets of positive and negative samples can be obtained: the litter detection positive sample, the litter detection negative sample, the litter segmentation positive sample, and the litter segmentation negative sample.

[0092] Then, the litter detection positive sample, the litter detection negative sample, the litter segmentation positive sample, and the litter segmentation negative sample, and the probability value of the training sample image being a litter image are used as the training input data of the noise reduction model, the noise reduction model is trained, and a trained noise reduction model is obtained.

[0093] Finally, the detection segmentation model and the noise reduction model obtained by training are cascaded, and the litter detection model is obtained.

[0094] In an embodiment of the present disclosure, the method can further include the following steps:

[0095] The road surface collection image used when the litter detection model is used to detect litter and the corresponding litter detection result are added to the litter training sample data set as new training data, and the litter detection model is retrained.

[0096] The road surface collection image refers to a collected image that may or may not contain litter and is intended to be detected by the litter detection model.

[0097] In order to improve the completeness of the litter training sample data set as the training data of the litter detection model and guarantee the comprehensiveness and completeness of the litter detection learning training result, in this embodiment, a feedback mechanism is used to iteratively train the litter detection model, that is, after the litter detection model is used to detect litter on the road surface collection image, the road surface collection image, the corresponding litter detection region detection result, and the litter contour region detection result are added to the litter training sample data set of the litter detection model as new training input data, and then the litter detection model is retrained to enrich the training data, improve the accuracy of the litter detection model, and obtain a more complete litter detection model for the next litter detection.

[0098] Figure 2 A flow chart of a litter detection method according to an embodiment of the present disclosure is shown in FIG. 2. As shown in FIG. 2, the litter detection method includes the following steps S201-S202: Figure 2

[0099] In step S201, a road surface collection image to be recognized is acquired.

[0100] In step S202, the road surface collection image to be recognized is input into the litter detection model trained by the training method, to obtain a probability value of the road surface collection image being a litter image. When the probability value of the road surface collection image being a litter image is greater than or equal to a preset probability threshold, it is determined that the road surface collection image includes litter.

[0101] As mentioned above, with the development and progress of society, there are more and more vehicles on the road. Litter on the road is a dangerous item that affects driving safety, especially on roads such as expressways, national highways, and provincial highways, where vehicles are allowed to drive at high speed. The influence of litter on driving safety is greater. Road maintenance departments usually remove litter through artificial inspection by maintenance vehicles. In the process of artificial inspection, since litter is a low-frequency disease, the process of discovering litter is the most time-consuming. Therefore, it is necessary to use the algorithm model to realize automatic identification of litter. However, since litter is a low-frequency disease and the types of litter are diverse, it is difficult to find sufficient and diverse litter samples in real life to train the algorithm model, resulting in a low accuracy rate of using the model to identify litter. Therefore, there is an urgent need to provide a new solution with a high identification accuracy rate for automatic identification (discovery) of litter.

[0102] In view of the above-mentioned defects, in this embodiment, a litter detection method is proposed, which uses a litter detection model including a litter detection segmentation sub-model and a litter detection denoising sub-model to improve the richness and diversity of litter samples and improve the detection recall rate and accuracy of litter, thereby realizing efficient and accurate detection of litter. This technical solution can effectively improve the detection recall rate and accuracy of litter and improve the automation level of litter detection.

[0103] In an embodiment of the present disclosure, the litter detection method can be applied to a computer, a computing device, an electronic device, a server, a service cluster, and the like for detecting litter.

[0104] ​In the above embodiment, after obtaining the to-be-identified road surface collection image requiring to-be-identified litter detection, the to-be-identified road surface collection image is input into the litter detection model trained according to the training method, to obtain a probability value of the road surface collection image being a litter image, and finally, the litter detection result of the road surface collection image is determined by means of a comparison result of a preset probability threshold and the probability value of the road surface collection image being a litter image. Specifically, the probability value of the road surface collection image being a litter image is compared with the preset probability threshold. If the probability value of the road surface collection image being a litter image is greater than or equal to the preset probability threshold, it can be determined that the road surface collection image includes litter. Otherwise, if the probability value of the road surface collection image being a litter image is less than the preset probability threshold, it can be determined that the road surface collection image does not include litter.

[0105] Further, in order to facilitate subsequent data use, when it is determined that the road surface collection image includes litter, the detection result of the detection region where the litter is located and the detection result of the litter contour region corresponding to the road surface collection image are also output. Subsequently, the detection result of the detection region where the litter is located and the detection result of the litter contour region can be used for cleaning, data statistics and other processing.

[0106] Figure 2 The technical terms and technical features involved in the embodiments shown and related embodiments are the same as or similar to those in the embodiments shown Figure 1 The technical terms and technical features mentioned in the embodiments shown and related embodiments are the same as or similar to those in the embodiments shown Figure 2 The explanations and descriptions of the technical terms and technical features involved in the embodiments shown and related embodiments can refer to the explanations and descriptions of the embodiments shown Figure 1 The explanations and descriptions of the embodiments shown and related embodiments can refer to the explanations and descriptions of the embodiments shown

[0107] Figure 3 The overall flowchart of the litter detection method according to an embodiment of the present disclosure is shown in FIG. 1. As shown in FIG. 1, in the litter detection method, the following steps are performed: Figure 3 As shown in FIG. 1, in the litter detection method, the following steps are performed:

[0108] Firstly, based on the pre-acquired road surface image data and the scattering object image data, a training sample data set of diversified scattering objects is generated. The training sample data set of the scattering objects includes training sample images, detection region recognition results of scattering objects corresponding to the training sample images, scattering object contour region segmentation results, and probability values of the training sample images being scattering object images. The training sample images include training positive sample images and training negative sample images. When generating the training positive sample images, firstly, a first road surface acquisition image is acquired, semantic segmentation is performed on the first road surface acquisition image to obtain a road surface drivable region in the first road surface acquisition image; then, multi-class data enhancement processing is performed on pre-acquired scattering object samples to obtain diversified scattering object samples corresponding to the scattering object samples. When acquiring the scattering object samples, firstly, a second road surface acquisition image is acquired, the second road surface acquisition image is labeled in a scattering object region, and the labeled scattering object region is taken as a first scattering object sample; object samples that are likely to be scattering objects are screened from an object sample database and taken as second scattering object samples; then, the first scattering object samples and the second scattering object samples are combined to be taken as the scattering object samples; finally, the scattering object samples and the road surface drivable region in the first road surface acquisition image are synthesized to generate the training positive sample images.

[0109] Then, the training sample images, the detection region recognition results of scattering objects corresponding to the training sample images, and the scattering object contour region segmentation results are taken as training input data of the detection segmentation model to train the detection segmentation model; the training sample images are input into the trained detection segmentation model to obtain detection region detection results of scattering objects and scattering object contour region detection results output by the detection segmentation model; labeled results of the detection region detection results of scattering objects output by the detection segmentation model and labeled results of the scattering object contour region detection results are acquired. The labeled results include scattering object detection positive samples and scattering object detection negative samples, scattering object segmentation positive samples and scattering object segmentation negative samples. The scattering object detection positive samples, the scattering object detection negative samples, the scattering object segmentation positive samples, the scattering object segmentation negative samples, and the probability values of the training sample images being scattering object images are taken as training input data of the noise reduction model to train the noise reduction model. The detection segmentation model and the noise reduction model are cascaded to obtain a scattering object detection model for predicting a scattering object probability.

[0110] Finally, a road surface acquisition image to be recognized is acquired; the road surface acquisition image to be recognized is input into the trained scattering object detection model to obtain a probability value of the road surface acquisition image being a scattering object image. When the probability value of the road surface acquisition image being a scattering object image is greater than or equal to a preset probability threshold, it is determined that the road surface acquisition image includes a scattering object.

[0111] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein.

[0112] Figure 4 This diagram illustrates a structural block diagram of a projectile detection model training device according to an embodiment of the present disclosure. This device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 4 As shown, the projectile detection model training device includes:

[0113] The generation module 401 is configured to generate a diverse training sample data set of spilled objects based on pre-collected road surface image data and spilled object image data. The training sample data set of spilled objects includes training sample images, the identification result of the detection area where the spilled object is located corresponding to the training sample image, the segmentation result of the spilled object contour region, and the probability value that the training sample image is a spilled object image. The training sample images include training positive sample images and training negative sample images.

[0114] Training module 402 is configured to train a preset neural network model based on the training sample data set. The neural network model includes a cascaded detection and segmentation model and a noise reduction model to obtain a projectile detection model for predicting the probability of projectiles.

[0115] As mentioned above, with social development and progress, the number of vehicles on the road is increasing. Spilled debris on the road is a dangerous substance that affects driving safety, especially on highways, national roads, and provincial roads where high speeds are permitted. Road maintenance departments typically remove spilled debris through manual inspections using maintenance vehicles. During manual inspections, detecting spilled debris is the most time-consuming step, as it is a low-frequency problem. Therefore, an algorithmic model is needed for automatic identification of spilled debris. However, due to the low frequency and variety of spilled debris, it is difficult to find a sufficient number and variety of samples in real life to train the algorithm, resulting in low accuracy in identifying spilled debris. Therefore, there is an urgent need for a new solution with high accuracy for the automated identification (discovery) of spilled debris.

[0116] In view of the above defects, in this embodiment, a diversity of litter training sample generation and litter detection model training method is provided, which can not only generate diverse litter training samples, but also achieve efficient and accurate detection of litter by means of the joint of the detection segmentation model and the noise reduction model. This technical solution can not only improve the richness and diversity of litter samples, but also improve the training efficiency of the litter detection model, improve the detection recall rate and accuracy of the litter, and improve the automation degree of litter detection.

[0117] In an embodiment of the present disclosure, the litter detection model training device can be implemented as a computer, a computing device, an electronic device, a server, a service cluster, etc. for training the litter detection model.

[0118] Figure 5 A structural block diagram of a litter detection device according to an embodiment of the present disclosure is shown, which can be implemented as part or all of an electronic device by software, hardware or a combination of both. As shown in Figure 5 The litter detection device includes:

[0119] The acquisition module 501 is configured to acquire a road surface collection image to be recognized.

[0120] The detection module 502 is configured to input the road surface collection image to be recognized into the litter detection model trained by the training device of claim 7, to obtain a probability value of the road surface collection image being a litter image, and determine that the road surface collection image includes litter when the probability value of the road surface collection image being a litter image is greater than or equal to a preset probability threshold.

[0121] As mentioned above, with the development and progress of society, there are more and more vehicles on the road, and litter on the road is a dangerous item that affects driving safety, especially on roads such as highways, national roads and provincial roads that allow vehicles to drive at high speed. The influence of litter on driving safety is greater. The road maintenance department usually removes litter through artificial inspection by maintenance vehicles. In the process of artificial inspection, since litter is a low-frequency disease, the process of finding litter is the most time-consuming, so the algorithm model is needed to realize automatic identification of litter. However, since litter is a low-frequency disease and the types of litter are rich, it is difficult to find sufficient and diverse litter samples in real life to train the algorithm model, resulting in low accuracy of using the model to identify litter. Therefore, for automatic identification (finding) of litter, it is urgent to provide a new solution with high identification accuracy.

[0122] In view of the above defects, in this embodiment, a litter detection device is proposed, which uses a litter detection model including a litter detection segmentation sub-model and a litter detection denoising sub-model to improve the richness and diversity of litter samples, improve the detection recall rate and accuracy of litter, and realize efficient and accurate detection of litter. This technical solution can effectively improve the detection recall rate and accuracy of litter and improve the automation level of litter detection.

[0123] In an embodiment of the present disclosure, the litter detection device can be implemented as a computer, a computing device, an electronic device, a server, a service cluster, or the like for detecting litter.

[0124] The technical terms and technical features involved in the above device-related embodiments are the same as or similar to the technical terms and technical features mentioned in the above method-related embodiments. For the explanation and description of the technical terms and technical features involved in the above device-related embodiments, reference can be made to the above explanation and description of the method-related embodiments, which will not be repeated here.

[0125] The present disclosure also discloses an electronic device, Figure 6 A structural block diagram of an electronic device according to an embodiment of the present disclosure is shown as follows, Figure 6 As shown, the electronic device 600 includes a memory 601 and a processor 602; wherein,

[0126] The memory 601 is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor 602 to implement the above method steps.

[0127] Figure 7 is a structural schematic diagram of a computer system suitable for implementing the litter detection model training method and / or litter detection method according to an embodiment of the present disclosure.

[0128] As shown in Figure 7 The computer system 700 includes a processing unit 701, which can execute various processes in the above embodiments according to programs stored in a read-only memory (ROM) 702 or loaded from a storage portion 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the computer system 700 are also stored. The processing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0129] The following components are connected to the I / O interface 705: an input part 706 including a keyboard, a mouse, and the like; an output part 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage part 708 including a hard disk, and the like; and a communication part 709 including a network interface card such as a LAN card, a modem, and the like. The communication part 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as necessary. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is attached to the drive 710 as necessary, so that a computer program read out therefrom is installed in the storage part 708 as necessary. Among them, the processing unit 701 can be implemented as a CPU, a GPU, a TPU, an FPGA, an NPU, and the like processing unit.

[0130] In particular, according to embodiments of the present disclosure, the method described above can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program tangibly embodied on a non-transitory computer readable medium, the computer program containing program code for executing the method. In such embodiments, the computer program can be downloaded and installed from a network via the communication part 709, and / or installed from the removable medium 711.

[0131] The flow and block diagrams in the drawings show possible architectures, functional and operational, of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0132] The units or modules described in the embodiments of the present disclosure can be implemented by software, or can be implemented by hardware. The described units or modules can also be provided in a processor, and the names of these units or modules do not constitute a limitation on the units or modules themselves in some cases.

[0133] As another aspect, the disclosure also provides a computer readable storage medium, which can be the computer readable storage medium included in the apparatus described in the above embodiments; or can exist separately and not be assembled into the apparatus. The computer readable storage medium stores one or more programs for being executed by one or more processors to perform the method described in the disclosure.

[0134] The above description is merely the preferred embodiments of the disclosure and the explanation of the principles of the applied technologies. It should be understood by those skilled in the art that the inventive scope of the disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by the combinations of the above technical features or equivalent features without departing from the inventive concept. For example, the technical solutions formed by the mutual replacement of the above features and the technical features disclosed in the disclosure (but not limited to) with similar functions.

Claims

1. A method for training a litter detection model, comprising: generating a set of training sample data of litters with diversity based on pre-collected road image data and litter image data, wherein the set of training sample data of litters comprises training sample images, detection region recognition results of litters corresponding to the training sample images, contour region segmentation results of litters, and probability values of the training sample images being litter images, and the training sample images comprise training positive sample images and training negative sample images; training a preset neural network model based on the set of training sample data to obtain a litter detection model for predicting litter probability, wherein the neural network model comprises a detection and segmentation model and a noise reduction model in cascade, the noise reduction model is trained with the training positive sample images, the training negative sample images, the training positive sample images of litter segmentation, the training negative sample images of litter segmentation, and the probability values of the training sample images being litter images as training input data, and the training positive sample images, the training negative sample images, the training positive sample images of litter segmentation, and the training negative sample images of litter segmentation are outputs of the detection and segmentation model.

2. The method of claim 1, wherein, The method further comprises generating training positive sample images, comprising: obtaining a first road collection image, performing semantic segmentation on the first road collection image to obtain a drivable region of the first road collection image; obtaining litters with diversity by randomly changing pre-obtained litter samples; synthesizing the litters with diversity and the drivable region of the first road collection image to generate the training positive sample images. 3.The method of claim 2, wherein the obtaining litters with diversity by randomly changing pre-obtained litter samples comprises: performing multi-class data enhancement processing on the pre-obtained litter samples to obtain litters with diversity corresponding to the litter samples. 4.The method of any one of claims 1-3, wherein the training a preset neural network model based on the set of training sample data to obtain a litter detection model for predicting litter probability comprises: training the detection and segmentation model with the training sample images, the detection region recognition results of litters corresponding to the training sample images, and the contour region segmentation results of litters as training input data; inputting the training sample images into the trained detection and segmentation model to obtain detection region detection results of litters and contour region detection results of litters output by the detection and segmentation model; obtaining annotation results of the detection region detection results of litters output by the detection and segmentation model and the contour region detection results of litters, wherein the annotation results comprise training positive sample images and training negative sample images, training positive sample images of litter segmentation, and training negative sample images of litter segmentation. The detection and segmentation model and the noise reduction model are cascaded to obtain a litter detection model for predicting litter probability.

5. The method of claim 4, further comprising: adding a road surface collection image used when litter is detected by the litter detection model and a corresponding litter detection result obtained thereby as new training data to the litter training sample data set, and retraining the litter detection model.

6. A litter detection method, comprising: obtaining a road surface collection image to be identified; inputting the road surface collection image to be identified into a litter detection model trained by the training method of any one of claims 1-5 to obtain a probability value that the road surface collection image is a litter image, when the probability value that the road surface collection image is a litter image is greater than or equal to a preset probability threshold, determining that the road surface collection image includes litter. When the probability value that the road surface collection image is a litter image is greater than or equal to a preset probability threshold, the litter detection model also outputs a detection result of a detection region of the litter corresponding to the road surface collection image and a detection result of a litter contour region.

7. The method of claim 6, wherein, 8. A litter detection model training apparatus, comprising: a generation module configured to generate a diversity litter training sample data set based on pre-collected road surface image data and litter image data, wherein the litter training sample data set includes training sample images, detection region identification results of litters corresponding to the training sample images, segmentation results of litter contour regions, and probability values that the training sample images are litter images, and the training sample images include training positive sample images and training negative sample images; a training module configured to train a preset neural network model based on the training sample data set, the neural network model including a cascaded detection and segmentation model and a noise reduction model, to obtain a litter detection model for predicting litter probability, wherein the noise reduction model is trained using the litter detection positive sample, the litter detection negative sample, the litter segmentation positive sample, the litter segmentation negative sample, and the probability value that the training sample image is a litter image as training input data of the noise reduction model, the litter detection positive sample, the litter detection negative sample, the litter segmentation positive sample, and the litter segmentation negative sample being outputs of the detection and segmentation model.

9. A litter detection apparatus, comprising: an acquisition module configured to obtain a road surface collection image to be identified; ​ The detection module is configured to input the road surface collection image to be identified into the litter detection model trained by the training device in claim 8, to obtain a probability value of the road surface collection image being a litter image, and to determine that the road surface collection image includes litter when the probability value of the road surface collection image being a litter image is greater than or equal to a preset probability threshold.

10. An electronic device comprising a memory and at least one processor; wherein, The memory is configured to store one or more computer instructions, and the one or more computer instructions are executed by the at least one processor to implement the method steps in any one of claims 1-7.

Images