Method and system for detecting opening of spilled objects on road surface

Abstract

The invention discloses a road surface spilled object open detection method and detection system, and the method comprises the following steps: S1, constructing a spilled object detection data set which comprises known spilled object and non-spilled object backgrounds; s2, classifying all known spilled objects into a spilled object class without category distinguishing, and training a YOLO network to identify road spilled objects, so as to realize detection of known spilled objects in a data set and unknown spilled objects outside the data set; s3, training an OfficientNet network, and combining an incremental learning algorithm to realize picture classification of known spilled objects and unknown spilled objects for all detected spilled objects; and S4, grading the detection result through the processing platform, and finally performing early warning according to grading. According to the invention, detection and classification of unlimited unknown spilled objects on the road surface are realized.

Description

Technical Field

[0001] The invention relates to a method and a system for detecting open road spillage. Background Art

[0002] Transportation is a solid "runway" for economic and social development, and is increasingly playing an important supporting role in promoting high-quality development of regional economies by promoting urban development. However, the corresponding road surface safety inspection is relatively weak, and there are great safety hazards, especially spilled objects on the road have become one of the main factors threatening road safety.

[0003] Traditionally, road detection is basically done through manual detection and calibration, which is time-consuming and labor-intensive and can only be used for detection of key sections of roads. With the continuous development of information technology, the most commonly used method for road detection is to use sensors on the detection vehicle to sense the road conditions and then send them back for background detection, so the road condition system updates slowly.

[0004] At present, some companies such as Hikvision and Zhejiang Dahua have launched event detection products for highways. The detection methods for spilled objects are basically realized by extracting moving objects and analyzing their static states, and using the five-frame difference method and the inter-frame difference adaptive method to detect spilled objects on highways. SF Express has disclosed a spilled object detection method (CN114913488A), which performs semantic segmentation on road images, synthesizes new views using adversarial networks, and determines whether there are spilled objects on the road by comparing the original image with the new view.

[0005] In terms of detecting unknown spilled objects, the above methods mainly include image-based algorithms and deep learning-based algorithms. Image-based algorithms usually use image difference to detect spilled objects by comparing changes in inter-frame areas. However, this method cannot detect static targets and has application limitations.

[0006] The target detection algorithm based on deep learning is not affected by the target's motion state, but it is closed-loop. Its ability to identify unknown types of objects is directly related to the richness of the annotations in the data set. However, there are many kinds of objects spilled on the road. The contradiction between the closed nature of the training data and the limited categories and the openness of the real scene and the inexhaustible categories brings great difficulties to this type of detection.

[0007] Most of these algorithms are batch learning models. When training is completed, the learning process is terminated. They cannot identify target types outside the training set and no longer learn new knowledge. However, in practical applications, it is usually impossible to obtain all training samples at once. If all data needs to be relearned after new samples arrive, it will take a lot of time and space. Therefore, the existing related spilled object detection algorithms cannot meet this demand.

[0008] Whenever a new sample appears, the model needs to be trained from scratch, which is a computationally intensive and overly expensive process. What’s more serious is that due to the small number of unknown spilled samples, the retrained model will be overfitted to a large extent. Therefore, with the increasing mileage of highways, in order to update the road condition system as quickly as possible, reduce the time of safety hazards, and meet the requirements of efficient and rapid road surface detection, it is urgent to develop a rapid road surface detection method. Summary of the invention

[0009] The purpose of the present invention is to provide an open detection method and detection system for road spills, so as to achieve the classification of unlimited unknown spills.

[0010] To this end, the present invention provides an open detection method for road spills, comprising the following steps: S1, constructing a spill detection data set, the data set including known spills and non-spill background; S2, classifying all known spills into one category without distinguishing categories, training a YOLO network to identify road spills, and realizing the detection of known spills in the data set and unknown spills outside the data set; S3, training an EfficientNet network, combining an incremental learning algorithm, realizing image classification of known spills and unknown spills for all detected spills; S4, classifying the detection results through a disposal platform, and finally issuing an early warning based on the classification.

[0011] The present invention also provides a road spillage detection system, comprising an edge device on which a computer program is stored, and when the program is executed, it is used to implement each step in the road spillage detection method.

[0012] The present invention also provides a computer program product, including a computer program, which is used to implement each step in the road spillage detection method when executed.

[0013] Compared with the prior art, the present invention has the following technical effects: The present invention provides a method for detecting unknown objects, which can detect target objects outside the data set based on a limited data set, reduce the algorithm's dependence on the data set, and reduce data collection and annotation costs.

[0014] The present invention provides an open detection system for road spills, which is mounted on an edge mobile device and can replace road inspectors to perform long-term inspections and the need for intensive deployment of fixed detection equipment, thereby reducing a large amount of unnecessary waste of manpower and material resources. It can autonomously detect, identify and warn of spills on the road, dispose of them as quickly as possible, and minimize losses. It can also continuously optimize the recognition range based on online updated learning content.

[0015] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting a part of the present application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings: Figure 1 is a flow chart of the road spillage open detection method of the present invention; Figure 2 It is an architecture diagram of the improved YOLO network of the present invention; Figure 3 It is a flow chart of the unknown spilled matter detection of the present invention; Figure 4 It is a flowchart of the incremental learning algorithm; Figure 5 This is the schematic diagram of the incremental learning of the EfficientNet network; Figure 6 It is an architectural diagram of the road spillage open detection system of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0018] Traditional machine learning methods and deep neural networks require a large number of labeled samples as support in the process of training models. However, labeling a large number of samples is a very time-consuming process, and the real scene is unpredictable. The sources of road spills are uncertain and there are many types. It is unrealistic to obtain labeled samples of all categories.

[0019] Existing deep learning-based spilled object detection is trained on a fixed-category and fixed-number spilled object dataset. In practical applications, if it is necessary to add recognition of new categories of targets, it is necessary to merge the new and old datasets and start training from scratch.

[0020] In this invention, the classification and regression tasks are decoupled, regression is performed first, and then classification: 1. Train YOLO to identify spilled objects that are not on the road. Do not classify the spilled objects first, but only perform regression positioning (frame them with a rectangular frame). This can break through the category restrictions in the data set and identify spilled objects outside the data set.

[0021] 2. When training EfficientNet, the targets in the data set are cropped according to the rectangular frame and then trained. This ensures that there is only one target in a picture, and it can be classified by picture without the need for positioning tasks, so as to achieve classification of known spilled objects in the data set.

[0022] 3. When YOLO returns the cropped spilled object target, if it is a known spilled object in the dataset, EfficientNet can directly classify it. If it is an unknown target outside the dataset, EfficientNet will perform incremental learning and only perform small sample training on the new category of spilled objects to increase the number of recognized spilled objects.

[0023] The present invention introduces open set detection, which can break through the limitation of labeled samples and is more in line with realistic scenarios. The established model can not only classify the categories that appear during training, but also effectively handle unseen categories. It gets rid of the contradiction between the closed nature of existing training data on spilled objects and limited categories and the open nature of real scenarios and the inexhaustible categories.

[0024] The present invention introduces incremental learning, which can continuously process the continuous information flow in the real world, retaining or even integrating and optimizing the old knowledge while absorbing new knowledge. Under the condition of limited computing and storage resources, only the unknown part of the target needs to be calibrated and trained, and the new and old targets can be detected simultaneously in the same network model, and the types of spilled objects can be expanded with minimal cost.

[0025] Combined with reference Figure 1 The road spillage detection method of the present invention comprises the following steps S1-S4.

[0026] S1. Construct a spilled object detection dataset.

[0027] In view of the problems that the current road spillage detection dataset is relatively scarce, with few target types and insufficient scene richness, we plan to build a new road spillage detection dataset.

[0028] The dataset mainly covers known spilled objects and non-spilled backgrounds, which are used to guide the learning direction of the neural network.

[0029] For known spilled objects and background interference objects, a deep learning network is provided to learn positive sample known target features and negative sample suppressed object features.

[0030] (1) It is known that spilled objects are the positive sample objects to be learned in the data set. The wider the types and coverage, the higher the overall detection accuracy. The key spilled object detection objects are established for the data set. For example: a. Regular garbage: randomly discarded waste such as paper, plastic bags, bottles, cans, fast food cartons, beverage cups and other food packaging.

[0031] b. Vehicle parts: tire fragments, bumpers, headlights, etc.

[0032] c. Construction materials: waste materials during the construction process such as wooden boards, bricks, and steel bars.

[0033] (2) Non-spillage background is the negative sample object to be suppressed in the data set. It is a common object that does not need to be paid attention to in the road detection scene. The coverage of the scene needs to be as wide as possible to exclude other targets besides spillage and prevent false alarm detection. For example: a. Vehicles: cars, trucks, motorcycles, electric cars, bicycles, buses, etc.

[0034] b. Traffic signs: road markings, traffic signs, manhole covers, floor drains, roadblocks, etc.

[0035] c. Pedestrians: Pedestrians of different ages and in different clothing.

[0036] d. Buildings: prominent buildings, signs, trees, etc. along the road.

[0037] S2. Classify all known objects as spilled objects without distinguishing their categories, train the YOLO network to identify road spills, and detect unknown spills outside the dataset.

[0038] Since the types of spilled objects that may appear in road scenes cannot be exhaustively listed, and the categories that can be collected and annotated in the dataset are limited, for the spilled objects that do not appear in the dataset, an unknown spilled object detection algorithm is constructed based on YOLO to supplement the detection.

[0039] The objects are other significant unknown objects that do not appear in the dataset, that is, non-annotated and non-background objects in the dataset, which meet the recognition requirements for all targets in the road scene and realize the detection of all spilled objects.

[0040] On the basis of YOLO, in order to detect multi-scale targets in road scenes, the receptive field module RFB is used to replace the original SPPF to avoid the loss of detail features caused by the maximum pooling in the SPPF module. The convolution layer with multi-scale convolution kernels is used to form a multi-branch parallel structure and introduce a hole convolution layer to increase the network receptive field without changing the network resolution, simulate the principle of human visual perception, and increase the network's multi-scale target detection ability. At the same time, BIFPN_Add is used to replace the original Concat module to increase the ability to fuse different depth features.

[0041] exist Figure 2Among them, RFB (Receptive Field Block): receptive field module; CSP (Cross Stage Partial): cross-stage local module; CBS (Conv; Batch Normalization; Silu): convolution; batch normalization; Silu activation function; BiFPN (Bidirectional Feature Pyramid Network): weighted bidirectional feature pyramid network.

[0042] During training, all positive sample targets in the constructed data set are classified into the same category, namely, road surface objects to be detected. All negative sample targets in the data set are classified into the same category, namely, background.

[0043] Focusing on the “spillage” and “background” division of the data set, the neural network is trained to distinguish whether there are objects on the road and whether they are negative sample objects, rather than identifying the specific types of objects on the road.

[0044] By distinguishing the spatial features of known targets and road interference targets, the network's ability to extract features of known targets and suppress features of interference targets is enhanced, the utilization of known target features is strengthened, and the influence of interference target features on unknown target features is suppressed.

[0045] Let the network blur the feature distinction between positive sample targets, strengthen the ability to extract positive sample features and suppress negative sample features. The focus is to let the network learn to distinguish whether the objects on the road are spilled objects or background, so as to achieve the goal of identifying an infinite number of objects from a limited data set.

[0046] During detection, YOLO can solve the problem of identifying known and unknown spills after being trained using the above method. The detected targets and categories are uniformly labeled as spills and marked with a rectangular frame. With a limited training data set, it can detect an unlimited number of unknown spills.

[0047] S3. The detected objects are cropped and transmitted back to the local machine. Based on EfficientNet, the types of spilled objects are continuously learned through incremental learning to achieve the classification of spilled object images.

[0048] On the local platform, the spilled objects in the constructed spilled object detection dataset are cropped according to the marked rectangular frame. Each cropped spilled object image contains only a single spilled object, and each image is labeled with the type of spilled object to construct a training dataset. EfficientNet is trained to be able to classify images and classify known spilled objects in the dataset.

[0049] Based on the detection of unknown objects by edge devices, cropped images of objects in the detection frame are transmitted back to the platform. The EfficientNet network combined with the incremental learning algorithm can classify newly added unknown targets.

[0050] On the basis of the detection of unknowns, the specific classification of unknowns is achieved by fine-tuning the features of small samples through open learning. First, the parameters of the last layer of the model are removed while keeping the parameters of the entire feature extractor unchanged. Then, the randomly initialized weights of the newly added class are assigned to the classifier of the last layer of the model for data fine-tuning. When performing incremental training, only the newly added class data set is input, the weight parameters of the feature extraction network are kept frozen, and the corresponding weight parameters of the newly added class classification nodes are updated through training. Finally, the values ​​of all the newly added classification layer nodes are normalized to achieve the classification of the newly added types of targets.

[0051] like Figure 5 As shown in , the solid circle represents the numerical node corresponding to the original class, and the solid arrow represents the weight parameter corresponding to the original model. This part of the weight parameters remains unchanged and remains frozen during incremental training. The dotted part represents the newly added nodes and the corresponding weight parameters. This part of the weight parameters is the variable parameter during incremental training.

[0052] Let the last layer of the feature extraction network be Layer, there are nodes, the first The node values ​​are: ; Classification level The node values ​​are: ; The corresponding weight parameters are: ; Let the last layer of the feature extraction network be Layer, there are nodes, the first The node values ​​are: ; Classification level The node values ​​are: ; The corresponding weight parameters are: ; The first The node values ​​corresponding to the class are: ; The corresponding weights are: .

[0053] Then add a new class, namely The node values ​​corresponding to the class are: ; The corresponding weights are: .

[0054] The corresponding relationship is: .

[0055] Under limited computing resources, the algorithm can continuously process the continuous information flow in the real world, retaining the ability to integrate and optimize old knowledge while absorbing new knowledge. This method can gradually update knowledge, and can correct and strengthen previous knowledge, so that the updated knowledge can adapt to the newly arrived data without having to re-learn all the data.

[0056] After labeling the unknown objects, the model can be continuously updated to expand the types of spilled objects that can be identified, reducing the time and computing costs of training the model and better meeting actual requirements. In a network model of EfficientNet, the classification of new and old targets of different categories is achieved, and the types of spilled objects that can be detected are expanded at the minimum cost.

[0057] S4. The detection results are classified into different levels through the platform, and finally an early warning is processed.

[0058] The basic disposal process of the present invention mainly includes two links: data collection and decision management. The data collection link completes the detection of road spills and the uploading of related information. This step S4 is the decision management link, and the platform dumps and records the latest detection results uploaded by the edge device. At the same time, the platform can record and classify unknown objects, optimize network weights through model incremental learning and optimization strategies, and update the weights of the classification algorithm on the platform side.

[0059] In addition, the management platform notifies and dispatches road users and road maintenance personnel based on the type of spilled objects detected. The classification is based on the degree of hazard of the spilled objects, which are divided into two levels: Level 1 is general hazard objects, such as napkins, snack packaging bags, tiny stones, etc.; Level 2 is major hazard objects, such as tires, large wooden boards, broken glass, etc.

[0060] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting road spillage, characterized in that: The following steps are involved: S1. Constructing a spilled object detection data set, wherein the data set includes known spilled objects and non-spilled object background; S2, classify all known spilled objects into one category without distinguishing their categories, train the YOLO network to identify road spills, and detect known spilled objects in the data set and unknown spilled objects outside the data set; S3: Train the EfficientNet network and combine it with the incremental learning algorithm to classify the images of all detected spills into known and unknown ones. S4. Classify the test results through the disposal platform, and finally issue warnings based on the classification.

2. The road spill detection method according to claim 1, characterized in that: The known spilled objects are the positive sample objects to be learned in the dataset, including: regular garbage, vehicle parts, and building materials. The non-spilled background is the negative sample objects to be suppressed in the dataset, which are common objects that do not need to be paid attention to in highway detection scenes, including vehicles, traffic signs, pedestrians, and buildings.

3. The road spill detection method according to claim 1, characterized in that: The YOLO algorithm includes a backbone network and a feature fusion network. The backbone network adopts a receptive field module RFB to replace the SPPF module, and the feature fusion network adopts a BIFPN_Add module to replace the Concat module.

4. The road spillage detection method according to claim 1, characterized in that: In step S2, when training the YOLO network, all positive sample targets are not classified, and their labels are uniformly classified as spilled objects, confusing the intra-class features of the spilled objects. The network is trained to distinguish the features of the spilled objects from the background features, and detect the known spilled objects in the data set and the unknown spilled objects outside the data set on the road surface. After YOLO detects the spilled object target, it is marked with a rectangular frame, and the target image in the rectangular frame is cropped and transmitted back to the disposal platform.

5. The road spill detection method according to claim 1, characterized in that: In step S3, when training the EfficientNet network, the spilled objects marked in the data set are cropped according to the rectangular frame, and the EfficientNet is trained to classify the images to achieve the ability to classify the known objects in the data set. When the EfficientNet network is detected, the spilled object images detected and cropped in step S2 and returned to the platform are input into the trained EfficientNet to achieve the classification of the spilled objects detected on the road surface.

6. The road spill detection method according to claim 5, characterized in that: The EfficientNet network classifies the newly added types of targets, including the following steps: S31, while keeping the parameters of the entire feature extractor unchanged, remove the parameters of the last layer of the model, and then assign the randomly initialized weights of the newly added class to the classifier of the last layer of the model to perform data fine-tuning; S32, only input the newly added class data set, keep the feature extraction network weight parameters frozen, and train and update the corresponding weight parameters of the newly added class classification nodes; S33, normalizing the values ​​of all newly added classification layer nodes, so as to achieve classification of the newly added types of targets.

7. The road spill detection method according to claim 1, characterized in that: The detection results are classified into two levels according to the degree of hazard of the spilled objects: level one is general hazard; level two is major hazard, and the warning according to the classification level includes notifying and dispatching road drivers and road maintenance personnel.

8. The road spill detection method according to claim 1, characterized in that: All spills are detected on the edge device and are cropped and transmitted back to the disposal platform, where they are recorded and classified. When new categories of spills are added, the network weights are optimized through model incremental learning and optimization strategies, and the weights of the classification algorithm are updated on the disposal platform.

9. A road spillage detection system, characterized in that: It comprises an edge device and a disposal platform in communication connection with the edge device, on which a computer program is stored, which is used to implement steps S1-S4 in the open detection method for road spillage according to any one of claims 1 to 8 when executed.

10. A computer program product, characterized in that It comprises a computer program which, when executed, is used to implement the steps of the road spillage detection method according to any one of claims 1 to 8.

Images