A road spill detection method based on contrast clustering self-learning

Abstract

The present invention discloses a road spilled object detection method based on contrast clustering self-learning, and constructs a road spilled object detection network model, wherein the road spilled object detection network model includes a feature extraction module, a candidate area module, an unknown label generation module, and a spilled object recognition module based on a maximum logical value. The road spilled object detection network model is trained using training samples, and the trained road spilled object detection network model is used to detect an image to be detected, and a detection result is output. The present invention can identify spilled objects stationary on a highway, and has a strong anti-interference ability, and can detect spilled object targets that are not included in the spilled object category of a data set, and therefore has more practical value.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a method for detecting road spilled objects based on contrast clustering self-learning. Background Art

[0002] In recent years, highways have played an increasingly important role in transportation and logistics, and trucks account for a high proportion of vehicles passing through many sections of the road. However, the spilled objects on the highway have seriously affected driving safety and have become one of the main causes of traffic accidents on the current highways. At the same time, in the current process of handling spilled objects on highways, there has always been a problem of difficulty in timely detection of spilled objects. There are several reasons: ① The occurrence of spilled objects is random. In many cases, after the spilled object incident occurs, the driver himself is unaware of it, let alone timely alarm. ② Due to the limitations of current video surveillance technology, it is difficult to detect through monitoring, especially under special conditions such as night and rainy days. ③ Conventional road inspections are inefficient, especially for spilled objects that occur during inspections. Fortunately, with the continuous advancement of computer vision technology, it has gradually become a reality to use real-time image information collected by cameras to detect spilled object incidents, which is of great significance for the timely detection and subsequent disposal of spilled objects.

[0003] Currently, methods for detecting spilled objects using videos captured by cameras can be divided into two types: indirect detection and direct detection.

[0004] First, indirect detection refers to a method of detecting spilled objects based on background modeling. First, the background modeling algorithm is used to distinguish the foreground and background in the captured image, and then the foreground area of ​​the suspected spilled object is located. Finally, the traditional image features are used to determine the possibility that the suspected spilled object area is true. However, this type of solution has obvious defects: ① The highway is an open scene with a complex and changeable environment. Therefore, the camera will generate a lot of background noise in the process of collecting images, which will cause strong interference to the background modeling algorithm and affect the accuracy of the modeling. ② The background modeling algorithm assumes that the spilled object is a moving object. However, the movement process of the spilled object will not be captured by the camera in many scenarios, or the spilled object is just stationary on the road surface. In this case, the spilled object cannot be detected.

[0005] Second, the direct detection method refers to the spilled object detection method based on deep learning. First, several categories of spilled object data sets are collected, then a general target detection model or instance segmentation model is used for training, and finally the trained model is used for detection. Although the current detection method based on deep learning has high detection accuracy, its main drawback is that it can only detect the types of spilled objects contained in the data set. However, on real highways, there are many types of spilled objects, and the number of types far exceeds the number of types of spilled objects contained in general data sets, which greatly limits the practical application of such detection methods. Summary of the invention

[0006] The purpose of this application is to propose a road spilled object detection method based on contrast clustering self-learning to overcome the technical problems raised in the above background technology.

[0007] In order to achieve the above purpose, the technical solution of this application is as follows:

[0008] A road spill detection method based on contrast clustering self-learning, comprising:

[0009] Constructing a road spilled object detection network model, wherein the road spilled object detection network model includes a feature extraction module, a candidate area module, an unknown label generation module, and a spilled object recognition module based on a maximum logical value;

[0010] Using training samples to train the road spilled object detection network model, using the trained road spilled object detection network model to detect the image to be detected, and outputting the detection result;

[0011] Wherein, when training the road spilled object detection network model, or using the trained road spilled object detection network model to detect the image to be detected, it includes:

[0012] Inputting the training sample image or the image to be detected into the feature extraction module to extract the first feature map;

[0013] Inputting the first feature map into the candidate region module to obtain a potential candidate box set, and filtering out a general candidate box set and a potential unknown candidate box set from the potential candidate box set according to a preset rule;

[0014] Inputting the first feature map and the potential unknown candidate box set into the unknown label generation module to obtain the unknown candidate box set;

[0015] The unknown candidate box set and the general candidate box set are merged into a total candidate box set, and then the first feature map and the total candidate box set are input into a spilled object recognition module based on the maximum logical value to obtain a detection result.

[0016] Furthermore, when training the road spilled object detection network model, the general candidate frame set and the potential unknown candidate frame set are screened out from the potential candidate frame set according to the preset rules, including:

[0017] Filter out the candidate boxes whose confidence is greater than or equal to the first threshold from the potential candidate box set P to form a set P a , then, the set P a The candidate boxes in the image and the bounding boxes in the corresponding labels are combined in pairs to calculate the intersection-over-union ratio and generate an IOU matrix.

[0018] According to the IOU matrix, from the set P a The candidate boxes whose iou values ​​with any bounding box in the label are less than or equal to the second threshold are selected to form a set Pb;

[0019] From the set P a The candidate boxes whose iou values ​​with any bounding box in the label are greater than the second threshold are selected to form the general candidate box set P1;

[0020] The set P b The candidate boxes in are sorted from high to low according to the confidence, and a preset number of candidate boxes are selected from high to low according to the confidence to form a potential unknown candidate box set P2.

[0021] Furthermore, when the trained road spilled object detection network model is used to detect the image to be detected, the general candidate frame set and the potential unknown candidate frame set are screened out from the potential candidate frame set according to the preset rules, including:

[0022] From the potential candidate box set P, select the candidate boxes whose confidence is greater than or equal to the third threshold to form a set P1, and the candidate boxes whose confidence is less than the third threshold to form a set P b ;

[0023] The set P b The candidate boxes in are sorted from high to low according to the confidence, and a preset number of candidate boxes are selected from high to low according to the confidence to form a potential unknown candidate box set P2.

[0024] Furthermore, the spilled object recognition module based on the maximum logical value includes a feature storage set F store , used to store each unknown category feature vector, the spilled object category feature vector and the non-sprinkled object target category feature vector, the spilled object recognition module based on the maximum logical value also includes a maximum logical value storage set E store , the maximum logical value storage set E store Includes a queue q for storing the maximum logical value of each unknown category feature vector u , and a queue q for storing the maximum logical value of the known category feature vector k ;

[0025] The spilled object recognition module based on the maximum logical value performs the following operations during training:

[0026] The candidate boxes are taken out one by one from the total candidate box set, and then the region of interest is extracted from the first feature map according to the candidate boxes, and the region of interest is convolved to generate a feature vector to be classified, and the feature vector to be classified is stored in the feature storage set F according to the category to which the feature vector to be classified belongs. store In the corresponding queue;

[0027] Performing regression analysis on the feature vector to be classified, predicting the bounding box of the feature vector to be classified, calculating the intersection-and-union ratio of the bounding box and the label in turn, and if the maximum value of the intersection-and-union ratio is greater than or equal to a fourth threshold, calculating the loss function and updating the parameters of the road spilled object detection network model;

[0028] According to whether the feature vector to be classified belongs to an unknown category or a known category, the maximum logical value is taken from the feature vector to be classified and stored in the maximum logical value storage set E. store in the corresponding queue;

[0029] Calculate the maximum logical value storage set E respectively store The average of the two queues in, where queue q u The mean value is μ u , and queue q k The mean value is μ k , determine the classification decision function α(logit j ):

[0030]

[0031] Among them, logit j is the feature vector to be classified j The maximum logical value, α(logit j )=0 represents the feature vector to be classified f j belongs to the unknown category, and α(logit j )=1 represents the feature vector to be classified f j belongs to a known category;

[0032] The spilled object recognition module based on the maximum logical value performs the following operations when recognizing the image to be detected:

[0033] Taking candidate boxes one by one from the total candidate box set, then extracting the region of interest from the first feature map according to the candidate boxes, performing a convolution operation on the region of interest, generating a feature vector to be classified, and predicting a bounding box through regression analysis;

[0034] Using the classification decision function α(logitj ) Determine whether the currently extracted feature vector to be classified belongs to an unknown category or a known category. If the feature vector to be classified belongs to an unknown category, output the first prediction result. If the feature vector to be classified belongs to a known category, calculate the Euclidean distance between the feature vector to be classified and the prototype vector in the feature prototype set in turn, take the category corresponding to the minimum Euclidean distance as the recognition result, and output the second prediction result.

[0035] Furthermore, the loss function of the road spilled object detection network model includes: regression loss L reg and contrast clustering loss L cont , the contrastive clustering loss L cont The calculation process is as follows:

[0036] Using the feature storage set F store Each FIFO queue q of length L in i The stored feature vector to be classified f l Calculate the corresponding prototype vector p i , and all prototype vectors p i Combined into a category prototype set V = {p0,...,p H};

[0037] Prototype vector p i Calculated according to the following formula:

[0038]

[0039] Then, for storage into queue q i The feature vector f to be classified corresponds to a picture in j , calculate f j Contrasting clustering loss of:

[0040]

[0041] Among them, l(f j , p i ) is defined as:

[0042]

[0043] Where D represents the Euclidean distance, m α and m β are two distance parameters, w θ is the weighting coefficient.

[0044] Furthermore, the step of inputting the first feature map and the potential unknown candidate box set into an unknown label generation module to obtain the unknown candidate box set includes:

[0045] The second feature map is obtained by taking the average value in the direction of the number of channels of the first feature map;

[0046] Take the average value of all feature values ​​in the candidate box in the second feature map as the score of the corresponding candidate box;

[0047] In order from high to low scores, a preset number of candidate boxes are selected to form a set of unknown candidate boxes.

[0048] The present application proposes a road spill detection method based on contrast clustering self-learning, which has the following beneficial effects:

[0049] On the one hand, compared with the general spilled object detection method based on background modeling, it can identify the scattered objects stationary on the highway and has a strong anti-interference ability. On the other hand, compared with the general spilled object detection method based on deep learning, it can detect the spilled object targets that are not included in the spilled object category of the data set, so it is more practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of the road spilled object detection method based on comparative clustering self-learning in this application;

[0051] Figure 2 This is a schematic diagram of the road spilled object detection network model for this application;

[0052] Figure 3 This is a schematic diagram of a candidate area module in an embodiment of the present application;

[0053] Figure 4 This is a schematic diagram of a spilled object recognition module based on a maximum logical value according to an embodiment of the present application;

[0054] Figure 5 This is a classification flow chart in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In one embodiment, Figure 1 As shown in the figure, a road spilled object detection method based on contrast clustering self-learning is proposed, including:

[0057] Step S1, constructing a road spilled object detection network model, wherein the road spilled object detection network model includes a feature extraction module, a candidate area module, an unknown label generation module and a spilled object recognition module based on a maximum logical value.

[0058] This application constructs a road spill detection network model for spill detection, such as Figure 2 As shown, it includes a feature extraction module, a candidate area module, an unknown label generation module and a spilled object recognition module based on a maximum logical value.

[0059] The feature extraction module is used to extract feature maps; the candidate area module is used to generate general candidate boxes and potential unknown candidate boxes; the unknown label generation module is used to generate unknown candidate boxes; and the spilled object recognition module based on the maximum logical value is used to recognize spilled objects and output a predicted box.

[0060] Step S2: Use training samples to train the road spilled object detection network model, use the trained road spilled object detection network model to detect the image to be detected, and output the detection result.

[0061] During training, we first construct a training data set N = {X, Y}, where X and Y represent the image set and label set respectively, and X = {I1,...,I M} indicates that there are M training images in total; and Y = {Y1,...,Y M}, where Y i ={y1,y2,...,y K} indicates picture I i The categories and positions of the K objects in .

[0062] Specifically, y k =[v k ,x k ,y k ,w k ,h k ], x k ,y k and w k ,h k They are the center coordinates and width and height of the bounding box respectively.

[0063] All training samples are labeled as follows: and non-spillage target categories The total number of categories H = k1 + k2.

[0064] For example, the dataset contains a total of 10 categories, including 5 categories of spilled objects: plastic bags, cartons, mineral water bottles, tires, and stones, coded as 1, 2, 3, 4, and 5 in order; 5 categories of non-spilled objects: pedestrians, cars, bicycles, motorcycles, and triangular cones, coded as 6, 7, 8, 9, and 10 in order. In particular, code 0 represents an unknown category. 3,000 images are collected and annotated for each category, and the bounding box is encoded in relative coordinates.

[0065] The training samples are then input into the road spilled object detection network model for training, the loss is calculated, and the network parameters are updated until the training is completed.

[0066] After the training is completed, the image to be detected is input into the trained road spilled object detection network model to detect the image to be detected and output the detection result.

[0067] When the present application trains the road spilled object detection network model or uses the trained road spilled object detection network model to detect the image to be detected, the road spilled object detection network model performs the following operations on the input training sample image or the image to be detected:

[0068] Step F1: input the training sample image or the image to be detected into the feature extraction module to extract the first feature map.

[0069] The feature extraction module of this embodiment adopts the ResNetXt-50 network, the input is the image Ii, and the output is the feature map F1, that is, the first feature map.

[0070] Assuming the height and width of the input image are h×w (608×608), the output of each stage of ResNetXt-50 is shown in the following table:

[0071]

[0072] Table 1

[0073] The feature maps output by Conv3, Conv4 and Conv5 of the ResNetXt-50 network are C3, C4 and C5 respectively, and finally the feature map F1=C5 is output. The feature map F1 is output to the candidate region module, the unknown label generation module and the spilled object recognition module based on the maximum logical value. The ResNetXt-50 network is a relatively mature technology and will not be described here.

[0074] Step F2: input the first feature map into the candidate area module to obtain a potential candidate box set, and filter out a general candidate box set and a potential unknown candidate box set from the potential candidate box set according to preset rules.

[0075] The candidate region module of this embodiment takes feature map F1 as input and outputs general candidate frame set P1 and potential unknown candidate frame set P2. The set of all potential candidate frames generated by the candidate region module is P, and satisfies P <P1∪P2, Denote any candidate box as p j =[α j ,x j ,y j ,w j ,h j ], where α jis the confidence of the candidate box, and x j ,y j and w j ,h j They are the center coordinates and width and height of the candidate box respectively.

[0076] like Figure 3 As shown, the candidate region module of this embodiment performs the following operations:

[0077] Anchor frame generation: The input image size is 608 pixels × 608 pixels, and the downsampling multiple of the feature extraction module is 32. Because 608 divided by 32 is 19, the size of the feature map F1 is 19 × 19, and the anchor frame grid size is also set to 19 × 19. This embodiment sets 9 anchor frame sizes, and each anchor frame size is a relative coordinate with the intersection of the anchor frame grid as the origin. For example, the anchor frame with sequence number 1 is relative to the origin (0, 0), and its coordinates on the input image are (-70, -30, 90, 50), where the coordinates represent the coordinates of the upper left corner of the anchor frame (-70, -30) and the coordinates of the lower right corner of the anchor frame (90, 50), and the same applies to other anchor frame sizes. In this embodiment, the anchor frame grid has a total of 361 intersections, and each intersection generates 9 anchor frame coordinates, so a total of 3249 anchor frame coordinates are generated, and one anchor frame coordinate corresponds to one anchor frame.

[0078] Regression: The regression branch is used to generate unprocessed candidate boxes. The regression branch predicts the coordinates that each anchor box needs to offset, and the coordinates of the anchor box after offset are the candidate boxes.

[0079] Classification: The classification branch is used to predict the probability that any candidate box is a foreground. For example, for a candidate box p j , the classification branch will predict the confidence α that this candidate box is the foreground j .

[0080] NMS classification: The input of the NMS part is the unprocessed candidate box predicted by the regression branch and the confidence of each candidate box predicted by the classification branch. The candidate boxes with a high degree of overlap are screened out, and the remaining candidate boxes form the potential candidate box set P.

[0081] It should be noted that processing the feature map to obtain a candidate frame set is a relatively mature technology in the technical field and will not be described in detail here. In a specific embodiment, when training a road spilled object detection network model, a general candidate frame set and a potential unknown candidate frame set are screened from the candidate frame set according to preset rules, including:

[0082] Filter out the candidate boxes whose confidence is greater than or equal to the first threshold from the potential candidate box set P to form a set P a , then, the set P aThe candidate boxes in the image and the bounding boxes in the corresponding labels are combined in pairs to calculate the intersection-over-union ratio and generate an IOU matrix.

[0083] According to the IOU matrix, from the set P a The candidate boxes whose IOU values ​​with any bounding box in the label are less than or equal to the second threshold are selected to form a set P b ;

[0084] From the set P a The candidate boxes whose iou values ​​with any bounding box in the label are greater than the second threshold are selected to form the general candidate box set P1;

[0085] The set P b The candidate boxes in are sorted from high to low according to the confidence, and a preset number of candidate boxes are selected from high to low according to the confidence to form a potential unknown candidate box set P2.

[0086] For example, for the input image I i , whose corresponding label is Y i , filter out the confidence α from the candidate box set P j The candidate boxes with a value greater than or equal to 0.65 form a set P a , that is to say, P a The candidate boxes in the middle are those with a high probability of enclosing the target. Then, the set P a Candidate boxes and labels Y i The bounding boxes in the set P are combined to calculate the intersection-over-union ratio and form an IOU matrix. The value in the IOU matrix is ​​represented by iou. The larger the iou value, the better the overlap between the candidate box and the bounding box in the label. Finally, according to the IOU matrix, a Filter out and label Y i The candidate boxes whose iou values ​​of any bounding boxes are less than or equal to 0.5 form a set P b , which is generally considered to contain no targets of known categories. a Filter out and label Y i The candidate boxes whose IOU values ​​of any bounding boxes are greater than 0.5 are selected as positive examples to form the general candidate box set P1. b The candidate box in is represented by the confidence α j Sort in descending order, and then select the top Q candidate boxes to form the potential unknown candidate box set P2.

[0087] In a specific embodiment, when the road spilled object detection network model is used for detection, a general candidate frame set and a potential unknown candidate frame set are screened out from the potential candidate frame set according to a preset rule, including:

[0088] From the potential candidate box set P, select the candidate boxes whose confidence is greater than or equal to the third threshold to form a set P1, and the candidate boxes whose confidence is less than the third threshold to form a set P b ;

[0089] The set P b The candidate boxes in are sorted from high to low according to the confidence, and a preset number of candidate boxes are selected from high to low according to the confidence to form a potential unknown candidate box set P2.

[0090] It is easy to understand that the set P b The candidate boxes in may contain objects that are not labeled in the input image because the confidence given by the model is very high. Then, the set P b The candidate boxes in are sorted from high to low by confidence, and the top five candidate boxes are selected to form a potential unknown candidate box set P2.

[0091] It should be noted that, in this embodiment, the third threshold value may be equal to the first threshold value, or a new third threshold value may be set, and the specific value may be determined through experimental results.

[0092] Step F3: input the first feature map and the potential unknown candidate box set into the unknown label generation module to obtain the unknown candidate box set.

[0093] The unknown label generation module of this embodiment inputs the feature map F1 and the potential unknown candidate box set P2, and outputs the unknown candidate box set P3.

[0094] In a spatial region, the degree of activation value on the feature map reflects the possibility that the spatial region contains the target, so it can be used to evaluate the confidence of the feature map. Specifically, it includes:

[0095] The second feature map is obtained by taking the average value in the direction of the number of channels of the first feature map;

[0096] Take the average value of all feature values ​​in the candidate box in the second feature map as the score of the corresponding candidate box;

[0097] In order from high to low scores, a preset number of candidate boxes are selected to form a set of unknown candidate boxes.

[0098] For example, for the feature map (D is the number of channels), take the average value in the direction of D to generate the feature map Then, the score s of each candidate box in P2 is calculated according to the following formula: j :

[0099]

[0100] Finally, the ratings jThe higher the value, the higher the probability that the candidate box contains an object. j Select E candidate boxes from the highest to the lowest order to form a set P3, and record the total candidate box set P all =P1∪P3, and the set P all Enter the spill recognition module based on the maximum logical value.

[0101] Step F4: merge the unknown candidate box set and the general candidate box set into a total candidate box set, and then input the first feature map and the total candidate box set into a spilled object recognition module based on the maximum logical value to obtain a prediction result.

[0102] The spilled object recognition module in this embodiment is based on the maximum logical value, and its input is the feature map F1 and the total candidate box P all , whose output is the prediction O i ={o1,o2,...,o m}.

[0103] Specifically, m =[cls m ,x m ,y m ,w m ,h m ], where cls m is equal to 0 or 1 to represent the type of the spill, 0 represents an unknown type of spill, 1 represents a known type of spill, and x m ,y m and w m ,h m They are the center coordinates and width and height of the prediction box respectively.

[0104] In a specific embodiment, the spilled object recognition module based on the maximum logic value includes a feature storage set F store , which is used to store the feature vectors of each unknown category, the feature vectors of the spilled object category, and the feature vectors of the non-sprinkled object target category.

[0105] Specifically, the spilled object recognition module based on the maximum logical value includes a feature storage set F store ={q0,...,q H}, each q i Each represents a first-in-first-out queue of fixed length L for storing feature vectors of a specific category, which stores all feature vectors of the specific category, and a total of L feature vectors can be stored. For any feature vector, this application uses f l Specifically, queue q0 is used to store feature vectors of unknown categories, and queue A feature vector for storing the category of the spilled object, and A feature vector storing the non-spillage target categories.

[0106] In another specific embodiment, the spilled object identification module based on the maximum logic value further includes a maximum logic value storage set E store , the maximum logical value storage set E store Includes a queue q for storing the maximum logical value of each unknown category feature vector u , and a queue q for storing the maximum logical value of the known category feature vector k .

[0107] Specifically, the spilled object recognition module based on the maximum logic value also includes a maximum logic value storage set E store = {q u ,q k}, queue q u It is used to store the maximum logical value of the unknown category feature vector, and the queue q k Used to store the maximum logical value of the known category feature vector, q u and q k There is no length limit.

[0108] In specific operations, such as Figure 5 As shown, from the total candidate box set P all Take out candidate boxes p one by one j , then according to p j Extract the region of interest r from the feature map F1 j . j Perform convolution operation (input convolution Conv e ), generate the feature vector f j In this application, the feature vector f j It is used to represent the feature vector corresponding to the currently processed image. Next, according to the feature vector f j The category is stored in F store The corresponding queue q in j Then, according to f j Belongs to an unknown category or a known category, from the feature vector f j Take the maximum logical value logit j Store the maximum logical value storage set E store The queue q u or q k In particular, the eigenvector f j The value of each dimension in is called a logical value, so logit j is to take the eigenvector f j The largest value among all dimensions in .

[0109] In a specific embodiment, the spilled object recognition module based on the maximum logic value performs the following operations during training:

[0110] The candidate boxes are taken out one by one from the total candidate box set, and then the region of interest is extracted from the first feature map according to the candidate boxes, and the region of interest is convolved to generate a feature vector to be classified, and the feature vector to be classified is stored in the feature storage set F according to the category to which the feature vector to be classified belongs. store In the corresponding queue;

[0111] Performing regression analysis on the feature vector to be classified, predicting the bounding box of the feature vector to be classified, calculating the intersection-and-union ratio of the bounding box and the label in turn, and if the maximum value of the intersection-and-union ratio is greater than or equal to a fourth threshold, calculating the loss function and updating the parameters of the road spilled object detection network model;

[0112] According to whether the feature vector to be classified belongs to an unknown category or a known category, the maximum logical value is taken from the feature vector to be classified and stored in the maximum logical value storage set E. store in the corresponding queue;

[0113] Calculate the maximum logical value storage set E respectively store The average of the two queues in, where queue q u The mean value is μ u , and queue q k The mean value is μ k , determine the classification decision function α(logit j ):

[0114]

[0115] Among them, logit j is the feature vector to be classified j The maximum logical value, α(logit j )=0 represents the feature vector to be classified f j belongs to the unknown category, and α(logit j )=1 represents the feature vector to be classified f j belongs to a known category. In particular, the eigenvector f j The value of each dimension in is called a logical value, so logit j is to take the eigenvector f j The largest value among all dimensions in .

[0116] See also Figure 4 , Figure 5 During training, the input training sample pictures are selected from the total candidate box set P all Take out candidate boxes p one by one j , then according to p jExtract the region of interest r from the feature map F1 j . j Perform convolution operation (input convolution Conv e ), generate the feature vector f j The training samples have been labeled, so we can use the feature vector f j The category to which it belongs is stored in the feature storage set F store In the corresponding cohort in , the contrastive clustering loss is calculated.

[0117] The regression part is used to predict the feature vector f j Bounding box representing the target j =[x j ,y j ,w j ,h j ], during training, input image I i The label set is Y i , label set Y i Any label in is y k , where k = 1, 2, ..., K and K represents Y i The number of labels in . Calculate the bounding box bbox in turn j and label y k The intersection-and-union ratio generates the IOU matrix, IOU = {iou1,…,iou K}. If the maximum value iou in the IOU matrix m ≥0.5, the bbox is calculated using the L1smooth loss function j and m The regression loss is then updated through the gradient descent algorithm. If the maximum value iou in the IOU matrix m If <0.5, no processing is performed.

[0118] Then, according to whether the feature vector to be classified belongs to an unknown category or a known category, the maximum logical value is taken from the feature vector to be classified and stored in the maximum logical value storage set E. store During training, the classification process calculates the maximum logical value storage set E store The average of the two queues in, where queue q u The mean value is μ u , and queue q k The mean value is μ k , determine the classification decision function α(logit j ).

[0119] In a specific embodiment, the spilled object recognition module based on the maximum logic value performs the following operations when recognizing the image to be detected:

[0120] Taking candidate boxes one by one from the total candidate box set, then extracting the region of interest from the first feature map according to the candidate boxes, performing a convolution operation on the region of interest, generating a feature vector to be classified, and predicting a bounding box through regression analysis;

[0121] Using the classification decision function α(logit j ) Determine whether the currently extracted feature vector to be classified belongs to an unknown category or a known category. If the feature vector to be classified belongs to an unknown category, output the first prediction result. If the feature vector to be classified belongs to a known category, calculate the Euclidean distance between the feature vector to be classified and the prototype vector in the feature prototype set in turn, take the category corresponding to the minimum Euclidean distance as the recognition result, and output the second prediction result.

[0122] Specifically, after the image is input into the model, the feature vector f is obtained j , j = 1, 2, ..., M, where M is the number of eigenvectors. At the same time, the eigenvector f is predicted by regression analysis j The bounding box of the target represented by j ,y j ,w j ,h j Then, calculate the maximum logit value of the eigenvector j Next, the classification process calculates the classification decision function α (logit j ) value. If α(logit j )=0, then the eigenvector f j The target is a thrown object of unknown category and outputs the predicted bounding box [x j ,y j ,w j ,h j ] and its category is “unknown category spilled objects”; if α(logit j )=1, then the eigenvector f j The target represented is of known category, that is, the category of this target is one of plastic bags, cartons, mineral water bottles, tires, stones, pedestrians, cars, bicycles, motorcycles and triangular cones.

[0123] If α(logit j )=1, then according to the Euclidean distance formula D, the feature vector f is calculated in sequence j and the prototype vector p in the category prototype set V i The distance d i , forming a distance set M = {d1,...,d H}; then take the minimum value d in the distance set M m, where m∈{1,2,…,H}. If m∈{1,2,…,k1}, where k1 is the number of known categories of spilled objects mentioned above; then the feature vector f j The target represented by is a known category of spilled objects, that is, the category of this target is one of plastic bags, cartons, mineral water bottles, tires and stones; at this time, the output predicted bounding box [x j ,y j ,w j ,h j ] and its category is "unknown category of spilled objects". If m∈{k1+1,…,k1+k2}, where k2 is the number of known non-spilled object categories mentioned above; then the feature vector f j The target represented is a known non-scattered object category, that is, the category of this target is one of pedestrians, cars, bicycles, motorcycles and triangular cones. At this time, no prediction is output.

[0124] This embodiment uses the classification decision function α(logit j ) Determine the currently extracted feature f j Is it an unknown category? If the feature vector f j Belongs to an unknown category, then output prediction o j =[0,x j ,y j ,w j ,h j ]. Otherwise, if f j is a known category, and according to the Euclidean distance formula D, f is calculated in sequence j and the prototype vector p in the category prototype set i The distance d i , we get the distance set M = {d1,...,d H}, take the category corresponding to the minimum value in M ​​as v n Finally, if v n ∈S, then the feature f j represents the category of scattered objects contained in the data set, then the output prediction o j =[1,x j ,y j ,w j ,h j ].

[0125] In a specific embodiment, the loss function of the road spilled object detection network model of the present application includes: regression loss L reg and contrast clustering loss L cont . The regression loss L reg This is a relatively mature technology in this field, so I will not go into details here. The following mainly describes the comparison of clustering loss L cont .

[0126] The contrastive clustering loss L cont The calculation process is as follows:

[0127] Using the feature storage set F store Each FIFO queue q of length L in i The stored feature vector to be classified f l Calculate the corresponding prototype vector p i , and all prototype vectors p i Combined into a category prototype set V = {p0,...,p H};

[0128] Prototype vector p i Calculated according to the following formula:

[0129]

[0130] Then, for storage into queue q i The feature vector f to be classified corresponds to a picture in j , calculate f j Contrasting clustering loss of:

[0131]

[0132] Among them, l(f j , p i ) is defined as:

[0133]

[0134] Where D represents the Euclidean distance, m α and m β are two distance parameters, w θ is the weighting coefficient. The distance parameter and weighting parameter are determined by experiments, for example, m α Set to 3, m β Set to 10, w θ Set to 0.65. H is the total number of categories. Calculate the clustering loss so that the features f extracted by the model j It can be separated as much as possible in the feature space according to its category, so that the prototype vector in the category prototype set P can better represent its corresponding category and the maximum logical value Logit j It can be better used to distinguish known categories from unknown categories.

[0135] Finally, the regression loss L reg and contrast clustering loss L conr The weighted sum of is taken as the total loss, and then the gradient descent method is used to update the parameters of the network model, which will not be repeated here.

[0136] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A road spill detection method based on contrast clustering self-learning, characterized in that: The road spilled object detection method based on contrast clustering self-learning includes: Constructing a road spilled object detection network model, wherein the road spilled object detection network model includes a feature extraction module, a candidate area module, an unknown label generation module, and a spilled object recognition module based on a maximum logical value; Using training samples to train the road spilled object detection network model, using the trained road spilled object detection network model to detect the image to be detected, and outputting the detection result; Wherein, when training the road spilled object detection network model, or using the trained road spilled object detection network model to detect the image to be detected, it includes: Inputting the training sample image or the image to be detected into the feature extraction module to extract the first feature map; Inputting the first feature map into the candidate region module to obtain a potential candidate box set, and filtering out a general candidate box set and a potential unknown candidate box set from the potential candidate box set according to a preset rule; Inputting the first feature map and the potential unknown candidate box set into the unknown label generation module to obtain the unknown candidate box set; The unknown candidate frame set and the general candidate frame set are merged into a total candidate frame set, and then the first feature map and the total candidate frame set are input into a spilled object recognition module based on a maximum logical value to obtain a detection result; The spilled object recognition module based on the maximum logical value includes a feature storage set F store , used to store each unknown category feature vector, the spilled object category feature vector and the non-sprinkled object target category feature vector, the spilled object recognition module based on the maximum logical value also includes a maximum logical value storage set E store , the maximum logical value storage set E store Includes a queue q for storing the maximum logical value of each unknown category feature vector u , and a queue q for storing the maximum logical value of the known category feature vector k .

2. The road spilled object detection method based on contrast clustering self-learning according to claim 1 is characterized in that: When training the road spilled object detection network model, the general candidate frame set and the potential unknown candidate frame set are screened out from the potential candidate frame set according to the preset rules, including: Filter out the candidate boxes whose confidence is greater than or equal to the first threshold from the potential candidate box set P to form a set P a , then, the set P a The candidate boxes in the image and the bounding boxes in the corresponding labels are combined in pairs to calculate the intersection-over-union ratio and generate an IOU matrix. According to the IOU matrix, from the set P a The candidate boxes whose IOU values ​​with any bounding box in the label are less than or equal to the second threshold are selected to form a set P b ; From the set P a The candidate boxes whose iou values ​​with any bounding box in the label are greater than the second threshold are selected to form the general candidate box set P1; The set P b The candidate boxes in are sorted from high to low according to the confidence, and a preset number of candidate boxes are selected from high to low according to the confidence to form a potential unknown candidate box set P2.

3. The road spilled object detection method based on contrast clustering self-learning according to claim 1 is characterized in that: When the trained road spilled object detection network model is used to detect the image to be detected, the general candidate frame set and the potential unknown candidate frame set are screened out from the potential candidate frame set according to the preset rules, including: From the potential candidate box set P, select the candidate boxes whose confidence is greater than or equal to the third threshold to form a set P1, and the candidate boxes whose confidence is less than the third threshold to form a set P b ; The set P b The candidate boxes in are sorted from high to low according to the confidence, and a preset number of candidate boxes are selected from high to low according to the confidence to form a potential unknown candidate box set P2.

4. The road spilled object detection method based on contrast clustering self-learning according to claim 1 is characterized in that: The spilled object recognition module based on the maximum logical value performs the following operations during training: The candidate boxes are taken out one by one from the total candidate box set, and then the region of interest is extracted from the first feature map according to the candidate boxes, and the region of interest is convolved to generate a feature vector to be classified, and the feature vector to be classified is stored in the feature storage set F according to the category to which the feature vector to be classified belongs. store In the corresponding queue; Performing regression analysis on the feature vector to be classified, predicting the bounding box of the feature vector to be classified, calculating the intersection-and-union ratio of the bounding box and the label in turn, and if the maximum value of the intersection-and-union ratio is greater than or equal to a fourth threshold, calculating the loss function and updating the parameters of the road spilled object detection network model; According to whether the feature vector to be classified belongs to an unknown category or a known category, the maximum logical value is taken from the feature vector to be classified and stored in the maximum logical value storage set E. store in the corresponding queue; Calculate the maximum logical value storage set E respectively store The average of the two queues in u The mean value is μ u , and queue q k The mean value is μ k , determine the classification decision function α(logit j ): Among them, logit j is the feature vector to be classified j The maximum logical value, α(logit j )=0 represents the feature vector to be classified f j belongs to the unknown category, and α(logit j )=1 represents the feature vector to be classified f j belongs to a known category; The spilled object recognition module based on the maximum logical value performs the following operations when recognizing the image to be detected: Taking candidate boxes one by one from the total candidate box set, then extracting the region of interest from the first feature map according to the candidate boxes, performing a convolution operation on the region of interest, generating a feature vector to be classified, and predicting a bounding box through regression analysis; Using the classification decision function α(logit j ) Determine whether the currently extracted feature vector to be classified belongs to an unknown category or a known category. If the feature vector to be classified belongs to an unknown category, output the first prediction result. If the feature vector to be classified belongs to a known category, calculate the Euclidean distance between the feature vector to be classified and the prototype vector in the feature prototype set in turn, take the category corresponding to the minimum Euclidean distance as the recognition result, and output the second prediction result.

5. The road spilled object detection method based on contrast clustering self-learning according to claim 4 is characterized in that: The loss function of the road spilled object detection network model includes: regression loss L reg and contrast clustering loss L cont , the contrastive clustering loss L cont The calculation process is as follows: Using the feature storage set F store Each FIFO queue q of length L in i The stored feature vector to be classified f l Calculate the corresponding prototype vector p i , and all prototype vectors p i Combined into a category prototype set V = {p0,...,p H }; Prototype vector p i Calculated according to the following formula: Then, for storage into queue q i The feature vector f to be classified corresponds to a picture in j , calculate f j Contrasting clustering loss of: in, Defined as: Where D represents the Euclidean distance, m α and m β are two distance parameters, w θ is the weighting coefficient.

6. The road spilled object detection method based on contrast clustering self-learning according to claim 1 is characterized in that: The step of inputting the first feature map and the potential unknown candidate box set into the unknown label generation module to obtain the unknown candidate box set includes: The second feature map is obtained by taking the average value in the direction of the number of channels of the first feature map; Take the average value of all feature values ​​in the candidate box in the second feature map as the score of the corresponding candidate box; In order from high to low scores, a preset number of candidate boxes are selected to form a set of unknown candidate boxes.

Images