Typical scene-based rare traffic sign instance generation method

Abstract

The invention discloses a rare traffic sign instance generation method based on typical scenes, which uses an unsupervised scene classification algorithm to divide pictures in a traffic sign data set into a plurality of scenes, and designs an'acquisition-coverage 'technology to acquire a target instance of a traffic sign and cover the target instance on other traffic signs. Data augmentation of rare traffic signs in a traffic data set is realized; according to the method, a data set related to traffic signs is input, for each data set picture, firstly, the traffic scene classification where the picture is located is calculated through an unsupervised classification algorithm, then after rare traffic sign instances are obtained through an obtaining-covering method, the instances are pasted and covered on target instances of other traffic scenes, and finally, the target instances of other traffic scenes are obtained. And a new data set sample is formed, and data augmentation of rare traffic signs is realized. According to the method provided by the invention, data augmentation of rare traffic signs in a traffic scene can be effectively realized.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and pattern recognition, and in particular relates to a method for generating rare traffic sign instances based on typical scenarios. Background Art

[0002] Traffic sign detection for autonomous driving is a basic visual task in which the on-board camera uses visible light information to record vehicle road information and discover the type and location of traffic signs on the road. It can provide support for other downstream tasks of autonomous driving, and it can also help drivers understand and make decisions on the road, thereby improving driving safety. It is of great significance to promote the industrialization of autonomous driving systems.

[0003] The long-tail problem in traffic sign detection for autonomous driving refers to the fact that in the traffic sign dataset, a few traffic sign categories have a large number of samples, while most traffic sign categories have only a small number of samples, which will lead to poor detection performance of the autonomous driving system for traffic sign categories with sparse samples. Early research methods dealt with the long-tail problem by roughly adjusting the number of samples and the weights between categories. This approach easily led to a seesaw problem, that is, the performance of the detection network for rare categories improved, while the performance for frequent categories decreased. In more serious cases, it will cause the detection network to overfit for rare categories. In the traffic environment, issues such as traffic scene understanding and traffic sign context information also need to be considered. Therefore, how to make the traffic sign detector learn in a balanced way in a long-tail environment is a challenging and meaningful task. Summary of the invention

[0004] The object of the present invention is to provide a method for generating rare traffic sign instances based on typical scenarios.

[0005] The technical solution to realize the present invention is: a method for generating rare traffic sign instances based on typical scenarios, comprising the following steps:

[0006] Step 1: Use an image classification algorithm based on unsupervised learning. The algorithm first presets the number of scenes, then uses proxy tasks to continuously train the neural network, uses the loss value of the proxy task to back propagate, continuously optimizes the neural network, and finally uses the neural network to divide the pictures in the traffic dataset into several scenes. The algorithm finally outputs the scene number corresponding to each picture;

[0007] Step 2: Record all traffic sign scene graphs corresponding to each scene number;

[0008] Step 3: Count the number of each traffic sign class in the traffic sign dataset, take the six traffic signs with the least number as rare traffic signs, and obtain a subset of images containing rare traffic signs from the traffic sign dataset;

[0009] Step 4: For a picture containing sparse traffic signs in step 3, select all sparse traffic sign target instances in the picture;

[0010] Step 4: Using the result in step 1, get the scene number where the rare traffic target instance in step 4 is located

[0011] Step 6: Randomly select a scene number for the traffic sign instance selected in step 4;

[0012] Step 7: Compare the random number obtained in step 6 with the scene number corresponding to the traffic sign instance in step 5. If they are equal, repeat step 6 to obtain a new scene number. If they are not equal, continue to step 8.

[0013] Step 8: Using the scene number obtained in step 6, randomly select a traffic sign scene image with the same scene number as that in step 6 from the results recorded in step 2;

[0014] Step 9: Using the “get-cover” technique, the traffic sign instance in step 4 and the traffic sign scene image selected in step 8 are combined into a new traffic sign augmented image, which is added to the traffic sign augmented dataset;

[0015] Step 10: Jump to step 4 and repeat until all rare traffic sign images are augmented;

[0016] Step 11: Merge the traffic sign augmented dataset with the original traffic sign dataset. This completes the data augmentation algorithm.

[0017] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for generating rare traffic sign instances based on typical scenarios is implemented.

[0018] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for generating rare traffic sign instances based on typical scenarios.

[0019] A computer program product includes a computer program, which implements the above-mentioned method for generating rare traffic sign instances based on typical scenarios when executed by a processor.

[0020] Compared with the prior art, the present invention has the following significant advantages:

[0021] (1) The present invention aims to augment and expand the training set to solve the problem of poor performance in sparse traffic sign detection without any modification to the neural network structure, so it will not cause any time burden when deploying inference.

[0022] (2) The present invention provides a general augmentation method for traffic sign datasets. In addition to solving the problem of poor performance in detecting sparse traffic signs, the method of the present invention can be migrated to other target detection datasets to solve the performance problem of sparse categories in other datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Flow chart of the scarce traffic sign data augmentation method of the present invention.

[0024] Figure 2 This is the effect diagram of the copy-paste method. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0026] The present invention proposes a method for generating rare traffic sign instances based on typical scenarios, which mainly includes calculating and analyzing the belonging scene of each traffic scene image, obtaining the target instance of the traffic sign, and synthesizing and outputting a new traffic scene augmented image according to the scene of the traffic sign instance and the randomly acquired picture scene.

[0027] Combination Figure 1 As shown, the implementation of this method generally includes the following 11 steps:

[0028] Step A1, using an image classification algorithm based on unsupervised learning, the algorithm first presets the number of scenes, and then uses the proxy task to reduce the value of the loss function, and divides the pictures in the traffic data set into several scenes, such as highways, crosswalks, green streets, and bridge passages. The algorithm finally outputs the scene number corresponding to each picture.

[0029] Step A2, recording all traffic sign scene graphs corresponding to each scene number.

[0030] Step A3: Count the number of each traffic sign category in the traffic sign dataset, take the six traffic signs with the least number as rare traffic signs, and obtain a subset of images containing traffic signs from the traffic sign dataset.

[0031] Step A4: for a picture containing rare traffic signs in step A3, select all rare traffic sign target instances in the picture.

[0032] Step A5, using the result in step A1, obtain the scene number where the sparse traffic target instance in step A4 is located.

[0033] Step A6, randomly selecting a scene number for the traffic sign instance selected in step A4.

[0034] Step A7, compare the random number obtained in step A6 with the scene number corresponding to the traffic sign instance in step A5, if they are equal, repeat step A6 to obtain a new scene number, if they are not equal, continue to step A8.

[0035] Step A8, using the scene number obtained in step A6, randomly select a traffic sign scene image with the same scene number as that in step A6 from the results recorded in step A2.

[0036] Step A9, using the “get-cover” technique, the traffic sign instance in step A4 and the traffic sign scene image selected in step A8 are combined into a new traffic sign augmented image, which is added to the traffic sign augmented dataset. Some augmented images are as follows: Figure 2 The red boxes represent the signs augmented by the copy-paste method, a) pasting a 70km / h maximum speed limit sign into the picture, b) pasting a 5m height limit sign into the picture, and c) pasting a dual-use motorcycle prohibition sign into the picture.

[0037] Step A10, jump to step A4 and repeat until all rare traffic sign images are augmented.

[0038] Step A11: merge the traffic sign augmented data set with the original traffic sign data set, and the result of the merging is the content of the original traffic sign data set and the content of the augmented traffic sign data set. The data augmentation algorithm ends here.

[0039] In the above method, the specific steps of A1 are:

[0040] 1) Based on the proxy task, the proxy loss is designed to minimize the loss to complete the representation learning of the neural network. The proxy loss formula is:

[0041]

[0042] Among them, X i represents the original image, T[X i ] represents the image after affine transformation, θ represents the weight of the neural network, Φ θ (·) represents the image features obtained under the mapping of the neural network with weight θ, and d(·) represents the cosine distance between two image features;

[0043] 2) Using the prior knowledge that "samples and semantically similar samples have a high probability of belonging to the same category", we design a clustering loss and minimize the loss to complete a fine-tuning of the neural network. The formula for clustering loss is:

[0044]

[0045] Where D represents the total number of images. In the first term of the equation, X represents a certain image, and N x represents the semantically similar sample set of image X, k represents an image in the semantically similar sample set of image X, <·> represents the dot product, η represents the weight of the neural network, Φ η (·) represents the image clustering result obtained under the mapping of the neural network with weight η, <Φ η (X),Φ η (k)> means to perform dot multiplication on the neural network prediction results of the two images. The second term of the equation is a regular term in the form of information entropy. λ is a hyperparameter used to control the weight of the second term of the equation. In this method, it is 5. C is a hyperparameter, which indicates the pre-set number of clusters. c indicates one of the clusters in the number of clusters. It represents the probability that the function assigns sample X to cluster c.

[0046] 3) Using the prior knowledge that "a well-initialized network has a high probability of predicting a sample with high confidence and matching the true label", we construct pseudo labels, design self-label loss, and minimize the loss to fine-tune the neural network again. The formula for self-label loss is:

[0047] Loss=∑log(Φ η (T s [D])·Y

[0048] D=X i where max(Φ η (X i ))>Threshold

[0049] Y=Φ η (D)

[0050] where Φ η (·) represents the network’s predicted distribution of sample categories, T s [·] represents the enhanced transformation of the sample, Threshold is a hyperparameter representing the high confidence threshold mentioned above, and D represents a high confidence sample.

[0051] 4) After two fine-tunings of the neural network, the unsupervised classification task is completed. By inputting a traffic sign scene graph, the scene classification number of the scene graph can be obtained.

[0052] In the above method, the specific steps of A9 are:

[0053] 1) Use the slice operation to copy the source traffic sign instance, complete the "get" operation, and obtain the width and height of the target traffic sign instance and the four corner coordinates (x1, y1, x2, y2) in the real image.

[0054] 2) Use downsampling to scale the image, or use upsampling to expand the source traffic sign instance to the size of the target traffic sign instance.

[0055] 3) Use the slicing operation to set the slicing starting point of the target traffic instance, assign the source traffic sign instance to the target traffic sign instance, and complete the "overwrite" operation.

[0056] In response to the long-tail distribution problem of traffic signs in traffic sign detection scenarios, the present invention extracts each target instance containing a rare traffic sign category based on unsupervised scene classification and an "acquire-cover" image augmentation strategy, and further synthesizes a new traffic scene augmented map based on the scene information of the traffic sign dataset. Specifically, it includes: first, using a slicing operation to obtain a target instance of a rare category of traffic signs, then using the scene where the target instance is located to compare with a randomly selected scene map, and then using a slicing assignment operation to embed the target instance into the randomly selected scene map to complete the augmentation operation of the traffic sign dataset. Using the method proposed in the present invention, the scene diversity of rare categories of traffic signs can be increased, the detection level of the target detector on rare categories of traffic signs can be improved, and the long-tail problem of traffic sign distribution in traffic scenes can be effectively solved.

Claims

1. A method for generating rare traffic sign instances based on typical scenarios, characterized in that: The steps include: Step 1: Use an image classification algorithm based on unsupervised learning. The algorithm first presets the number of scenes, then uses proxy tasks to continuously train the neural network, uses the loss value of the proxy task to back propagate, and continuously optimizes the neural network. Finally, the neural network is used to divide the pictures in the traffic data set into several scenes. The algorithm finally outputs the scene number corresponding to each picture; Step 2: Record all traffic sign scene graphs corresponding to each scene number; Step 3: Count the number of each traffic sign class in the traffic sign dataset, take the six traffic signs with the least number as rare traffic signs, and obtain a subset of images containing rare traffic signs from the traffic sign dataset; Step 4: For a picture containing sparse traffic signs in step 3, select all sparse traffic sign target instances in the picture; Step 4: Using the result in step 1, obtain the scene number where the sparse traffic target instance in step 4 is located; Step 6: Randomly select a scene number for the traffic sign instance selected in step 4; Step 7: Compare the random number obtained in step 6 with the scene number corresponding to the traffic sign instance in step 5. If they are equal, repeat step 6 to obtain a new scene number. If they are not equal, continue to step 8. Step 8: Using the scene number obtained in step 6, randomly select a traffic sign scene image with the same scene number as that in step 6 from the results recorded in step 2; Step 9: Using the "get-cover" technique, combine the traffic sign instance in step 4 and the traffic sign scene image selected in step 8 into a new traffic sign augmented image and add it to the traffic sign augmented dataset; Step 10: Jump to step 4 and repeat until all rare traffic sign images are augmented; Step 11: Merge the traffic sign augmented dataset with the original traffic sign dataset. This completes the data augmentation algorithm.

2. The method according to claim 1, characterized in that The step 1 is specifically as follows: Step 1-1: Based on the proxy task, design the proxy loss to minimize the loss to complete the representation learning of the neural network. The proxy loss formula is: Among them, X i represents the original image, T[X i ] represents the image after affine transformation, θ represents the weight of the neural network, Φ θ (·) represents the image features obtained under the mapping of the neural network with weight θ, and d(·) represents the cosine distance between two image features; Step 1-2: Design a clustering loss and minimize the loss to complete a fine-tuning of the neural network in step 1-3. The formula for clustering loss is: Where D represents the total number of images. In the first term of the equation, X represents a certain image, and N x represents the semantically similar sample set of image X, k represents an image in the semantically similar sample set of image X, <·> represents the dot product, η represents the weight of the neural network, Φ η (·) represents the image clustering result obtained under the mapping of the neural network with weight η, <Φ η (x),Φ η (k)> means to perform dot multiplication on the neural network prediction results of the two images; the second term of the equation is a regular term in the form of information entropy, λ is a hyperparameter used to control the weight of the second term of the equation, C is a hyperparameter, indicating the pre-set number of clusters, c indicates one of the clusters in the number of clusters, Represents the probability that the function assigns sample X to cluster c; Step 1-3: Construct pseudo labels, design self-label loss, and minimize the loss to complete another fine-tuning of the neural network. The formula for self-label loss is: Loss=∑log(Φ η (T s [D]))·Y D=X i wheremax(Φ η (X i ))>Threshold Y= Φ the(D) where Φ η (·) represents the network’s predicted distribution of sample categories, T s [·] represents the enhanced transformation of the sample, Threshold is a hyperparameter representing the high confidence threshold, and D represents the high confidence sample. Step 1-4: After two fine-tuning of the neural network, the unsupervised classification task is completed, and the traffic sign scene graph is input to obtain the scene classification number of the scene graph.

3. The method according to claim 2, characterized in that The value of λ is 5.

4. The method according to claim 1, characterized in that The specific steps of step 9 are: Step 9-1: Use the slice operation to copy the source traffic sign instance, complete the "get" operation, and obtain the width and height of the target traffic sign instance and the four corner coordinates (x1, y1, x2, y2) in the real image; Step 9-2: scaling the image by downsampling, or expanding the source traffic sign instance to the size of the target traffic sign instance by upsampling; Step 9-3: Use the slicing operation to set the slicing starting point of the target traffic instance, assign the source traffic sign instance to the target traffic sign instance, and complete the "overwrite" operation.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

7. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method described in any one of claims 1 to 4 is implemented.

Images