Generation method and device of generalized small sample pavement anomaly detection model

Abstract

The embodiment of the invention discloses a generation method and device of a generalized small sample road surface anomaly detection model. A specific embodiment of the method comprises the following steps: performing base class and new class division on a data set; performing base class matching on each new class of data; performing initial round training to obtain an initial base class loss value; based on a preset round, executing the following training steps: inputting the base class new class matching pair into the initial road surface anomaly detection model to obtain a base class loss value of the current round; determining a dynamic tradeoff operator; determining the total loss value of the current round; performing parameter updating on the initial pavement anomaly detection model to obtain an updated pavement anomaly model; if the training round is equal to the preset round, determining the updated pavement anomaly model as a pavement anomaly detection model; and if the training round is smaller than the preset round, taking the updated pavement anomaly model as the initial pavement anomaly detection model to execute the training step again. According to the embodiment, the pavement anomaly detection model can be generated, and the accuracy of pavement anomaly detection is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method and device for generating a generalized small sample road surface anomaly detection model. Background Art

[0002] The emergence of road anomalies such as potholes, missing road markings and obstacles poses a serious threat to traffic safety. At the same time, the emergence of road anomalies is accidental and relatively scarce, resulting in a scarcity of road anomaly data. At present, when generating a detection model for road anomalies, the commonly used method is: small sample target detection method. Among them, the small sample target detection method often assumes that there are base class data with abundant data volume and new class data with scarce data volume in the data set. By using the base class data, the model can learn to locate and identify new class information of road anomalies that are rarely annotated.

[0003] However, when the above method is used to generate a road anomaly detection model, the following technical problems often occur:

[0004] The small sample target detection method only evaluates the performance of the model in detecting new classes, but ignores the performance of the basic classes, which is prone to false detection and missed detection, resulting in poor robustness in road anomaly detection, and thus causing the model to have low detection accuracy for road anomalies.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose a method and device for generating a generalized small sample road surface anomaly detection model to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide a method for generating a generalized small sample road anomaly detection model, the method comprising: dividing a preset road anomaly data set into base classes and new classes to obtain a base class data set and a new class data set; performing base class matching on each new class data in the above new class data set according to the base class data set to obtain a base class and new class matching pair set; inputting the above base class and new class matching pair set into an initial road anomaly detection model for initial round training to obtain an initial base class loss value; based on a preset round, performing the following model training steps: inputting the above base class and new class matching pair set into the initial road anomaly detection model to obtain the base class loss value of the current round and the new class loss value of the current round. loss value; determine the dynamic trade-off operator according to the above-mentioned initial base class loss value and the above-mentioned current round base class loss value; determine the total loss value of the current round according to the above-mentioned dynamic trade-off operator, the above-mentioned current round base class loss value, and the above-mentioned current round new class loss value; according to the above-mentioned current round total loss value, back-propagate parameter update of the above-mentioned initial road anomaly detection model to obtain an updated road anomaly detection model; in response to determining that the training round is equal to the preset round, determine the above-mentioned updated road anomaly detection model as the road anomaly detection model; in response to determining that the training round is less than the preset round, use the above-mentioned updated road anomaly detection model as the initial road anomaly detection model to execute the above-mentioned model training step again.

[0009] In a second aspect, some embodiments of the present disclosure provide a device for generating a generalized small sample road anomaly detection model, the device comprising: a base class and new class division unit, configured to divide a preset road anomaly data set into base classes and new classes to obtain a base class data set and a new class data set; a base class matching unit, configured to perform base class matching on each new class data in the above new class data set according to the base class data set to obtain a base class and new class matching pair set; an initial round training unit, configured to input the above base class and new class matching pair set into an initial road anomaly detection model for initial round training to obtain an initial base class loss value; a model training unit, configured to perform the following model training steps based on preset rounds: input the above base class and new class matching pair set into the initial road anomaly detection model; In the detection model, the base class loss value of the current round and the new class loss value of the current round are obtained; according to the above-mentioned initial base class loss value and the above-mentioned base class loss value of the current round, the dynamic trade-off operator is determined; according to the above-mentioned dynamic trade-off operator, the above-mentioned base class loss value of the current round and the above-mentioned new class loss value of the current round, the total loss value of the current round is determined; according to the above-mentioned total loss value of the current round, the parameters of the above-mentioned initial road surface anomaly detection model are back-propagated and updated to obtain an updated road surface anomaly detection model; in response to determining that the training round is equal to the preset round, the above-mentioned updated road surface anomaly detection model is determined as the road surface anomaly detection model; in response to determining that the training round is less than the preset round, the above-mentioned updated road surface anomaly detection model is used as the initial road surface anomaly detection model to perform the above-mentioned model training steps again.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the generation method of the generalized small sample road anomaly detection model of some embodiments of the present disclosure, the detection accuracy of the model for road anomalies can be improved. Specifically, the reason for the low detection accuracy of the model is that the small sample target detection method only evaluates the performance of the model in detecting new classes, but ignores the performance of the basic classes, which is prone to false detection and missed detection, resulting in poor robustness of road anomaly detection. Based on this, the generation method of the generalized small sample road anomaly detection model of some embodiments of the present disclosure, first, divides the preset road anomaly data set into base classes and new classes to obtain a base class data set and a new class data set. A large number of annotated base class data sets help the model learn common features, so that the model can better extract and classify features from new classes with a small amount of data. Secondly, according to the base class data set, base class matching is performed on each new class data in the above new class data set to obtain a base class and new class matching pair set. As a result, the model can mine the combined information of the base class and the new class, and give the model more category common knowledge and individual knowledge on limited data. Then, the above base class and new class matching pair set is input into the initial road anomaly detection model for initial round training to obtain the initial base class loss value. Finally, based on the preset round, the following model training steps are performed: the above base class and new class matching pair set is input into the initial road anomaly detection model to obtain the current round base class loss value and the current round new class loss value; according to the above initial base class loss value and the above current round base class loss value, a dynamic trade-off operator is determined; according to the above dynamic trade-off operator, the above current round base class loss value, and the above current round new class loss value, the current round total loss value is determined; according to the above current round total loss value, the above initial road anomaly detection model is back-propagated to update the parameters to obtain an updated road anomaly detection model; in response to determining that the training round is equal to the preset round, the above updated road anomaly detection model is determined as the road anomaly detection model; in response to determining that the training round is less than the preset round, the above updated road anomaly detection model is used as the initial road anomaly detection model to perform the above model training steps again. The dynamic trade-off operator is used to achieve adaptive parameter adjustment during the model training process, and at the same time, the model's detection performance for base class targets and new class targets is improved, thereby improving the robustness of the model and further improving the model's detection accuracy for road anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1is a flow chart of some embodiments of a method for generating a generalized small sample road surface anomaly detection model according to the present disclosure;

[0015] Figure 2 is a visualization diagram of road surface anomaly detection according to some embodiments of the method for generating a generalized small sample road surface anomaly detection model disclosed in the present invention;

[0016] Figure 3 is a schematic structural diagram of some embodiments of a device for generating a generalized small sample road surface anomaly detection model according to the present disclosure;

[0017] Figure 4 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 The process 100 of some embodiments of the method for generating a generalized small sample road surface anomaly detection model according to the present disclosure is shown. The method for generating a generalized small sample road surface anomaly detection model comprises the following steps:

[0025] Step 101 , dividing a preset road anomaly data set into a base class and a new class to obtain a base class data set and a new class data set.

[0026] In some embodiments, the execution entity may divide the preset road anomaly data set into a base class and a new class to obtain a base class data set and a new class data set.

[0027] Optionally, the above road anomaly dataset is constructed by the following steps:

[0028] The first step is to obtain a sample road surface anomaly data set. The sample road surface anomaly data includes a sample road surface anomaly image and a corresponding sample road surface anomaly label group. Here, the sample road surface anomaly image includes at least one road surface anomaly area. Secondly, the sample road surface anomaly label group includes sample road surface anomaly labels corresponding to each road surface anomaly area. Usually, the sample road surface anomaly label consists of a rectangular frame defining the road surface anomaly area and a category label representing the road surface anomaly category.

[0029] In practice, an open source road anomaly dataset with rich data volume and high annotation quality can be selected as a sample road anomaly dataset.

[0030] As an example, the open source road anomaly dataset may include but is not limited to at least one of the following: a road damage dataset (Road Damage Detector, RDD), a road lost cargo dataset (Lost and Found).

[0031] The second step is to perform data enhancement operations on each sample road anomaly data in the above sample road anomaly data set to obtain an enhanced road anomaly data set. Among them, the sample road anomaly images contained in each sample road anomaly data in the above sample road anomaly data set can be data enhanced by operations such as image rotation, image flipping, image cropping and color transformation to obtain an enhanced road anomaly image set. Here, one sample road anomaly image can correspond to multiple enhanced road anomaly images. At the same time, the sample road anomaly label group corresponding to the sample road anomaly image is subjected to the same image rotation, image flipping and image cropping operations to obtain an enhanced road anomaly label group set. Finally, the above enhanced road anomaly image set and the corresponding enhanced road anomaly label group set are determined as the enhanced road anomaly data set.

[0032] The third step is to determine the enhanced road anomaly data set as a road anomaly data set. The road anomaly data includes a road anomaly image and a corresponding road anomaly label group. Here, the road anomaly label group includes road anomaly labels corresponding to each road anomaly area. Usually, the road anomaly label consists of a rectangular frame defining the road anomaly area and a category label representing the road anomaly category. Here, the enhanced road anomaly image corresponding to each enhanced road anomaly data in the enhanced road anomaly data set can be determined as a road anomaly image. Secondly, the enhanced road anomaly label group corresponding to the enhanced road anomaly data is determined as a road anomaly label group.

[0033] In practice, data augmentation of sample road anomaly datasets can improve the generalization ability of the trained model.

[0034] In some optional implementations of some embodiments, the execution subject divides the preset road anomaly data set into base classes and new classes to obtain the base class data set and the new class data set, which may include the following steps:

[0035] The first step is to determine each road anomaly category corresponding to each road anomaly data in the road anomaly data set to obtain a road anomaly category set. In which, each road anomaly label in the road anomaly label group corresponding to each road anomaly data in the road anomaly data set corresponds to a road anomaly category in the road anomaly category set. Here, one road anomaly data corresponds to at least one road anomaly category.

[0036] For example, a road surface anomaly data may contain road surface anomaly categories such as "pothole", "zebra crossing defect" and "obstacle".

[0037] The second step is to determine the category data volume of each road anomaly category in the road anomaly category set according to the road anomaly label corresponding to each road anomaly data in the road anomaly data set, and obtain the category data volume set. Among them, the category data volume of each road anomaly category in the road anomaly category set can be determined by the number of each road anomaly label in the road anomaly label group corresponding to each road anomaly data in the road anomaly data set, and the category data volume set can be obtained. For example, the number of road anomaly labels with the road anomaly label "pothole" is the category data volume of the road anomaly category "pothole".

[0038] The third step is to determine the road anomaly category corresponding to the category data volume greater than or equal to the preset threshold in the above category data volume set as the basic category to obtain the basic category set. The preset threshold can be 10000. Here, the basic category is the road anomaly category with a larger data volume in the road anomaly data set.

[0039] In practice, road anomaly categories with a data volume greater than 10,000 are selected from the RDD dataset as the base categories. The data volume of these categories is sufficient to support the full training of the model. Because the sample size of each category in the Lost and Found dataset does not exceed 500 and is relatively rare, the six categories of data in the RDD dataset are uniformly selected as the base categories.

[0040] In the fourth step, the road anomaly category corresponding to the category data volume less than the preset threshold in the above category data volume set is determined as a newly added category to obtain a newly added category set. Among them, the newly added category is the newly discovered road anomaly category. Here, the newly added category is the road anomaly category with less data volume in the road anomaly data set.

[0041] In practice, categories with sparse data are selected as new categories. These categories have less data and need to be transferred through the knowledge of the base class. Four types of data were selected as new categories from both RDD and Lost and Found. However, in order to ensure the reliability and stability of the experimental results, some embodiments of the present invention follow the data partitioning method in the field of small sample learning, and set three different data set partitioning schemes on both data sets. That is, three classes are selected from the four new classes as new classes, forming a total of three different data subsets. The final experimental results will be averaged over the three data subsets to reduce the randomness of the experimental results.

[0042] In the fifth step, the road anomaly data corresponding to the above basic category in the road anomaly data set is determined as base class data to obtain a base class data set. According to each road anomaly label corresponding to each road anomaly data in the road anomaly data set, each road anomaly label and each road anomaly data corresponding to the basic category are taken as base class data to obtain a base class data set. Here, the base class data set is a set of each road anomaly data corresponding to the basic category in the road anomaly data set.

[0043] In the sixth step, the road surface anomaly data corresponding to the newly added category in the road surface anomaly data set is determined as new category data to obtain a new category data set. According to each road surface anomaly label corresponding to each road surface anomaly data in the road surface anomaly data set, each road surface anomaly label and each road surface anomaly data corresponding to the newly added category are used as new category data to obtain a new category data set. Here, the new category data set is a set of each road surface anomaly data corresponding to the newly added category in the road surface anomaly data set.

[0044] In practice, it is found that when using the two-stage training strategy of pre-training and fine-tuning, the following technical problems exist:

[0045] The model pre-trained on the base class may regard the new class as the background, and it is difficult to correct such recognition errors in the subsequent fine-tuning stage, thereby reducing the model's detection accuracy for the new class. The training strategy that combines the base class and the new class matching pair can fully tap the advantages of category-related knowledge and category-irrelevant knowledge, and the following solution can be decided.

[0046] Step 102, performing base class matching on each new class data in the new class data set according to the base class data set, to obtain a base class and new class matching pair set;

[0047] In some embodiments, the execution entity may perform base class matching on each new class data in the new class data set to obtain a base class and new class matching pair set.

[0048] In some optional implementations of some embodiments, the execution subject performs base class matching on each new class data in the new class data set to obtain a base class and new class matching pair set, which may include the following steps:

[0049] In the first step, each newly added category in the newly added category set is matched with each basic category in the basic category set to generate a base class and new class category pair set. The base class and new class category pair is a combination of the newly added category and the basic category. Here, the base class and new class category pair set contains all possible combinations of the newly added category and the basic category.

[0050] In the second step, data is allocated to each base class and new class category pair in the above base class and new class category pair set according to a preset ratio to obtain a base class and new class matching pair set. Among them, according to the base class and new class category pairs, a preset number of base class data and new class data can be randomly sampled from the base class data set and the new class data set respectively according to a preset ratio to obtain a base class data subset and a new class data subset. Afterwards, the above base class data subset and new class data subset are combined into a base class and new class matching pair.

[0051] In practice, it is assumed that the pavement transverse crack class in the base class and the manhole cover damage class in the new class form a base class and new class category pair. According to a 1:1 balanced ratio, 8 base class data with labels corresponding to pavement transverse cracks can be randomly sampled from the base class data set, and 8 new class data with labels corresponding to manhole cover damage can be randomly sampled from the new class data set to form a base class and new class matching pair of size 16. Here, 16 can be the size of the mini-batch data in model training. In this way, the number of base classes and new classes in each round can be ensured to be equal, thereby avoiding the problem of data imbalance.

[0052] The above step 102 and its related contents, as an inventive point of an embodiment of the present disclosure, solve the above technical problem "the model has a low detection accuracy for new classes". The factors that lead to the above technical problems are often as follows: the model pre-trained on the base class may regard the new class as the background, and it is difficult to correct such recognition errors in the subsequent fine-tuning stage. If the above factors are solved, the detection accuracy of the model for new classes can be improved. In order to achieve this effect, first, all category combinations are traversed to construct all possible base class and new class category pairs. In this way, category-related knowledge and category-irrelevant knowledge can be fully mined to improve the recognition ability of new class data. Afterwards, a balanced ratio of base class and new class matching pairs is constructed. Ensure that the number of base classes and new classes in each round is equal to avoid data imbalance problems. At the same time, small batch training of base class and new class matching pairs can enable the model to better jump out of the local optimum, thereby accelerating the model convergence speed. Therefore, the above steps can improve the detection accuracy of the model for new classes.

[0053] Step 103 , input the base class and new class matching pair set into the initial road surface anomaly detection model for initial round training to obtain the initial base class loss value.

[0054] In some embodiments, the execution entity may input the base class and new class matching pair set into an initial road anomaly detection model for initial round training to obtain an initial base class loss value.

[0055] In some optional implementations of some embodiments, the execution subject inputs the base class and new class matching pair set into the initial road surface anomaly detection model for initial round training to obtain the initial base class loss value, which may include the following steps:

[0056] The first step is to determine the initial parameters of the initial road anomaly detection model. The initial road anomaly detection model is a preset target detection model. Here, the initial parameters of the initial road anomaly detection model can be determined by the parameters of a public pre-trained model, or the initial parameters of the initial road anomaly detection model can be set to random numbers.

[0057] As an example, the preset target detection model may include but is not limited to at least one of the following: Faster R-CNN (Faster Regions-Convolutional Neural Network), SSD (Single Shot MultiBox Detector), etc.

[0058] In practice, some embodiments of the present disclosure are a universal network model learning strategy used to train network models, which is independent of the selection of a network framework.

[0059] In the second step, each base class and new class matching pair in the above base class and new class matching pair set is sequentially input into the above initial road anomaly detection model to obtain a base class detection result set. Among them, the base class data set corresponding to each base class and new class matching pair is input into the above initial road anomaly detection model to obtain a base class detection result set. Here, the base class detection result is the road anomaly detection result made by the initial road anomaly detection model for the road anomaly image in the base class data. Secondly, the base class detection result includes at least one result rectangular box defining the road anomaly area and the corresponding road anomaly result category.

[0060] The third step is to determine the loss value between each base class detection result in the above base class detection result set and the road anomaly label group corresponding to each base class data in the above base class data set to obtain the initial base class loss value. Here, the above initial base class loss value represents the loss value obtained for each base class data after the model has been trained for the initial round. Among them, the loss value may include a classification loss value and a regression loss value. Here, the classification loss value between the road anomaly result category and the true category label can be determined by the cross-entropy loss. Afterwards, the regression loss value between the result rectangular box and the true rectangular box can be determined by the smooth L1 loss.

[0061] In practice, the initial base class loss value is expressed as l b (θ 0 ). Where l represents the loss value. b is the base class identifier (base). b Represents the base class loss value. θ represents the parameters of the model. θ 0 Represents the parameters of the initial round model.

[0062] Step 104: Based on the preset rounds, perform the following model training steps:

[0063] Step 1041, input the base class and new class matching pair set into the initial road anomaly detection model to obtain the base class loss value of the current round and the new class loss value of the current round. Among them, by inputting the base class and new class matching pair set into the initial road anomaly detection model, the base class loss value of the current round and the new class loss value of the current round can be obtained through the loss value determination step in step 103. I will not go into details here. Here, the above-mentioned current round base class loss value represents the loss value obtained for each base class data after a round of training of the model. Secondly, the above-mentioned current round new class loss value represents the loss value obtained for each new class data after a round of training of the model.

[0064] Step 1042, determine the dynamic trade-off operator according to the initial base class loss value and the current round base class loss value.

[0065] In some embodiments, the execution entity may determine a dynamic trade-off operator based on the initial base class loss value and the current round base class loss value.

[0066] In some optional implementations of some embodiments, the execution subject determines the dynamic trade-off operator according to the initial base class loss value and the current round base class loss value, which may include the following steps:

[0067] The first step is to perform a hyperbolic tangent transformation on the initial base class loss value and the current round base class loss value to obtain the current round base class training degree value. Among them, the current round base class training degree value represents the base class training degree measured according to the change of the base class training loss value. Here, the initial base class loss value and the current round base class loss value can be transformed by a hyperbolic tangent transformation to obtain the current round base class training degree value through the following formula:

[0068]

[0069] Where t represents the current training round. p represents the training level value. b Indicates the base class training degree value. Indicates the base class training level value of the current t round. tanh() represents the hyperbolic tangent function. b (θ t ) represents the base class loss value of the current tth round.

[0070] In practice, the overall trend of loss value decreases with the increase of training rounds, so the difference between the initial base class loss value and the current round base class loss value can reflect the degree of model training. In order to more accurately quantify the value of the current round base class training degree, the initial base class loss value and the current round base class loss value are normalized by using the hyperbolic tangent function and then divided, which can reflect the current round of base class training degree.

[0071] The second step is to determine the dynamic trade-off operator based on the current round base class training degree value. The dynamic trade-off operator is determined by the base class training degree value and can be used to guide the model training value. The dynamic trade-off operator can be determined by the following formula based on the current round base class training degree value:

[0072]

[0073] Where λ represents the dynamic trade-off operator. t Represents the dynamic trade-off operator of the current round t. α is a parameter used in multi-task learning to control the sensitivity of the trade-off operator to changes in training progress, and is usually between 0.5 and 2.

[0074] Step 1043, determine the total loss value of the current round according to the dynamic trade-off operator, the base class loss value of the current round, and the new class loss value of the current round. The total loss value of the current round can be determined according to the dynamic trade-off operator, the base class loss value of the current round, and the new class loss value of the current round by the following formula:

[0075] l main =λ t l n (θ t )+l b (θ t )

[0076] Among them, l main Represents the total loss value. n is the identifier of the new class (new). n (θ t ) represents the loss value of the new class in the current tth round.

[0077] Step 1044, according to the total loss value of the current round, the initial road anomaly detection model is back-propagated to update the parameters to obtain an updated road anomaly detection model. The gradients of each layer of the initial road anomaly detection model can be determined by a back-propagation algorithm. Then, according to the gradients of each layer, the parameters of the initial road anomaly detection model are updated to obtain an updated road anomaly detection model.

[0078] Step 1045, in response to determining that the training round is equal to the preset round, determine the updated road surface anomaly detection model as the road surface anomaly detection model. If the training round has reached the preset round, the updated road surface anomaly detection model is used as the road surface anomaly detection model.

[0079] Step 1046: In response to determining that the number of training rounds is less than the preset number of rounds, the updated road anomaly detection model is used as the initial road anomaly detection model to perform the above model training step again. If the number of training rounds does not reach the preset number of rounds, the updated road anomaly detection model is used as the initial road anomaly detection model to perform the above model training step again.

[0080] Optionally, after the above step 104, the following steps are further included:

[0081] The first step is to obtain an abnormal road surface image to be tested, wherein the abnormal road surface image to be tested can be an image captured from a traffic scene, or can be an image frame in a traffic monitoring video.

[0082] The second step is to input the above-mentioned road anomaly image to be tested into the above-mentioned road anomaly detection model to obtain a road anomaly detection result map. The above-mentioned road anomaly detection model is generated by the above-mentioned steps 101 to 104. Secondly, the above-mentioned road anomaly detection result map contains a road anomaly area annotation box and a corresponding road anomaly category.

[0083] In practice, the above road anomaly detection model and the current advanced algorithm TFA-cos (Two-stage Fine-tuning Approach-cosine similarity, two-stage fine-tuning algorithm based on cosine similarity classifier) ​​for road anomaly detection are visualized as follows: Figure 2 As shown. Among them, Figure 2 Each column corresponds to the test sample image used in the test experiment. The first row is the test sample image and the corresponding ground truth (GT) label. The second row is the road anomaly detection result diagram output by the road anomaly detection model Ours w / DLW (Ours with Dynamic Lexicographic Weighting, this model based on the dynamic trade-off strategy), which includes a road anomaly area annotation box and the corresponding road anomaly category. The third row is the TFA-cos road anomaly detection result diagram output by the TFA-cos model. It can be seen that the road anomaly detection model (Ours w / DLW) generated by some embodiments of the present disclosure can detect road anomalies that are not detected by TFA-cos, with fewer false detections and missed detections, and the detection results are closer to the true value, which indicates that the road anomaly detection model generated by some embodiments of the present disclosure has a higher accuracy rate.

[0084] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the generation method of the generalized small sample road anomaly detection model of some embodiments of the present disclosure, the detection accuracy of the road anomaly detection model can be improved. Specifically, the reason for the low detection accuracy of the model is that the small sample target detection method only evaluates the performance of the model in detecting new classes, but ignores the performance of the basic classes, which is prone to false detection and missed detection, resulting in poor robustness of road anomaly detection. Based on this, the generation method of the generalized small sample road anomaly detection model of some embodiments of the present disclosure, first, divides the preset road anomaly data set into base classes and new classes to obtain a base class data set and a new class data set. A large number of annotated base class data sets help the model learn common features, so that the model can better extract and classify features from new classes with small data volumes. Secondly, according to the base class data set, base class matching is performed on each new class data in the above new class data set to obtain a base class and new class matching pair set. As a result, the model can mine the combined information of the base class and the new class, and give the model more category common knowledge and individual knowledge on limited data. Then, the above base class and new class matching pair set is input into the initial road anomaly detection model for initial round training to obtain the initial base class loss value. Finally, based on the preset round, the following model training steps are performed: the above base class and new class matching pair set is input into the initial road anomaly detection model to obtain the current round base class loss value and the current round new class loss value; according to the above initial base class loss value and the above current round base class loss value, a dynamic trade-off operator is determined; according to the above dynamic trade-off operator, the above current round base class loss value, and the above current round new class loss value, the current round total loss value is determined; according to the above current round total loss value, the above initial road anomaly detection model is back-propagated to update the parameters to obtain an updated road anomaly detection model; in response to determining that the training round is equal to the preset round, the above updated road anomaly detection model is determined as the road anomaly detection model; in response to determining that the training round is less than the preset round, the above updated road anomaly detection model is used as the initial road anomaly detection model to perform the above model training steps again. The dynamic trade-off operator is used to achieve adaptive parameter adjustment during the model training process, and at the same time, the model's detection performance for base class targets and new class targets is improved, thereby improving the robustness of the model and further improving the detection accuracy of the road anomaly detection model.

[0085] Further references Figure 3 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a device for generating a generalized small sample road anomaly detection model. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0086] like Figure 3As shown, some embodiments of the generalized small sample road anomaly detection model generation device 300 include: a base class and new class division unit 301, a base class matching unit 302, an initial round training unit 303, and a model training unit 304. The base class and new class division unit 301 is configured to divide the preset road anomaly data set into base classes and new classes to obtain a base class data set and a new class data set; the base class matching unit 302 is configured to perform base class matching on each new class data in the above new class data set according to the base class data set to obtain a base class and new class matching pair set; the initial round training unit 303 is configured to input the above base class and new class matching pair set into the initial road anomaly detection model for initial round training to obtain an initial base class loss value; the model training unit 304 is configured to perform the following model training steps based on the preset rounds: input the above base class and new class matching pair set into the initial road anomaly detection model to obtain the base class of the current round loss value and the new class loss value of the current round; determine the dynamic trade-off operator according to the above initial base class loss value and the above current round base class loss value; determine the total loss value of the current round according to the above dynamic trade-off operator, the above current round base class loss value, and the above current round new class loss value; perform back propagation parameter update on the above initial road anomaly detection model according to the above current round total loss value to obtain an updated road anomaly detection model; in response to determining that the training round is equal to the preset round, determine the above updated road anomaly detection model as the road anomaly detection model; in response to determining that the training round is less than the preset round, use the above updated road anomaly detection model as the initial road anomaly detection model to execute the above model training step again.

[0087] It is understood that the units described in the device 300 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 300 and the units contained therein, and will not be described in detail here.

[0088] Reference below Figure 4 , which shows a structural schematic diagram of an electronic device (eg, a computing device) 400 suitable for implementing some embodiments of the present disclosure. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0089] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory 402 or a program loaded from a storage device 408 into a random access memory 403. Various programs and data required for the operation of the electronic device 400 are also stored in the random access memory 403. The processing device 401, the read-only memory 402, and the random access memory 403 are connected to each other via a bus 404. An input / output interface 405 is also connected to the bus 404.

[0090] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0091] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the read-only memory 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0092] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0093] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0094] The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or it may exist independently without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: divides the preset road anomaly data set into base classes and new classes to obtain a base class data set and a new class data set; performs base class matching on each new class data in the above-mentioned new class data set according to the above-mentioned base class data set to obtain a base class and new class matching pair set; inputs the above-mentioned base class and new class matching pair set into the initial road anomaly detection model for initial round training to obtain an initial base class loss value; based on the preset rounds, performs the following model training steps: inputs the above-mentioned base class and new class matching pair set into the initial road anomaly detection model to obtain the base class loss value of the current round and the current round new class loss value; determine the dynamic trade-off operator according to the above-mentioned initial base class loss value and the above-mentioned current round base class loss value; determine the total loss value of the current round according to the above-mentioned dynamic trade-off operator, the above-mentioned current round base class loss value, and the above-mentioned current round new class loss value; according to the above-mentioned current round total loss value, back-propagate parameter update of the above-mentioned initial road anomaly detection model to obtain an updated road anomaly detection model; in response to determining that the training round is equal to the preset round, determine the above-mentioned updated road anomaly detection model as the road anomaly detection model; in response to determining that the training round is less than the preset round, use the above-mentioned updated road anomaly detection model as the initial road anomaly detection model to execute the above-mentioned model training step again.

[0095] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0096] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0097] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including a base class and a new class division unit, a base class matching unit, an initial round training unit, and a model training unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the base class matching unit may also be described as a "unit for performing base class matching on new classes".

[0098] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0099] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A method for generating a generalized small sample road anomaly detection model, comprising: Divide the preset road anomaly data set into base classes and new classes to obtain a base class data set and a new class data set; According to the base class data set, base class matching is performed on each new class data in the new class data set to obtain a base class and new class matching pair set; Inputting the base class and new class matching pair set into the initial road surface anomaly detection model for initial round training to obtain an initial base class loss value; Based on the preset rounds, the following model training steps are performed: Inputting the base class and new class matching pair set into the initial road anomaly detection model to obtain the current round base class loss value and the current round new class loss value; Determining a dynamic trade-off operator according to the initial base class loss value and the current round base class loss value; Determine the total loss value of the current round according to the dynamic trade-off operator, the base class loss value of the current round, and the new class loss value of the current round; According to the total loss value of the current round, back-propagation parameters of the initial road anomaly detection model are updated to obtain an updated road anomaly detection model; In response to determining that the training round is equal to the preset round, determining the updated road surface anomaly detection model as the road surface anomaly detection model; In response to determining that the training round is less than the preset round, the model training step is performed again using the updated road surface anomaly detection model as the initial road surface anomaly detection model.

2. The method according to claim 1, wherein: The road surface anomaly dataset is constructed by the following steps: Acquire a sample road surface anomaly data set, wherein the sample road surface anomaly data includes a sample road surface anomaly image and a corresponding sample road surface anomaly label group, the sample road surface anomaly image includes at least one road surface anomaly area, and the sample road surface anomaly label group includes sample road surface anomaly labels corresponding to each road surface anomaly area; Performing a data enhancement operation on each sample road surface anomaly data set in the sample road surface anomaly data set to obtain an enhanced road surface anomaly data set; The enhanced road anomaly data set is determined as a road anomaly data set, wherein the road anomaly data includes a road anomaly image and a corresponding road anomaly label group.

3. The method according to claim 1, wherein: The method of dividing the preset road anomaly data set into a base class and a new class to obtain a base class data set and a new class data set includes: Determine each road surface anomaly category corresponding to each road surface anomaly data in the road surface anomaly data set to obtain a road surface anomaly category set, wherein one road surface anomaly data corresponds to at least one road surface anomaly category; According to the pavement anomaly label corresponding to each pavement anomaly data in the pavement anomaly data set, determining the category data volume of each pavement anomaly category in the pavement anomaly category set, to obtain a category data volume set; Determine the road surface abnormality category corresponding to the category data amount greater than or equal to a preset threshold in the category data amount set as a basic category to obtain a basic category set; Determine the road surface abnormality category corresponding to the category data amount in the category data amount set that is less than a preset threshold as a newly added category to obtain a newly added category set; Determine the road surface anomaly data corresponding to the basic category in the road surface anomaly data set as base class data to obtain a base class data set; The road surface anomaly data corresponding to the newly added category in the road surface anomaly data set is determined as new category data to obtain a new category data set.

4. The method according to claim 1, wherein: The inputting the base class and new class matching pair set into the initial road anomaly detection model for initial round training to obtain the initial base class loss value includes: Determining initial parameters of the initial road surface anomaly detection model, wherein the initial road surface anomaly detection model is a preset target detection model; Inputting each base class and new class matching pair in the base class and new class matching pair set into the initial road surface anomaly detection model in sequence to obtain a base class detection result set; Determine the loss value between each base class detection result in the base class detection result set and the road surface anomaly label group corresponding to each base class data in the base class data set to obtain an initial base class loss value.

5. The method according to claim 1, wherein: The determining of the dynamic trade-off operator according to the initial base class loss value and the current round base class loss value includes: Performing a hyperbolic tangent transformation on the initial base class loss value and the current round base class loss value to obtain a current round base class training degree value, wherein the current round base class training degree value represents the base class training degree measured according to the change of the base class training loss value; A dynamic trade-off operator is determined according to the base class training degree value of the current round.

6. The method according to claim 1, wherein: The method further comprises: Acquire an abnormal image of the road surface to be tested; The road anomaly image to be tested is input into the road anomaly detection model to obtain a road anomaly detection result map, wherein the road anomaly detection result map includes a road anomaly area marking box and a corresponding road anomaly category.

7. A device for generating a generalized small sample road anomaly detection model, comprising: The base class and new class division unit is configured to divide the preset road anomaly data set into base class and new class to obtain a base class data set and a new class data set; A base class matching unit is configured to perform base class matching on each new class data in the new class data set according to the base class data set to obtain a base class and new class matching pair set; An initial round training unit is configured to input the base class and new class matching pair set into an initial road surface anomaly detection model for initial round training to obtain an initial base class loss value; The model training unit is configured to perform the following model training steps based on a preset round: Inputting the base class and new class matching pair set into the initial road anomaly detection model to obtain the current round base class loss value and the current round new class loss value; Determining a dynamic trade-off operator according to the initial base class loss value and the current round base class loss value; Determine the total loss value of the current round according to the dynamic trade-off operator, the base class loss value of the current round, and the new class loss value of the current round; According to the total loss value of the current round, back-propagation parameters of the initial road anomaly detection model are updated to obtain an updated road anomaly detection model; In response to determining that the training round is equal to the preset round, determining the updated road surface anomaly detection model as the road surface anomaly detection model; In response to determining that the training round is less than the preset round, the model training step is performed again using the updated road surface anomaly detection model as the initial road surface anomaly detection model.

8. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images