Method and apparatus for training defect detection model and detecting defects

Abstract

The present disclosure provides a method and device for training a defect detection model, relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning and the like, and can be applied to scenarios such as automatic driving and intelligent transportation. The specific implementation scheme is as follows: obtaining a sample and a defect detection model, wherein the sample includes an image and defect annotation information, and the defect detection model includes a target detection model and a segmentation model; inputting the image in the sample into the target detection model and the segmentation model respectively to obtain a target detection result and a segmentation result; dividing the target detection result and the segmentation result into a first result and a second result represented by small boxes of a predetermined size respectively; using the first result to supplement the second result to obtain a fusion result; and adjusting network parameters of the target detection model and the segmentation model based on the difference between the fusion result and the defect annotation information in the sample. The defect detection model trained by the implementation manner can efficiently, conveniently and economically perform defect detection.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning and the like, and can be applied to scenarios such as automatic driving and smart traffic, and in particular to a method and device for training a defect detection model and a method and device for detecting defects. BACKGROUND

[0002] In recent years, defect detection based on image recognition has been applied in various fields. In particular, in a smart highway maintenance and inspection system, detecting pavement diseases is a very important task. Pavement diseases refer to the damage to the structure or performance of the highway pavement caused by various reasons, such as cracks, potholes, broken plates and the like. These diseases not only affect the flatness and comfort of the highway, but also reduce the carrying capacity and service life of the highway, increase the driving resistance and fuel consumption of vehicles, and even cause traffic accidents. Therefore, timely detection and repair of highway pavement diseases is an important measure to ensure highway safety and prolong highway life.

[0003] Existing defect detection models usually use a single object detection (Object Detection) or semantic segmentation (Semantic Segmentation) model, while defects in images can have a very large number of categories, and their respective features are different. Using only one solution will cause the model to have poor generalization between different defects. SUMMARY

[0004] The present disclosure provides a method and device for training a defect detection model and detecting defects, equipment, a storage medium and a computer program product.

[0005] According to a first aspect of the present disclosure, a method for training a defect detection model is provided, comprising: obtaining a sample and a defect detection model, wherein the sample comprises an image and defect annotation information, and the defect detection model comprises an object detection model and a segmentation model; inputting the image in the sample into the object detection model and the segmentation model respectively to obtain an object detection result and a segmentation result; dividing the object detection result and the segmentation result into a first result and a second result represented by a small box of a predetermined size respectively; supplementing the first result based on the second result to obtain a fusion result; and adjusting network parameters of the object detection model and the segmentation model based on a difference between the fusion result and the defect annotation information in the sample.

[0006] According to a second aspect of the present disclosure, a method for detecting defects is provided, comprising: obtaining an image to be detected; inputting the image into a defect detection model trained according to the method of any one of the first aspect to output defect information.

[0007] According to a third aspect of the present disclosure, an apparatus for training a defect detection model is provided, comprising: an obtaining unit configured to obtain a sample and a defect detection model, wherein the sample comprises an image and defect annotation information, and the defect detection model comprises a target detection model and a segmentation model; a detecting unit configured to input the image in the sample into the target detection model and the segmentation model respectively to obtain a target detection result and a segmentation result; a segmenting unit configured to divide the target detection result and the segmentation result into a first result and a second result represented by small boxes of a predetermined size respectively; a fusing unit configured to obtain a fusion result by using the first result to supplement the second result; and an adjusting unit configured to adjust network parameters of the target detection model and the segmentation model based on a difference between the fusion result and the defect annotation information in the sample.

[0008] According to a fourth aspect of the present disclosure, an apparatus for detecting defects is provided, comprising: an obtaining unit configured to obtain an image to be detected; and a detecting unit configured to input the image into a defect detection model trained by the apparatus according to any one of the third aspect to output defect information.

[0009] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of the first aspect and the second aspect.

[0010] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the method according to any one of the first aspect and the second aspect.

[0011] According to a seventh aspect of the present disclosure, an autonomous vehicle is provided, configured to perform the method according to any one of the first aspect and the second aspect.

[0012] According to an eighth aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any one of the first aspect and the second aspect.

[0013] The embodiments of the present disclosure provide a method and apparatus for training a defect detection model and detecting defects, which combine the result of target detection and the result of semantic segmentation to solve the problem of poor generalization of the model between different defects, and improve the accuracy of defect detection.

[0014] It should be appreciated that the description set forth in this section is not intended to identify key or essential features of an embodiment of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the disclosure will be apparent from review of the disclosure, both hereinafter and in the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of this specification. Illustrations in the drawings are for purposes of illustrating an example and do not limit the present disclosure, in which:

[0016] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure can be applied;

[0017] Figure 2 is a flowchart of one embodiment of a method for training a defect detection model according to the present disclosure;

[0018] Figures 3a-3e is a schematic diagram of one application scenario of a method for training a defect detection model according to the present disclosure;

[0019] Figure 4 is a flowchart of one embodiment of a method for detecting defects according to the present disclosure;

[0020] Figure 5 is a structural schematic diagram of one embodiment of an apparatus for training a defect detection model according to the present disclosure;

[0021] Figure 6 is a structural schematic diagram of one embodiment of an apparatus for detecting defects according to the present disclosure;

[0022] Figure 7 is a structural schematic diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present disclosure are described herein with reference to the accompanying drawings, which are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification. Embodiments of the present disclosure described herein should be considered in a descriptive sense only and not for purposes of limitation. Therefore, various modifications and changes can be made to the embodiments described herein without departing from the scope and spirit of the disclosure. Similarly, it should be appreciated that the specifications and examples given herein are only exemplary and are not intended to limit the scope of the disclosure. As such, other specifications and examples can be developed without departing from the scope and spirit of the disclosure. In order to make the disclosure more clearly understood, the description of well-known functions and structures will be omitted hereinafter.

[0024] Figure 1 An exemplary system architecture 100 of a vehicle to which the method and apparatus for training a defect detection model and detecting defects according to the embodiments of the present disclosure can be applied is shown.

[0025] As Figure 1As shown, the system architecture 100 can include terminal devices 101, 102, a network 103, a database server 104 and a server 105. The terminal device can be an electronic device with a camera installed, such as a mobile phone, an unmanned vehicle (also known as an autonomous vehicle), etc. The network 103 is a medium for providing a communication link between the terminal devices 101, 102, the database server 104 and the server 105. The network 103 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0026] A camera can be installed in the unmanned vehicle to collect road images. A driving control device and a device for collecting point cloud data such as a laser radar and a millimeter wave radar can also be installed. The driving control device (also known as an on-board brain) is responsible for intelligent control of the unmanned vehicle. The driving control device can be a separately arranged controller, such as a programmable logic controller (PLC), a single-chip microcomputer, an industrial control computer, etc.; can also be a device composed of other electronic devices with input / output ports and having operation control functions; or can also be a computer device installed with vehicle driving control application.

[0027] It should be noted that in practice, at least one sensor can also be installed in the unmanned vehicle, such as a gravity sensor, a wheel speed sensor, etc. In some cases, a GNSS (Global Navigation Satellite System) device and a SINS (Strap-down Inertial Navigation System) can also be installed in the unmanned vehicle, etc.

[0028] The database server 104 can be a database server providing various services. For example, the image set and the defect detection model can be stored in the database server. In this way, the user can store the collected images into the database server 104 through the terminal device 101, 102, and can also obtain the defect detection model from the database server 104 for detecting defects in the images.

[0029] The server 105 can also be a server providing various services, such as a background server providing support for various applications displayed on the terminal device 101, 102. The background server can train the image set and the initial defect detection model stored in the database server 104, and return the trained defect detection model to the terminal device for detecting defects in the images. The terminal device can also send the collected images to the server for detecting defects in the images.

[0030] The database server 104 and the server 105 in the system architecture 100 can be hardware or software. When they are hardware, they can be implemented as a distributed server cluster composed of multiple servers or as a single server. When they are software, they can be implemented as multiple software or software modules (for example, to provide distributed services) or as a single software or software module. No specific limitation is made herein. The database server 104 and the server 105 can also be servers of a distributed system or servers combined with a blockchain. The database server 104 and the server 105 can also be cloud servers or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0031] It should be noted that the method for training a defect detection model and detecting defects provided in the embodiments of the present application is generally executed by the server 105. Correspondingly, the apparatus for training a defect detection model and detecting defects is also generally provided in the server 105.

[0032] It should be noted that in the case where the server 105 can implement the functions of the database server 104, the database server 104 can not be provided in the system architecture 100.

[0033] It should be understood that Figure 1 The number of terminal devices, networks, database servers, and servers in the system architecture 100 is merely illustrative. According to the implementation needs, there can be any number of terminal devices, networks, database servers, and servers.

[0034] Continuing to refer to Figure 2 , a flow 200 of one embodiment of the method for training a defect detection model of a vehicle according to the present application is shown. The method for training a defect detection model of a vehicle can include the following steps:

[0035] In step 201, samples and a defect detection model are obtained.

[0036] In the present embodiment, the execution subject (for example, the server 105 shown in Figure 1 ) of the method for training a defect detection model can obtain samples in multiple ways. For example, the execution subject can obtain existing samples stored in a database server (for example, the database server 104 shown in Figure 1 ) from the database server in a wired connection manner or a wireless connection manner. For another example, a user can collect samples through a terminal device (for example, the terminal device 101 or 102 shown in Figure 1 ). In this way, the execution subject can receive the samples collected by the terminal device and store these samples locally, thereby generating a sample set. Part of the samples in the sample set can be selected each time for training.

[0037] The sample can include an image and defect annotation information. The defect annotation information is used to annotate the type, position, size, and the like of the defect. The annotation can be performed in pixels or in a fixed-size grid.

[0038] The defect detection model includes a target detection model and a segmentation model, and further includes a granularity conversion module and a result fusion module, as shown in Figure 3a The target detection model and the segmentation model are two parallel branches, which can share a backbone network or use respective backbone networks. Each of them corresponds to a granularity conversion module. The converted granularities of the two granularity conversion modules can be the same or different. Whether to share the backbone network can be selected according to the application scenario. For example, for a scene with high latency requirement (for example, during the driving process of an unmanned vehicle), in order to improve the detection speed, a shared backbone network is used. For a scene with low latency requirement (for example, during the defect detection process in a wooden toy pipeline), in order to improve the detection accuracy, respective backbone networks are used. The granularity conversion module is used to perform step 203, and the result fusion module is used to perform step 204. The output result of the defect detection model is the fusion result of the granularity conversion result of the target detection result and the granularity conversion result of the segmentation result.

[0039] In step 202, the image in the sample is input into the target detection model and the segmentation model respectively to obtain a target detection result and a segmentation result.

[0040] In this embodiment, the target detection model performs the following steps: feature extraction: inputting the preprocessed image into a backbone network (for example, a CNN) to obtain feature maps of different scales; feature fusion: fusing the feature maps of different scales through cross-layer connection and upsampling operations to obtain a unified feature pyramid; and detection: inputting each feature map in the feature pyramid into a detection head to obtain a target detection result at each position and scale.

[0041] The segmentation model performs the following steps: feature extraction: inputting the preprocessed image into a backbone network (which can be a Transformer-based visual large model) to obtain feature maps of different scales; segmentation mask generation (generation layer): inputting the feature maps into a Transformer encoder network to obtain a segmentation mask at each position and scale; and segmentation result recovery (detection layer): inputting the segmentation mask into a Decoder network to obtain a final segmentation result, including a class label and an instance label.

[0042] In step 203, the target detection result and the segmentation result are respectively divided into a first result and a second result represented by small boxes of a predetermined size.

[0043] In this embodiment, the outputs of the target detection model and the segmentation model can be converted into small boxes of a relatively coarse granularity, which can eliminate the differences between the two models and reduce the amount of calculation and noise. The generation method of the small boxes is as follows: for the output result of the target detection model, the bounding box predicted by the target detection model can be directly divided into small boxes of equal proportions, as shown in FIG. 6, 1 large box is converted into 3*4 small boxes; for the output result of the segmentation model, the minimum circumscribed rectangle of a predetermined size can be calculated according to the mask predicted by the segmentation model, as shown in FIG. 7, the segmentation result in pixel points is converted into 20*20 small boxes. If the entire image is 1000*1000, 50*50 small boxes can be divided, and if there is a pixel point of a defect in each small box, the entire small box is marked as defective. Figure 3c Figure 3b

[0044] Step 204, based on the second result, the first result is used for supplement to obtain a fusion result.

[0045] In this embodiment, the small boxes of the segmentation model are supplemented by the small boxes of the target detection model. Specifically, taking road disease detection as an example, the segmentation small boxes of each pit and broken plate and the like can be traversed, if it is found that there is a large overlapping area between the target detection small box and the segmentation small box, and the area of the target detection small box is greater than the area of the segmentation small box, the target detection small box is used to supplement the segmentation small box, as shown in FIG. 8. In this way, the better detection capability of the target detection model for pits and broken plates and the like can be utilized to avoid missed detection or under-segmentation. Figure 3d

[0046] Step 205, based on the difference between the fusion result and the defect annotation information in the sample, the network parameters of the target detection model and the segmentation model are adjusted.

[0047] In this embodiment, the loss value can be calculated according to the difference between the fusion result and the defect annotation information in the sample, and then the network parameters of the target detection model and the segmentation model are adjusted according to the loss value until the loss value converges, and then the model training is completed. The model training can also be terminated by setting a maximum number of iterations.

[0048] The method for training the defect detection model in this embodiment combines the advantages of the target detection model and the segmentation model. The target detection model can better handle diseases with obvious boundaries and large areas, such as pits and broken plates, and the segmentation model can better handle diseases with long and irregular shapes, such as cracks and joints. Therefore, the outputs of the two models are fused to obtain more accurate and complete disease detection and segmentation results.

[0049] ​​​In some optional implementations of the embodiment, the step of obtaining the fusion result based on the second result and supplemented by the first result comprises: obtaining a pre-trained supplementary coefficient matrix, wherein each element in the supplementary coefficient matrix corresponds to a small box of a predetermined size in the image in the sample, and the initial value of each element is 0; if the element value is 1, it indicates that the first result is used to supplement the second result; and fusing the second result and the first result based on the supplementary coefficient matrix to obtain the fusion result.

[0050] The gamma is used as the supplementary coefficient matrix in the fusion, and the initial value of the gamma is 0 by default. The result after segmentation is used as the base, and the small box result after target detection is fused. The gamma determines whether the result of target detection is used in the fusion.

[0051] The small box after fusion is used as the final output, and the position and range of the defect are indicated.

[0052] For example, if the image is 1000*1000, and each small box is 20*20, the supplementary coefficient matrix is 50*50. The initial value of the supplementary coefficient matrix is 0. At this time, only the coarse-grained result of the segmentation model (i.e., the second result) is used. After the loss value is calculated to be large, the network parameters of the target detection model and the segmentation model are adjusted, and the supplementary coefficient matrix is also adjusted, so that some elements become 1. When the sample image is trained again, the coarse-grained result of the target detection model (i.e., the first result) is integrated. Assuming that the small box positions of the defects annotated in the defect annotation information are (0, 0), (1, 1), and (0, 1), the defect is detected at the small box position (1, 1) by the segmentation model, and the defect is detected at the small box positions (0, 0), (1, 1), (0, 1), and (1, 0) by the target detection model. The elements corresponding to the small box positions (0, 0), (1, 1), and (0, 1) in the supplementary coefficient matrix are 1. Therefore, the output result of the target detection model is used to supplement the output result of the segmentation model, and the fusion result (0, 0), (1, 1), and (0, 1) can be obtained. Both the missed detection problem of the segmentation model and the problem of too large detection range of the target detection model can be solved, so that the accuracy of the detection result is improved. The mathematical expression is as follows:

[0053] M = S + O*T, wherein M represents the fusion result, S represents the segmentation result of the small box, O represents the target detection result of the small box, and T represents the supplementary coefficient matrix.

[0054] In some optional implementations of the embodiment, the method further comprises adjusting the supplementary coefficient matrix based on a difference between the fusion result and defect annotation information in the sample. The supplementary coefficient matrix can be jointly trained with network parameters of the target detection model and the segmentation model. The result fusion module contains the supplementary coefficient matrix, which is equivalent to a full connection layer, and is used to fuse the target detection result and the segmentation result after the granularity conversion.

[0055] In some optional implementations of the embodiment, the target detection model and the segmentation model share a backbone network, and the backbone network is a visual large model based on Swin Transformer.

[0056] The visual large model based on Swin Transformer has shown strong performance in image classification and other tasks, which has aroused people's interest in using the visual large model as the backbone for segmentation tasks. The visual large model is mainly based on the self-attention mechanism (Self-Attention), which can use global information to extract image features and can improve the model's generalization ability through fine-tuning. It has the following advantages:

[0057] It can handle any segmentation task, whether it is panoptic segmentation, instance segmentation or semantic segmentation, and can be completed with the same model without the need to design a specific architecture or loss function.

[0058] It can generate different segmentation results according to different input prompts such as points, boxes or text without pre-defining object categories or instance quantities.

[0059] It can be pre-trained and fine-tuned on large-scale datasets to improve the model's capacity and generalization ability. It has been pre-trained on 11M images and 11B masks and achieved state-of-the-art performance on multiple public datasets.

[0060] In summary, using the visual large model as the backbone network as described above can make good use of the prior knowledge of the large model to provide high-quality feature representation for defect detection.

[0061] In some optional implementations of the embodiment, the backbone network of the target detection model is a convolutional neural network, and the backbone network of the segmentation model is a visual large model based on Swin Transformer. The target detection task and the segmentation task can use the best backbone network respectively to obtain the best defect detection model.

[0062] In some optional implementations of the embodiment, the method further includes: if the intersection over union between the fusion result and the defect annotation information is less than a predetermined threshold, extracting the defects in the image by the segmentation model and pasting them into a blank image as an augmented sample.

[0063] The IOU evaluation is calculated in the manner of the small box mentioned before. For each scene and defect, the IOU value between it and the real annotation small box is calculated as the accuracy and recall rate. For the scene and defect with accuracy and recall rate lower than a certain threshold, it is regarded as a poor effect sample and needs to be augmented.

[0064] The poor effect sample is randomly fused to generate a new scene and defect. Specifically, the defects in the image are extracted by the segmentation model and then pasted into a blank image as an augmented sample. Using a pure background can avoid the influence of too complex background on the detection result. In the case of insufficient sample data, the existing samples can be fully utilized to increase the sample quantity, thereby improving the effectiveness of training.

[0065] In some optional implementations of the embodiment, the pasting into a blank image as an augmented sample includes: adjusting at least one of the following parameters of the defect before pasting into a blank image as an augmented sample: type, position, size, direction. Randomly generating defects that are not accurately identified, such as cracks and pits. Random sampling can be performed according to the parameters of the type, position, size, and direction of the defect to increase the diversity of the data. The generated defects are fused with the original image to obtain a new scene image. This can avoid overfitting of the detection result.

[0066] In some optional implementations of the embodiment, the extracting the defects in the image by the segmentation model includes: extracting the road surface in the image by the segmentation model. This can avoid the influence of the background other than the road surface on the detection result.

[0067] In some optional implementations of the embodiment, the method further includes: randomly generating defects that are not accurately identified on the road surface. This solves the problem of being unable to accurately identify defects due to inaccurate road surface identification. According to the generated defects, corresponding annotation information such as category and bounding box is generated, as shown in Figure 3e

[0068] ​In some optional implementations of the embodiment, the method further includes: adding the enhanced sample to the training set for next-stage training; and dynamically adjusting the proportion and parameters of the enhanced sample in the training set according to the training progress and performance of the defect detection model. If the training progress is too slow, no enhanced sample is generated. If the training progress is faster than expected and the performance does not meet the expectation, the enhanced sample can be generated during the training process. The performance can be improved and the training progress can be ensured by controlling the proportion and parameters of the enhanced sample. Here, the proportion refers to the proportion of the enhanced sample in the total sample, and the parameters refer to the type, number, position, and the like of the defects. For example, if the performance of detecting the fine slit defect is poor, the fine slit sample can be increased, and the number of samples can be increased by randomly generating a plurality of fine slit pictures.

[0069] The fused new sample is added to the training set for next-stage training. The proportion and parameters of data enhancement can be dynamically adjusted according to the training progress and performance of the model to achieve the best effect. The process is repeated until the model achieves a relatively good effect.

[0070] To improve the generalization ability and adaptability of the model, the data engine module is implemented by the above method, which can online fuse the scenes and diseases with poor effect after each training of the model and supplement them into the training set for next-stage training. The feedback information of the model itself can be effectively used to enhance the data set and improve the detection and segmentation effect of the model on different scenes and defects.

[0071] See Figure 4 , which shows a flow 400 of one embodiment of the method for detecting defects provided by the present disclosure. The method for detecting defects can include the following steps:

[0072] Step 401, acquiring an image to be detected.

[0073] In the embodiment, the execution subject of the method for detecting defects (for example, the server 105 shown in Figure 1 The execution subject can acquire the image of the detection object in various ways. For example, the execution subject can acquire the image stored in the database server (for example, the database server 104 shown in Figure 1 ) from the database server in a wired connection manner or a wireless connection manner. For another example, the execution subject can also receive the image collected by the terminal device (for example, the terminal device 101, 102 shown in Figure 1 ) or other devices.

[0074] In the embodiment, the detection object can be an object image with defects, for example, a road image collected by an unmanned vehicle, an image of a wooden toy collected by a camera on a production line, and the like. The image can be a color image and / or a grayscale image, and the like. Moreover, the format of the image is not limited in the present disclosure.

[0075] Step 402: Input the image into the defect detection model and output the defect information.

[0076] In this embodiment, the defect detection model can be as described above. Figure 2 This was generated using the method described in the embodiments. For details of the generation process, please refer to [link / reference needed]. Figure 2 The relevant descriptions of the embodiments will not be repeated here. The output defect information is a fusion result of the target detection model output and the segmentation model output after granularity conversion. The defect information includes information such as the type, size, and location of the defects.

[0077] It should be noted that the defect detection method in this embodiment can be used to test the defect detection models generated in the above embodiments. Furthermore, the defect detection models can be continuously optimized based on the test results. This method can also be a practical application of the defect detection models generated in the above embodiments. Using the defect detection models generated in the above embodiments for defect detection helps improve the performance of defect detection, such as finding more defects or finding more accurate defect information.

[0078] In some optional implementations of this embodiment, the method further includes: inputting the image segment corresponding to the defect information into the segmentation model in the defect detection model to obtain fine-grained defect information. If a finer-grained defect contour is needed, it can be segmented again based on the existing detection results, which can reduce the detection range of the segmentation model and improve the detection speed. It can also avoid missed detections caused by using only the segmentation results of the segmentation model.

[0079] In some optional implementations of this embodiment, the method further includes: calculating the defect area based on the defect information. Since the final result also requires outputting the defect area, using this small box method is more conducive to calculating and outputting the defect area.

[0080] See also Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of an apparatus for training a defect detection model. This apparatus embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0081] like Figure 5As shown, the device 500 for training a defect detection model in the embodiment can include an acquisition unit 501, a detection unit 502, a segmentation unit 503, a fusion unit 504, and an adjustment unit 505. The acquisition unit 501 is configured to acquire samples and a defect detection model, wherein the samples include images and defect annotation information, and the defect detection model includes a target detection model and a segmentation model. The detection unit 502 is configured to input the images in the samples into the target detection model and the segmentation model respectively to obtain target detection results and segmentation results. The segmentation unit 503 is configured to divide the target detection results and the segmentation results into first results and second results represented by small boxes of a predetermined size respectively. The fusion unit 504 is configured to use the first results to supplement the second results based on the second results to obtain fusion results. The adjustment unit 505 is configured to adjust network parameters of the target detection model and the segmentation model based on differences between the fusion results and the defect annotation information in the samples.

[0082] In some optional implementations of the embodiment, the fusion unit 504 is further configured to acquire a pre-trained supplementary coefficient matrix, wherein each element in the supplementary coefficient matrix corresponds to a divided small box of a predetermined size in an image in the samples one by one, and an initial value of each element is 0. If the element value is 1, it indicates that the first results are used to supplement the second results. The fusion unit 504 is further configured to fuse the second results and the first results based on the supplementary coefficient matrix to obtain fusion results.

[0083] In some optional implementations of the embodiment, the adjustment unit 505 is further configured to adjust the supplementary coefficient matrix based on differences between the fusion results and the defect annotation information in the samples.

[0084] In some optional implementations of the embodiment, the target detection model and the segmentation model share a backbone network, and the backbone network is a visual large model based on a Swin Transformer.

[0085] In some optional implementations of the embodiment, the backbone network of the target detection model is a convolutional neural network, and the backbone network of the segmentation model is a visual large model based on a Swin Transformer.

[0086] In some optional implementations of the embodiment, the device further includes an enhancement unit (not shown in the drawings) configured to, if an intersection over union between the fusion results and the defect annotation information is less than a predetermined threshold, extract defects in the images by the segmentation model and paste the defects into a blank image as enhancement samples.

[0087] In some optional implementations of the present embodiment, the enhancement unit is further configured to: adjust at least one of the following parameters of the defect and map it into the blank image as an enhanced sample: type, position, size, and direction.

[0088] In some optional implementations of the present embodiment, the enhancement unit is further configured to: extract the road surface in the image by the segmentation model.

[0089] In some optional implementations of the present embodiment, the enhancement unit is further configured to: randomly generate a defect that is not accurately identified on the road surface.

[0090] In some optional implementations of the present embodiment, the enhancement unit is further configured to: add the enhanced sample to the training set for the next stage of training; and dynamically adjust the proportion and parameters of the enhanced samples in the training set according to the training progress and performance of the defect detection model.

[0091] Continuing to refer to Figure 6 , as an implementation of the method shown in the above figures, the present disclosure provides one embodiment of a device for detecting defects. The device embodiment corresponds to the method embodiment shown in Figure 4 , and the device can be applied to various electronic devices.

[0092] As shown in Figure 6 , the device 600 for detecting defects of the present embodiment can include an acquisition unit 601 and a detection unit 602. The acquisition unit 601 is configured to acquire an image to be detected, and the detection unit 602 is configured to input the image into a defect detection model trained by the device 500 and output defect information.

[0093] In some optional implementations of the present embodiment, the detection unit 602 is further configured to: input an image segment corresponding to the defect information into a segmentation model in the defect detection model to obtain fine-grained defect information.

[0094] In some optional implementations of the present embodiment, the device 600 further includes a calculation unit (not shown in the figure) configured to: calculate a defect area according to the defect information.

[0095] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution comply with relevant laws and regulations and do not violate public order and good customs.

[0096] According to the embodiments of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, an autonomous vehicle, and a computer program product.

[0097] An electronic device comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of flow 200 or 400.

[0098] A non-transitory computer readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method of flow 200 or 400.

[0099] An autonomous vehicle configured to perform the method of flow 200 or 400.

[0100] A computer program product comprising a computer program which, when executed by a processor, implements the method of flow 200 or 400.

[0101] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0102] As shown in Figure 7 Device 700 includes a computing unit 701, which can perform various suitable actions and processes in accordance with computer programs stored in a read-only memory (ROM) 702 or loaded into a random access memory (RAM) 703 from a storage unit 708. Various programs and data used by device 700 can also be stored in the RAM 703 as needed. The computing unit 701, ROM 702, and RAM 703 are connected to each other by a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0103] A plurality of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0104] The computing unit 701 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the method of training a defect detection model. For example, in some embodiments, the method of training a defect detection model can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of the method of training a defect detection model described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the method of training a defect detection model by any other appropriate means, such as by means of firmware.

[0105] The various implementations of the systems and techniques described above herein can be realized in a digital electronic circuit system, an integrated circuit system, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip system (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0106] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0107] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0108] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0109] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0110] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions between them occurring over a communication network. The relationship between client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers incorporating blockchain.

[0111] It should be understood that the steps shown in the various forms above can be reordered, added to, or deleted from. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, which is not limited herein.

[0112] The specific embodiments described above are not intended to be limiting, and persons skilled in the art will appreciate that various modifications, combinations, sub-combinations and alternatives can be made to the specific embodiments without departing from the spirit and principles of the disclosure. Any further modifications, equivalents and / or alternatives thereof are also included within the scope of the disclosure.

Claims

1. A method for training a defect detection model, comprising: obtaining a sample and a defect detection model, wherein the sample comprises an image and defect annotation information, the defect detection model comprises a target detection model and a segmentation model, and whether the target detection model and the segmentation model share a backbone network is selected according to an application scenario; inputting the image in the sample into the target detection model and the segmentation model respectively to obtain a target detection result and a segmentation result; dividing the target detection result and the segmentation result into a first result and a second result represented by small boxes of a predetermined size respectively, wherein for the target detection result, a predicted bounding box is directly divided into small boxes of equal proportions; and for the segmentation result, a minimum circumscribed rectangle of the predetermined size is calculated according to a predicted mask; supplementing the first result based on the second result to obtain a fusion result; adjusting network parameters of the target detection model and the segmentation model based on a difference between the fusion result and the defect annotation information in the sample.

2. The method of claim 1, wherein, The step of supplementing the first result based on the second result to obtain a fusion result comprises: obtaining a pre-trained supplementary coefficient matrix, wherein each element in the supplementary coefficient matrix corresponds to a divided small box of a predetermined size in the image in the sample, and an initial value of each element is 0; if the element value is 1, it indicates that the first result is used to supplement the second result; fusing the second result and the first result based on the supplementary coefficient matrix to obtain a fusion result.

3. The method of claim 2, wherein, The method further comprises: adjusting the supplementary coefficient matrix based on a difference between the fusion result and the defect annotation information in the sample.

4. The method of claim 1, wherein, The target detection model and the segmentation model share a backbone network, and the backbone network is a visual large model based on Swin Transformer.

5. The method of claim 1, wherein, The backbone network of the target detection model is a convolutional neural network, and the backbone network of the segmentation model is a visual large model based on Swin Transformer.

6. The method of claim 1, wherein, The method further comprises: if an intersection over union between the fusion result and the defect annotation information is less than a predetermined threshold, extracting a defect in the image through the segmentation model and pasting the defect into a blank image as an enhanced sample.

7. The method of claim 6, wherein, The step of pasting the defect into a blank image as an enhanced sample comprises: adjusting at least one of the following parameters of the defect: type, position, size, and direction, and then pasting the defect into a blank image as an enhanced sample.

8. The method of claim 6, wherein, The step of extracting the defect in the image through the segmentation model comprises: extracting a road surface in the image through the segmentation model.

9. The method of claim 8, wherein, The method further comprises: randomly generating a defect that is not accurately recognized on the road surface.

10. The method of claim 6, wherein, The method further comprises: adding the enhanced sample to a training set for next-stage training; dynamically adjusting a proportion and parameters of the enhanced sample in the training set according to a training progress and performance of the defect detection model.

11. A method for detecting a defect, comprising: obtaining an image to be detected; inputting the image into a defect detection model trained according to the method of any one of claims 1-10, and outputting defect information.

12. The method of claim 11, wherein, The method further comprises: inputting the image segment corresponding to the defect information into a segmentation model in the defect detection model to obtain fine-grained defect information.

13. The method of claim 11, wherein, The method further comprises: calculating a defect area according to the defect information.

14. An apparatus for training a defect detection model, comprising: an acquisition unit configured to acquire samples and a defect detection model, wherein the samples comprise images and defect annotation information, the defect detection model comprises a target detection model and a segmentation model, and whether the target detection model and the segmentation model share a backbone network is selected according to an application scenario; a detection unit configured to input the images in the samples into the target detection model and the segmentation model respectively to obtain target detection results and segmentation results; a segmentation unit configured to divide the target detection results and the segmentation results into first results and second results represented by small boxes of a predetermined size respectively, wherein for the target detection results, the predicted bounding boxes are directly divided into small boxes of equal proportions; and for the segmentation results, a minimum circumscribed rectangle of the predetermined size is calculated according to a predicted mask; a fusion unit configured to use the first results to supplement the second results on the basis of the second results to obtain fusion results; an adjustment unit configured to adjust network parameters of the target detection model and the segmentation model based on differences between the fusion results and the defect annotation information in the samples.

15. The apparatus of claim 14, wherein, The fusion unit is further configured to: acquire a pre-trained supplementary coefficient matrix, wherein each element in the supplementary coefficient matrix corresponds to a divided small box of a predetermined size in an image in the samples one by one, and an initial value of each element is 0; if the element value is 1, it indicates that the first results are used to supplement the second results; fuse the second results and the first results based on the supplementary coefficient matrix to obtain fusion results.

16. The apparatus of claim 15, wherein, The adjustment unit is further configured to: adjust the supplementary coefficient matrix based on differences between the fusion results and the defect annotation information in the samples.

17. The apparatus of claim 14, wherein, The target detection model and the segmentation model share a backbone network, and the backbone network is a visual large model based on Swin Transformer.

18. The apparatus of claim 14, wherein, The backbone network of the target detection model is a convolutional neural network, and the backbone network of the segmentation model is a visual large model based on Swin Transformer.

19. The apparatus of claim 14, wherein, The apparatus further comprises an enhancement unit configured to: if an intersection over union between the fusion results and the defect annotation information is less than a predetermined threshold, extract defects in the images through the segmentation model and map them to a blank image as enhanced samples.

20. The apparatus of claim 19, wherein, The enhancement unit is further configured to: map the defects to the blank image as enhanced samples after adjusting at least one of the following parameters of the defects: type, position, size, and direction.

21. The apparatus of claim 19, wherein, The enhancement unit is further configured to: extract a road surface in the images through the segmentation model.

22. The apparatus of claim 21, wherein, The enhancement unit is further configured to: randomly generate defects that are not accurately recognized on the road surface.

23. The apparatus of claim 19, wherein, The enhancement unit is further configured to: add the enhanced samples to a training set for next-stage training; The proportion and parameters of the enhanced samples in the training set are dynamically adjusted according to the training progress and performance of the defect detection model.

24. An apparatus for detecting defects, comprising: an acquisition unit configured to acquire an image to be detected; a detection unit configured to input the image into a defect detection model trained by the apparatus according to any one of claims 14-23, and output defect information.

25. The apparatus of claim 24, wherein, The detection unit is further configured to: input an image segment corresponding to the defect information into a segmentation model in the defect detection model to obtain fine-grained defect information.

26. The apparatus of claim 24, wherein, The apparatus further comprises a calculation unit configured to: calculate a defect area according to the defect information.

27. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

28. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-13.

29. An autonomous vehicle configured to perform the method of any one of claims 1-13.

30. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-13.

Images