Defect detection method and related apparatus, device, storage medium

Abstract

The application discloses a defect detection method and related devices, equipment and storage media. The defect detection method comprises the following steps: obtaining a target image to be subjected to defect detection; determining a target network model for defect detection of the target image based on a structure type corresponding to a target structure in which a defect to be detected in the target image is located, wherein the target network model corresponding to different structure types is different; and performing defect detection on the target image by using the target network model. The above scheme can improve the accuracy of defect detection.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a defect detection method and related devices, equipment, and storage media. Background Art

[0002] To improve the protection of equipment in an environment, defect detection is often required to ensure timely detection of defects and facilitate maintenance. Currently, defect detection relies on manual inspection of images captured by cameras to determine if defects are present. This manual defect detection method is prone to missed or incorrect detections due to human negligence. Therefore, how to conduct defect detection to reduce the rate of missed detections and improve defect detection accuracy is a critical issue. Summary of the Invention

[0003] This application at least provides a defect detection method and related devices, equipment, and storage media.

[0004] The present application provides a defect detection method, including: obtaining a target image to be subjected to defect detection; determining a target network model for performing defect detection on the target image based on the structure type corresponding to the target structure where the defect to be detected is located in the target image, wherein different target network models correspond to different structure types; and performing defect detection on the target image using the target network model.

[0005] Therefore, by using the target network model to detect defects in the target image, the technical solution provided by this application has a higher accuracy in defect detection compared to manual detection. In addition, different target network models are selected for defect detection of different structural types, further improving the accuracy of defect detection.

[0006] Among them, the structure type is related to the size of the target structure and / or the positional relationship between the target structure and the adjacent structures. Before determining the target network model for defect detection on the target image based on the structure type corresponding to the target structure where the defect to be detected is located in the target image, the method also includes: receiving a structure type setting instruction; setting the structure type for the target structure corresponding to the target image in response to the structure type setting instruction; or, determining the similarity between the target image and each historical image in the database, the historical image is an image that has undergone defect detection, and the database records the structure type corresponding to the target structure where the defect to be detected is located in each historical image; using the structure type corresponding to the target structure where the defect to be detected is located in the target historical image whose similarity meets the similarity requirement as the structure type corresponding to the target structure in the target image.

[0007] Therefore, the structural type corresponding to the defect to be detected is determined by the positional relationship between the target structure and adjacent structures, enabling the target network model for defect detection to be determined based on structures other than the defect to be detected. Furthermore, by receiving a structure type setting instruction and setting the structure type for the target structure corresponding to the target image based on that instruction, user autonomy is enhanced. Furthermore, the structural type corresponding to the target structure in the target image is determined by the similarity between the target image and each historical image, eliminating the need for the user to set the corresponding structure type for each image.

[0008] Among them, based on the structure type corresponding to the target structure where the defect to be detected is located in the target image, a target network model for defect detection on the target image is determined, including: judging whether the structure type is a first target type, and obtaining a first judgment result, the first target type is a type in which the pixel ratio of the target structure in the target image is less than a first preset value; based on the first judgment result, determining the target network model for defect detection on the target image.

[0009] Therefore, by determining the corresponding target network model according to the pixel ratio of the target structure in the target image, the corresponding target network model can be selected for defect detection in combination with the size of the target structure, thereby improving the accuracy of defect detection.

[0010] Among them, based on the first judgment result, determining the target network model for defect detection of the target image includes: in response to the structure type being the second target type, determining the target network model to be the first target network model corresponding to the second target type, the second target type being a type in which the pixel ratio of the target structure in the target image is greater than or equal to a first preset value; or, in response to the structure type being the first target type, judging whether the first target type is a multi-level structure type, and obtaining a second judgment result, the multi-level structure type is an adjacent structure with a relatively fixed position to the target structure; based on the second judgment result, determining the target network model for defect detection of the target image.

[0011] Therefore, by selecting the corresponding target network model according to whether the structure type of the target structure is a multi-level structure type, the positional relationship between the target structure and the adjacent structures can be considered, thereby improving the accuracy of defect detection.

[0012] Among them, based on the second judgment result, the target network model for defect detection of the target image is determined, including: in response to the first target type being a single-level structure type, determining that the target network model is a second target network model corresponding to the single-level structure type, and the single-level structure type is that there is no adjacent structure whose position is relatively fixed with the target structure; in response to the first target type being a multi-level structure type, determining that the target network model is a third target network model corresponding to the multi-level structure type.

[0013] Therefore, by selecting the corresponding target network model according to whether the structure type of the target structure is a multi-level structure type, the positional relationship between the target structure and the adjacent structures can be considered, thereby improving the accuracy of defect detection.

[0014] Among them, the first target network model includes a first target detection sub-model and a first defect detection sub-model, the second target network model includes a second target detection sub-model and a second defect detection sub-model, and the target network model is used to perform defect detection on the target image, including: using the first target detection sub-model or the second target detection sub-model to perform target detection on the target image to obtain several target detection areas; using the first defect detection sub-model or the second target detection sub-model to perform defect detection on the target detection area to obtain the defect detection result of the target image.

[0015] Therefore, by first performing target detection on larger structures or single-stage structures to obtain several target detection areas, and then performing defect detection on the target detection areas, the two-stage defect detection method can further improve the accuracy of defect detection.

[0016] Among them, the third target network model includes a third target detection sub-model, a fourth target detection sub-model and a third defect detection sub-model, and the target network model is used to perform defect detection on the target image, including: using the third target detection sub-model to perform target detection on the target image to obtain several candidate detection areas containing adjacent structures; using the fourth target detection sub-model to perform target detection on the candidate detection areas to obtain target detection areas containing target structures; using the third defect detection sub-model to perform defect detection on the target detection areas to obtain defect detection results on the target image.

[0017] Therefore, by first determining the candidate detection area for the multi-level structure type, then determining the target detection area from the candidate detection area, and then performing defect detection on the target detection area, this three-level defect detection method can further improve the accuracy of defect detection.

[0018] Among them, the target network model includes a target detection submodel and a defect detection submodel, the target detection submodel is used to perform target detection on the target image to obtain a target detection result, and the defect detection submodel is used to perform defect detection on the target image based on the target detection result to obtain a defect detection result of the target image. The method includes a training process for each target network model, and the training process includes: for each structure type, obtaining a first sample image containing a target structure corresponding to the structure type, and using the first sample image to train the target detection submodel corresponding to the structure type, the target structure is the structure where the defect to be detected is located; and, obtaining a second sample image, and annotating different substructures of the target structure in the second sample image to obtain an annotated image; using the annotated image to train the image segmentation model; using the subnetwork used for feature extraction in the image segmentation model as the feature extraction subnetwork of the defect detection submodel; using the third sample image to train the defect detection submodel, and the third sample image contains the defect to be detected.

[0019] Therefore, by first training the image segmentation model and then using the subnetwork for feature extraction in the image segmentation model as the feature extraction subnetwork of the defect detection submodel, the defect detection submodel can take into account the structural characteristics of the structure where the target defect is located.

[0020] The present application provides a defect detection device, including: an image acquisition module, used to acquire a target image to be subjected to defect detection; a model determination module, used to determine a target network model for performing defect detection on the target image based on the structure type corresponding to the target structure where the defect to be detected is located in the target image, wherein different target network models correspond to different structure types; and a defect detection module, used to perform defect detection on the target image using the target network model.

[0021] The present application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-mentioned defect detection method.

[0022] The present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned defect detection method when the program instructions are executed by a processor.

[0023] The above-mentioned solution uses a target network model to detect defects in target images. Compared with manual detection, the technical solution provided by this application has higher accuracy in defect detection. In addition, different target network models are selected for defect detection of different structural types, further improving the accuracy of defect detection.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0026] Figure 1 This is a flow chart of an embodiment of the defect detection method of the present application;

[0027] Figure 2 This is a partial flow chart showing step S12 in an embodiment of the defect detection method of the present application;

[0028] Figure 3 This is another flowchart of an embodiment of the defect detection method of the present application;

[0029] Figure 4 This is a schematic structural diagram of an embodiment of a defect detection device of the present application;

[0030] Figure 5 This is a structural diagram of an embodiment of an electronic device of the present application;

[0031] Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0032] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0033] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0034] The term "and / or" in this article is simply a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0035] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of the defect detection method of the present application.

[0036] Specifically, the following steps may be included:

[0037] Step S11: Acquire a target image to be inspected for defects.

[0038] In some disclosed embodiments, the target image to be inspected can be obtained by capturing it using a device executing the defect detection method provided by the disclosed embodiments, or by capturing it using an image acquisition device that is in communication with the executing device. In the disclosed embodiments, the target image is obtained by the executing device from an image acquisition device.

[0039] Exemplarily, an image acquisition device captures a target image in a specific environment. For example, the image acquisition device may be an imaging device on a high-speed railway inspection vehicle. The imaging device captures an image of the contact network to obtain a target image. The image acquisition device then transmits the target image to an execution device. The execution device performs defect detection on the target image transmitted by the image acquisition device according to the method described in the embodiments of the present disclosure, thereby obtaining a detection result of whether various components on the contact network have defects. Exemplarily, possible defects in the contact network include loose pipe caps, loose anchor angle steel jacking bolts, loose bolts, loose nuts, loose jacking bolt thin nuts, and loose U-shaped latches.

[0040] Step S12: determining a target network model for defect detection on the target image based on the structure type corresponding to the target structure where the defect to be detected is located in the target image, wherein different target network models correspond to different structure types.

[0041] In some disclosed embodiments, the structural type corresponding to the target structure where the defect to be detected is located can be determined based on the size of the structure where the defect to be detected is located, or the structural type corresponding to the target structure where the defect to be detected is located can be determined based on the structural characteristics of the structure where the defect to be detected is located, or the structural type corresponding to the target structure where the defect to be detected is located can be determined based on both the size of the structure where the defect to be detected is located and the structural characteristics of the structure where the defect to be detected is located.

[0042] The target network models corresponding to different structure types may be the same in structure but different in network parameters, or the target network models may be different in structure.

[0043] In some disclosed embodiments, the target network model corresponding to a target image may be one or multiple different target network models. For example, if two target network models correspond to a target image, the target image is input into each of the two target network models for defect detection, resulting in two defect detection results. Defect detection on the target image by each target network model may be performed in parallel.

[0044] Step S13: Utilize the target network model to perform defect detection on the target image.

[0045] Specifically, the target image is used as the input of the target network model, and the target image is processed by the target network model to obtain the defect detection result of the target image.

[0046] The above-mentioned solution uses a target network model to detect defects in target images. Compared with manual detection, the technical solution provided by this application has higher accuracy in defect detection. In addition, different target network models are selected for defect detection of different structural types, further improving the accuracy of defect detection.

[0047] In some disclosed embodiments, the structure type is related to the size of the target structure and / or the positional relationship between the target structure and the adjacent structures. In some application scenarios, the structure type is related to the size of the target structure. In some application scenarios, the structure type is related to the positional relationship between the target structure and the adjacent structures. In some application scenarios, the structure type is related to the size of the target structure and the positional relationship between the target structure and the adjacent structures. Among them, the adjacent structure refers to a structure that is relatively fixed in position with respect to the target structure. For example, a loose U-shaped pin defect generally occurs on the load-bearing cable clamp of a suspension string. Therefore, the target structure where the U-shaped pin loose defect is located is the U-shaped pin, and the adjacent structure can be the load-bearing cable clamp of a suspension string. Before executing step S12, the following steps can also be performed:

[0048] A structure type setting instruction is received. In response to the structure type setting instruction, a structure type is set for a target structure corresponding to the target image. The target structure is a structure where a defect to be detected is located in the target image.

[0049] In some application scenarios, the structure type of the target structure is set to a large target structure type or a small target structure type according to the size of the target structure. Among them, the small target structure type is a structure in which the pixel ratio of the target structure in the target image is less than a preset ratio. The large target structure type is a structure in which the pixel ratio of the target structure in the target image is greater than or equal to a preset ratio. In some application scenarios, the structure type of the target structure is set to a multi-level structure type or a single-level structure type according to the positional relationship between the target structure and the adjacent structure. Among them, the multi-level structure type may be an adjacent structure that has a relatively fixed position with respect to the target structure, such as the above-mentioned U-shaped pin example. The single-level structure type may be an adjacent structure that does not have a relatively fixed position with respect to the target structure, such as a loose nut defect. Nut parts are widely used in the entire contact network, so it is inconvenient to summarize the adjacent structures that appear fixedly with respect to the target structure.

[0050] In some application scenarios, the target structure is assigned a structure type based on its size and type, the size of the target structure, and the positional relationship between the target structure and adjacent structures. For example, the target structure type can be divided into a large target structure type, a single-level structure small target type, and a multi-level structure small target type.

[0051] For example, due to the presence of multiple loose defects in high-speed rail contact lines, each defect type exhibits distinct characteristics in the captured images and belongs to different structures. Defect types where the defective component accounts for less than 0.5% of the pixels in the overall target image are classified as "small target types," such as loose nuts. Other types are classified as "large target types," such as loose pipe caps. The structural characteristics of small target types are then analyzed. If the defective component has fixed adjacent components in the contact line, it is classified as a "multi-level structure small target type." For example, a loose U-shaped pin defect only occurs on the catenary wire clamp of the dropper. If the defective component cannot be identified as a fixed adjacent component in the contact line, it is classified as a "single-level structure small target type." For example, a loose nut defect is widely used throughout the contact line, and its fixed parent component cannot be identified. Thus, defects in high-speed rail contact lines are divided into three categories: "large target type defects," "multi-level structure small target type defects," and "single-level structure small target type defects."

[0052] In some disclosed embodiments, before executing step S12, the following steps may be further executed:

[0053] Determine the similarity between the target image and each historical image in the database. The historical image is an image that has been defect-detected, and the database records the structural type of the target structure where the defect to be detected is located in each historical image. The database can be set in the execution device. Then, the structural type corresponding to the target structure where the defect to be detected is located in the target historical image whose similarity meets the similarity requirement is used as the structural type corresponding to the target structure in the target image. The similarity calculation method of the two images can refer to conventional technology and will not be repeated here. The similarity requirement can be that the similarity with the target image is the highest, and the similarity is greater than a preset similarity threshold. For example, there are 3 historical images in total, the preset similarity threshold is 0.7, and the similarities between each historical image and the target image are 0.5, 0.6, and 0.8 respectively. Then, the historical image with a similarity of 0.8 with the target image is used as the target historical image, and the structural type corresponding to the target structure where the defect to be detected is located in the target historical image is the structural type corresponding to the target structure where the defect to be detected is located in the target image. In other disclosed embodiments, if no target historical image exists that satisfies the similarity requirement with the target image, a prompt is issued, allowing the user to issue a structure type setting instruction based on the prompt. The execution device then responds to the structure type setting instruction and sets the structure type for the target structure corresponding to the target image. The specific method for setting the structure type for the target structure corresponding to the target image in response to the structure type setting instruction can be found in the previous embodiment and will not be repeated here.

[0054] By determining the target network model for defect detection on the target image based on the structural type of the structure where the defect to be detected is located, the structural information of the structure where the defect to be detected is referenced, thereby improving the accuracy of defect detection on the target image.

[0055] Please also see Figure 2 , Figure 2 This is a partial flow chart showing step S12 in an embodiment of the defect detection method of the present application. Figure 2 As shown, the above step S12 may specifically include the following steps:

[0056] Step S121: Determine whether the structure type is the first target type, and obtain a first determination result.

[0057] The first target type is a type in which the pixel ratio of the target structure in the target image is less than a first preset value. The first preset value can be defined by the user. Specifically, the first target type can be the small target type described above.

[0058] Step S122: Based on the first judgment result, determine a target network model for performing defect detection on the target image.

[0059] By determining the corresponding target network model based on the pixel ratio of the target structure in the target image, the corresponding target network model can be selected for defect detection in combination with the size of the target structure, thereby improving the accuracy of defect detection.

[0060] In some disclosed embodiments, based on the first judgment result, a method for determining a target network model for performing defect detection on a target image may be:

[0061] In response to the structure type being the second target type, determining the target network model to be the first target network model corresponding to the second target type. The second target type is a type in which the pixel ratio of the target structure in the target image is greater than or equal to a first preset value. Specifically, the second target type can be the large target type described above.

[0062] In response to the structure type being the first target type, a determination is made as to whether the first target type is a multi-level structure type, obtaining a second determination result. A multi-level structure type is defined as an adjacent structure having a relatively fixed position relative to the target structure. A target that is both the first target type and a multi-level structure type is considered a multi-level structure small target type, while a target that is both the first target type and a single-level structure type is considered a single-level structure small target type.

[0063] Then, based on the second judgment result, a target network model for performing defect detection on the target image is determined.

[0064] By selecting a corresponding target network model according to whether the target structure is a multi-level structure, the positional relationship between the target structure and adjacent structures can be considered, thereby improving the accuracy of defect detection.

[0065] Wherein, based on the second judgment result, a method of determining a target network model for performing defect detection on a target image may include:

[0066] In response to the first target type being a single-level structure type, the target network model is determined to be a second target network model corresponding to the single-level structure type. The single-level structure type refers to a structure in which there are no adjacent structures with fixed relative positions to the target structure. That is, if the target structure is a small single-level structure type, the target network model for defect detection in the target image is determined to be the second target network model.

[0067] In response to the first target type being a multi-level structure type, the target network model is determined to be a third target network model corresponding to the multi-level structure type.

[0068] That is, when the target structure is the above-mentioned multi-level structure small target type, the target network model for defect detection on the target image is determined to be the third target network model.

[0069] By selecting a corresponding target network model according to whether the target structure is a multi-level structure, the positional relationship between the target structure and adjacent structures can be considered, thereby improving the accuracy of defect detection.

[0070] The first target network model includes a first target detection sub-model and a first defect detection sub-model. The second target network model includes a second target detection sub-model and a second defect detection sub-model. The third target network model includes a third target detection sub-model, a fourth target detection sub-model, and a third defect detection sub-model. The network parameters in each target detection sub-model can be the same or different, and the network parameters in each defect detection sub-model can be the same or different.

[0071] In some disclosed embodiments, step S13 may include the following steps:

[0072] Wherein, after determining that the target network model used for defect detection on the target image includes the first target network model, the target image is subjected to target detection using the first target detection sub-model to obtain a number of target detection areas. Wherein, the several specifically described in the embodiment of the present disclosure may be one or more, for example, the target detection area may be one, or may be two, etc. Specifically, the target detection area refers to the image area containing the target structure. That is, multiple image areas containing target structures can be obtained by performing target detection. The target detection area is subjected to defect detection using the first defect detection sub-model to obtain a defect detection result of the target image. That is, a number of target detection areas are input into the first defect detection sub-model, and the first defect detection sub-model performs defect detection on each target detection area to obtain a defect detection result of each target detection area, and then the defect detection results of each target detection area are spliced ​​to obtain a defect detection result of the target image.

[0073] Among them, after determining that the target network model used for defect detection of the target image includes the second target network model, the second target detection sub-model is used to perform target detection on the target image to obtain several target detection areas. Specifically, the target detection area refers to the image area containing the target structure. That is, by performing target detection, multiple image areas containing target structures can be obtained. The second defect detection sub-model is used to perform defect detection on the target detection area to obtain the defect detection result of the target image. That is, several target detection areas are input into the second defect detection sub-model, and the second defect detection sub-model performs defect detection on each target detection area to obtain the defect detection result of each target detection area, and then the defect detection results of each target detection area are spliced ​​to obtain the defect detection result of the target image. Among them, the second target detection sub-model in the second target network model has a stronger detection capability for target structures of small target types.

[0074] By first performing target detection on larger structures or single-stage structures to obtain several target detection areas, and then performing defect detection on the target detection areas, the two-stage defect detection method can further improve the accuracy of defect detection.

[0075] After determining that the target network model for performing defect detection on the target image includes the third target network model, step S13 may include the following steps:

[0076] The target image is detected using the third target detection sub-model to obtain several candidate detection areas containing adjacent structures. The adjacent structures here refer to the adjacent structures whose positions are relatively fixed with respect to the target structure. The third target detection sub-model performs target detection on the target image, determines the positions of the adjacent structures, and then determines the size of the candidate detection area based on the positional relationship between the adjacent structures and the target structure, the size of the adjacent structures, and the size of the target structure. This allows the candidate detection area to contain both the target structure and the adjacent structures, facilitating target detection on the candidate detection area by the fourth target detection sub-model.

[0077] Then, the fourth object detection sub-model is used to perform object detection on the candidate detection regions to obtain the target detection regions containing the target structure. Specifically, each candidate detection region is used as the input of the fourth object detection sub-model. After performing object detection on each candidate detection, the fourth object detection sub-model obtains the position of each target structure in the candidate detection region. Based on the position of each target structure, the candidate detection region is cropped to obtain the corresponding target detection region.

[0078] Finally, the third defect detection sub-model is used to perform defect detection on the target detection area to obtain a defect detection result for the target image. Each target detection area is used as input to the third defect detection sub-model, which performs defect detection on each target detection area to obtain a defect detection result for each target detection area. The defect detection results corresponding to each target detection area are then spliced ​​together to obtain a defect detection result for the target image.

[0079] By first determining the candidate detection area for the multi-level structure type, then determining the target detection area from the candidate detection area, and then performing defect detection on the target detection area, this three-level defect detection method can further improve the accuracy of defect detection.

[0080] For example, for "large target types," a general target detection algorithm (i.e., the first target detection sub-model) is used to locate the component (i.e., the target structure) where the defect is located. For example, for a loose pipe cap defect, the trained general target detection algorithm can be directly used to detect all pipe cap components in the input image. For "single-level structure small target types," a small target detection algorithm (i.e., the second target detection sub-model) with stronger small target detection capabilities is used to locate the component (i.e., the target structure) where the defect is located. For example, for a loose nut defect, the trained small target detection algorithm is used to detect all nuts in the input image. For "multi-level structure small target types," a general target detection algorithm (i.e., the third target detection sub-model) is first used to detect the adjacent components (i.e., the adjacent structure) of the defective component. The general target detection algorithm (i.e., the fourth target detection sub-model) is then used to detect the component where the defect is located on the adjacent component detection result sub-graph. For example, for a loose U-shaped latch defect, the suspension string is first detected, and then all the load-bearing cable and suspension string clamp components where the U-shaped latch is located are detected on the suspension string sub-graph.

[0081] In some application scenarios, the first, third, and fourth object detection sub-models can be the same object detection sub-model capable of localizing multiple object structures. That is, the object detection sub-models are trained using training samples containing multiple structures, enabling the trained object detection sub-models to detect multiple structures.

[0082] As described above, the target network model includes a target detection sub-model and a defect detection sub-model. The target detection sub-model is used to perform target detection on the target image and obtain target detection results. The target detection results include multiple target detection regions. The defect detection sub-model is used to perform defect detection on the target image based on the target detection results and obtain defect detection results for the target image. In other words, the defect detection sub-model is used to perform defect detection on multiple target detection regions and obtain defect detection results for the target image.

[0083] In some disclosed embodiments, the defect detection method provided by the disclosed embodiments further includes a training process for each target network model. The training process may include the following steps:

[0084] For each structure type, a first sample image containing a target structure corresponding to the structure type is obtained, and the target detection sub-model corresponding to the structure type is trained using the first sample image. As mentioned above, the target structure is the structure where the defect to be detected is located. The target structure contained in the first sample image may or may not contain a defect to be detected. For the training of the target detection sub-model, the first sample image needs to be labeled with the target structure, but does not need to be labeled with the defect to be detected. As mentioned above, the target detection sub-model can be a model for target detection of the target structure, or a model for target detection of the adjacent structure of the target structure. Therefore, the sample image containing the adjacent structure can be used to train the target detection sub-model for detecting the adjacent structure. The target structure corresponding to each structure type can be one or more. That is, different target structures belonging to the same structure type can be used to train the target detection sub-model.

[0085] A second sample image is obtained. Different substructures of the target structure in the second sample image are annotated to obtain an annotated image. The annotated image is then used to train an image segmentation model. For example, in the case of a loose nut defect, the screw structure and the nut structure in the nut part where the defect is located are segmented and annotated as two different objects, and the image segmentation model is trained. To better understand the different substructures of the target structure, annotated images are obtained.

[0086] The subnetwork used for feature extraction in the image segmentation model is used as the feature extraction subnetwork of the defect detection submodel. In other words, the subnetwork used for feature extraction in the image segmentation model is used as a pre-trained model for the defect detection submodel to train the defect detection submodel.

[0087] The defect detection sub-model corresponding to the defect to be detected is trained using the third sample image. The third sample image can be cropped from the first sample image or from other training images. The third sample image is annotated with flag information indicating whether the third sample image contains the defect to be detected. Exemplarily, the flag information can be a label, where the label for the third sample image containing the defect to be detected is "yes", and the label for the third sample image without the defect to be detected is "no". In some application scenarios, if the third sample image contains multiple target structures, a label can be set for each target structure separately, and the label is used to indicate whether the corresponding target structure has the defect to be detected.

[0088] By using samples corresponding to the defects to be detected, the target network model corresponding to each defect to be detected is trained, so that the trained target network model can perform defect detection on the corresponding defect to be detected.

[0089] For example, a high-definition imaging device installed on an inspection vehicle is used to capture an image of a high-speed rail contact network. The target structure and / or adjacent structures of the target structure are annotated on the high-speed rail contact network. The image is then input into a target detection sub-model in a trained target network model. The target detection sub-model then obtains a target detection region containing the target structure or a candidate target detection region containing adjacent structures. After the landmark information of the target detection region containing the target structure is annotated, the image is used as the third sample image to input into the corresponding defect detection sub-model for training the defect detection sub-model. Alternatively, after the target structure in the candidate target detection region is annotated, the image is used as the first sample image of the next-level target detection sub-model for training.

[0090] By first training the image segmentation model, and then performing feature extraction on the sub-network in the image segmentation model.

[0091] As a feature extraction subnetwork of the defect detection submodel, the defect detection submodel is able to consider the structural characteristics of the structure where the defect to be detected is located.

[0092] To better understand the defect detection method provided by the embodiment of the present disclosure, please also refer to Figure 3 , Figure 3 This is another flow chart of an embodiment of the defect detection method of the present application. Figure 4 As shown, the defect detection method may include the following steps:

[0093] Step S21: Acquire a target image to be inspected for defects.

[0094] The method of obtaining the target image to be inspected for defects is as described in the above step S11 and will not be repeated here.

[0095] Step S22: receiving a structure type setting instruction, and responding to the structure type setting instruction to set a structure type for a target structure corresponding to the target image.

[0096] The method of receiving the structure type setting instruction and setting the structure type for the target structure corresponding to the target image in response to the structure type setting instruction may be as follows: if the user currently wants to detect one or more defects to be detected, the structure type of the target structure in the target image is set to the corresponding structure type. For example, if the user currently wants to detect a loose nut defect, the structure type setting instruction for the target structure in the target image is set to a single-stage small target type.

[0097] Step S23: Determine whether the structure type is the first target type.

[0098] If the judgment result is no, step S24 is executed, and if the judgment result is yes, step S26 is executed.

[0099] Step S24: In response to the structure type being the second target type, determining that the target network model is the first target network model corresponding to the second target type.

[0100] Step S25: Utilize the first target network model to perform defect detection on the target image.

[0101] Step S26: In response to the structure type being the first target type, determining whether the first target type is a multi-level structure type.

[0102] If the judgment result is no, step S27 and its subsequent steps are executed; if the judgment result is yes, step S29 and its subsequent steps are executed.

[0103] Step S27: In response to the first target type being a single-level structure type, determining that the target network model is a second target network model corresponding to the single-level structure type.

[0104] Step S28: Utilize the second target network model to perform defect detection on the target image.

[0105] Step S29: In response to the first target type being a multi-level structure type, determining that the target network model is a third target network model corresponding to the multi-level structure type.

[0106] Step S30: Utilize the third target network model to perform defect detection on the target image.

[0107] In some application scenarios, the high-speed rail contact network images obtained by the high-definition imaging equipment on the inspection vehicle are input into the trained component positioning model (i.e., the target detection sub-model) to obtain all defective components that may have loose defects. Then all the component sub-images are input into the trained defect detection sub-model to obtain the model's classification prediction results on whether the component has defects, and the pre-set threshold is used to determine whether the component has the corresponding defect.

[0108] The above-mentioned solution uses a target network model to detect defects in target images. Compared with manual detection, the technical solution provided by this application has higher accuracy in defect detection. In addition, different target network models are selected for defect detection of different structural types, further improving the accuracy of defect detection.

[0109] In addition, the sub-network used for feature extraction in the defect detection sub-model is trained using pixel-level annotations of different parts structures, allowing the defect detection sub-model to pay attention to the structural information of the parts, thereby improving the accuracy of defect detection.

[0110] In addition, different component positioning processes are designed according to the characteristics of the components where the defects to be detected are located, which effectively improves the positioning accuracy of the defects to be detected.

[0111] In addition, the overall defect detection method is simple and efficient, and can efficiently meet the detection needs of different loose defects in high-speed rail contact network application scenarios. For example, for loose defects that have appeared in high-speed rail contact networks, such as loose bolts, loose nuts, loose pipe caps, loose anchor angle steel tightening bolts, loose U-shaped pins, and loose tightening bolts and thin nuts, this method can simply and efficiently detect these defects. Of course, for other industrial production scenarios involving loose defects, the method mentioned in the embodiment of the present disclosure can also be used to improve the accuracy of defect detection. Alternatively, for new types of loose defects in high-speed rail contact networks, the method described in the embodiment of the present disclosure is used to locate and classify the defects, which can ensure a higher defect detection rate.

[0112] In some application scenarios, the defect detection method provided by the embodiments of the present disclosure can be applied to high-speed rail contact network inspection systems or other defect detection systems.

[0113] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0114] The defect detection method may be executed by a defect detection device, for example, a terminal device, a server, or other processing device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or the like. In some possible implementations, the defect detection method may be implemented by a processor invoking computer-readable instructions stored in a memory.

[0115] See also Figure 4 , Figure 4 Schematic diagram of the structure of an embodiment of a defect detection device of the present application. Defect detection device 40 includes an image acquisition module 41, a model determination module 42, and a defect detection module 43. Image acquisition module 41 is used to acquire a target image for defect detection; model determination module 42 is used to determine a target network model for defect detection in the target image based on the structure type corresponding to the target structure where the defect to be detected is located in the target image, where different target network models correspond to different structure types; and defect detection module 43 is used to perform defect detection on the target image using the target network model.

[0116] The above-mentioned solution uses a target network model to detect defects in target images. Compared with manual detection, the technical solution provided by this application has higher accuracy in defect detection. In addition, different target network models are selected for defect detection of different structural types, further improving the accuracy of defect detection.

[0117] In some disclosed embodiments, the structure type is related to the size of the target structure and / or the positional relationship between the target structure and adjacent structures. Based on the structure type corresponding to the target structure where the defect to be detected is located in the target image, before determining the target network model for defect detection on the target image, the model determination module 42 is also used to: receive a structure type setting instruction; set the structure type for the target structure corresponding to the target image in response to the structure type setting instruction; or, determine the similarity between the target image and each historical image in the database, the historical image is an image that has undergone defect detection, and the database records the structure type corresponding to the target structure where the defect to be detected is located in each historical image; use the structure type corresponding to the target structure where the defect to be detected is located in the target historical image whose similarity meets the similarity requirements as the structure type corresponding to the target structure in the target image.

[0118] The above scheme determines the structural type corresponding to the defect to be detected based on the positional relationship between the target structure and adjacent structures, enabling the comprehensive determination of the target network model for defect detection based on structures other than the defect to be detected. Furthermore, by receiving a structure type setting instruction and setting the structure type for the target structure corresponding to the target image based on that instruction, user autonomy is enhanced. Furthermore, the structural type corresponding to the target structure in the target image is determined based on the similarity between the target image and various historical images, eliminating the need for the user to set the corresponding structure type for each image.

[0119] In some disclosed embodiments, the model determination module 42 determines a target network model for defect detection on a target image based on a structure type corresponding to a target structure where a defect to be detected is located in the target image, including: determining whether the structure type is a first target type, and obtaining a first judgment result, where the first target type is a type in which the pixel ratio of the target structure in the target image is less than a first preset value; and determining a target network model for defect detection on the target image based on the first judgment result.

[0120] The above scheme determines the corresponding target network model according to the pixel ratio of the target structure in the target image, so that the corresponding target network model can be selected for defect detection in combination with the size of the target structure, thereby improving the accuracy of defect detection.

[0121] In some disclosed embodiments, the model determination module 42 determines a target network model for performing defect detection on a target image based on a first judgment result, including: in response to the structure type being the second target type, determining that the target network model is the first target network model corresponding to the second target type, the second target type being a type in which the pixel ratio of the target structure in the target image is greater than or equal to a first preset value; or, in response to the structure type being the first target type, determining whether the first target type is a multi-level structure type, and obtaining a second judgment result, the multi-level structure type being an adjacent structure with a relatively fixed position to the target structure; based on the second judgment result, determining the target network model for performing defect detection on the target image.

[0122] The above solution selects the corresponding target network model according to whether the target structure is a multi-level structure type, so that the positional relationship between the target structure and adjacent structures can be considered, thereby improving the accuracy of defect detection.

[0123] In some disclosed embodiments, the model determination module 42 determines a target network model for defect detection on a target image based on a second judgment result, including: in response to the first target type being a single-level structure type, determining that the target network model is a second target network model corresponding to the single-level structure type, where the single-level structure type does not have an adjacent structure whose position is relatively fixed to the target structure; in response to the first target type being a multi-level structure type, determining that the target network model is a third target network model corresponding to the multi-level structure type.

[0124] The above solution selects the corresponding target network model according to whether the target structure is a multi-level structure type, so that the positional relationship between the target structure and adjacent structures can be considered, thereby improving the accuracy of defect detection.

[0125] In some disclosed embodiments, the first target network model includes a first target detection sub-model and a first defect detection sub-model, the second target network model includes a second target detection sub-model and a second defect detection sub-model, and the defect detection module 43 uses the target network model to perform defect detection on the target image, including: using the first target detection sub-model or the second target detection sub-model to perform target detection on the target image to obtain several target detection areas; using the first defect detection sub-model or the second target detection sub-model to perform defect detection on the target detection area to obtain a defect detection result of the target image.

[0126] The above scheme first performs target detection on larger structures or single-stage structures to obtain several target detection areas, and then performs defect detection on the target detection areas. The two-stage defect detection method can further improve the accuracy of defect detection.

[0127] In some disclosed embodiments, the third target network model includes a third target detection sub-model, a fourth target detection sub-model and a third defect detection sub-model, and the defect detection module 43 uses the target network model to perform defect detection on the target image, including: using the third target detection sub-model to perform target detection on the target image to obtain several candidate detection areas containing adjacent structures; using the fourth target detection sub-model to perform target detection on the candidate detection areas to obtain target detection areas containing target structures; using the third defect detection sub-model to perform defect detection on the target detection areas to obtain defect detection results on the target image.

[0128] The above scheme first determines the candidate detection area for the multi-level structure type, then determines the target detection area from the candidate detection area, and then performs defect detection on the target detection area. This three-level defect detection method can further improve the accuracy of defect detection.

[0129] In some disclosed embodiments, the target network model includes a target detection sub-model and a defect detection sub-model. The target detection sub-model is used to perform target detection on a target image to obtain a target detection result. The defect detection sub-model is used to perform defect detection on the target image based on the target detection result to obtain a defect detection result for the target image. The defect detection device includes a training module (not shown) that is used to perform training steps for each target network model. The training steps include: for each structure type, obtaining a first sample image containing a target structure corresponding to the structure type, and using the first sample image to train the target detection sub-model corresponding to the structure type, where the target structure is the structure where the defect to be detected is located; obtaining a second sample image, and annotating different sub-structures of the target structure in the second sample image to obtain an annotated image; using the annotated image to train an image segmentation model; using the sub-network for feature extraction in the image segmentation model as the feature extraction sub-network of the defect detection sub-model; and training the defect detection sub-model using a third sample image, where the third sample image contains the defect to be detected.

[0130] The above scheme first trains the image segmentation model and then uses the subnetwork for feature extraction in the image segmentation model as the feature extraction subnetwork of the defect detection submodel, so that the defect detection submodel can consider the structural characteristics of the structure where the defect to be detected is located.

[0131] See also Figure 5 , Figure 51 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 50 includes a memory 51 and a processor 52. The processor 52 is configured to execute program instructions stored in the memory 51 to implement the steps of the above-described defect detection method embodiment. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer and a server. Furthermore, the electronic device 50 may also include mobile devices such as laptops and tablet computers, which are not limited herein.

[0132] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps in any of the above-mentioned defect detection method embodiments. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0133] The above-mentioned solution uses a target network model to detect defects in target images. Compared with manual detection, the technical solution provided by this application has higher accuracy in defect detection. In addition, different target network models are selected for defect detection of different structural types, further improving the accuracy of defect detection.

[0134] See also Figure 6 , Figure 6 The computer-readable storage medium 60 stores program instructions 601, which, when executed by a processor, are used to implement the steps of the above-mentioned defect detection method embodiment.

[0135] The above-mentioned solution uses a target network model to detect defects in target images. Compared with manual detection, the technical solution provided by this application has higher accuracy in defect detection. In addition, different target network models are selected for defect detection of different structural types, further improving the accuracy of defect detection.

[0136] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0137] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0139] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A defect detection method, characterized in that: include: Acquire a target image to be inspected for defects; Determining a target network model for defect detection on the target image based on a structure type corresponding to a target structure where a defect to be detected is located in the target image, including: determining whether the structure type is a first target type to obtain a first judgment result, where the first target type is a type where the pixel ratio of the target structure in the target image is less than a first preset value; determining a target network model for defect detection on the target image based on the first judgment result, where different target network models correspond to different structure types; Performing defect detection on the target image using the target network model; Wherein, determining a target network model for performing defect detection on the target image based on the first judgment result includes: In response to the structure type being a second target type, determining that the target network model is a first target network model corresponding to the second target type, the second target type being a type in which a pixel ratio of the target structure in the target image is greater than or equal to a first preset value; Or, in response to the structure type being the first target type, determining whether the first target type is a multi-level structure type, and obtaining a second determination result, the multi-level structure type is that there is an adjacent structure that is relatively fixed in position with respect to the target structure; The target network model for performing defect detection on the target image based on the second judgment result is determined, including: in response to the first target type being a single-level structure type, determining that the target network model is a second target network model corresponding to the single-level structure type, and the single-level structure type is that there is no adjacent structure with a relatively fixed position to the target structure; in response to the first target type being a multi-level structure type, determining that the target network model is a third target network model corresponding to the multi-level structure type.

2. The method according to claim 1, characterized in that The structure type is related to the size of the target structure and / or the positional relationship between the target structure and adjacent structures. Before determining the target network model for defect detection on the target image based on the structure type corresponding to the target structure where the defect to be detected is located in the target image, the method further includes: Receive structure type setting instructions; In response to the structure type setting instruction, setting a structure type for a target structure corresponding to the target image; Alternatively, determining a similarity between the target image and each historical image in a database, wherein the historical image is an image that has undergone defect detection, and the database records a structure type corresponding to a target structure where a defect to be detected is located in each of the historical images; The structure type corresponding to the target structure where the defect to be detected is located in the target historical image whose similarity meets the similarity requirement is used as the structure type corresponding to the target structure in the target image.

3. The method according to claim 1, characterized in that The first target network model includes a first target detection sub-model and a first defect detection sub-model, the second target network model includes a second target detection sub-model and a second defect detection sub-model, and performing defect detection on the target image using the target network model includes: Performing target detection on the target image using the first target detection sub-model or the second target detection sub-model to obtain a plurality of target detection areas; Defect detection is performed on the target detection area using the first defect detection sub-model or the second target detection sub-model to obtain a defect detection result of the target image.

4. The method according to claim 1, wherein The third target network model includes a third target detection sub-model, a fourth target detection sub-model, and a third defect detection sub-model. The performing defect detection on the target image using the target network model includes: Performing target detection on the target image using the third target detection sub-model to obtain a plurality of candidate detection regions containing the adjacent structures; Performing target detection on the candidate detection area using the fourth target detection sub-model to obtain a target detection area containing a target structure; Defect detection is performed on the target detection area using the third defect detection sub-model to obtain a defect detection result for the target image.

5. The method according to any one of claims 1 to 4, characterized in that The target network model includes a target detection sub-model and a defect detection sub-model. The target detection sub-model is used to perform target detection on the target image to obtain a target detection result. The defect detection sub-model is used to perform defect detection on the target image based on the target detection result to obtain a defect detection result of the target image. The method includes a training process for each target network model. The training process includes: For each of the structural types, obtaining a first sample image containing a target structure corresponding to the structural type, and using the first sample image to train a target detection sub-model corresponding to the structural type, wherein the target structure is the structure where the defect to be detected is located; and Acquire a second sample image, and annotate different substructures of the target structure in the second sample image to obtain an annotated image; Training an image segmentation model using the labeled image; Using the subnetwork for feature extraction in the image segmentation model as the feature extraction subnetwork of the defect detection submodel; The defect detection sub-model is trained using a third sample image, where the third sample image contains the defect to be detected.

6. A defect detection device, characterized in that: include: An image acquisition module is used to acquire a target image to be inspected for defects; a model determination module, configured to determine a target network model for defect detection on the target image based on a structure type corresponding to a target structure where a defect to be detected is located in the target image, comprising: determining whether the structure type is a first target type, and obtaining a first determination result, wherein the first target type is a type in which a pixel ratio of the target structure in the target image is less than a first preset value; and determining a target network model for defect detection on the target image based on the first determination result, wherein different target network models correspond to different structure types; A defect detection module, configured to perform defect detection on the target image using the target network model; Wherein, the model determination module is used to determine the target network model for defect detection on the target image based on the first judgment result, including: in response to the structure type being the second target type, determining that the target network model is the first target network model corresponding to the second target type, the second target type being a type in which the pixel ratio of the target structure in the target image is greater than or equal to a first preset value; or, in response to the structure type being the first target type, judging whether the first target type is a multi-level structure type, and obtaining a second judgment result, the multi-level structure type being that there are adjacent structures whose positions are relatively fixed to the target structure; wherein, based on the second judgment result, determining the target network model for defect detection on the target image includes: in response to the first target type being a single-level structure type, determining that the target network model is the second target network model corresponding to the single-level structure type, the single-level structure type being that there are no adjacent structures whose positions are relatively fixed to the target structure; in response to the first target type being a multi-level structure type, determining that the target network model is the third target network model corresponding to the multi-level structure type.

7. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images