Bolt defect controllable generation method based on view angle and attribute guidance

Abstract

The invention discloses a visual angle and attribute guidance-based bolt defect controllable generation method, which comprises the following steps of: constructing a generative adversarial network, selecting a FastGAN generator as a basic model of a generation part, introducing visual angle and attribute information into the generator as conditions to guide generation, and proposing a residual error type cross-layer excitation ResSLE. The gradient information flow is enhanced; the generation controllability is increased; a U-Net self-supervised reconstruction discriminator (USRD) is proposed in a discrimination part, and a unique local cutting mode Shape-crop is designed for the T-shaped bolt, so that the capabilities of generating bolt texture details and extracting features are enhanced. The bolt defect generation method can controllably generate a bolt defect image with a specific view angle and attribute under the condition of a small number of samples, and improves the accuracy of a bolt defect identification network in an auxiliary manner from the data level.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and in particular to a method for controllable generation of bolt defects based on viewing angle and attribute guidance. Background Art

[0002] As an important part of the power system, the stable, safe and efficient operation of transmission lines is crucial to ensure the continuity and reliability of power supply. Among the many components of transmission lines, bolts play a vital role, responsible for fixing and supporting various equipment and structures of the line. However, since the operating conditions of transmission lines are usually harsh, bolts are easily affected by environmental factors and produce defects. Therefore, timely and accurate detection of bolt defects has become the key to ensure the safe operation of the line.

[0003] In recent years, UAV intelligent inspection technology has been widely used in power transmission lines. It has the advantages of high safety and high efficiency, and can be combined with deep learning target detection and intelligent recognition algorithms to achieve intelligent processing. However, in actual transmission lines, due to various reasons, defective samples of bolts are often very scarce. This few-sample problem poses a huge challenge to the automatic detection and classification of bolt defects. Traditional machine learning and deep learning methods usually perform well when faced with a large number of samples, but their performance is often severely limited in few-sample scenarios. Therefore, how to improve the performance of the model under few-sample conditions is one of the problems that need to be solved urgently.

[0004] In the case of limited data, how to make full use of and expand samples has become a research focus. Traditional data enhancement methods, such as geometric transformation, color transformation, and pixel transformation, expand the data set and optimize the image quality through image processing technology. However, these methods are not suitable for all data enhancement operations and face great application limitations in some cases. Generative adversarial networks are an excellent data enhancement strategy. Goodfellow first proposed generative adversarial networks (GAN) in 2014. GAN constructs a set of generators and discriminators for adversarial training and reaches Nash equilibrium in the game process. As one of the fastest-growing models in recent years, GAN and its variants have achieved great success in image generation, style transfer, image super-resolution, data enhancement, etc. However, GAN still has great challenges in the generation of transmission line bolt defects.

[0005] 1) Insufficient training set for GAN: GAN model training requires a large number of real bolt defect images, but in actual transmission lines, there are too few bolts with defects. Therefore, it is easy to lose information when training GAN, causing the model to crash during training;

[0006] 2) Bolts from different perspectives have large visual differences: Usually, GAN classifies bolts with the same attributes into the same category during the generation process, ignoring the visual differences between bolt images, which may cause the generated images to deviate from the actual application scenarios. Due to differences in the installation position, environmental factors, and shooting perspectives of the bolts, the bolt images in the transmission line often have large visual differences, which further increases the complexity of the generation task, thus affecting the generation trend and resulting in the generation of erroneous samples.

[0007] Therefore, under the above background, introducing generative adversarial networks into the field of power systems to solve the current problem of insufficient bolt defect data on transmission lines, expand the bolt defect data set, and assist in improving the accuracy of bolt defect identification is a top priority to meet the current industrialization needs in the field of power systems. Summary of the invention

[0008] The purpose of this invention is to provide a controllable generation method of bolt defects based on perspective and attribute guidance to solve the problems of insufficient data of transmission line bolt defects and poor downstream task effects caused by them. By introducing perspective and attribute information, and proposing residual cross-layer excitation and U-Net self-supervised reconstruction discriminator USRD, a novel generative adversarial network is designed to controllably generate bolt defect images with specific perspectives and attributes under few sample conditions, thereby improving the accuracy of bolt defect recognition.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A controllable generation method of bolt defects based on view and attribute guidance includes the following steps:

[0011] S1, constructs the dataset Dataset 1 for training the bolt perspective label generator, which contains the front and side perspectives of normal bolts; constructs the few-sample bolt defect dataset Dataset 2, which contains three categories of bolt defect images, including normal bolts, missing pins, and missing nuts.

[0012] S2 adopts the contrastive language-image pre-training CLIP model and proposes a bolt perspective label generator based on CLIP fine-tuning. It automatically assigns perspective labels to the few-sample bolt defect dataset Dataset 2 and constructs a bolt defect dataset Dataset 3 containing perspective and attribute labels to provide data support for subsequent image generation tasks.

[0013] S3 adopts the architecture of conditional generative adversarial networks, introduces perspectives and attributes as additional conditional inputs into the generator, and proposes residual cross-layer excitation ResSLE to build a generator with stronger gradient information flow and higher generation controllability. Among them, ResSLE introduces cross-layer connections on the basis of cross-layer excitation SLE to strengthen the gradient signals between layers, and controls the intensity of excitation through the Lambda layer to improve the gradient flow of the generator.

[0014] S4, uses an encoder-decoder architecture to expand the discriminator to form U-Net, proposes a self-supervised reconstruction discriminator USRD based on U-Net, and designs a unique local cropping method Shape-crop for T-bolts, which enhances the discriminator's feature extraction and detail texture generation capabilities through global and local reconstruction.

[0015] Among them, S2 specifically includes:

[0016] First, in order to generate more accurate bolt images, it is proposed to introduce perspective information based on bolt defect attributes into GAN and divide the bolt perspective into two categories with the largest visual difference: front perspective 0° and side perspective 90°, to ensure the feasibility and accuracy of the task.

[0017] Next, a bolt perspective label generator based on CLIP fine-tuning is proposed and trained on Dataset 1 to learn the features of the two perspectives. The weighted space integration method WiSE-FT is used to fine-tune CLIP. The formula is as follows:

[0018] wse(x,α)=f(x,(1-α)·θ0+α·θ1) (1)

[0019] Among them, x represents the input data, f is the classification prediction function of the model, θ0 represents the original model parameters, θ1 represents the model parameters obtained by standard fine-tuning, α is the mixing coefficient that controls the weight, and the image encoder E of the fine-tuned CLIP is extracted as the bolt view label generator.

[0020] Finally, the bolt defect dataset Dataset 2 with attribute labels is input into the bolt perspective label generator to assign a perspective label to each image. By combining these perspective labels with the attribute labels of the bolts, the perspective-attribute bolt dataset Dataset 3 is constructed. This process is defined as follows:

[0021] Label(D3)=E(D2)+Attribute(D2) (2)

[0022] Among them, E represents the bolt perspective label generator, Attribute is the attribute label of Dataset 2, and the categories of Dataset 3 include front-normal bolt, side-normal bolt, front-pin missing, side-pin missing, front-nut missing, side-nut missing, a total of 6 categories. The images in Dataset 2 and Dataset 3 are the same, only the category labels are different. The bolt perspective label generator reduces human bias and improves label consistency.

[0023] Among them, the architecture of the conditional generative adversarial network is adopted, the perspective and attribute are introduced as additional conditional inputs in the generator, and the residual cross-layer excitation ResSLE is proposed to construct a generator with stronger gradient information flow and higher generation controllability, which specifically includes:

[0024] First, an n-dimensional conditional vector is added to the input N-dimensional random noise vector. The two are concatenated and input into the generator. The input definition is as follows:

[0025] Z=concat(z,y) (3)

[0026] Among them, Z is the input vector of the generator, z is the noise vector, and y is the conditional vector. The labels containing perspectives and attributes are used as conditional information to assist in the generation, which increases the prior knowledge of the generator and improves the controllability of GAN generation. Another reason for introducing conditions is that different bolt defect images usually have certain commonalities. Although the properties or perspectives of the bolts are different, the main body of the bolts is similar, which is conducive to the learning of the generator.

[0027] In addition, the generator of the few-shot generation model FastGAN is used as the basic model, and the architecture of the generator is improved. Residual cross-layer excitation ResSLE is proposed. ResSLE introduces cross-layer connections on the basis of SLE to strengthen the gradient signal between layers, and controls the intensity of the excitation through the Lambda layer, which effectively improves the gradient flow of the generator. The definition of ResSLE is as follows:

[0028] ResSLE=SLE(x)+λ·U(x) (4)

[0029] Among them, x represents the input feature, U() represents upsampling, and λ is used to control the excitation intensity. The improvement of the generator increases its own information content, aiming to alleviate the unequal confrontation between the generator and the discriminator, which helps to improve the generated performance indicators and training stability.

[0030] The encoder-decoder architecture is used to expand the discriminator to form U-Net, a self-supervised reconstruction discriminator USRD based on U-Net is proposed, and a unique local cropping method Shape-crop is designed for T-bolts. The feature extraction and detail texture generation capabilities of the discriminator are enhanced through global and local reconstruction, including:

[0031] In the training of GAN, the information of the generator and the discriminator is not equal, which often causes the discriminator to rely too much on non-target features to judge the truth, affecting the generation effect and falling into a vicious circle. To solve this problem, a self-supervised reconstruction discriminator USRD based on U-Net is proposed. The encoder-decoder architecture is adopted. USRD uses the feature extraction network of the discriminator as the encoder and reverses its structure to construct a decoder to form a U-Net structure, which includes an encoder D enc , global decoder D dec1 and local decoder D dec2 .

[0032] First, for the underlying architecture encoder D enc , using the traditional CNN network to build, D enc The upper layer architecture of the proposed method consists of a traditional adversarial head and a conditional classification head. Specifically, the underlying architecture of the discriminator acts as a feature extractor to extract the features of the input image and sends them to the adversarial head and the classification head respectively. Similar to general GANs, the adversarial head uses the sigmoid activation function to determine whether the input image is real or fake; the classification head uses the softmax activation function to classify the category of the input image and compare it with the input condition of the generator, hoping that the two belong to the same category.

[0033] Second, the global decoder D dec1 By encoder D enc The output bottleneck layer is used as input and jump-connected with the feature map of the encoder to ensure that the generated image is effectively aligned with the original data distribution on a global scale, which means that the generated image is similar in overall structure and layout. Through global reconstruction, the model can learn to capture the diversity and accuracy of data distribution from a global perspective. The local decoder D dec2 The architecture and global decoder D dec1 Similar, but different in that D dec2 The input is from the encoder D enc The final output features are clipped instead of being directly input from the bottleneck layer. It should be noted that the encoder D enc With local decoder D dec2The jump connections between them need to go through corresponding cropping operations. The local decoder mainly focuses on the local areas of the generated image to ensure that these areas have high-quality textures and details. Another important reason for using local reconstruction is that the colors of the bolts and their background hardware are similar, and it is sometimes difficult to distinguish them well under lighting conditions. The reconstruction of local pixels can highlight the shape characteristics of the bolts, which is conducive to the accurate generation of the bolt shape by the generator. The global and local reconstruction losses are defined as follows:

[0034]

[0035] Where I represents the input image, L global Use the L2 norm to optimize significant differences in global reconstruction to ensure that the overall image is close to the original image; local Using the L1 norm, we focus on the error of each pixel in local reconstruction, emphasizing the accuracy of details and textures.

[0036] Among them, it also includes: according to the shape characteristics of T-bolts, a shape cropping method Shape-crop is proposed, which performs cross-centered cropping on the bolt image, intercepts four squares from the middle of the horizontal and vertical directions, and increases the central area containing the most bolt information. The area of ​​the cropped area by the two methods is the same, but Shape-crop extracts significantly more bolt pixels than the random cropping method to improve D dec2 reconstruction quality.

[0037] Among them, it also includes: using the controllable generation method of bolt defects to assist in improving the accuracy of the bolt defect recognition network, and further verifying the effectiveness of the controllable generation method of bolt defects, specifically including:

[0038] In order to verify the effectiveness of generating a few-sample bolt defect images, experiments were conducted on the classification task. The experimental process is divided into three steps: first, a large number of bolt defect images are generated using the generation method; then, the dataset is augmented using these generated images; finally, the bolt defect classification model is trained using the augmented dataset, and the classification accuracy of the model is evaluated on the test set. The final results significantly improve the accuracy of the bolt defect classification model, proving the effectiveness of the generation method.

[0039] The present application also provides a device for controlling generation of bolt defects based on perspective and attribute guidance, the device comprising:

[0040] The S1 module is used to construct the dataset Dataset 1 for training the bolt perspective label generator, which contains the front and side perspectives of normal bolts; and to construct the few-sample bolt defect dataset Dataset 2, which contains three categories of bolt defect images, including normal bolts, missing pins, and missing nuts.

[0041] The S2 module is used to pre-train the CLIP model using contrastive language-image, propose a bolt perspective label generator based on CLIP fine-tuning, automatically assign perspective labels to the few-sample bolt defect dataset Dataset 2, and construct a bolt defect dataset Dataset 3 containing perspective and attribute labels to provide data support for subsequent image generation tasks.

[0042] The S3 module is used to adopt the architecture of conditional generative adversarial networks, introduce perspectives and attributes as additional conditional inputs in the generator, and propose residual cross-layer excitation ResSLE to build a generator with stronger gradient information flow and higher generation controllability; among them, ResSLE introduces cross-layer connections on the basis of cross-layer excitation SLE to strengthen the gradient signals between layers, and controls the intensity of excitation through the Lambda layer to improve the gradient flow of the generator.

[0043] The S4 module is used to expand the discriminator with an encoder-decoder architecture to form a U-Net. A self-supervised reconstruction discriminator USRD based on U-Net is proposed, and a unique local cropping method Shape-crop is designed for T-bolts. The feature extraction and detail texture generation capabilities of the discriminator are enhanced through global and local reconstruction.

[0044] The S2 module specifically includes:

[0045] First, in order to generate more accurate bolt images, it is proposed to introduce perspective information based on bolt defect attributes into GAN and divide the bolt perspective into two categories with the largest visual difference: front perspective 0° and side perspective 90°, to ensure the feasibility and accuracy of the task.

[0046] Next, a bolt perspective label generator based on CLIP fine-tuning is proposed and trained on Dataset 1 to learn the features of the two perspectives. The weighted space integration method WiSE-FT is used to fine-tune CLIP. The formula is as follows:

[0047] wse(x,α)=f(x,(1-α)·θ0+α·θ1) (1)

[0048] Among them, x represents the input data, f is the classification prediction function of the model, θ0 represents the original model parameters, θ1 represents the model parameters obtained by standard fine-tuning, α is the mixing coefficient that controls the weight, and the image encoder E of the fine-tuned CLIP is extracted as the bolt view label generator.

[0049] Finally, the bolt defect dataset Dataset 2 with attribute labels is input into the bolt perspective label generator to assign a perspective label to each image. By combining these perspective labels with the attribute labels of the bolts, the perspective-attribute bolt dataset Dataset 3 is constructed. This process is defined as follows:

[0050] Label(D3)=E(D2)+Attribute(D2) (2)

[0051] Among them, E represents the bolt perspective label generator, Attribute is the attribute label of Dataset 2, and the categories of Dataset 3 include front-normal bolt, side-normal bolt, front-pin missing, side-pin missing, front-nut missing, side-nut missing, a total of 6 categories. The images in Dataset 2 and Dataset 3 are the same, only the category labels are different. The bolt perspective label generator reduces human bias and improves label consistency.

[0052] The present application also provides an electronic device, which includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, and the program is used to implement the controllable generation method of bolt defects based on perspective and attribute guidance.

[0053] The present application also provides a storage medium, in which computer executable instructions are stored, and the computer executable instructions are used to execute the controllable generation method of bolt defects based on perspective and attribute guidance.

[0054] The present invention discloses the following technical effects: a method, device, electronic device and storage medium for controllable generation of bolt defects based on perspective and attribute guidance, specifically, a bolt perspective label generator based on contrast language-image pre-training is designed to automatically assign perspective labels to bolt data sets, thereby alleviating the visual difference problem of bolt images and providing data support for image generation tasks; a generative adversarial network is constructed, the generator of FastGAN is selected as the basic model of the generation part, perspective and attribute information are introduced as conditions in the generator to guide generation, and residual cross-layer excitation ResSLE is proposed to strengthen the gradient information flow and increase the controllability of generation; in the discriminant part, a U-Net self-supervised reconstruction discriminator USRD is proposed, and a unique local cropping method Shape-crop is designed for T-bolts, thereby enhancing the generation and feature extraction capabilities of bolt texture details. The bolt defect generation method can controllably generate bolt defect images with specific perspectives and attributes under the condition of a small number of samples, and assists in improving the accuracy of the bolt defect recognition network from the data level. The present invention applies generative adversarial networks to the generation of few-sample bolt defect images. By combining perspective and attributes as conditions to guide the generation of GAN, the quality and controllability of bolt defect image generation are effectively improved, meeting the data requirements of downstream tasks of few-sample bolt defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0056] Figure 1 It is a flow chart of a method for controllable generation of bolt defects based on perspective and attribute guidance according to an embodiment of the present invention;

[0057] Figure 2 is a schematic diagram of the structure of a bolt perspective label generator according to an embodiment of the present invention;

[0058] Figure 3 is a schematic diagram of the structure of residual cross-layer excitation according to an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of a shape cutting method according to an embodiment of the present invention;

[0060] Figure 5 It is a schematic diagram of the overall structure of an embodiment of the present invention;

[0061] Figure 6 The effect diagram of the bolt defect is generated for the embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The purpose of this invention is to provide a controllable generation method of bolt defects based on perspective and attribute guidance to solve the problems of insufficient data of transmission line bolt defects and poor downstream task effects caused by them. By introducing perspective and attribute information, and proposing residual cross-layer excitation and U-Net self-supervised reconstruction discriminator USRD, a novel generative adversarial network is designed to controllably generate bolt defect images with specific perspectives and attributes under few sample conditions, thereby achieving the purpose of improving bolt defect recognition.

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] like Figure 1 As shown, the present invention provides a method for controlling generation of bolt defects based on perspective and attribute guidance, comprising the following steps:

[0066] S1, constructs the dataset Dataset 1 for training the bolt perspective label generator, which only contains the front and side perspectives of normal bolts; constructs the few-sample bolt defect dataset Dataset 2, which contains three categories of bolt defect images, including normal bolts, missing pins, and missing nuts.

[0067] S2 adopts contrastive language-image pre-training and proposes a bolt perspective label generator based on CLIP fine-tuning. It automatically assigns perspective labels to the few-sample bolt defect dataset Dataset 2 and constructs a bolt defect dataset Dataset 3 containing perspective and attribute labels to provide data support for subsequent image generation tasks.

[0068] S3 adopts the architecture of conditional generative adversarial networks, introduces perspectives and attributes as additional conditional inputs into the generator, and proposes residual cross-layer excitation ResSLE to build a generator with stronger gradient information flow and higher generation controllability. Among them, ResSLE introduces cross-layer connections on the basis of cross-layer excitation SLE to strengthen the gradient signals between layers, and controls the intensity of excitation through the Lambda layer to improve the gradient flow of the generator.

[0069] S4, uses the encoder-decoder architecture to expand the discriminator to form U-Net, proposes a self-supervised reconstruction discriminator based on U-Net, and designs a unique local cropping method Shape-crop for T-bolts, which enhances the feature extraction and detail texture generation capabilities of the discriminator through global and local reconstruction;

[0070] S5, the controllable generation method of bolt defects is used to assist in improving the accuracy of the bolt defect recognition network, and further verify the effectiveness of the controllable generation method of bolt defects.

[0071] The basic flow chart of the present invention is as follows Figure 1 shown.

[0072] The deep learning model requires a large number of dataset image samples as support during training. Since the drone aerial inspection images are usually global images of the transmission line, they need to be cropped and preprocessed to unify the size of the dataset, and the defect categories of the bolts need to be further divided and labeled. In addition, different datasets need to be constructed for different tasks in the present invention. Therefore, in the step S1, a dataset Dataset 1 for training the bolt perspective label generator is constructed, which only contains the front and side perspectives of normal bolts; a small sample bolt defect dataset Dataset 2 is constructed, which contains bolt defect images of different categories, specifically including:

[0073] Collect the inspection images of transmission lines taken by drones, crop the areas containing bolts, clean them, and select the images with clear images and more types and numbers of bolt defects. Perform data preprocessing, unify the scale of bolt images, and annotate their defect types. Then, construct datasets for different tasks: 1) Dataset 1 contains 1000 normal bolt images from the front view (0°) and 1000 side view (90°) for training the bolt view label generator; 2) Dataset 2 contains 2260 images of three categories: normal bolts, missing pins, and missing nuts.

[0074] The structural schematic diagram of the bolt view label generator in the present invention is as follows: Figure 2 shown.

[0075] In the present invention, considering the significant differences in the visual morphology of bolts in transmission lines, it is necessary to introduce the perspective and attribute information of the bolts into the generative adversarial network. At the same time, since the human perspective division has subjective bias, it is necessary to adopt a model-based method to realize the automatic division of the bolt perspective. This method proposes a bolt perspective label generator based on CLIP fine-tuning to reduce human bias and improve label consistency. Among them, in the step S2, a contrastive language-image pre-training model is adopted, and a bolt perspective label generator based on CLIP fine-tuning is proposed to automatically assign perspective labels to the few-sample bolt defect dataset Dataset 2, and construct a bolt defect dataset Dataset 3 containing perspective and attribute labels to provide data support for subsequent image generation tasks, specifically including:

[0076] First, in transmission lines, due to the different shooting angles of drones and the positions of bolts, the visual forms of bolts are different and are often mistakenly classified as the same visual words, which affects the feature extraction performance, especially for GAN. Most existing methods rely on the classification of bolt defect attributes and ignore the perspective factor. In order to generate more accurate bolt images, it is proposed to introduce perspective information based on bolt defect attributes in GAN. Due to the lack of standardized inspection data sets, the present invention divides the bolt perspective into two categories with the largest visual differences: front perspective (0°) and side perspective (90°) to ensure the feasibility and accuracy of the task.

[0077] Next, since the manual division of the intermediate perspective has subjective bias, a bolt perspective label generator based on CLIP fine-tuning is proposed and trained on Dataset 1 to learn the characteristics of the two perspectives. The weighted space integration method WiSE-FT is used to fine-tune CLIP, and the formula is as follows:

[0078] wse(x,α)=f(x,(1-α)·θ0+α·θ1) (1)

[0079] Among them, x represents the input data, f is the classification prediction function of the model, θ0 represents the original model parameters, θ1 represents the model parameters obtained by standard fine-tuning, and α is the mixing coefficient that controls the weight. The image encoder E of the fine-tuned CLIP is extracted as the bolt view label generator.

[0080] Finally, the bolt defect dataset Dataset 2 with attribute labels is input into the bolt view label generator to assign a view label to each image. By combining these view labels with the attribute labels of the bolts, the view-attribute bolt dataset Dataset 3 is constructed. This process is defined as follows:

[0081] Label(D3)=E(D2)+Attribute(D2) (2)

[0082] Among them, E represents the bolt perspective label generator, and Attribute is the attribute label of Dataset 2. The categories of Dataset 3 include 6 categories: front-normal bolt, side-normal bolt, front-pin missing, side-pin missing, front-nut missing, side-nut missing. Note that the images in Dataset 2 and Dataset 3 are the same, only the category labels are different. The bolt perspective label generator reduces human bias and improves label consistency, which provides reliable data support for subsequent generation tasks.

[0083] The structural diagram of the residual cross-layer excitation in the present invention is as follows: Figure 3 shown.

[0084] In step S3, the conditional generative adversarial network architecture is adopted to introduce perspective and attribute as additional conditional inputs in the generator, and residual cross-layer excitation ResSLE is proposed to construct a generator with stronger gradient information flow and higher generation controllability, which specifically includes:

[0085] First, in order to solve the problem of bolt defect generation with a small number of samples, more useful information is input into the generator. On the basis of the input N-dimensional random noise vector, an n-dimensional conditional vector is added, and the two are concatenated and input into the generator. The input definition is as follows:

[0086] Z=concat(z,y) (3)

[0087] Among them, Z is the input vector of the generator, z is the noise vector, and y is the conditional vector. Using labels containing perspectives and attributes as conditional information to assist in generation increases the prior knowledge of the generator and improves the controllability of GAN generation. Another reason for introducing conditions is that different bolt defect images usually have certain commonalities. Although the properties or perspectives of the bolts are different, the main body of the bolts is similar, which is conducive to the learning of the generator.

[0088] In addition, the present invention adopts the generator of the few-sample generation model FastGAN as the basic model, and improves the architecture of the generator. FastGAN uses the lightweight design of the SLE model in the generator, which can alleviate the problem of gradient disappearance, but it still has some problems. SLE loses a certain amount of information in exchange for a reduction in the amount of calculation, which is necessary for the generation of 1024×1024 high-resolution images. However, it is not optimal for the task of bolt defect generation. Bolt images usually have a lower resolution, which requires less computing resources and has higher requirements for gradient information. Based on this discovery, residual cross-layer excitation is proposed. ResSLE introduces cross-layer connections on the basis of SLE to strengthen the gradient signals between layers, and controls the intensity of the excitation through the Lambda layer, which effectively improves the gradient flow of the generator. The definition of ResSLE is as follows:

[0089] ResSLE=SLE(x)+λ·U(x) (4)

[0090] Among them, x represents the input feature, U() represents upsampling, and λ is used to control the intensity of the excitation. The improvement of the generator increases its own information content, aiming to alleviate the unequal confrontation between the generator and the discriminator, and helps to improve the generated performance indicators and training stability.

[0091] The schematic diagram of the shape cutting method in the present invention is as follows Figure 4 shown.

[0092] In step S4, the discriminator is extended by using the encoder-decoder architecture to form U-Net, a self-supervised reconstruction discriminator USRD based on U-Net is proposed, and a unique local cropping method Shape-crop is designed for T-bolts. The feature extraction and detail texture generation capabilities of the discriminator are enhanced through global and local reconstruction, specifically including:

[0093] In the training of GAN, the information of the generator and the discriminator is not equal, which often causes the discriminator to rely too much on non-target features to judge the truth, affecting the generation effect and falling into a vicious circle. To solve this problem, the present invention proposes a self-supervised reconstruction discriminator USRD based on U-Net, which adopts an encoder-decoder architecture. USRD uses the feature extraction network of the discriminator as an encoder and reverses its structure to construct a decoder to form a U-Net structure, which includes an encoder D enc , global decoder D dec1 and local decoder D dec2 .

[0094] First, for the underlying architecture encoder D enc , constructed using the traditional CNN network. encThe upper layer architecture of the GAN consists of a traditional adversarial head and a conditional classification head. Specifically, the underlying architecture of the discriminator acts as a feature extractor to extract the features of the input image and send them to the adversarial head and the classification head respectively. Similar to general GANs, the adversarial head uses the sigmoid activation function to discriminate the authenticity of the input image; the classification head uses the softmax activation function to classify the category of the input image and compare it with the input condition of the generator, hoping that the two belong to the same category.

[0095] Second, the global decoder D dec1 By encoder D enc The output bottleneck layer is used as input and jump-connected to the feature maps of the encoder. This ensures that the generated image is effectively aligned with the original data distribution on a global scale, which means that the generated image is similar in overall structure and layout. Through global reconstruction, the model can learn to capture the diversity and accuracy of data distribution from a global perspective. Local decoder D dec2 The architecture and global decoder D dec1 Similar, but different in that D dec2 The input is from the encoder D enc The final output features are clipped instead of being directly input from the bottleneck layer. It should be noted that the encoder D enc With local decoder D dec2 The jump connections between them need to go through corresponding cropping operations. The local decoder focuses on the local areas of the generated image to ensure that these areas have high-quality textures and details. Another important reason for using local reconstruction is that the colors of the bolts and their background hardware are similar, and it is sometimes difficult to distinguish them well under lighting conditions. The reconstruction of local pixels can highlight the shape characteristics of the bolts, which is conducive to the generator's accurate generation of the bolt shape. The global and local reconstruction losses are defined as follows:

[0096]

[0097] Where I represents the input image. global Use the L2 norm to optimize significant differences in global reconstruction to ensure that the overall image is close to the original image; local Using the L1 norm, we focus on the error of each pixel in local reconstruction, emphasizing the accuracy of details and textures.

[0098] Finally, in the local reconstruction part of the discriminator, the traditional random cropping method often captures areas containing a large amount of useless background, which affects the extraction of local information of the bolt defect image, especially when the bolt size is small and the key information is concentrated in the center. To solve this problem, a shape cropping method Shape-crop is proposed based on the shape characteristics of T-bolts. It performs cross-centered cropping on the bolt image, captures four squares from the middle of the horizontal and vertical directions, and increases the central area containing the most bolt information. The two methods crop the same area, but Shape-crop extracts significantly more bolt pixels than random cropping, thereby improving D dec2 reconstruction quality.

[0099] USRD reconstructs real images in a self-supervised manner, and the encoder is forced to learn more useful information about the distribution of real image data to improve the overall performance of GAN. At the same time, during the training process of GAN, the decoder's reconstruction of real images can serve as an additional regularization term to help stabilize training.

[0100] In step S5, the controllable generation method of bolt defects is used to assist in improving the accuracy of the bolt defect recognition network, and further verify the effectiveness of the controllable generation method of bolt defects, which specifically includes:

[0101] In order to verify the effectiveness of the present invention in generating a few-sample bolt defect images, an experiment was conducted on the classification task. The experimental process is divided into three steps: first, a large number of bolt defect images are generated using the method proposed in this study; then, the dataset is augmented using these generated images; finally, the bolt defect classification model is trained using the augmented dataset, and the classification accuracy of the model is evaluated on the test set. The final result significantly improves the accuracy of the bolt defect classification model, proving the effectiveness of the method proposed in this invention.

[0102] The present invention discloses a controllable generation method of bolt defects based on perspective and attribute guidance. The network structure of the generation method is as follows: Figure 5 shown.

[0103] The bolt defect generation effect of the method of the present invention is as follows: Figure 6As shown. The present invention designs a bolt perspective label generator based on contrast language-image pre-training, automatically assigns perspective labels to bolt data sets, thereby alleviating the visual difference problem of bolt images and providing data support for image generation tasks; constructs a generative adversarial network, selects the generator of FastGAN as the basic model of the generation part, introduces perspective and attribute information as conditions in the generator to guide generation, and proposes residual cross-layer excitation ResSLE to strengthen the gradient information flow and increase the controllability of generation; proposes a U-Net self-supervised reconstruction discriminator USRD in the discriminant part, and designs a unique local cropping method Shape-crop for T-bolts, thereby enhancing the generation and feature extraction capabilities of bolt texture details. The bolt defect generation method can controllably generate bolt defect images with specific perspectives and attributes under the condition of a small number of samples, and assists in improving the accuracy of the bolt defect recognition network from the data level. The present invention applies the generative adversarial network to the generation of bolt defect images with a small number of samples. By combining perspectives and attributes as conditions to guide the generation of GAN, the quality and controllability of bolt defect image generation are effectively improved, and the data requirements of the downstream tasks of bolt defects with a small number of samples are met.

[0104] Based on the method for controlling generation of bolt defects based on view angle and attribute guidance provided in the above embodiment, an embodiment of the present invention correspondingly provides a device for controlling generation of bolt defects based on view angle and attribute guidance, the device comprising:

[0105] The S1 module is used to construct the dataset Dataset 1 for training the bolt perspective label generator, which contains the front and side perspectives of normal bolts; and to construct the few-sample bolt defect dataset Dataset 2, which contains three categories of bolt defect images, including normal bolts, missing pins, and missing nuts.

[0106] The S2 module is used to pre-train the CLIP model using contrastive language-image, propose a bolt perspective label generator based on CLIP fine-tuning, automatically assign perspective labels to the few-sample bolt defect dataset Dataset 2, and construct a bolt defect dataset Dataset 3 containing perspective and attribute labels to provide data support for subsequent image generation tasks.

[0107] The S3 module is used to adopt the architecture of conditional generative adversarial networks, introduce perspectives and attributes as additional conditional inputs in the generator, and propose residual cross-layer excitation ResSLE to build a generator with stronger gradient information flow and higher generation controllability; among them, ResSLE introduces cross-layer connections on the basis of cross-layer excitation SLE to strengthen the gradient signals between layers, and controls the intensity of excitation through the Lambda layer to improve the gradient flow of the generator.

[0108] The S4 module is used to expand the discriminator with an encoder-decoder architecture to form a U-Net. A self-supervised reconstruction discriminator USRD based on U-Net is proposed, and a unique local cropping method Shape-crop is designed for T-bolts. The feature extraction and detail texture generation capabilities of the discriminator are enhanced through global and local reconstruction.

[0109] The S2 module specifically includes:

[0110] Firstly, in order to generate more accurate bolt images, it is proposed to introduce perspective information based on bolt defect attributes into GAN and divide the bolt perspective into two categories with the largest visual difference: front perspective 0° and side perspective 90°, to ensure the feasibility and accuracy of the task.

[0111] Next, a bolt perspective label generator based on CLIP fine-tuning is proposed and trained on Dataset 1 to learn the features of the two perspectives. The weighted space integration method WiSE-FT is used to fine-tune CLIP. The formula is as follows:

[0112] wse(x,α)=f(x,(1-α)·θ0+α·θ1) (1)

[0113] Among them, x represents the input data, f is the classification prediction function of the model, θ0 represents the original model parameters, θ1 represents the model parameters obtained by standard fine-tuning, α is the mixing coefficient that controls the weight, and the image encoder E of the fine-tuned CLIP is extracted as the bolt view label generator.

[0114] Finally, the bolt defect dataset Dataset 2 with attribute labels is input into the bolt perspective label generator to assign a perspective label to each image. By combining these perspective labels with the attribute labels of the bolts, the perspective-attribute bolt dataset Dataset 3 is constructed. This process is defined as follows:

[0115] Label(D3)=E(D2)+Attribute(D2) (2)

[0116] Among them, E represents the bolt perspective label generator, Attribute is the attribute label of Dataset 2, and the categories of Dataset 3 include front-normal bolt, side-normal bolt, front-pin missing, side-pin missing, front-nut missing, side-nut missing, a total of 6 categories. The images in Dataset 2 and Dataset 3 are the same, only the category labels are different. The bolt perspective label generator reduces human bias and improves label consistency.

[0117] Based on the controllable generation method of bolt defects based on perspective and attribute guidance provided in the above embodiment, an embodiment of the present invention correspondingly provides an electronic device, which includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, and the program is used to implement the above-mentioned controllable generation method of bolt defects based on perspective and attribute guidance.

[0118] Based on the controllable generation method of bolt defects based on perspective and attribute guidance provided in the above embodiment, an embodiment of the present invention correspondingly provides a storage medium, in which computer executable instructions are stored, and the computer executable instructions are used to execute the above-mentioned controllable generation method of bolt defects based on perspective and attribute guidance.

[0119] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

[0120] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0121] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device that includes a series of elements is inherent to the elements, or also includes elements inherent to these processes, methods, articles or devices. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0122] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A controllable generation method of bolt defects based on perspective and attribute guidance, characterized in that: The following steps are involved: S1, construct the dataset Dataset 1 for training the bolt view label generator, which contains the front and side views of normal bolts; construct the few-sample bolt defect dataset Dataset 2, which contains three categories of bolt defect images, including normal bolts, missing pins, and missing nuts; S2, using contrastive language-image pre-training CLIP model, proposed a bolt perspective label generator based on CLIP fine-tuning, automatically assigned perspective labels to the few-sample bolt defect dataset Dataset 2, and constructed a bolt defect dataset Dataset 3 containing perspective and attribute labels, providing data support for subsequent image generation tasks; S3, adopts the architecture of conditional generative adversarial network, introduces perspective and attributes as additional conditional inputs in the generator, and proposes residual cross-layer excitation ResSLE to build a generator with stronger gradient information flow and higher generation controllability. Among them, ResSLE introduces cross-layer connections on the basis of cross-layer excitation SLE to strengthen the gradient signal between layers, and controls the intensity of excitation through the Lambda layer to improve the gradient flow of the generator; S4, uses the encoder-decoder architecture to expand the discriminator to form U-Net, proposes a self-supervised reconstruction discriminator USRD based on U-Net, and designs a unique local cropping method Shape-crop for T-bolts, which enhances the discriminator's feature extraction and detail texture generation capabilities through global and local reconstruction; Among them, S2 specifically includes: Firstly, in order to generate more accurate bolt images, it is proposed to introduce perspective information based on bolt defect attributes into GAN and divide the bolt perspective into two categories with the largest visual difference: front perspective 0° and side perspective 90°, to ensure the feasibility and accuracy of the task. Next, a bolt perspective label generator based on CLIP fine-tuning is proposed and trained on Dataset 1 to learn the features of the two perspectives. The weighted space integration method WiSE-FT is used to fine-tune CLIP. The formula is as follows: wse(x,α)=f(x,(1-α)·θ0+α·θ1) (1) Among them, x represents the input data, f is the classification prediction function of the model, θ0 represents the original model parameters, θ1 represents the model parameters obtained by standard fine-tuning, α is the mixing coefficient of the control weight, and the image encoder E of the fine-tuned CLIP is extracted as the bolt view label generator; Finally, the bolt defect dataset Dataset 2 with attribute labels is input into the bolt perspective label generator to assign a perspective label to each image. By combining these perspective labels with the attribute labels of the bolts, the perspective-attribute bolt dataset Dataset 3 is constructed. This process is defined as follows: Label(D3)=E(D2)+Attribute(D2) (2) Among them, E represents the bolt perspective label generator, Attribute is the attribute label of Dataset 2, and the categories of Dataset 3 include front-normal bolt, side-normal bolt, front-pin missing, side-pin missing, front-nut missing, side-nut missing, a total of 6 categories. The images in Dataset 2 and Dataset 3 are the same, only the category labels are different. The bolt perspective label generator reduces human bias and improves label consistency.

2. The generation method according to claim 1, characterized in that: The conditional generative adversarial network architecture is adopted to introduce perspective and attributes as additional conditional inputs in the generator, and residual cross-layer excitation ResSLE is proposed to construct a generator with stronger gradient information flow and higher generation controllability, including: First, an n-dimensional conditional vector is added to the input N-dimensional random noise vector. The two are concatenated and input into the generator. The input definition is as follows: Z=concat(z,y) (3) Among them, Z is the input vector of the generator, z is the noise vector, and y is the conditional vector. The labels containing the perspective and attributes are used as conditional information to assist in the generation, which increases the prior knowledge of the generator and improves the controllability of GAN generation. Another reason for introducing conditions is that different bolt defect images usually have certain commonalities. Although the properties or perspectives of the bolts are different, the main body of the bolts is similar, which is conducive to the learning of the generator; In addition, the generator of the few-shot generation model FastGAN is used as the basic model, and the architecture of the generator is improved. Residual cross-layer excitation ResSLE is proposed. ResSLE introduces cross-layer connections on the basis of SLE to strengthen the gradient signal between layers, and controls the intensity of the excitation through the Lambda layer, which effectively improves the gradient flow of the generator. The definition of ResSLE is as follows: ResSLE=SLE(x)+λ·U(x) (4) Among them, x represents the input feature, U() represents upsampling, and λ is used to control the excitation intensity. The improvement of the generator increases its own information content, aiming to alleviate the unequal confrontation between the generator and the discriminator, which helps to improve the generated performance indicators and training stability.

3. The generation method according to claim 1, characterized in that: The encoder-decoder architecture is used to expand the discriminator to form U-Net, and a self-supervised reconstruction discriminator USRD based on U-Net is proposed. A unique local cropping method Shape-crop is designed for T-bolts. The feature extraction and detail texture generation capabilities of the discriminator are enhanced through global and local reconstruction, including: In the training of GAN, the information of the generator and the discriminator is not equal, which often causes the discriminator to rely too much on non-target features to judge the truth, affecting the generation effect and falling into a vicious circle. To solve this problem, a self-supervised reconstruction discriminator USRD based on U-Net is proposed. The encoder-decoder architecture is adopted. USRD uses the feature extraction network of the discriminator as the encoder and reverses its structure to construct a decoder to form a U-Net structure, which includes an encoder D enc , global decoder D dec1 and local decoder D dec2 ; First, for the underlying architecture encoder D enc , using the traditional CNN network to build, D enc The upper layer architecture of the proposed method consists of a traditional adversarial head and a conditional classification head. Specifically, the underlying architecture of the discriminator acts as a feature extractor to extract the features of the input image and sends them to the adversarial head and the classification head respectively. Similar to general GAN, the adversarial head uses the sigmoid activation function to discriminate the authenticity of the input image; the classification head uses the softmax activation function to classify the category of the input image and compare it with the input condition of the generator, hoping that the two belong to the same category. Second, the global decoder D dec1 By encoder D enc The output bottleneck layer is used as input and jump-connected with the feature map of the encoder to ensure that the generated image is effectively aligned with the original data distribution on a global scale, which means that the generated image is similar in overall structure and layout. Through global reconstruction, the model can learn to capture the diversity and accuracy of data distribution from a global perspective. The local decoder D dec2 The architecture and global decoder D dec1 Similar, but different in that D dec2 The input is from the encoder D enc The final output features are clipped instead of being directly input from the bottleneck layer. It should be noted that the encoder D enc With local decoder D dec2 The jump connections between them need to go through corresponding cropping operations. The local decoder mainly focuses on the local areas of the generated image to ensure that these areas have high-quality textures and details. Another important reason for using local reconstruction is that the colors of the bolts and their background hardware are similar, and it is sometimes difficult to distinguish them well under lighting conditions. The reconstruction of local pixels can highlight the shape characteristics of the bolts, which is conducive to the accurate generation of the bolt shape by the generator. The global and local reconstruction losses are defined as follows: Where I represents the input image, L global Use the L2 norm to optimize significant differences in global reconstruction to ensure that the overall image is close to the original image; local Using the L1 norm, we focus on the error of each pixel in local reconstruction, emphasizing the accuracy of details and textures.

4. The generation method according to claim 3, characterized in that: Also includes: According to the shape characteristics of T-bolts, a shape cropping method Shape-crop is proposed. It performs cross-centered cropping on the bolt image, intercepts four squares from the middle of the horizontal and vertical directions, and increases the central area containing the most bolt information. The two methods crop the same area, but Shape-crop extracts significantly more bolt pixels than the random cropping method to improve D dec2 reconstruction quality.

5. The generation method according to claim 1, characterized in that: Also includes: The controllable generation method of bolt defects is used to assist in improving the accuracy of the bolt defect recognition network and further verify the effectiveness of the controllable generation method of bolt defects, including: In order to verify the effectiveness of generating a few-sample bolt defect images, experiments were conducted on the classification task. The experimental process is divided into three steps: first, a large number of bolt defect images are generated using the generation method; then, the dataset is augmented using these generated images; finally, the bolt defect classification model is trained using the augmented dataset, and the classification accuracy of the model is evaluated on the test set. The final results significantly improve the accuracy of the bolt defect classification model, proving the effectiveness of the generation method.

6. A controllable generation device for bolt defects based on perspective and attribute guidance, characterized in that: The device comprises: The S1 module is used to construct the dataset Dataset 1 for training the bolt view label generator, which contains the front and side views of normal bolts; construct the few-sample bolt defect dataset Dataset 2, which contains three categories of bolt defect images, including normal bolts, missing pins, and missing nuts; The S2 module is used to pre-train the CLIP model using contrastive language-image, propose a bolt view label generator based on CLIP fine-tuning, automatically assign view labels to the few-sample bolt defect dataset Dataset 2, and construct a bolt defect dataset Dataset 3 containing view and attribute labels to provide data support for subsequent image generation tasks; The S3 module is used to adopt the architecture of conditional generative adversarial networks, introduce perspectives and attributes as additional conditional inputs in the generator, and propose residual cross-layer excitation ResSLE to build a generator with stronger gradient information flow and higher generation controllability. Among them, ResSLE introduces cross-layer connections on the basis of cross-layer excitation SLE to strengthen the gradient signal between layers, and controls the intensity of excitation through the Lambda layer to improve the gradient flow of the generator; The S4 module is used to expand the discriminator using an encoder-decoder architecture to form a U-Net. A self-supervised reconstruction discriminator USRD based on U-Net is proposed, and a unique local cropping method Shape-crop is designed for T-bolts. The feature extraction and detail texture generation capabilities of the discriminator are enhanced through global and local reconstruction. The S2 module specifically includes: Firstly, in order to generate more accurate bolt images, it is proposed to introduce perspective information based on bolt defect attributes into GAN and divide the bolt perspective into two categories with the largest visual difference: front perspective 0° and side perspective 90°, to ensure the feasibility and accuracy of the task. Next, a bolt perspective label generator based on CLIP fine-tuning is proposed and trained on Dataset 1 to learn the features of the two perspectives. The weighted space integration method WiSE-FT is used to fine-tune CLIP. The formula is as follows: wse(x,α)=f(x,(1-α)·θ0+α·θ1) (1) Among them, x represents the input data, f is the classification prediction function of the model, θ0 represents the original model parameters, θ1 represents the model parameters obtained by standard fine-tuning, α is the mixing coefficient of the control weight, and the image encoder E of the fine-tuned CLIP is extracted as the bolt view label generator; Finally, the bolt defect dataset Dataset 2 with attribute labels is input into the bolt perspective label generator to assign a perspective label to each image. By combining these perspective labels with the attribute labels of the bolts, the perspective-attribute bolt dataset Dataset 3 is constructed. This process is defined as follows: Label(D3)=E(D2)+Attribute(D2) (2) Among them, E represents the bolt perspective label generator, Attribute is the attribute label of Dataset 2, and the categories of Dataset 3 include front-normal bolt, side-normal bolt, front-pin missing, side-pin missing, front-nut missing, side-nut missing, a total of 6 categories. The images in Dataset 2 and Dataset 3 are the same, only the category labels are different. The bolt perspective label generator reduces human bias and improves label consistency.

7. An electronic device, characterized in that: The electronic device includes: at least one memory and at least one processor; the memory stores a program, and the processor calls the program stored in the memory, and the program is used to implement the controllable generation method of bolt defects based on perspective and attribute guidance as described in any one of claims 1-5.

8. A storage medium, characterized in that: The storage medium stores computer executable instructions, and the computer executable instructions are used to execute the controllable generation method of bolt defects based on perspective and attribute guidance as described in any one of claims 1-5.

Images