Method and device for generating images of surface defects of a steel wire rope

Abstract

The present application relates to a kind of steel wire rope surface defect image generation method and device.It includes: the reference image without defect is loaded to basic feature extraction network, to obtain the hidden vector without defect reference by basic feature extraction network, and the hidden vector without defect reference extracted is loaded to fusion decoding network;Defect feature text and defect reference image are loaded to defect feature generation network, to generate text guided defect hidden vector by defect feature generation network, and the text guided defect hidden vector generated is loaded to fusion decoding network;The hidden vector without defect reference and text guided defect hidden vector are fused and decoded using fusion decoding network, to generate steel wire rope target defect image after fusion decoding processing.This application can realize the generation of high-quality steel wire rope surface defect image, improve the efficiency and reliability of steel wire rope surface defect image generation.

Description

Technical Field

[0001] This invention relates to an image generation method and apparatus, and more particularly to a method and apparatus for generating images of surface defects in steel wire ropes. Background Technology

[0002] Steel wire ropes are widely used in various industries such as construction, mining, transportation, and petroleum due to their excellent strength, toughness, and adaptability. They not only perform well in terms of load-bearing capacity but can also work for long periods of time in harsh environments, making them a core component of many heavy equipment and machinery.

[0003] Although steel wire ropes possess excellent performance, they can still experience various types of damage over time. Common types of damage include broken wires, wear, deformation, and corrosion. Understandably, if problems in the steel wire rope are not detected in time, the damage will increase with working hours. If no action is taken after the damage reaches a certain level, it can lead to safety accidents.

[0004] Traditional methods for detecting defects in wire ropes include magnetic induction testing, ultrasonic testing, and eddy current testing. Magnetic induction testing detects broken wires and cracks by changing magnetic flux, making it suitable for detecting internal defects in wire ropes, but the equipment is expensive. Ultrasonic testing uses the reflection and attenuation characteristics of waves to detect changes in the internal structure of wire ropes, offering high accuracy, but signal processing is complex. Eddy current testing quickly detects surface and shallow defects by changing conductivity, making it suitable for online detection, but it is not sensitive to deep defects in wire ropes.

[0005] With the development of machine learning, researchers have proposed visual inspection methods for surface defects in wire ropes. Visual inspection methods include manual visual inspection and machine vision inspection. Manual visual inspection relies on the experience of the inspectors, is low-cost, but inefficient and highly subjective. Machine vision inspection, using industrial cameras and algorithms (such as YOLO and Mask R-CNN), can automatically detect defects such as broken wires and wear in wire ropes, offering high precision and speed, but requiring a higher initial equipment investment. Therefore, overall, machine vision has advantages in automation and accuracy, and is the mainstream method for modern wire rope appearance inspection.

[0006] However, in real-world scenarios, researchers face the challenge of data scarcity due to the scarcity and difficulty in labeling wire rope defect data. To meet the needs of machine vision inspection, data augmentation can be applied to wire rope defect data. Traditional data augmentation methods mainly include geometric transformations (such as rotation, flipping, scaling, cropping, and translation), color transformations (such as brightness, contrast, and saturation adjustments), and adding noise. These methods expand the dataset through simple image transformations, improving the robustness of the detection model to common deformations. However, their limitations include: the generated samples lack diversity, cannot simulate complex defect features in real-world scenarios (such as random deformations and combinations of multiple defects), and have limited coverage of data distribution, which may result in limited improvement in actual defect detection performance.

[0007] To address this issue, generating high-quality images of wire rope defects has become an important research task. By generating realistic defect images, the training effect of the detection model can be significantly improved, and the accuracy and robustness of defect detection can be enhanced.

[0008] Currently, the methods used to generate defect images are mainly divided into traditional defect image generation methods and deep learning-based generation methods, each with its own characteristics and limitations. In traditional machine learning, the generative model is a concept distinct from the discriminative model. The generative model aims to learn the essential characteristics of the data distribution and generate new data that conforms to that distribution.

[0009] In the research of generative models, Wang Xinyan proposed a diffusion model-based defect sample simulation generation solution to address the problem that the difficulty in obtaining defect samples leads to ineffective training of deep learning models. He improved the diffusion model by incorporating an attention mechanism and DDIM accelerated sampling technology, achieving rapid image generation. Peng Yi, addressing the issues of limited X-ray image samples and class imbalance, introduced the diffusion model into the field of weld defect image enhancement. He proposed an improved diffusion model algorithm to expand the sample size and enhance the detection capability of rare classes of defects. To improve the skip-layer connections in the noise estimation network of the IDDPM model, he proposed the IDDPM_CCT model.

[0010] Defect image generation methods based on deep learning and generative models have shown great potential in improving data diversity and quality; however, due to the special characteristics of steel wire ropes, how to effectively generate high-quality steel wire rope defect images is a technical problem that urgently needs to be solved. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method and apparatus for generating surface defect images of steel wire ropes, which can generate high-quality surface defect images of steel wire ropes and improve the efficiency and reliability of steel wire rope surface defect image generation.

[0012] According to the technical solution provided by the present invention, a method for generating images of surface defects in steel wire ropes is provided, the method comprising:

[0013] The system provides conditions for generating defect images and loads these conditions into the constructed defect image generation model.

[0014] The defect image generation model includes a basic feature extraction network, a defect feature generation network, and a fusion decoding network.

[0015] The defect image generation conditions include a flawless reference image, defect feature text, and a defect reference image, wherein the type of wire rope in the flawless reference image is consistent with the type of wire rope in the defect reference image.

[0016] The flawless reference image is loaded into the basic feature extraction network to extract the flawless reference latent vector, and the extracted flawless reference latent vector is loaded into the fusion decoding network.

[0017] The defect feature text and defect reference image are loaded into the defect feature generation network to generate a text-guided defect latent vector, and the generated text-guided defect latent vector is loaded into the fusion decoding network.

[0018] A fusion decoding network is used to fuse and decode the flawless reference latent vector and the text-guided defect latent vector to generate a target defect image of the wire rope. The defect state of the target defect image of the wire rope is consistent with the defect state described by the defect feature text.

[0019] The basic feature extraction network is constructed based at least on the Stable Diffusion feature extraction model and the IP-Adapter module, wherein,

[0020] For the loaded intact reference image, the variational autoencoder in the Stable Diffusion feature extraction model is used to perform variational autoencoding on the intact reference image to generate an intact coded latent vector.

[0021] The IP-Adapter module uses a decoupled cross-attention mechanism to convert the missing encoded latent vector into a missing reference cross-attention feature, and then injects the missing reference cross-attention feature into the U-Net module of the Stable Diffusion feature extraction model. After denoising by the U-Net module, a missing reference latent vector is generated.

[0022] The defect feature generation network is constructed based at least on the Stable Diffusion defect generation model and the ControlNet module, wherein,

[0023] The defect reference image is loaded into the Controlnet module to generate defect constraints, and the defect constraints are then loaded into the defect generation Stable Diffusion model.

[0024] The defect reference image and defect feature text are simultaneously loaded into the defect generation Stable Diffusion model, so that the defect generation Stable Diffusion model can generate text-guided defect latent vectors under defect constraints.

[0025] The fusion decoding network includes a latent vector fusion unit and a latent vector decoder, wherein...

[0026] The implicit vector fusion unit is used to perform implicit vector fusion processing on the unblemished reference implicit vector and the text-guided defect implicit vector, so as to generate the wire rope fused implicit vector after implicit vector fusion processing.

[0027] The latent vector decoder decodes the fused latent vectors of the wire rope to generate a target image of the wire rope.

[0028] When performing latent vector fusion processing, the following are included:

[0029] Align the intact reference latent vector and the text-guided defect latent vector in the latent space to generate an intact aligned latent vector and a text-guided defect aligned latent vector after alignment in the latent space.

[0030] Frequency domain features are extracted from the above-mentioned unblemished alignment latent vector and text-guided defect alignment latent vector to obtain unblemished alignment frequency domain features and text-guided defect alignment frequency domain features, respectively. The unblemished alignment frequency domain features include unblemished alignment low-frequency features and unblemished alignment high-frequency features, and the text-guided defect alignment frequency domain features include defect alignment low-frequency features and defect alignment high-frequency features.

[0031] The unblemished alignment low-frequency features and the defective alignment low-frequency features are fused at low frequencies to generate aligned fused low-frequency features.

[0032] The high-frequency features of flawless alignment and the high-frequency features of defective alignment are fused at high frequency to generate alignment fused high-frequency features.

[0033] Alignment fusion low-frequency features and alignment fusion high-frequency features are combined to generate alignment fusion frequency domain features. The generated alignment fusion frequency domain features are then transformed in the time domain to generate wire rope fusion latent vectors.

[0034] When aligning the flawless reference latent vector with the text-guided defect latent vector in the latent space, we have:

[0035]

[0036] Among them, h IP h is the unambiguous aligned latent vector. CN To align latent vectors for text-guided defects, W IP The mapping matrix W for the incomplete reference latent vectors CN b is the mapping matrix for the latent vectors of text-guided defects. IP The bias term for the incomplete reference latent vector, b CN z is the bias term of the latent vector of text-guided defects. IP For the incomplete reference latent vector, z CN This is a hidden vector for text-guided defects.

[0037] When generating aligned combined frequency domain features, we have:

[0038]

[0039] Among them, Z fuse To align the combined frequency domain features, To align and fuse low-frequency features, To align and fuse high-frequency features, α is the background fusion modulation coefficient, and β is the defect enhancement coefficient. For seamless alignment of low-frequency characteristics, For flawless alignment of high-frequency features, Align low-frequency features with defects. High-frequency features are aligned to defects.

[0040] When providing a defect reference image, the method of providing it includes:

[0041] Provide a reference image of the defect.

[0042] The defect reference source image is converted to HSV color space and Lab color space respectively. Then, binarization operation is performed in HSV color space and Lab color space respectively to generate HSV color space binarized image and Lab color space binarized image respectively after the binarization operation.

[0043] Perform a union operation on the HSV color space binarized image and the Lab color space binarized image to form a spatially fused binarized image;

[0044] Based on the HSV color space of the defective reference source image and the spatially fused binarized image, the hue variation state in the HSV color space is calculated and determined.

[0045] Based on the Lab color space of the defect reference source image and the spatial fusion binarized image, the brightness change state in the Lab color space is calculated and determined.

[0046] Based on the calculated and determined hue and brightness change states, among which,

[0047] When the hue change is significant, hue separation is performed in the HSV color space of the defect reference source image, and a defect reference image is generated after separation.

[0048] When the brightness change is significant, the brightness is enhanced in the Lab color space of the defect reference source image, and the defect reference image is generated after the brightness enhancement.

[0049] When constructing a defect image generation model, the construction methods include:

[0050] A defect image generation basic model is constructed, which includes a basic feature extraction network after preliminary training, a defect feature generation network after preliminary training, and an untrained fusion decoding network.

[0051] Construct a basic model training dataset for training the base model of defect image generation, wherein...

[0052] The basic model training dataset includes a defect-free training sample set and a defective training sample set.

[0053] The defect-free training sample set includes several defect-free training samples, and each defect-free training sample includes a defect-free Class I wire rope training image.

[0054] The defect training sample set includes several defect training samples. Each defect training sample includes a second type of wire rope training image with defects and training image text used to characterize the defect state of the second type of wire rope training image.

[0055] The basic model for generating defect images is trained based on the configured basic model training conditions.

[0056] During model training, a defect-free training sample is loaded into the basic feature extraction network, and a defective training sample is loaded into the defect feature generation network. Subsequently, the wire rope training defect image is generated by the fusion decoding network.

[0057] The loss function of the training model is calculated based on the steel wire rope training defect image and the second type of steel wire rope training image in the current defect training sample. The network parameters of the defect image generation base model are adjusted based on the calculated loss function until the defect image generation base model is trained to the target state.

[0058] Based on the training of the model to reach the target state, the defect image generation model is generated.

[0059] An apparatus for generating images of surface defects in steel wire ropes includes a defect image generation device and a defect image generation model deployed within the defect image generation device, wherein...

[0060] The conditions for generating defect images of steel wire ropes are obtained. The defect image generation device performs defect image generation processing using the method described above, and generates a target defect image of the steel wire rope after the defect image generation processing.

[0061] The advantages of this invention are as follows: A flawless reference image is loaded into a basic feature extraction network to extract a flawless reference latent vector; defect feature text and defect reference image are loaded into a defect feature generation network to generate a text-guided defect latent vector; and then, a fusion decoding network is used to fuse and decode the flawless reference latent vector and the text-guided defect latent vector to generate a steel wire rope target defect image.

[0062] A basic feature extraction network is constructed based on the Stable Diffusion model for feature extraction and the IP-Adapter module. At the same time, when a defect feature generation network is constructed based on the Stable Diffusion model for defect generation and the Controlnet module, the conflict arising from combining the above with a single Stable Diffusion model can be effectively resolved. However, this approach can better achieve the generation of surface defect images of steel wire ropes, thereby improving the generation quality, efficiency, and reliability of target defect images of steel wire ropes. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of an embodiment of the generation of surface defect images of steel wire ropes according to the present invention.

[0064] Figure 2 This is a structural block diagram of an embodiment of the defect image generation model of the present invention.

[0065] Figure 3 This is a schematic diagram of one embodiment of the present invention, showing a complete reference image.

[0066] Figure 4 This is a schematic diagram of a first embodiment of the defect reference image of the present invention.

[0067] Figure 5 This is a schematic diagram of a first embodiment of the present invention for generating target defect images of steel wire ropes. Detailed Implementation

[0068] The present invention will be further described below with reference to specific accompanying drawings and embodiments.

[0069] To achieve the generation of high-quality images of surface defects in steel wire ropes and improve the efficiency and reliability of such generation, this invention provides a method for generating images of surface defects in steel wire ropes. Specifically, the generation method includes:

[0070] The system provides conditions for generating defect images and loads these conditions into the constructed defect image generation model.

[0071] The defect image generation model includes a basic feature extraction network, a defect feature generation network, and a fusion decoding network.

[0072] The defect image generation conditions include a flawless reference image, defect feature text, and a defect reference image, wherein the type of wire rope in the flawless reference image is consistent with the type of wire rope in the defect reference image.

[0073] The flawless reference image is loaded into the basic feature extraction network to extract the flawless reference latent vector, and the extracted flawless reference latent vector is loaded into the fusion decoding network.

[0074] The defect feature text and defect reference image are loaded into the defect feature generation network to generate a text-guided defect latent vector, and the generated text-guided defect latent vector is loaded into the fusion decoding network.

[0075] A fusion decoding network is used to fuse and decode the flawless reference latent vector and the text-guided defect latent vector to generate a target defect image of the wire rope. The defect state of the target defect image of the wire rope is consistent with the defect state described by the defect feature text.

[0076] As explained above, a wire rope surface defect image specifically refers to an image of a wire rope containing defects, and these defects should be distributed across the surface of the wire rope within the image. The types of wire rope defects generally include broken wires, wear, corrosion, and / or deformation. To improve the quality of generated wire rope surface defect images, Figure 1 This paper illustrates an embodiment of the present invention for generating surface defect images of wire ropes. As shown in the figure, when generating surface defect images of wire ropes, defect image generation conditions should be provided and loaded into the defect image generation model. The defect image generation conditions and defect image generation model are explained in detail below.

[0077] In practice, the conditions for generating the defect image include a flawless reference image, defect feature text, and the defect reference image itself. Specifically, the flawless reference image refers to an image in which the surface of the wire rope is free of defects. Figure 3 As shown; the defect reference image specifically refers to an image of the surface of a wire rope with defects, such as... Figure 4 , Figure 5 This is one embodiment of generating a target defect image for a steel wire rope. Generally, the corresponding image sizes of the defect-free reference image and the defect reference image are the same.

[0078] It should be noted that the defects present in the defect reference image generally include one or more of the aforementioned broken wires, wear, corrosion, and deformation. Generally, the type of wire rope in the defect-free reference image is consistent with the type of wire rope in the defect reference image; that is, the same type of wire rope should be used as the generation condition. The defect feature text is mainly used to guide the generation of the distribution state of defects on the surface defect image. The defect feature text should include the defect type, the number of defects corresponding to the defect type, and / or the distribution location of the defects on the wire rope surface defect image, etc.

[0079] Figure 2 The figure illustrates an embodiment of the defect image generation model of the present invention. As shown in the figure, the defect image generation model should include at least a basic feature extraction network, a defect feature generation network, and a fusion decoding network. The basic feature extraction network and the defect feature generation network should both be connected to the fusion decoding network. It should be noted that when loading the defect image generation conditions into the defect image generation model, specifically, the flawless reference image is loaded into the basic feature extraction network, and the defect feature text and the defect reference image are simultaneously loaded into the defect feature generation network. At this time, the fusion decoding network serves as the output layer of the defect image generation model. Therefore, the fusion decoding network can generate a surface defect image of the wire rope. Specifically, the surface defect image of the wire rope generated by the fusion decoding network is the target defect image of the wire rope.

[0080] It should be understood that the defect state of the target defect image of the wire rope is consistent with the defect state described in the defect feature text. The defect state described in the defect feature text can be referred to the above explanation of the corresponding defect feature text. Therefore, the defect state contained in the target defect image of the wire rope can generally include the type of defect, the number of corresponding defect types, and the distribution, etc.

[0081] After loading the flawless reference image into the basic feature extraction network, the flawless reference latent vector can be extracted by the basic feature extraction network. After loading the defect feature text and the defect reference image into the defect feature generation network, the defect feature generation network can generate the text-guided defect latent vector. Subsequently, the fusion decoding network performs fusion decoding processing on the flawless reference latent vector and the text-guided defect latent vector, and finally generates the target defect image of the wire rope. The specific method and process of generating the target defect image of the wire rope will be explained in detail below.

[0082] In one embodiment of the present invention, the basic feature extraction network is constructed based at least on the StableDiffusion feature extraction model and the IP-Adapter module, wherein,

[0083] For the loaded intact reference image, the variational autoencoder in the Stable Diffusion feature extraction model is used to perform variational autoencoding on the intact reference image to generate an intact coded latent vector.

[0084] The IP-Adapter module uses a decoupled cross-attention mechanism to convert the missing encoded latent vector into a missing reference cross-attention feature, and then injects the missing reference cross-attention feature into the U-Net module of the Stable Diffusion feature extraction model. After denoising by the U-Net module, a missing reference latent vector is generated.

[0085] To extract the flawless reference latent vector from the flawless reference image, the basic feature extraction network can be constructed based on the Stable Diffusion feature extraction model and the IP-Adapter module. The Stable Diffusion feature extraction model can adopt existing commonly used Stable Diffusion models. It should be noted that when using existing commonly used Stable Diffusion models, the Stable Diffusion feature extraction model can include variational autoencoders (VAEs), text encoders, and U-Net modules for image encoding. The IP-Adapter (Image Projection Adapter) module is an extension module of the Stable Diffusion model, designed to more accurately control the image generation process by combining image and text prompts, which can enhance the model's responsiveness to specific input prompts (prompt words or other auxiliary information).

[0086] In specific implementation, when constructing a basic feature extraction network based on the Stable Diffusion feature extraction model and the IP-Adapter module, the connection and cooperation state between the IP-Adapter module and the Stable Diffusion feature extraction model can be consistent with existing technologies. For example, the IP-Adapter module can be connected to the variational autoencoder and U-Net module within the Stable Diffusion model. As can be seen from the above description, when generating a flawless reference latent vector using the basic feature extraction network, only a flawless reference image needs to be used.

[0087] To meet the requirement of generating intact reference latent vectors, a decoupled cross-attention mechanism should be adopted to separate the text encoder and variational autoencoder within the Stable Diffusion feature extraction model. This allows the basic feature extraction network, with text input to the text encoder, to process the intact reference image and generate intact reference latent vectors. Specifically, the method and process of separating the text encoder and variational autoencoder using the decoupled cross-attention mechanism can be consistent with existing techniques, and will not be elaborated upon here.

[0088] During operation, the variational autoencoder within the Stable Diffusion feature extraction model receives a flawless reference image and performs variational autoencoding on it to generate a flawless encoded latent vector. This generated flawless encoded latent vector is then used by the IP-Adapter module to generate flawless reference cross-attention features. These features are then injected into the U-Net module of the Stable Diffusion feature extraction model. After denoising by the U-Net module, flawless reference latent vectors are generated. Therefore, the basic feature extraction network of this invention can extract and enhance the visual features of flawless reference images and generate clear, realistic, and flawless flawless reference latent vectors.

[0089] It should be understood that when the basic feature extraction network adopts the above-described structure, the generated flawless reference latent vector is the representation of the flawless reference image in a low-dimensional abstract space. All latent vectors in this invention express the same meaning, as detailed in the description herein.

[0090] It is understandable that the methods and processes of variational autoencoder performing variational autoencoding on the intact reference image, IP-Adapter module generating intact reference cross-attention features, and U-Net module generating intact reference latent vectors after denoising are all consistent with existing technologies, and will not be elaborated here. After generating the intact reference latent vectors, since self-decoding is not required, the self-decoder in the Stable Diffusion model can be omitted.

[0091] In one embodiment of the present invention, the defect feature generation network is constructed based at least on the defect generation StableDiffusion model and the ControlNet module, wherein,

[0092] The defect reference image is loaded into the Controlnet module to generate defect constraints, and the defect constraints are then loaded into the defect generation Stable Diffusion model.

[0093] The defect reference image and defect feature text are simultaneously loaded into the defect generation Stable Diffusion model, so that the defect generation Stable Diffusion model can generate text-guided defect latent vectors under defect constraints.

[0094] To generate text-guided defect latent vectors, the defect feature extraction network can include a defect generation Stable Diffusion model and a ControlNet module. The defect generation Stable Diffusion model can employ existing, commonly used Stable Diffusion models; therefore, the details of the defect generation Stable Diffusion model can be found in the aforementioned descriptions and will not be repeated here. The ControlNet module is a neural network-based architecture primarily used to control the defect generation Stable Diffusion model through additional input, achieving precise control over the generated content. The ControlNet module within the defect feature generation network can also adopt existing, commonly used forms. Therefore, when using the ControlNet module to control the defect generation Stable Diffusion model, the connection and cooperation between the ControlNet module and the defect generation Stable Diffusion model can be consistent with existing technologies, specifically designed to satisfy the requirement of controlling the generated content of the defect generation Stable Diffusion model.

[0095] When generating text-guided defect latent vectors, the defect reference image should be loaded into the ControlNet module, and simultaneously, the defect reference image and defect feature text should be loaded into the Stable Diffusion model for defect generation. The ControlNet module processes the defect reference image to generate defect constraints; the specific method and process for generating these constraints can be consistent with existing technologies and will not be elaborated here. When the defect reference image and defect feature text are simultaneously loaded into the Stable Diffusion model, the variational autoencoder within the model performs variational autoencoding on the defect reference image to generate a defect-encoded latent vector. A text encoder performs text encoding on the defect feature text to generate a defect-feature text-encoded latent vector. An attention mechanism is used to embed the defect constraints with the defect-encoded latent vector. Subsequently, the defect-feature text-encoded latent vector is used to control the denoising process of the U-Net module within the Stable Diffusion model, thus generating text-guided defect latent vectors.

[0096] The above only illustrates one embodiment of the process for generating text-guided defect latent vectors using a defect feature generation network. It is understood that when a defect generation Stable Diffusion model and Controlnet module are used to construct the defect feature generation network, the detailed process for generating text-guided defect latent vectors can be consistent with existing technologies, and will not be elaborated here. In specific implementations, since there is no need to directly decode the text-guided defect latent vectors, the defect feature generation network should not include the self-decoder of the Stable Diffusion model.

[0097] As explained above, the Controlnet module enables structural control (such as edge detection, pose control, and depth mapping), while the IP-Adapter module primarily handles the style or features of the reference image. Therefore, combining the Controlnet and IP-Adapter modules with a single Stable Diffusion model may cause conflicts during feature extraction in the U-Net module. For example, the Controlnet module might enforce shape constraints, while the IP-Adapter module might want to adjust the shape to better match the reference style. This could lead to conflicts within the U-Net module of the Stable Diffusion model during feature extraction. In one embodiment of this invention, a basic feature extraction network is constructed based on the Stable Diffusion model and the IP-Adapter module. Simultaneously, a defect feature generation network is constructed based on the Stable Diffusion model and the Controlnet module. This effectively resolves the conflicts arising from combining both with a single Stable Diffusion model, while simultaneously achieving better generation of surface defect images of the wire rope.

[0098] In one embodiment of the present invention, the fusion decoding network includes a latent vector fusion unit and a latent vector decoder, wherein,

[0099] The implicit vector fusion unit is used to perform implicit vector fusion processing on the unblemished reference implicit vector and the text-guided defect implicit vector, so as to generate the wire rope fused implicit vector after implicit vector fusion processing.

[0100] The latent vector decoder decodes the fused latent vectors of the wire rope to generate a target image of the wire rope.

[0101] It should be understood that after generating the intact reference latent vector and the text-guided defect latent vector, a fusion decoding process is required. To achieve this fusion decoding process, the fusion decoding network should include a latent vector fusion unit and a latent vector decoder connected to the fusion unit. The latent vector fusion unit is connected to a basic feature extraction network and a defect feature generation network, enabling it to perform latent vector fusion processing on the intact reference latent vector and the text-guided defect latent vector, generating a fused latent vector for the steel wire rope. Subsequently, the latent vector decoder decodes the fused latent vector for the steel wire rope, generating a target image of the steel wire rope after decoding.

[0102] In practical implementation, the latent vector decoder can adopt the self-decoder commonly used in the Stable Diffusion model. Therefore, the method and process of decoding the wire rope fused latent vectors using the latent vector decoder can be consistent with existing technologies. The following will provide a detailed explanation of the method and process of latent vector fusion processing by the latent vector fusion unit.

[0103] To achieve fusion decoding, the outputs of the U-Net module in the Stable Diffusion model for feature extraction and the U-Net module in the Stable Diffusion model for defect generation should be connected to the latent vector fusion unit. This allows the latent vector fusion unit to perform latent vector fusion processing on the unblemished reference latent vector and the text-guided defect latent vector.

[0104] In one embodiment of the present invention, the latent vector fusion process includes:

[0105] Align the intact reference latent vector and the text-guided defect latent vector in the latent space to generate an intact aligned latent vector and a text-guided defect aligned latent vector after alignment in the latent space.

[0106] Frequency domain features are extracted from the above-mentioned unblemished alignment latent vector and text-guided defect alignment latent vector to obtain unblemished alignment frequency domain features and text-guided defect alignment frequency domain features, respectively. The unblemished alignment frequency domain features include unblemished alignment low-frequency features and unblemished alignment high-frequency features, and the text-guided defect alignment frequency domain features include defect alignment low-frequency features and defect alignment high-frequency features.

[0107] The unblemished alignment low-frequency features and the defective alignment low-frequency features are fused at low frequencies to generate aligned fused low-frequency features.

[0108] The high-frequency features of flawless alignment and the high-frequency features of defective alignment are fused at high frequency to generate alignment fused high-frequency features.

[0109] Alignment fusion low-frequency features and alignment fusion high-frequency features are combined to generate alignment fusion frequency domain features. The generated alignment fusion frequency domain features are then transformed in the time domain to generate wire rope fusion latent vectors.

[0110] It should be understood that, since the latent space distributions of the intact reference latent vector and the text-guided defect latent vector may differ, directly fusing the intact reference latent vector and the text-guided defect latent vector by summing them may lead to feature mismatch. This may result in the latent vector decoder generating a wire rope target image that does not meet the generation requirements. Therefore, it is necessary to align the intact reference latent vector and the text-guided defect latent vector in the latent space (Latent Space Alignment, LSA).

[0111] In practice, linear mapping based on statistical analysis can be used to process the intact reference latent vector and the text-guided defect latent vector separately, and then alignment is performed after the linear mapping process. Specifically, when aligning the intact reference latent vector and the text-guided defect latent vector in the latent space, we have:

[0112]

[0113] Among them, h IP h is the unambiguous aligned latent vector. CN To align latent vectors for text-guided defects, W IP The mapping matrix W for the incomplete reference latent vectors CN b is the mapping matrix for the latent vectors of text-guided defects. IP The bias term for the incomplete reference latent vector, b CN z is the bias term of the latent vector of text-guided defects. IP For the incomplete reference latent vector, z CN This is a hidden vector for text-guided defects.

[0114] It should be noted that when using a linear mapping based on statistical analysis for latent spatial alignment, the mapping matrix W mentioned above should be calculated. IP Mapping matrix W CN Bias term b IP and bias term b CN The following example illustrates the specific calculation process:

[0115]

[0116] Among them, Ψ IP,i Let μ be the i-th positional feature vector within the flawless alignment latent vector, and N be the number of positional feature vectors within the flawless alignment latent vector and the text guidance defect latent vector. IP Ψ is the mean of the N positional eigenvectors within the unaligned latent vector. CN,i Let μ be the feature vector at the i-th position within the latent vector of the text-guided defect. CN Let Σ be the mean of the N positional feature vectors within the latent vector of the text-guided defect. IP Let Σ be the covariance matrix of the unambiguous aligned latent vectors. CN Let be the covariance matrix of the latent vector of text-guided defects.

[0117] It should be understood that by using the same U-Net backbone structure for both the basic feature extraction network and the defect generation network, the number of positional feature vectors within the flawless alignment latent vector and the text-guided defect latent vector should be the same. Furthermore, after obtaining the flawless alignment latent vector and the text-guided defect latent vector, the corresponding positional feature vectors can be obtained using techniques commonly used in this technical field.

[0118] In practical implementation, based on the above calculation results, we have:

[0119]

[0120] It is understandable that the above mapping matrix W is calculated. IP Mapping matrix W CN Bias term b IP and bias term b CN Then, the above calculation method can be used to align the flawless reference latent vector and the text-guided defect latent vector in the latent space, and the flawless aligned latent vector h can be obtained respectively. IP Text-guided defect alignment of latent vector h CN .

[0121] When extracting frequency domain features from the aforementioned unblemished alignment latent vectors and text-guided defect alignment latent vectors, Fourier transform can be used to map and transform them to the frequency domain for frequency domain feature extraction. Specifically:

[0122]

[0123] Among them, Z IP For the unblemished aligned latent vector h IP The frequency domain characteristics of Z; CN F represents the frequency domain features of the latent vector aligned to the text guidance defect, where F is the Fourier transform.

[0124] It should be noted that for the missing alignment latent vector h IP Perform a Fourier transform and obtain the frequency domain feature Z. IP The method and process are consistent with existing technologies; the method and process of Fourier transform will not be elaborated here. The frequency domain feature Z is obtained. IP and frequency domain feature Z CN Then, the frequency features can be decomposed into low-frequency features representing global background information and high-frequency features representing local details and defects according to the frequency range. Specifically:

[0125]

[0126] Where Z is the frequency domain feature Z IP or frequency domain feature Z CN When Z is a frequency domain feature Z IP Then, by using LowPass(Z), the flawless aligned low-frequency characteristics can be obtained. Subsequently, the corresponding flawless aligned high-frequency features can be calculated. When Z is a frequency domain feature Z CNWhen this is the case, the defect alignment low-frequency characteristics can be obtained through LowPass(Z). Subsequently, the corresponding high-frequency features of defect alignment can be calculated.

[0127] Specifically, LowPass(Z) represents low-pass filtering, where the low-frequency cutoff frequency and high-frequency start frequency should be determined during low-pass filtering. One embodiment for the low-frequency cutoff frequency and high-frequency start frequency is as follows:

[0128]

[0129] Where: f max This represents the maximum frequency, f. max It can be related to the image size; one possible calculation method is:

[0130]

[0131] Where M is a parameter related to the size of the flawless or defective reference image. For example, if the size of the flawless or defective reference image is M×M, then the corresponding maximum frequency f can be calculated. max .

[0132] In one embodiment of the present invention, when generating aligned combined frequency domain features, the following applies:

[0133]

[0134] Among them, Z fuse To align the combined frequency domain features, To align and fuse low-frequency features, To align and fuse high-frequency features, α is the background fusion modulation coefficient, and β is the defect enhancement coefficient. For seamless alignment of low-frequency characteristics, For flawless alignment of high-frequency features, Align low-frequency features with defects. High-frequency features are aligned to defects.

[0135] Specifically, when performing low-frequency fusion of the unblemished aligned low-frequency features and the defective aligned low-frequency features, we have: Where α∈[0,1]; when performing high-frequency fusion of the high-frequency features of flawless alignment and the high-frequency features of defect alignment, we have: The defect enhancement coefficient β also ranges from [0,1]. It can be understood that the low-frequency fusion state can be controlled by the background fusion control coefficient α, and the high-frequency fusion state can be controlled by the defect enhancement coefficient β. That is, the weighted fusion state of low-frequency features and high-frequency features can be controlled by the values ​​of the background fusion control coefficient α and the defect enhancement coefficient β, so as to ensure the balanced expression of background and defects.

[0136] After obtaining the aligned combined frequency domain features using the above method, the generated aligned combined frequency domain features can be transformed into the time domain to generate the wire rope fusion latent vector. It can be understood that the generated wire rope fusion latent vector can comprehensively represent the normal background and defect features, that is, it can improve the quality of the wire rope target defect image generated by the latent vector decoder.

[0137] In one embodiment of the present invention, when providing a defect reference image, the method of providing the image includes:

[0138] Provide a reference image of the defect.

[0139] The defect reference source image is converted to HSV color space and Lab color space respectively. Then, binarization operation is performed in HSV color space and Lab color space respectively to generate HSV color space binarized image and Lab color space binarized image respectively after the binarization operation.

[0140] Perform a union operation on the HSV color space binarized image and the Lab color space binarized image to form a spatially fused binarized image;

[0141] Based on the HSV color space of the defective reference source image and the spatially fused binarized image, the hue variation state in the HSV color space is calculated and determined.

[0142] Based on the Lab color space of the defect reference source image and the spatial fusion binarized image, the brightness change state in the Lab color space is calculated and determined.

[0143] Based on the calculated and determined hue and brightness change states, among which,

[0144] When the hue change is significant, hue separation is performed in the HSV color space of the defect reference source image, and a defect reference image is generated after separation.

[0145] When the brightness change is significant, the brightness is enhanced in the Lab color space of the defect reference source image, and the defect reference image is generated after the brightness enhancement.

[0146] It should be noted that the defect reference source image generally refers to the image generated by image acquisition of the wire rope in the working scene. The working scene of the wire rope can generally include environments such as underground coal mines. The wire rope in the working scene is usually covered with oil and mud. Therefore, when the defect reference source image is obtained by image acquisition of the wire rope in the working scene and the defect reference image is directly generated from the defect reference source image, the quality of the defect reference image will be low, which will in turn lead to the low quality of the generated target defect image of the wire rope.

[0147] The color space of the defect reference source image is generally RGB. Using techniques commonly used in this field, the defect reference source image can be converted to HSV and Lab color spaces respectively. The specific conversion methods are consistent with existing technologies and will not be elaborated here. Based on the characteristics of steel wire rope, in the HSV color space, hue (H) can be used to distinguish between oil stains (yellowish, brownish) and mud stains (grayish brown), saturation (S) reflects the degree of contamination, and luminance (V) helps identify low-brightness contamination areas. In the Lab color space, the luminance (L) channel enhances the structural details of the steel wire rope, and the a / b channels can be used to distinguish between oil stains (a channel is reddish, b channel is yellowish) and mud stains (a / b channels are close to neutral gray). Therefore, after conversion to HSV and Lab color spaces, effective processing of the defect reference source image can be achieved, improving the quality of the generated defect reference image.

[0148] After conversion to the HSV color space, the HSV color space can be binarized to generate an HSV color space binarized image. Similarly, the Lab color space can be binarized to generate a Lab color space binarized image. The following example illustrates the binarization method and process using the characteristics of oil and mud on a steel wire rope.

[0149] Specifically, when performing binarization in the HSV color space, an HSV color space mask should be constructed. Then, the binarized value of each pixel within the HSV color space is determined using the HSV color space mask. Specifically, for the HSV color space mask:

[0150]

[0151] In the formula, M HSV For HSV color space masks, M oil,H For HSV color space oil stain mask, M mud,H For HSV color space mud mask, M H For the HSV color space tone mask, S low S is the first threshold for saturation. mid V is the second threshold for saturation. low V is the first threshold for brightness. high H is the second threshold for brightness. low H is the first threshold for hue. high ∪ represents the second threshold for hue, ∪ represents the union operation, and ∩ represents the intersection operation.

[0152] In practical implementation, the first saturation threshold S low It can be set to 20, the second saturation threshold S midIt can be set to 40, the first threshold value for brightness V. low It can be set to 80, the second brightness threshold V high It can be set to 200, the first threshold H for hue. low It can be set to 10, the first threshold H for hue. high It can be set to 40; of course, it can also be set to other values, depending on the working scenario of the defect reference source image.

[0153] Understandably, when performing binarization using an HSV color space mask, the corresponding channel values ​​for the hue (H), saturation (S), and lightness (V) channels of each pixel are determined. Subsequently, these channel values ​​are used to determine the corresponding HSV color space smudge mask M for the current pixel. oil,H HSV color space mud mask M mud,H and HSV color space tone mask M H The corresponding value can then be used to obtain the HSV color space mask M corresponding to the current pixel. HSV Value; specifically, when the HSV color space oil mask M oil,H HSV color space mud mask M mud,H and HSV color space tone mask M H When one of the values ​​is 1, then the HSV color space mask M corresponding to the current pixel... HSV The value should be 1 if and only if the HSV color space oil mask M oil,H HSV color space mud mask M mud,H and HSV color space tone mask M H When all values ​​are 0, then the HSV color space mask M corresponding to the current pixel is... HSV The value should be 0.

[0154] In practice, for each pixel in the HSV color space, a corresponding HSV color space mask M is determined. HSV Value, based on the HSV color space mask M for all pixels. HSV This means performing binarization processing on the HSV color space and obtaining a binarized image in the HSV color space.

[0155] When binarizing the Lab color space, the method is similar to that used for binarizing the HSV color space. One possible approach is as follows:

[0156] Construct a Lab color space mask, wherein the Lab color space mask is:

[0157]

[0158] Among them, M Lab For Lab color space masks, Moil,L For Lab color space oil stain mask, M mud,L For a mud mask in the Lab color space, L low The first threshold for channel L, L mid The second threshold for the L channel, a thresh For channel a, the threshold is... low The first threshold for channel b, b thresh Let |a| be the second threshold of channel b, |b| be the absolute value of channel a, and |b| be the absolute value of channel b.

[0159] In specific implementation, the first threshold L of channel L low It can be set to 40, the second threshold of the L channel. mid It can be set to 60, the threshold for channel a. thresh It can be set to 10, the first threshold of the b channel. low It can be set to 20, the second threshold for the b channel. thresh It can be set to 10.

[0160] After constructing the Lab color space mask, you can refer to the above method for determining the HSV color space mask value. That is, the method and process for generating the Lab color space binarized image can be referred to the corresponding description of the HSV color space binarized image, which will not be repeated here.

[0161] As explained above, both Lab color space binarized images and HSV color space binarized images are binarized images. When a union operation is performed on an HSV color space binarized image and a Lab color space binarized image, a spatially fused binarized image can be formed. Therefore, the spatially fused binarized image includes all pixels with a binarization value of 1 in both the Lab color space binarized image and the HSV color space binarized image.

[0162] After obtaining the spatially fused binarized image, the hue variation state in the HSV color space can be calculated and determined based on the HSV color space of the defect reference source image and the spatially fused binarized image. Simultaneously, the brightness variation state in the Lab color space can be calculated and determined based on the Lab color space of the defect reference source image and the spatially fused binarized image. Examples of how to calculate the hue variation state and brightness variation state are given below.

[0163] When calculating the hue variation in the HSV color space, one feasible calculation method is as follows:

[0164]

[0165] Wherein, ΔH represents the hue transformation state. The average hue of the polluted area. M represents the average hue of the background area.union (i,j) represents the value of the pixel at coordinate (i,j) in the spatially fused binarized image, and H(i,j) represents the value of the hue H channel of the pixel at coordinate (i,j) in the HSV color space.

[0166] When calculating the brightness variation in the Lab color space, one feasible calculation method is as follows:

[0167]

[0168] Wherein, ΔL represents the brightness change state. The average brightness of the polluted area. L is the average brightness of the background area, and L(i,j) is the value of the L channel of the pixel at coordinate (i,j) in the Lab color space.

[0169] After determining the hue and brightness change states using the above method, further judgment is needed, specifically:

[0170] When the hue change is significant, hue separation is performed in the HSV color space of the defect reference source image, and a defect reference image is generated after separation. When the hue change state ΔH > 30°, the hue change can be considered significant. Subsequently, hue separation can be performed using techniques commonly used in this technical field. That is, the hue separation method and process can be consistent with the existing technology, and will not be elaborated here.

[0171] When the brightness change is significant, the brightness is enhanced in the Lab color space of the defect reference source image, and the defect reference image is generated after the brightness enhancement. When the brightness change ΔL is greater than 20, the brightness change can be considered significant. After that, the brightness enhancement can be performed using existing commonly used techniques.

[0172] In practice, the hue change threshold and brightness change threshold can be determined based on the working environment of the wire rope. For example, images of the wire rope in the working environment can be collected, and the hue change threshold and brightness change threshold can be obtained statistically. Then, the calculated hue change state can be compared with the determined hue change threshold to determine whether the hue change is significant. Similarly, it can be determined whether the brightness change state is significant.

[0173] When the color tone change is determined to be significant, the defect reference source image needs to be color-separated in the HSV color space. After color separation, it can be converted to the RGB color space and a defect reference image can be generated. The color separation method can adopt the commonly used methods in existing technologies, which will not be elaborated here.

[0174] When a significant change in brightness is determined, brightness enhancement needs to be performed in the Lab color space of the defect reference source image, followed by conversion to the RGB color space to generate the defect reference image. It is understood that the method of brightness enhancement in the Lab color space can be consistent with existing techniques, and will not be elaborated upon here.

[0175] The above illustrates one embodiment of processing a defect reference source image to generate a defect reference image. Of course, other methods can also be used to process the defect reference source image, depending on whether a high-quality defect reference image can be obtained. Examples will not be given here.

[0176] In one embodiment of the present invention, the method for constructing a defect image generation model includes:

[0177] A defect image generation basic model is constructed, which includes a basic feature extraction network after preliminary training, a defect feature generation network after preliminary training, and an untrained fusion decoding network.

[0178] Construct a basic model training dataset for training the base model of defect image generation, wherein...

[0179] The basic model training dataset includes a defect-free training sample set and a defective training sample set.

[0180] The defect-free training sample set includes several defect-free training samples, and each defect-free training sample includes a defect-free Class I wire rope training image.

[0181] The defect training sample set includes several defect training samples. Each defect training sample includes a second type of wire rope training image with defects and training image text used to characterize the defect state of the second type of wire rope training image.

[0182] The basic model for generating defect images is trained based on the configured basic model training conditions.

[0183] During model training, a defect-free training sample is loaded into the basic feature extraction network, and a defective training sample is loaded into the defect feature generation network. Subsequently, the wire rope training defect image is generated by the fusion decoding network.

[0184] The loss function of the training model is calculated based on the steel wire rope training defect image and the second type of steel wire rope training image in the current defect training sample. The network parameters of the defect image generation base model are adjusted based on the calculated loss function until the defect image generation base model is trained to the target state.

[0185] Based on the training of the model to reach the target state, the defect image generation model is generated.

[0186] It should be understood that the constructed defect image generation basic model should be consistent with the defect image generation model described above. Please refer to the above explanation for details. Afterwards, the defect image generation basic model can be trained, and the defect image generation model can be obtained after training. In specific implementation, the construction of the defect image generation basic model should include a pre-trained basic feature extraction network, a pre-trained defect feature generation network, and an untrained fusion decoding network. That is, when constructing the defect image generation basic model, the basic feature extraction network and the defect feature generation network should be trained and obtained first. The following provides a detailed explanation of the pre-training of the basic feature extraction network and the defect feature generation network.

[0187] It should be noted that, in order to obtain a preliminarily trained basic feature extraction network, a basic feature extraction network should be constructed first, and then trained to obtain a preliminarily trained basic feature extraction network. The details of the basic feature extraction network can be found in the above description. The difference is that the basic feature extraction network should include a self-decoder, such as the self-decoder of the Stable Diffusion feature extraction model. Similarly, in order to obtain a preliminarily trained defect feature generation network, a basic defect feature generation network should be constructed first, and then trained to obtain a preliminarily trained defect feature generation network. It can be understood that the basic defect feature generation network also includes a self-decoder, such as the self-decoder of the Stable Diffusion defect generation model.

[0188] When training the basic feature extraction network, a defect-free training sample set should be used for initial training. This set includes training images of the first type of steel wire rope. The type of steel wire rope in these images should be the same as the type of steel wire rope corresponding to the defective and flawed reference images. During training, at least necessary training conditions, such as the network loss function, should be configured. These conditions may include the number of training iterations and optimization parameters, which can be selected as needed. For example, the batch size can be 2–4, the number of training iterations can be 150–200, the learning rate can be 2e-4, and the AdamW optimizer can be used with a weight decay value of 1e-6.

[0189] After each round of training the basic feature extraction network using an intact training sample set, the loss value of the extraction network loss function should be calculated. The extraction network loss function can be the Mean Squared Error (MSE) loss function. One feasible way to calculate the loss value is as follows:

[0190] The defect-free training samples are loaded into the basic feature extraction network to obtain defect-free feature extraction training images corresponding to the current defect-free training samples. These defect-free feature extraction training images are the output images of the self-decoder of the StableDiffusion feature extraction model. From all generated defect-free feature extraction training images, one is randomly selected, and its corresponding defect-free training sample is determined. Then, the loss value of the extraction network's loss function is calculated between the selected defect-free feature extraction training image and the corresponding first-class steel wire rope training image within the determined defect-free training sample. The result is:

[0191]

[0192] Where MSE1 is the loss value of the extraction network loss function, I(i,j) is the pixel value at coordinates (i,j) in the first type of wire rope training image, and I... o (i,j) represents the pixel value at coordinates (i,j) in the defect-free feature extraction training image, m represents the number of pixel rows corresponding to the defect-free feature extraction training image and the first type of wire rope training image, and n represents the number of pixel columns corresponding to the defect-free feature extraction training image and the first type of wire rope training image.

[0193] In practice, after calculating the loss value MSE1 of the feature extraction network loss function, the loss value MSE1 should be compared with the feature extraction training loss threshold. If the loss value MSE1 is greater than the feature extraction training loss threshold, the network parameters of the basic feature extraction network should be adjusted in reverse and retrained until the loss value MSE1 is not greater than the feature extraction training loss threshold.

[0194] Specifically, once the loss value MSE1 is no greater than the feature extraction training loss threshold, a manual visual judgment can be made by an operator to determine whether the image quality of the defect-free feature extraction training image meets the standard. If yes, training ends, thus obtaining the basic feature extraction network after preliminary training. If not, the feature extraction training loss threshold can be reduced, and the basic feature extraction network can be trained again until the loss value MSE1 is no greater than the feature extraction training loss threshold and the image quality of the defect-free feature extraction training image meets the standard. Generally, the feature extraction training loss threshold can be 0.1.

[0195] When training the basic network for defect feature generation, a defect training sample set should be used. Each defect training sample includes a second-type wire rope training image with defects and training image text used to characterize the defect state of the second-type wire rope training image. The type of wire rope corresponding to the second-type wire rope training image belongs to the same type as the type of wire rope in the aforementioned flawless reference image. The training image text describes the defect state in the second-type wire rope training image. The defect state can be referred to the corresponding description above.

[0196] When constructing the defect training sample set, images of defective wire ropes are acquired. These images are then processed using a method that generates defect reference images from the aforementioned defect reference source images. This process produces a second type of wire rope training image. After obtaining the second type of wire rope training images, a cue word is created for each defect within these images. This cue word describes the defect characteristics of the wire rope, thus obtaining the feature information of the wire rope defects, which is equivalent to obtaining the training image text. Therefore, it can be seen that there is a one-to-one correspondence between the defect features in the training image text and the defects in the second type of wire rope training images.

[0197] When constructing the defect training sample set, the defect types within the training samples should cover typical defect types, such as broken wires, wear, corrosion, and deformation. Training image text can be stored as a JSON file or a table (such as CSV).

[0198] To obtain defect training samples, each defect image should be meticulously annotated. Specifically, the annotation should at least indicate the number of defects in the image, such as "1 broken wire" or "2 rust spots"; determine the location of the defect in the image, such as "top left corner" or "central area"; and classify it as broken wire, wear, rust, or deformation. The annotation information should be stored in a structured format, including image paths and annotation feature information, and then defect prompts (i.e., feature text information) should be generated, containing the number of defects, defect location, and defect type. For example, Example 1: "There is one broken wire in the top left corner of the wire rope, approximately 5 mm in length.", Example 2: "There are 3 rust spots distributed evenly on the right side of the wire surface." Each image and its corresponding prompt form a one-to-one correspondence, facilitating storage and retrieval. Data should be stored in tabular (e.g., JSON) or file (e.g., JSON).

[0199] It should be understood that when training the basic network for defect feature generation, it is still necessary to configure necessary training conditions, including the loss function of the generation network. Necessary training conditions may include the number of training iterations and the optimizer, etc., which can be selected as needed. For example, you can refer to the training condition settings used when training the basic network for basic feature extraction.

[0200] During training, defect training samples are loaded into the defect feature generation basic network, which then generates corresponding defect generation training images. These defect generation training images are output by the self-decoder of the StableDiffusion defect generation model. After each round of training the defect feature generation basic network using the defect training sample set, one defect generation training image is randomly selected from all generated defect generation training images, and its corresponding defect training sample is determined. Subsequently, the loss value of the generation network loss function can be calculated between the selected defect generation training image and the corresponding second-type wire rope training image within the determined defect training sample. The generation network loss function can be the Mean Squared Error (MSE) loss function. When using the MES loss function, please refer to the relevant explanation above for extracting the network loss function.

[0201] In practice, after calculating the loss value of the generation network loss function, the loss value should be compared with the defect generation training loss threshold. If the loss value is greater than the defect generation training loss threshold, the network parameters of the defect feature generation network should be adjusted in reverse and retrained until the loss value is no greater than the defect generation training loss threshold.

[0202] Specifically, when the loss value is not greater than the defect generation training loss threshold, manual visual judgment can be made by the operator to determine whether the image quality of the defect generation training image meets the standard. If yes, training ends, thus obtaining the initially trained defect feature generation network. If not, the defect generation training loss threshold can be reduced, and the basic defect feature generation network can be trained again until the loss value is not greater than the defect generation training loss threshold, and the image quality of the defect generation training image meets the standard. Generally, the defect generation training loss threshold can be 0.1.

[0203] As can be seen from the above description, after obtaining the basic feature extraction network and the defect feature generation network after preliminary training, the autoencoder used in the preliminary training can be removed respectively, and then connected with the fusion decoding network according to the above description to construct the basic model for defect image generation.

[0204] Once the basic model for generating defective images is constructed, training conditions for the basic model should generally be configured. These conditions should include the loss function, number of iterations, and optimizer. The specific training conditions can be selected as needed, such as those used when training the basic feature extraction network.

[0205] When training the defect image generation basic model using the basic model training dataset, a defect-free training sample is loaded into the basic feature extraction network, and a defective training sample is loaded into the defect feature generation network. Then, a wire rope training defect image is generated via a fusion decoding network. After one round of training on the defect image generation basic model using the basic model training dataset, the loss function for model training is calculated based on the wire rope training defect image and a second type of wire rope training image within the current defect training sample. The network parameters of the defect image generation basic model are then adjusted based on the calculated loss function until the defect image generation basic model is trained to the target state. Specifically, the method for training the defect image generation basic model to the target state can be referred to the corresponding explanation above, and will not be repeated here.

[0206] It should be noted that the defect image generation model is generated based on the basic defect image generation model trained to the target state, thus realizing the construction of the defect image generation model. Of course, other methods can also be used to construct the defect image generation model, and the specific method can be selected as needed, which will not be elaborated here.

[0207] Furthermore, a device for generating images of surface defects in steel wire ropes can be obtained. In one embodiment of the present invention, it includes a defect image generation device, and a defect image generation model is deployed within the defect image generation device, wherein...

[0208] The conditions for generating defect images of steel wire ropes are obtained. The defect image generation device performs defect image generation processing using the method described above, and generates a target defect image of the steel wire rope after the defect image generation processing.

[0209] Specifically, the defect image generation device can be a commonly used existing computer device. The type of defect image generation device can be selected according to needs, based on its ability to generate target defect images of the wire rope. The defect image generation model can be deployed within the defect image generation device using methods commonly used in this technical field. Subsequently, the target defect image of the wire rope can be generated using the methods described above. For details, please refer to the above description; it will not be repeated here.

Claims

1. A method for generating images of surface defects in steel wire ropes, characterized in that, The generation method includes: The system provides conditions for generating defect images and loads these conditions into the constructed defect image generation model. The defect image generation model includes a basic feature extraction network, a defect feature generation network, and a fusion decoding network. The defect image generation conditions include a flawless reference image, defect feature text, and a defect reference image, wherein the type of wire rope in the flawless reference image is consistent with the type of wire rope in the defect reference image. The flawless reference image is loaded into the basic feature extraction network to extract the flawless reference latent vector, and the extracted flawless reference latent vector is loaded into the fusion decoding network. The defect feature text and defect reference image are loaded into the defect feature generation network to generate a text-guided defect latent vector, and the generated text-guided defect latent vector is loaded into the fusion decoding network. A fusion decoding network is used to fuse and decode the flawless reference latent vector and the text-guided defect latent vector to generate a target defect image of a steel wire rope. The defect state of the target defect image of the steel wire rope is consistent with the defect state described by the defect feature text. The fusion decoding network includes a latent vector fusion unit and a latent vector decoder, wherein... The implicit vector fusion unit is used to perform implicit vector fusion processing on the unblemished reference implicit vector and the text-guided defect implicit vector, so as to generate the wire rope fused implicit vector after implicit vector fusion processing. The latent vector decoder decodes the fused latent vectors of the steel wire rope to generate a target image of the steel wire rope after decoding. When performing latent vector fusion, the following steps are included: Align the intact reference latent vector and the text-guided defect latent vector in the latent space to generate an intact aligned latent vector and a text-guided defect aligned latent vector after alignment in the latent space. Frequency domain features are extracted from the above-mentioned unblemished alignment latent vector and text-guided defect alignment latent vector to obtain unblemished alignment frequency domain features and text-guided defect alignment frequency domain features, respectively. The unblemished alignment frequency domain features include unblemished alignment low-frequency features and unblemished alignment high-frequency features, and the text-guided defect alignment frequency domain features include defect alignment low-frequency features and defect alignment high-frequency features. The unblemished alignment low-frequency features and the defective alignment low-frequency features are fused at low frequencies to generate aligned fused low-frequency features. The high-frequency features of flawless alignment and the high-frequency features of defective alignment are fused at high frequency to generate alignment fused high-frequency features. Alignment fusion low-frequency features and alignment fusion high-frequency features are combined to generate alignment fusion frequency domain features. The generated alignment fusion frequency domain features are then transformed in the time domain to generate wire rope fusion latent vectors. When aligning the flawless reference latent vector with the text-guided defect latent vector in the latent space, we have: in, For the unblemished aligned latent vectors, Align latent vectors for text-guided defects. The mapping matrix for the incomplete reference latent vectors, The mapping matrix for the latent vectors of textual defects. The bias term for the incomplete reference latent vector, This is the bias term for the latent vector of text-guided defects. For a flawless reference latent vector, This is used to guide the hidden vector of defects in the text.

2. The method for generating surface defect images of steel wire ropes according to claim 1, characterized in that: The basic feature extraction network is constructed based at least on the Stable Diffusion feature extraction model and the IP-Adapter module, wherein, For the loaded intact reference image, the variational autoencoder in the Stable Diffusion feature extraction model is used to perform variational autoencoding on the intact reference image to generate an intact coded latent vector. The IP-Adapter module uses a decoupled cross-attention mechanism to convert the missing encoded latent vector into a missing reference cross-attention feature, and then injects the missing reference cross-attention feature into the U-Net module of the Stable Diffusion feature extraction model. After denoising by the U-Net module, a missing reference latent vector is generated.

3. The method for generating surface defect images of steel wire ropes according to claim 1, characterized in that: The defect feature generation network is constructed based at least on the Stable Diffusion defect generation model and the ControlNet module, wherein, The defect reference image is loaded into the Controlnet module to generate defect constraints, and the defect constraints are then loaded into the defect generation Stable Diffusion model. The defect reference image and defect feature text are simultaneously loaded into the defect generation Stable Diffusion model, so that the defect generation Stable Diffusion model can generate text-guided defect latent vectors under defect constraints.

4. The method for generating surface defect images of steel wire ropes according to claim 1, characterized in that, When generating aligned combined frequency domain features, we have: in, To align the combined frequency domain features, To align and fuse low-frequency features, To align and fuse high-frequency features, For background fusion control coefficients, The defect enhancement factor is... For seamless alignment of low-frequency characteristics, For flawless alignment of high-frequency features, Align low-frequency features with defects. High-frequency features are aligned to defects.

5. The method for generating images of surface defects in steel wire ropes according to any one of claims 1 to 4, characterized in that, When providing a defect reference image, include: Provide a reference image of the defect. The defect reference source image is converted to HSV color space and Lab color space respectively. Then, binarization operation is performed in HSV color space and Lab color space respectively to generate HSV color space binarized image and Lab color space binarized image respectively after the binarization operation. Perform a union operation on the HSV color space binarized image and the Lab color space binarized image to form a spatially fused binarized image; Based on the HSV color space of the defective reference source image and the spatially fused binarized image, the hue variation state in the HSV color space is calculated and determined. Based on the Lab color space of the defect reference source image and the spatial fusion binarized image, the brightness change state in the Lab color space is calculated and determined. Based on the calculated and determined hue and brightness change states, among which, When the hue change is significant, hue separation is performed in the HSV color space of the defect reference source image, and a defect reference image is generated after separation. When the brightness change is significant, the brightness is enhanced in the Lab color space of the defect reference source image, and the defect reference image is generated after the brightness enhancement.

6. The method for generating images of surface defects in steel wire ropes according to any one of claims 1 to 4, characterized in that, When constructing a defect image generation model, the construction methods include: A defect image generation basic model is constructed, which includes a basic feature extraction network after preliminary training, a defect feature generation network after preliminary training, and an untrained fusion decoding network. Construct a basic model training dataset for training the base model of defect image generation, wherein... The basic model training dataset includes a defect-free training sample set and a defective training sample set. The defect-free training sample set includes several defect-free training samples, and each defect-free training sample includes a defect-free Class I wire rope training image. The defect training sample set includes several defect training samples. Each defect training sample includes a second type of wire rope training image with defects and training image text used to characterize the defect state of the second type of wire rope training image. The basic model for generating defect images is trained based on the configured basic model training conditions. During model training, a defect-free training sample is loaded into the basic feature extraction network, and a defective training sample is loaded into the defect feature generation network. Subsequently, the wire rope training defect image is generated by the fusion decoding network. The loss function of the training model is calculated based on the steel wire rope training defect image and the second type of steel wire rope training image in the current defect training sample. The network parameters of the defect image generation basic model are adjusted based on the calculated loss function until the defect image generation basic model is trained to the target state. Based on the training of the model to reach the target state, the defect image generation model is generated.

7. A device for generating images of surface defects in a steel wire rope, characterized in that, This includes a defect image generation device, and a defect image generation model is deployed within the defect image generation device, wherein... The defect image generation conditions for the steel wire rope are obtained. The defect image generation device performs defect image generation processing using the method described in any one of claims 1 to 6, and generates a target defect image of the steel wire rope after the defect image generation processing.

Images