Surface defect positioning system and method for in-situ anticorrosion operation of hydraulic steel gate

Abstract

The invention provides a surface defect positioning system and method for in-situ anticorrosion operation of a hydraulic steel gate. The surface defect positioning system comprises a multi-modal image acquisition module, a gate position reading and timer module, an image splicing and fusion module, a defect identification and positioning module and a data output module. An RGB image and an infrared image of the surface of the hydraulic steel gate are acquired through the multi-modal image acquisition module, the multi-modal image acquisition module is controlled to start and stop through the gate position reading and timer module, the acquired RGB image and the infrared image are fused and spliced through the image splicing and fusing module, and a final image is formed. And the defect on the final image is identified and positioned through the defect identification and positioning module, and the identified defect information and position data thereof are output and displayed through the data output module. By acquiring the defects on the surface of the gate and reporting the types and the positions of the defects, data support is provided for subsequent fixed-point corrosion prevention operation of the hydraulic steel gate, and the operation efficiency is improved.

Description

Technical Field

[0001] The invention relates to the technical field of hydraulic steel gate anti-corrosion operations, and in particular to a surface defect positioning system and method for on-site hydraulic steel gate anti-corrosion operations. Background Art

[0002] For the hydraulic steel gates of the hydropower station in service, part of them are below the waterline, and the other part is exposed to the air. After a certain number of years of use, due to aquatic organisms and electrochemical corrosion, some components have defects such as peeling of anti-corrosion coatings, rust spots, rust pits, and rust scale stacking, which are densely packed or have holes, resulting in thinning of the cross-sectional thickness. In addition, the weld problems at the welding position are manifested as original defects such as leakage or incomplete welding in some welds, which weaken the welding strength of the gate components. Under the action of long-term water pressure and microorganisms, the welds may produce cracks, corrosion and other harmful hidden dangers, affecting the bearing capacity of the gate. These problems not only affect the appearance of the steel gate, but more importantly, it weakens the strength and rigidity of the steel structure, reduces the bearing capacity, and shortens the service life of the gate. Therefore, it is particularly important to find defects in the hydraulic steel gate before anti-corrosion operations. Summary of the invention

[0003] In order to solve the current technical problems, the main purpose of the present invention is to provide a surface defect positioning system and method for on-site anti-corrosion operations of hydraulic steel gates, which can automatically detect and locate defects on the surface of steel gates, and output them to the user end through an output module, providing accurate data support for anti-corrosion operations.

[0004] The technical solution adopted by the present invention is: a surface defect positioning system for on-site anti-corrosion operations of hydraulic steel gates, comprising: A multimodal image acquisition module, which is used to capture RGB images and infrared images of the surface of the hydraulic steel gate; A gate position reading and timer module, which is used to control the start and stop of the multimodal image acquisition module according to the operating status and position of the hydraulic steel gate; An image stitching and fusion module, which is used to fuse the acquired RGB image and infrared image, and then stitch the fused images to form a final image of the surface of the hydraulic steel gate; A defect recognition and positioning module, which is used to analyze the final image, identify defects on the surface of the hydraulic steel gate, and output specific coordinate position information of the defects in the final image; A data output module is used to output the identified defect information and its location data to a display device and / or a storage device.

[0005] It also includes a sample expansion module, which adds residual blocks, spectral norm regularization, replaces deconvolution layers, adds downsampling layers, and a gradient penalty mechanism on the basis of the generative adversarial network, and then combines the characteristics of surface defect samples of hydraulic steel gates to expand the defect sample data.

[0006] The multimodal image acquisition module includes an RGB camera and a near-infrared camera, and the RGB camera and the near-infrared camera synchronously capture images of the surface of the hydraulic steel gate at the same position.

[0007] The gate position reading and timer module includes a gate position reading unit and a timer unit. The gate position reading unit is used to obtain the specific position of the current hydraulic steel gate, and the timer unit is used to obtain the time required for the hydraulic steel gate to be lifted or lowered. When the hydraulic steel gate starts to descend or rise, the timer unit starts timing, and the multimodal image acquisition module starts to acquire images at the same time. When the operation of the hydraulic steel gate is suspended, the timer unit suspends timing, and the multimodal image acquisition module suspends acquiring images. When the hydraulic steel gate rises to the top or falls to the bottom, the timer unit stops timing and the multimodal image acquisition module stops acquiring images.

[0008] The image stitching and fusion module performs a fusion process on the acquired RGB image and infrared image to obtain an image with obvious features, and stitches the fused images through an image stitching algorithm. During the stitching, the overlapping parts between the images are removed.

[0009] The image stitching and fusion module first stitches the fused image into a plurality of segmented images through an image stitching algorithm, and finally stitches the plurality of segmented images again to form a final image.

[0010] The defect recognition and positioning module includes a defect recognition unit and a positioning unit. The defect recognition unit is used to learn and extract defect features. The defect recognition unit uses a visual database for pre-training in the early stage of training. The positioning unit adopts an automatic defect positioning model method based on deep learning to automatically extract surface features of hydraulic steel gates and generate a thermal map. By clustering defect features, single defects and multiple defects on the surface of hydraulic steel gates are positioned. The position coordinates of the defects use the coordinate system on the final image as a reference. A positioning method using the surface defect positioning system for on-site anti-corrosion operation of hydraulic steel gates comprises the following steps: By taking pictures on site or obtaining previously taken images of the surface of hydraulic steel gates, and finding images with defects, marking and classifying the images, determining whether there are defects and the specific types of defects, the defect recognition and positioning module is trained; The image acquisition module is deployed near the hydraulic steel gate to control the lifting or lowering of the hydraulic steel gate, and at the same time, the RGB image and infrared image of the surface of the hydraulic steel gate are collected until the image of the entire surface of the hydraulic steel gate is collected; The collected images are then input into the image stitching and fusion module. First, the acquired RGB images and infrared images are fused one-to-one, and then the image stitching algorithm is used to remove the overlapping parts between the images, and multiple images are stitched into multiple segmented images. Then, multiple segmented images are fused and spliced ​​through an image splicing and fusion module to obtain a complete surface image of the hydraulic steel gate and form a final image; The final image is transmitted to the defect recognition and positioning module for recognition and positioning processing; The data output module outputs whether there are defects in the final image. If so, it displays the number of defects in the final image, the defect type, the detection confidence, and the position of the defect in the final image coordinate system.

[0011] The present invention has the following beneficial effects: 1. The present invention proposes a surface defect positioning system and method for hydraulic steel gates. The RGB image and infrared image of the surface of the hydraulic steel gate are obtained through a multimodal image acquisition module. The gate position reading and timer module is used to control the start and stop of the multimodal image acquisition module. The image splicing and fusion module fuses and splices the acquired RGB image and infrared image to form a final image. The defect recognition and positioning module is used to identify and locate the defects on the final image. The data output module outputs and displays the identified defect information and its position data. By obtaining the defects on the gate surface and reporting the type and location of the defects, data support is provided for the subsequent fixed-point anti-corrosion operation of the hydraulic steel gate, thereby improving the efficiency of the operation.

[0012] 2. The present invention also includes a sample expansion module, which effectively solves the problems of small samples, gradient disappearance, gradient anomaly, and low quality of generated images encountered by GAN in the process of gate surface defect sample data expansion by adding residual blocks, spectral norm regularization, replacing deconvolution layers, adding downsampling layers, and adding gradient penalty mechanisms on the basis of generative adversarial networks. These improvement measures not only improve the stability of GAN and the quality of generated samples, but also provide strong support for the effective expansion of defect sample data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0014] Figure 1 It is a system block diagram of the present invention.

[0015] Figure 2 It is a flowchart of the method of the present invention. DETAILED DESCRIPTION

[0016] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] Embodiment 1: like Figure 1 As shown, the present invention provides a surface defect positioning system for on-site anti-corrosion operations of hydraulic steel gates, comprising: A multimodal image acquisition module, which is used to capture RGB images and infrared images of the surface of the hydraulic steel gate; A gate position reading and timer module, which is used to control the start and stop of the multimodal image acquisition module according to the operating status and position of the hydraulic steel gate; An image stitching and fusion module, which is used to fuse the acquired RGB image and infrared image, and then stitch the fused images to form a final image of the surface of the hydraulic steel gate; A defect recognition and positioning module, which is used to analyze the final image, identify defects on the surface of the hydraulic steel gate, and output specific coordinate position information of the defects in the final image; A data output module is used to output the identified defect information and its location data to a display device and / or a storage device.

[0018] The present invention combines computer vision technology, obtains RGB images and infrared images of the surface of hydraulic steel gates through a multimodal image acquisition module, controls the start and stop of the multimodal image acquisition module through a gate position reading and timer module, fuses and splices the acquired RGB images and infrared images through an image splicing and fusion module to form a final image, identifies and locates defects on the final image through a defect recognition and positioning module, and outputs and displays the identified defect information and its position data through a data output module. By obtaining defects on the gate surface and reporting the type and location of the defects, data support is provided for subsequent fixed-point anti-corrosion operations of hydraulic steel gates, thereby improving the efficiency of the operation.

[0019] In this embodiment, the multimodal image acquisition module includes an RGB camera and a near-infrared camera, and the RGB camera and the near-infrared camera synchronously capture images of the surface of the hydraulic steel gate at the same position.

[0020] In reality, hydraulic steel gates are in contact with the external environment, and their surface conditions are constantly changing, especially the surface illumination conditions. In different weather conditions, such as rainy days, foggy days, sunny days, cloudy days, etc., the illumination conditions are very different. It is difficult for algorithms to detect targets in dynamic environmental changes using only visible light sensor data, such as images taken by RGB cameras. Traditional visual systems hope to work in scenes with stable illumination conditions. Visible light images contain rich texture information but are sensitive to illumination, while infrared images have the opposite characteristics. In the present invention, RGB cameras and near-infrared cameras are used in combination, and by integrating the complementarity of different modalities, the perceptibility, reliability and robustness of the detection algorithm can be further improved. This module can use high-resolution RGB industrial cameras and near-infrared industrial cameras, combined with specific light sources and lenses, to ensure the clarity and accuracy of the image.

[0021] In this embodiment, the gate position reading and timer module includes a gate position reading unit and a timer unit. The gate position reading unit is used to obtain the specific position of the current hydraulic steel gate, and the timer unit is used to obtain the time required for the hydraulic steel gate to be lifted or lowered. When the hydraulic steel gate starts to descend or rise, the timer unit starts timing, and the multimodal image acquisition module starts to acquire images. When the operation of the hydraulic steel gate is suspended, the timer unit suspends timing, and the multimodal image acquisition module suspends acquiring images. When the hydraulic steel gate rises to the top or falls to the bottom, the timer unit stops timing and the multimodal image acquisition module stops acquiring images.

[0022] The current hydraulic steel gate is opened and closed by the gate opening and closing motor in the opening and closing room, and the opening and closing accuracy is difficult to control. Under the current circumstances, the on-site operators identify the position of the gate by naked eyes. The gate position reading unit is used to obtain the specific position of the current gate, which can avoid the situation where the image obtained by the multimodal image acquisition module has too many overlapping parts or the gate surface is missed; the timer unit is used to obtain the time required in the process of lifting or lowering the gate. When the gate starts to lower or rise, the timer starts timing, and the multimodal image acquisition module starts working and shooting. When the gate operation is suspended, the timer stops timing, and the multimodal image acquisition module stops shooting until the gate rises to the top or falls to the bottom, the timer stops timing, and the multimodal image acquisition module stops shooting. In this way, the state of the entire gate surface is photographed during the entire process of lifting or lowering the gate, and the corresponding time is also recorded.

[0023] In this embodiment, the image stitching and fusion module fuses the acquired RGB image and infrared image to obtain an image with obvious features, and stitches the fused images through an image stitching algorithm. During the stitching, the overlapping parts between the images are removed.

[0024] Among them, the image stitching and fusion module first stitches the fused image into multiple segmented images through the image stitching algorithm, and finally stitches the multiple segmented images again to form a final image.

[0025] Due to the large overall size of the hydraulic steel gate, the gate size of a large power station can reach 13 meters high and 11 meters wide, and the space on the shooting side is limited. The distance between the camera and the gate surface is small, usually not more than 1 meter, and the shooting field of view is limited. Therefore, a single shot cannot completely capture the entire gate surface, and can only capture a part of the gate. Therefore, the camera continuously captures multiple images during the process of lifting or lowering the gate, and through algorithm processing, the visible light and near-infrared images are fused to obtain an image with obvious features. The image splicing algorithm is then used to remove the overlapping parts between the images, and multiple images are spliced ​​to finally obtain a complete gate surface image. The acquired position and time can be used as prior knowledge of the image splicing and fusion module to accelerate the fusion process. In the image splicing stage, since there are a large number of overlapping parts between consecutive frames in the captured image, the algorithm needs to spend more time to remove these overlaps during the processing process. The parallel processing strategy is adopted in the present invention. First, multiple segmented images are spliced, and finally the above-mentioned segmented images are re-spliced ​​to form the final image.

[0026] In this embodiment, the defect recognition and positioning module includes a defect recognition unit and a positioning unit. The defect recognition unit is used to learn and extract defect features. The defect recognition unit uses a visual database for pre-training in the early stage of training. The positioning unit adopts an automatic defect positioning model method based on deep learning to automatically extract surface features of hydraulic steel gates and generate a thermal map. By clustering defect features, single defects and multiple defects on the surface of hydraulic steel gates are positioned. The position coordinates of the defects use the coordinate system on the final image as a reference.

[0027] The defect recognition and positioning module automatically analyzes the collected images based on deep learning and image processing technology, identifies various defects on the surface of the steel gate, such as rust, cracks, coating peeling, etc., and outputs the specific location information of the defects in the image. Specifically, the defect recognition and positioning module consists of two parts: one is the learning and extraction of defect features, and the other is the positioning of defects. Although the learning ability of convolutional neural networks is getting stronger and stronger, as its scale becomes larger and larger, it takes more and more time. Complex networks are not suitable for actual production. Therefore, a lightweight network is used to learn and extract defect features, and a visual database (ImageNet) is used for pre-training in the early stage of training. No large amount of data is required during training, which solves the problem of small sample data of gate surface defects. The automatic defect location model method (ADLM) based on deep learning is adopted. This method can automatically extract gate surface features and generate heat maps. By clustering defect features, the positioning function of single and multiple defects on the gate surface is realized. There is no need to manually locate the surface defects and range of the gate. It is only necessary to mark whether there are defects, which greatly reduces the initial labor costs. Once the defect is identified, the location coordinates of the defect use the image coordinate system in the upper left corner of the image as a reference. The identified defect position contains (x, y) coordinates.

[0028] The data output module outputs the identified defect information and its location data to a display device or storage device for reference and use by anti-corrosion operators.

[0029] Since the gate itself has a long design life and few defects, and the special location of the hydraulic steel gate of the hydropower station makes it difficult to collect samples of gate surface defects, so the number of defect samples that can actually be collected is small, resulting in this type of detection being a small sample detection. If there are not enough samples for network training, it is easy to cause the neural network to overfit, which is not conducive to improving the model detection performance.

[0030] Therefore, in this embodiment, a sample expansion module is also included. The function of the sample expansion module is to solve the problem of few samples and effectively expand the defect sample data by improving the existing generative adversarial network and combining the characteristics of gate surface defect samples. The stability of GAN and the quality of generated samples are improved by adding residual blocks, spectral norm regularization, replacing deconvolution layers, adding downsampling layers, and adding gradient penalty mechanisms.

[0031] Specifically, the purpose of adding residual blocks is to solve the gradient vanishing problem during deep neural network training.

[0032] Implementation method: Add residual blocks to the GAN network structure, and connect the input and output of the block through skip connections or identity mapping. This structure allows the gradient to flow directly back to the previous layer through skip connections during back propagation, thereby alleviating the gradient vanishing problem.

[0033] The purpose of spectral norm regularization is to solve the gradient anomaly problem and stabilize the GAN training process.

[0034] Implementation method: Apply spectral norm constraints to the weight matrix in GAN to limit the maximum singular value of the weight matrix, thereby controlling the amplitude of gradient change and avoiding gradient explosion or disappearance.

[0035] The purpose of replacing the deconvolution layer with upsampling plus convolution is to reduce artifacts when generating images and improve the quality of generated images.

[0036] Implementation: In the generator, upsampling is used, such as bilinear interpolation or nearest neighbor interpolation, to enlarge the feature map, and then a convolutional layer is applied for further processing. Compared with deconvolution, this method can better control the quality of the generated image and reduce problems such as uneven overlap.

[0037] The purpose of adding a downsampling layer to the discriminator is to reduce the amount of calculation and improve the efficiency of the discriminator.

[0038] Implementation method: Add a pooling layer (such as maximum pooling or average pooling) as a downsampling layer in the discriminator network structure to reduce the dimensionality of the input image. This can reduce the amount of calculation and increase the training speed while maintaining the discriminative ability.

[0039] The purpose of adding a gradient penalty mechanism based on the maximum and minimum regret method to the objective function is to limit the amplitude of gradient changes and further stabilize the training process of GAN.

[0040] Implementation: Introduce a gradient penalty term in the objective function of GAN, which is calculated based on the maximum-minimum regret method (or similar methods). By adjusting the weight of the penalty term, the gradient change amplitude of the discriminator near the real sample and the generated sample can be limited, thereby avoiding problems such as mode collapse.

[0041] Through a series of improvement measures, including adding residual blocks, spectral norm regularization, replacing deconvolution layers, adding downsampling layers, and adding gradient penalty mechanisms, the problems of small samples, gradient disappearance, gradient anomalies, and low quality of generated images encountered by GAN in the process of gate surface defect sample data expansion are effectively solved. These improvement measures not only improve the stability of GAN and the quality of generated samples, but also provide strong support for the effective expansion of defect sample data.

[0042] Embodiment 2: like Figure 2 As shown, a positioning method using the surface defect positioning system for on-site anti-corrosion operation of hydraulic steel gates comprises the following steps: First, take on-site photos or obtain previously taken surface images of the hydraulic steel gate, and look for images with defects. Mark and classify the images to determine whether there are defects and the specific types of defects, and train the defect recognition and positioning module.

[0043] Since images containing gate defects are currently difficult to obtain, a sample expansion module can be used to expand the samples of the sample images that have been obtained above, so as to obtain more sufficient samples to facilitate later training.

[0044] In actual use, the image acquisition module is deployed near the hydraulic steel gate to control the lifting or lowering of the hydraulic steel gate, and at the same time, the RGB image and infrared image of the surface of the hydraulic steel gate are collected until the image of the entire hydraulic steel gate surface is collected.

[0045] The collected images are then input into the image stitching and fusion module. First, the acquired RGB image and infrared image are fused one-to-one, and then the image stitching algorithm is used to remove the overlapping parts between the images, and multiple images are stitched into multiple segmented images. Then, the multiple segmented images are fused and stitched through the image stitching and fusion module to obtain a complete surface image of the hydraulic steel gate to form the final image.

[0046] The final image is passed to the defect recognition and positioning module for recognition and positioning processing.

[0047] The data output module outputs whether there are defects in the final image. If so, it displays the number of defects in the final image, the defect type, the detection confidence, and the position of the defect in the final image coordinate system.

Claims

1. A surface defect positioning system for on-site anti-corrosion work of hydraulic steel gates, characterized in that: include: A multimodal image acquisition module, which is used to capture RGB images and infrared images of the surface of the hydraulic steel gate; A gate position reading and timer module, which is used to control the start and stop of the multimodal image acquisition module according to the operating status and position of the hydraulic steel gate; An image stitching and fusion module, which is used to fuse the acquired RGB image and infrared image, and then stitch the fused images to form a final image of the surface of the hydraulic steel gate; A defect recognition and positioning module, which is used to analyze the final image, identify defects on the surface of the hydraulic steel gate, and output specific coordinate position information of the defects in the final image; A data output module is used to output the identified defect information and its location data to a display device and / or a storage device.

2. The surface defect positioning system for on-site anti-corrosion work of hydraulic steel gates according to claim 1 is characterized in that: It also includes a sample expansion module, which adds residual blocks, spectral norm regularization, replaces deconvolution layers, adds downsampling layers, and a gradient penalty mechanism on the basis of the generative adversarial network, and then combines the characteristics of surface defect samples of hydraulic steel gates to expand the defect sample data.

3. The surface defect positioning system for on-site anti-corrosion work of hydraulic steel gates according to claim 1 is characterized in that: The multimodal image acquisition module includes an RGB camera and a near-infrared camera, and the RGB camera and the near-infrared camera synchronously capture images of the surface of the hydraulic steel gate at the same position.

4. The surface defect positioning system for on-site anti-corrosion work of hydraulic steel gates according to claim 1 is characterized in that: The gate position reading and timer module includes a gate position reading unit and a timer unit. The gate position reading unit is used to obtain the specific position of the current hydraulic steel gate, and the timer unit is used to obtain the time required for the hydraulic steel gate to be lifted or lowered. When the hydraulic steel gate starts to descend or rise, the timer unit starts timing, and the multimodal image acquisition module starts to acquire images at the same time. When the operation of the hydraulic steel gate is suspended, the timer unit suspends timing, and the multimodal image acquisition module suspends acquiring images. When the hydraulic steel gate rises to the top or falls to the bottom, the timer unit stops timing and the multimodal image acquisition module stops acquiring images.

5. The surface defect positioning system for on-site anti-corrosion work of hydraulic steel gates according to claim 1 is characterized in that: The image stitching and fusion module performs a fusion process on the acquired RGB image and infrared image to obtain an image with obvious features, and stitches the fused images through an image stitching algorithm. During the stitching, the overlapping parts between the images are removed.

6. The surface defect positioning system for on-site anti-corrosion work of hydraulic steel gates according to claim 5 is characterized in that: The image stitching and fusion module first stitches the fused image into a plurality of segmented images through an image stitching algorithm, and finally stitches the plurality of segmented images again to form a final image.

7. The surface defect positioning system for on-site anti-corrosion work of hydraulic steel gates according to claim 1 is characterized in that: The defect recognition and positioning module includes a defect recognition unit and a positioning unit. The defect recognition unit is used to learn and extract defect features. The defect recognition unit uses a visual database for pre-training in the early stage of training. The positioning unit adopts an automatic defect positioning model method based on deep learning to automatically extract surface features of hydraulic steel gates and generate a thermal map. By clustering defect features, single defects and multiple defects on the surface of hydraulic steel gates are positioned. The position coordinates of the defects use the coordinate system on the final image as a reference.

8. A positioning method using the surface defect positioning system for on-site anti-corrosion operation of hydraulic steel gates according to any one of claims 1 to 7, comprising the following steps: By taking pictures on site or obtaining previously taken images of the surface of hydraulic steel gates, and finding images with defects, marking and classifying the images, determining whether there are defects and the specific types of defects, the defect recognition and positioning module is trained; The image acquisition module is deployed near the hydraulic steel gate to control the lifting or lowering of the hydraulic steel gate, and at the same time, the RGB image and infrared image of the surface of the hydraulic steel gate are collected until the image of the entire surface of the hydraulic steel gate is collected; The collected images are then input into the image stitching and fusion module. First, the acquired RGB images and infrared images are fused one-to-one, and then the image stitching algorithm is used to remove the overlapping parts between the images, and multiple images are stitched into multiple segmented images. Then, multiple segmented images are fused and spliced ​​through an image splicing and fusion module to obtain a complete surface image of the hydraulic steel gate and form a final image; The final image is transmitted to the defect recognition and positioning module for recognition and positioning processing; The data output module outputs whether there are defects in the final image. If so, it displays the number of defects in the final image, the defect type, the detection confidence, and the position of the defect in the final image coordinate system.

Images