A method for target detection of underwater foreign objects for power plants

Abstract

The present application belongs to the technical field of machine vision, and discloses a target detection method for underwater foreign matters of a power station. The method comprises the following steps: S1, constructing a two-dimensional and three-dimensional data set of underwater foreign matters of the power station; S2, constructing a two-dimensional prediction model and a three-dimensional prediction model, and training the two-dimensional prediction model and the three-dimensional prediction model by using the two-dimensional data set and the three-dimensional data set respectively; S3, collecting an image of an actual foreign matter, inputting the actual foreign matter image into the two-dimensional prediction model to obtain the approximate position and type of the actual foreign matter, collecting three-dimensional point cloud data according to the actual position of the foreign matter according to the predicted result, and inputting the collected three-dimensional point cloud data into the three-dimensional prediction model to obtain the accurate position and accurate type of the actual foreign matter. Through the present application, the problem of undetectable detection and low detection accuracy of underwater foreign matters of the power station is solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of machine vision, and more particularly relates to a target detection method for underwater foreign matters of a power station. BACKGROUND

[0002] The equipment and system of a power station operate in an underwater radiation environment, and the underwater foreign matters of the power station refer to articles that are inconsistent with the design of the system or equipment, such as fragments, broken or lost parts, tools, rags, chemicals, part abrasion slag and all articles that affect the operation of the system or equipment. In order to effectively prevent foreign matters from entering the equipment and system in the production and life of the power station and affecting the safe and stable operation of the equipment and unit, the prevention of foreign matters has always been the focus of attention in the overhaul work of the power station, especially the fuel loading and unloading operation. Before each core loading and refueling, the core foreign matter prevention inspection work must be performed to avoid the existence of foreign matters in the core and cause serious events such as fuel breakage after the core is started. In addition, after the completion of the refueling at overhaul, the identifiers of all fuel assemblies in the core need to be checked and recorded one by one by taking pictures to determine whether the core loading work is correct. Except for the preparation time, the process of checking and confirming 121 fuel assemblies one by one occupies about 1 hour of the main line time of the overhaul. How to quickly, comprehensively find, locate and identify foreign matters, and quickly perform the picture inspection after loading, can play a positive role in the safe and stable operation of the power station, the management of core foreign matter prevention and the shortening of the key path time.

[0003] In the traditional foreign matter detection method, Chen Wentao of Chongqing University (Chen Wentao. Tobacco foreign matter online high-speed pattern recognition and rejection technology research[D]. Chongqing University, 2003.) used three features of pixel color, pixel corrected gray, and first-order central moment (or standard deviation) of gray in the judgment unit to effectively identify tobacco and various foreign matters. Niu Buan of Nanjing University of Aeronautics and Astronautics (Niu Buan. Key technology research on airport runway video foreign matter detection and identification[D]. Nanjing University of Aeronautics and Astronautics, 2016.) installed a camera and information processing equipment on a patrol vehicle to form a mobile, multi-information fusion runway foreign matter video detection and identification system that can complete foreign matter detection and identification classification in a short time. Gai Zhengguo of China University of Mining and Technology (Gai Zhengguo. Key technology research on coal mine underground transportation foreign matter detection[D]. China University of Mining and Technology, 2018.) designed and developed a set of coal mine underground belt foreign matter detection prototype system to realize real-time detection, alarm, and image display of underground belt transportation foreign matters under multi-camera judgment. Wang Xingtao et al. (Wang Xingtao, Li Wenyong, Chen Liang. Intelligent detection method for hanging empty foreign matter in power inspection robot[J]. Machinery and Electronics, 2021, 39(12): 76-80.) proposed an intelligent detection method for hanging empty foreign matter in the circuit. Du Kai et al. (Du Kai, Zhu Guorong, Lu Jianghua, Pang Muye. Metal foreign matter detection method for electric vehicle wireless charging system[J]. Journal of Zhejiang University (Engineering Science), 2022, 56(01): 56-62+74.) proposed a method for detecting metal foreign matters in the electric vehicle wireless charging system. Traditional foreign matter detection methods have low robustness, and the detection accuracy is difficult to guarantee in the extreme underwater environment of power stations.

[0004] In the foreign matter detection method based on the deep learning method, Li Xingxin of Beijing Jiaotong University (Li Xingxin. Railway foreign matter intrusion detection algorithm based on scene understanding[D]. Beijing Jiaotong University, 2021.) proposed a blanking point and rail detection algorithm based on a deep multi-task convolutional network; Wang Wei of Beijing Jiaotong University (Wang Wei. Lightweight algorithm of deep neural network for railway foreign matter intrusion detection[D]. Beijing Jiaotong University, 2021.) studied the deep neural network algorithm for railway foreign matter intrusion detection; a sleeper foreign matter detection method based on a depth camera (Chinese patent authorization announcement number: CN202210561931, authorization announcement date: 2022.05.23) collects sleeper color images under different light position conditions through a camera, makes a sleeper dataset for training, and completes the foreign matter detection of the sleeper; Lu Xueming et al. (Lu Xueming, Yu ZaiChuan, Xu Shengqi. Coal mine belt foreign matter detection based on deep generative model[J]. Computer systems application, 2022, 31(05):358-363.) proposed a belt foreign matter detection method based on a deep generative model, which can accurately detect foreign matters on the coal mine belt conveyor; Zhang Hongmin et al. (Zhang Hongmin, Zhou Hao, Li ShunYuan, Li PingPing. Transmission line foreign matter detection method improved YOLOv3[J]. Laser Journal, 2022, 43(05):82-87.) used an improved YOLOv3 model to complete the detection of foreign matters on the transmission line. The performance of the deep learning model depends on the size of the dataset, and it is difficult to collect a large amount of underwater environment data in power stations, and a small dataset cannot fully exert the advantages of deep learning.

[0005] In summary, the traditional underwater detection method does not meet the detection requirements of the underwater foreign matter of the power station, and the method based on deep learning does not have an effective method for the subdivision field of underwater foreign matter detection of the power station. SUMMARY

[0006] In view of the above defects or improvement needs of the prior art, the present application provides a target detection method for underwater foreign matter of a power station, which solves the problems of undetectable and low detection precision of underwater foreign matter detection of a power station.

[0007] To achieve the above-mentioned purpose, according to the present application, a target detection method for underwater foreign matter of a power station is provided, which comprises the following steps:

[0008] S1, a dataset of underwater foreign matter of a power station is constructed, which includes a two-dimensional image dataset and a three-dimensional point cloud dataset, and a two-dimensional label and a three-dimensional label are respectively assigned to each two-dimensional image data and three-dimensional point cloud data in the dataset;

[0009] S2, a two-dimensional prediction model and a three-dimensional prediction model are constructed, and the two-dimensional prediction model and the three-dimensional prediction model are trained respectively by using the two-dimensional dataset and the three-dimensional dataset.

[0010] S3 collects an image of the actual foreign matter, inputs the actual foreign matter image into the two-dimensional prediction model, thereby obtaining the approximate location and type of the actual foreign matter, collects three-dimensional point cloud data from the location where the actual foreign matter is located according to the predicted result, and inputs the collected three-dimensional point cloud data into the three-dimensional prediction model, thereby obtaining the accurate location and accurate type of the actual foreign matter.

[0011] Further preferably, in step S1, the data set is constituted by collecting an actual data set and a simulated data set.

[0012] Further preferably, a CycleGAN model is used for domain adaptation for the simulated data set, so as to convert the simulated data into a style close to the actual data.

[0013] Further preferably, the loss function used by the CycleGAN model is a Wasserstein Loss.

[0014] Further preferably, the two-dimensional prediction model is a YOLOX model.

[0015] Further preferably, the loss function L in the YOLOX model is:

[0016] L = 1.5 * L cl + L c + L loc

[0017] wherein L cl , L c , and L loc are classification loss, confidence loss, and positioning loss, respectively.

[0018] Further preferably, in the training process of the YOLOX model, training is first performed on an ImageNet data set, and then formal training is performed using the two-dimensional data set.

[0019] Further preferably, the three-dimensional prediction model is a PointPillars model.

[0020] Further preferably, the position loss weight in the PointPillars model is 1.5.

[0021] Further preferably, in the PointPillars model, the coordinates of the three-dimensional point cloud data are first used as input and the normal vector is used as output to perform self-supervised learning training of the network feature extraction layer, and then formal training is performed using the data set.

[0022] Overall, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects:

[0023] 1. The improved CycleGAN model is used in the present application to convert the simulation data set into a style close to the actual data. By appropriately increasing the weight of the generator loss function in the CycleGAN model, the similarity between the generated data and the actual data can be improved. Through domain adaptation of the CycleGAN model, the amount of data required by the target detection model is greatly expanded, and the performance of the target detection model is greatly improved.

[0024] 2. In the present application, the pre-trained YOLOX model is used, and the data enhancement only uses the Mosaic method, no longer uses the MixUp enhancement, which can reduce the influence of the actual image quality blur on the model performance, and greatly improve the accuracy of the model detection by modifying the weight of the class loss in the loss function.

[0025] 3. In the present application, the PointPillars pre-training model obtained by self-supervised learning is used to improve the generalization performance of the model, and the CutPaste method is used for data enhancement to increase the number of targets and reduce the overfitting of the model. By modifying the position loss weight, the accuracy of position detection is enhanced.

[0026] 4. In the present application, a two-dimensional model is used for prediction first to fully utilize the advantage of fast two-dimensional detection speed to quickly find the rough position of foreign matter in the scene, and then a three-dimensional model is used for prediction to fully utilize the advantage of high-precision three-dimensional detection and avoid the weakness of low efficiency of three-dimensional detection, and obtain the accurate physical position of the foreign matter in the scene.

[0027] 5. In the present application, a synthetic data set is used to train the model. Due to the small possibility of actual foreign matter and the difficulty of collection, only a small amount of samples can be collected as an actual data set. In order to enrich the data set, the present application generates a simulation data set with a large number of samples by using simulation foreign matter and simulation scene. Considering that the simulation data set and the actual data set have large differences, the present application uses a domain adaptation method to convert the distribution of the simulation data set to the distribution of the actual data set, and mixes the converted simulation data set with the actual data set as a synthetic data set. The data set has sufficient and close-to-actual-scene samples, which can be used for model training and fully utilize the performance of the prediction model. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a flow chart of the target detection method for underwater foreign matter of a power plant constructed according to the preferred embodiment of the present application;

[0029] Figure 2 is a simulation foreign matter diagram constructed according to the preferred embodiment of the present application;

[0030] Figure 3 Foreign matter image collected by an industrial camera constructed according to a preferred embodiment of the present application;

[0031] Figure 4 Foreign matter point cloud collected by a planar array scanner constructed according to a preferred embodiment of the present application;

[0032] Figure 5 Underwater foreign matter image of an actual power plant constructed according to a preferred embodiment of the present application;

[0033] Figure 6 Improved CycleGAN model training graph constructed according to a preferred embodiment of the present application;

[0034] Figure 7 Improved YOLOX model training graph constructed according to a preferred embodiment of the present application;

[0035] Figure 8 Improved PointPillars model training graph constructed according to a preferred embodiment of the present application;

[0036] Figure 9 Detection result graph of an improved YOLOX model on a simulation image constructed according to a preferred embodiment of the present application;

[0037] Figure 10 Detection result graph of an improved PointPillars model on simulation point cloud data constructed according to a preferred embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0039] As shown in Figure 1 A target detection method for underwater foreign matter of a power plant, comprising the following steps:

[0040] S1: preparing each type of simulation foreign matter that may exist underwater of a power plant;

[0041] S2: collecting a large number of simulation foreign matter images by using an industrial camera, collecting a large number of point cloud data of simulation foreign matter by using a three-dimensional scanner, and making a simulation data set;

[0042] S3: Collect actual foreign object images in the actual power plant underwater environment using an industrial camera, and collect a large amount of point cloud data of simulated foreign objects in the actual power plant underwater environment using a three-dimensional scanner to make an actual data set;

[0043] S4: Convert the simulated data set into a pseudo-actual data set using the improved CycleGAN model, and mix the pseudo-actual data set with the actual data set into a synthetic data set;

[0044] S5: Train the improved YOLOX model on the image data of the synthetic data set;

[0045] S6: Train the improved PointPillars model on the point cloud data of the synthetic data set;

[0046] S7: Detect the actual foreign object images using the trained YOLOX model to obtain the rough location and type of the foreign object;

[0047] S8: Detect the actual foreign object point cloud data using the trained PointPillars model to obtain the accurate location of the foreign object;

[0048] Further, the foreign objects that may exist in the power plant underwater in step S1 include: bolts, nuts, cable ties, sponges, adhesive tapes, etc., and a large number of simulated foreign objects of different models and sizes need to be prepared in advance.

[0049] Further, in step S2, a large number of simulated foreign object images prepared in step S1 are collected using an RGB industrial camera, and 2D annotation boxes of bolts, nuts, cable ties, sponges, adhesive tapes, etc. are obtained through manual annotation; a large number of simulated foreign object point cloud data prepared in step S1 are collected using a surface array scanner, and 3D annotation boxes of bolts, nuts, cable ties, sponges, adhesive tapes, etc. are obtained through manual annotation to make a simulated data set.

[0050] Further, in step S3, actual foreign object images in the actual power plant underwater environment are collected using an RGB industrial camera, and 2D annotation boxes of bolts, nuts, cable ties, sponges, adhesive tapes, etc. are obtained through manual annotation; actual foreign object point cloud data in the actual power plant underwater environment are collected using a surface array scanner, and 3D annotation boxes of bolts, nuts, cable ties, sponges, adhesive tapes, etc. are obtained through manual annotation to make an actual data set.

[0051] Further, in step S4, the improved CycleGAN model is used to realize domain adaptation, convert the simulated data set into a style close to the actual data, i.e. a pseudo-actual data set, and then mix the pseudo-actual data set with the actual data set into a synthetic data set.

[0052] Furthermore, the improved CycleGAN model in step S4 is as follows: the simulation dataset obtained in S2 and the actual dataset obtained in S3 are used as two input domains. The purpose is to train a generator that transforms the simulation dataset into the actual dataset. In order to improve the stability of model convergence in the early stage of training, Wasserstein Loss is chosen as the loss function.

[0053] Furthermore, in step S5, the improved YOLOX model is trained on the synthetic dataset images obtained in step S4 to obtain the trained YOLOX model.

[0054] Furthermore, the improved YOLOX model in step S5 is as follows: First, the weight parameters of the YOLOX model pre-trained on ImageNet are transferred to the current model to enhance the model's generalization performance; second, considering that the background of the foreign object image is close and the quality is poor, data augmentation is only performed using the Mosaic method, and MixUp augmentation is no longer used; to prevent data augmentation from affecting the model's understanding of the original data distribution, data augmentation is canceled in the last 15 epochs of training; in the loss function, in order to enhance the accuracy of class detection, the class loss weight is increased from 1.0 to 1.5.

[0055] Furthermore, in step S6, the improved PointPillars model is trained on the synthetic dataset point cloud data obtained in step S4 to obtain the trained PointPillars model.

[0056] Furthermore, in step S6, the improved PointPillars model is used as follows: First, the coordinates of the original point cloud data are used as input and the normal vector is used as output to perform self-supervised learning training of the network feature extraction layer, giving the network layer a better initial weight; considering that the number of foreign objects in the dataset is small, the CutPaste method is used for data augmentation to increase the number of targets; to prevent data augmentation from affecting the model's understanding of the original data distribution, data augmentation is canceled in the last 15 epochs of training; in the loss function, in order to enhance the accuracy of position detection, the position loss weight is increased from 1.0 to 1.5.

[0057] The present invention will be further described below with reference to specific embodiments.

[0058] (1) Prepare simulated foreign objects that may exist underwater at the power station.

[0059] Potential foreign objects (FOOs) in the underwater power plant area include bolts, nuts, cable ties, sponges, and tape. A large quantity of simulated FEOs of different models and sizes needs to be prepared in advance. For example, cable ties... Figure 2 As shown.

[0060] (2) A large number of simulated foreign object images were acquired using industrial cameras to create a simulation dataset.

[0061] An RGB industrial camera was used to acquire images of prepared simulated foreign objects. 2D bounding boxes for various foreign objects, such as bolts, nuts, cable ties, sponges, and tape, were obtained through manual annotation. A large amount of point cloud data of the simulated foreign objects was acquired using an area scan scanner, and 3D bounding boxes for the same objects were obtained through manual annotation. This data was then used to create a simulation dataset. The simulated foreign object images acquired by the industrial camera are shown below. Figure 3 As shown, the simulated foreign object point cloud data acquired by the area array scanner is as follows: Figure 4 As shown.

[0062] (3) Use industrial cameras to collect images of foreign objects in the actual underwater environment of a power plant and create a real dataset.

[0063] RGB industrial cameras were used to acquire images of actual foreign objects in the underwater environment of a power plant. 2D bounding boxes for various foreign objects, such as bolts, nuts, cable ties, sponges, and tape, were obtained through manual annotation. A surface scan scanner was used to acquire point cloud data of the actual foreign objects in the underwater environment of a power plant. 3D bounding boxes for the same objects were then obtained through manual annotation, creating a dataset. Examples of actual underwater foreign object images from a power plant include... Figure 5 As shown.

[0064] (4) The simulation dataset is converted into a pseudo-real dataset using the improved CycleGAN model, and the pseudo-real dataset is mixed with the real dataset to form a synthetic dataset.

[0065] An improved CycleGAN model is used to achieve domain adaptation by transforming the simulated dataset into a pseudo-realistic dataset that closely resembles real-world data. This pseudo-realistic dataset is then blended with the real dataset to create a synthetic dataset. Using both the simulated and real datasets as input domains aims to train a generator that transforms the simulated dataset into the real-world dataset. To improve the model's convergence stability in the early stages of training, Wasserstein Loss is chosen as the loss function. A training diagram of the improved CycleGAN model is shown below. Figure 6 As shown. Wasserstein Loss is:

[0066] W(P r ,P g ) = sup fL≤1 E x～Pr [f(x)]-E y～Pg [f(y)]

[0067] Among them, P r ,P gare the distribution functions of the source domain and the target domain, respectively.

[0068] (5) Training the improved YOLOX model on the synthetic dataset images

[0069] The improved YOLOX model is trained on the obtained synthetic dataset to obtain the trained YOLOX model. In step S5, the improved YOLOX model is as follows: first, the pre-trained YOLOX model weight parameters on ImageNet are migrated to the current model to enhance the generalization performance of the model; second, during the training process, considering that the foreign object images have similar backgrounds and poor quality, only the Mosaic method is used for data enhancement, and the MixUp enhancement is no longer used; to prevent data enhancement from affecting the understanding of the model on the original data distribution, data enhancement is canceled for the last 15 epochs of training; in the loss function, in order to enhance the accuracy of class detection, the class loss weight is increased from 1.0 to 1.5. The training flowchart of the improved YOLOX model is shown in Figure 7 The loss function of the improved YOLOX model is as follows:

[0070] L = 1.5 * L cl + L c + L loc

[0071] wherein, L cl , L c , and L loc are the classification loss, the confidence loss, and the positioning loss, respectively.

[0072] (6) Training the improved PointPillars model on the synthetic dataset point cloud data

[0073] The improved PointPillars model is trained on the synthetic dataset point cloud data to obtain the trained PointPillars model. The improved PointPillars model is as follows: first, the coordinates of the original point cloud data are used as input, and the normal vector is used as output for self-supervised learning training of the network feature extraction layer, so that a better initial weight is trained for the network layer; considering that the number of foreign object targets in the dataset is small, the CutPaste method is used for data enhancement to increase the number of targets; to prevent data enhancement from affecting the understanding of the model on the original data distribution, data enhancement is canceled for the last 15 epochs of training; in the loss function, in order to enhance the accuracy of position detection, the position loss weight is increased from 1.0 to 1.5. The training flowchart of the improved PointPillars model is shown in Figure 8 The loss function of the improved PointPillars model is as follows:

[0074] L = L cl+L c +1.5*L loc

[0075] wherein, L cl ,L c ,L loc are classification loss, confidence loss and positioning loss respectively.

[0076] (7) The trained YOLOX model is used to detect the actual foreign object image to obtain the position and type of the foreign object

[0077] The trained YOLOX model is used to detect the actual foreign object image to obtain the position and type of the foreign object such as bolt, nut, cable tie, sponge and adhesive tape, and the results are shown in Table 6. Figure 9

[0078] (8) The trained PointPillars model is used to detect the actual foreign object point cloud data to obtain the position and type of the foreign object

[0079] The trained YOLOX model is used to detect the actual foreign object image to obtain the position and type of the foreign object such as bolt, nut, cable tie, sponge and adhesive tape, and the results are shown in Table 6. Figure 10

[0080] It is easy for those skilled in the art to understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.​​

Claims

1. A method for target detection of underwater foreign objects of a power plant, characterized in that, The method comprises the following steps: S1, constructing a data set of underwater foreign matters of a power station, the data set comprising a two-dimensional image data set and a three-dimensional point cloud data set, and a two-dimensional label and a three-dimensional label are respectively assigned to each two-dimensional image data and three-dimensional point cloud data in the data set; S2, constructing a two-dimensional prediction model and a three-dimensional prediction model, and training the two-dimensional prediction model and the three-dimensional prediction model by using the two-dimensional data set and the three-dimensional data set respectively; S3, collecting an image of an actual foreign matter, inputting the actual foreign matter image into the two-dimensional prediction model to obtain a rough position and a type of the actual foreign matter, collecting three-dimensional point cloud data according to the predicted result to the position of the actual foreign matter, and inputting the collected three-dimensional point cloud data into the three-dimensional prediction model to obtain an accurate position and an accurate type of the actual foreign matter; The two-dimensional prediction model is a YOLOX model; The loss function in the YOLOX model L is: wherein, are the classification loss, the confidence loss and the localization loss, respectively. The three-dimensional prediction model is a PointPillars model; The position loss weight in the PointPillars model is 1.

5.

2. A method for target detection of underwater foreign objects for a power plant according to claim 1, characterized in that, In step S1, the data set is composed of an actual data set and a simulated data set.

3. A method for target detection of underwater foreign objects of a power plant according to claim 2, characterized in that, The CycleGAN model is used to realize domain adaptation for the simulated data set, so as to convert the simulated data into a style close to the actual data.

4. A method for target detection of underwater foreign objects of a power plant according to claim 3, characterized in that, The loss function used in the CycleGAN model is a Wasserstein Loss.

5. A method for target detection of underwater foreign objects of a power plant according to claim 1, characterized in that, In the training process of the YOLOX model, training is first performed on an ImageNet data set, and then formal training is performed by using the two-dimensional data set.

6. A method for target detection of underwater foreign objects for a power plant according to claim 1, characterized in that, In the PointPillars model, the coordinates of the three-dimensional point cloud data are first used as input and the normal vector is used as output to perform self-supervised learning training of a network feature extraction layer, and then formal training is performed by using the data set.

Images