A large closed space defect identification method, device and medium

Abstract

The embodiment of the application discloses a large closed space defect identification method, device and medium, and belongs to the technical field of overhauling. The method comprises the following steps: collecting a visible light image and a polarized light image of a large closed space; outputting a plurality of to-be-fused visible light image features at a plurality of predetermined layers of a multi-layer convolutional neural network by using the multi-layer convolutional neural network; obtaining a plurality of to-be-fused polarized light image features by wavelet decomposing the polarized light image; fusing the to-be-fused visible light image features and the to-be-fused polarized light image features corresponding to the to-be-fused visible light image features to obtain a plurality of to-be-identified fusion image features; and identifying defects of the to-be-identified fusion image features by using a defect identification neural network model. The embodiment of the application can automatically obtain complete defect information of a large closed space, and further discover safety hazards in time, thereby guaranteeing the safe operation of a boiler, a chemical industry and other industrial processes.

Description

Technical Field

[0001] This invention relates to the field of maintenance technology for large enclosed spaces, and in particular to a method, device and medium for identifying defects in large enclosed spaces. Background Technology

[0002] To ensure the safe and stable operation of equipment in large, enclosed spaces, such as boilers, and to reduce the workload of personnel and the safety risks of working at heights, existing technologies are increasingly employing intelligent devices such as wall-climbing inspection robots or drones for large, enclosed space inspections to replace manual labor in identifying defects on the surfaces of industrial equipment. However, large industrial enclosed spaces often have poor lighting conditions. Darkness, dust, and other factors prevent visual inspection systems from capturing clear and usable images. Summary of the Invention

[0003] This invention provides a method, device, and medium for identifying defects in large enclosed spaces. By using a neural network combined with visible light and polarized light images of the large enclosed space for defect identification, more defect information can be automatically acquired, thereby timely detection of safety hazards and ensuring the safe operation of industrial equipment.

[0004] In a first aspect, embodiments of the present invention provide a method for defect identification in a large enclosed space, comprising: acquiring visible light images and polarized light images of a predetermined detection area of ​​the large enclosed space to obtain visible light images and polarized light images of the predetermined detection area; extracting features from the visible light images using a multi-layer convolutional neural network and outputting them at multiple predetermined layers of the multi-layer convolutional neural network to obtain multiple visible light image features to be fused; performing wavelet decomposition on the polarized light images and extracting and processing multiple polarized light image features of different bands that correspond one-to-one with each visible light image feature to be fused; fusing each visible light image feature to be fused with its corresponding polarized light image feature to be fused to obtain multiple fused image features to be identified in the predetermined detection area; and using a defect identification neural network model to identify defects in the multiple fused image features to be identified to obtain defect information of the predetermined detection area.

[0005] Secondly, embodiments of the present invention provide a large-scale enclosed space defect identification device, comprising: a target image acquisition module, used to acquire visible light images and polarized light images of a predetermined detection area of ​​a large enclosed space to obtain visible light images and polarized light images of the predetermined detection area; a first target image feature extraction module, used to extract features from the visible light images using a multi-layer convolutional neural network and output them at multiple predetermined layers of the multi-layer convolutional neural network to obtain multiple visible light image features to be fused; a second target image feature extraction module, used to perform wavelet decomposition on the polarized light images and extract and process multiple polarized light image features of different bands that correspond one-to-one with each target visible light image feature to be fused; a target image feature fusion module, used to fuse each target visible light image feature to be fused with the corresponding target polarized light image feature to be fused to obtain multiple target fused image features of the predetermined detection area; and an identification module, used to identify defects in the multiple target fused image features using a defect identification neural network model to obtain defect information of the predetermined detection area.

[0006] Thirdly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for identifying defects in large enclosed spaces as described in any of the embodiments of the present invention.

[0007] In this embodiment of the invention, a convolutional neural network can be used to output multiple visible light image features of a large enclosed space in multiple predetermined layers, and then fuse them with multiple polarized light image features of different bands obtained by polarized light wavelet decomposition. Subsequently, a neural network model is used to identify defects in the fused image features, so that the visible light image features and the polarized light image features complement each other in different dimensions, thereby automatically obtaining relatively complete defect information of a large enclosed space, and thus timely discovering safety hazards and ensuring the safe operation of industrial equipment. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating a method for identifying defects in a large enclosed space according to an embodiment of the present invention.

[0009] Figure 2 This is a flowchart illustrating a method for identifying defects in a large enclosed space, as provided in this invention.

[0010] Figure 3 This is a schematic diagram illustrating the fusion of polarized light image features and polarized light image features in a method for identifying defects in a large enclosed space provided by an embodiment of the present invention.

[0011] Figure 4 This is a schematic diagram of the model training process in a method for identifying defects in a large enclosed space provided in an embodiment of the present invention;

[0012] Figure 5 This is a schematic diagram of a large-scale enclosed space defect identification device provided in an embodiment of the present invention. Detailed Implementation

[0013] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0014] Wear and tear on large industrial equipment can lead to serious safety accidents. For example, thermal power plants need to start and stop quickly to adapt to the peak-shaving requirements of new power systems, resulting in large load fluctuations that make boilers prone to wear and tear. Therefore, the safety of large industrial equipment must be given high priority. To ensure the safe and stable operation of large industrial equipment, and to reduce the workload of personnel in confined spaces and the safety risks of working at heights, existing technologies are increasingly using intelligent equipment such as large confined space wall-climbing inspection robots or inspection drones to replace manual labor for defect identification on the surface of industrial equipment. However, in large industrial confined spaces, lighting conditions are poor. Darkness, dust, and other factors prevent visual inspection systems from acquiring clear and usable images.

[0015] Currently available visual inspection products cannot meet the recognition accuracy requirements of large-scale industrial production lines. This is mainly due to the following two reasons:

[0016] Large industrial enclosed spaces often have poor lighting conditions. Darkness, dust, and other factors prevent visual inspection systems from capturing clear and usable images. Furthermore, fixed and mobile supplemental lighting devices based on visible light cannot guarantee consistent illumination in terms of intensity and wavelength within the robot's field of vision, severely impacting the accuracy of defect identification.

[0017] Images captured under visible light lose much of their information compared to polarized light images. Therefore, how to fuse the information carried by polarized light images of defective surfaces with that of general visible light images to increase the amount of information obtained by defect detection algorithms is a pressing research topic in the field of industrial inspection.

[0018] To address the issues of illumination and information loss in defect detection within large, enclosed spaces, it is necessary to introduce polarized light sources and propose a defect identification method that fuses polarized imaging with visible light imaging. This allows for the reflection of information from multiple original images, leading to a more accurate and comprehensive analysis and judgment of the target and scene.

[0019] This invention provides a method, device, and medium for defect identification in large enclosed spaces. It utilizes a convolutional neural network combined with visible light and polarized light images of large enclosed spaces for defect identification and introduces an adaptive supplementary lighting mechanism to automatically acquire more defect information, thereby promptly detecting safety hazards and ensuring the safe operation of industrial equipment in large enclosed spaces.

[0020] Figure 1 This is a schematic diagram illustrating an application of a method for identifying defects in large enclosed spaces provided by an embodiment of the present invention on a boiler water-cooled wall. This method can be executed by a device for identifying defects in large enclosed spaces provided by an embodiment of the present invention, which can be implemented using software and / or hardware. In a specific embodiment, the device can be integrated into an electronic device, such as a computer or server. The following embodiments will illustrate this using the integration of the device into an electronic device as an example. (Reference) Figure 1 The method may specifically include the following steps:

[0021] exist Figure 1 In an embodiment of the method for identifying defects in a large enclosed space, the method includes: Step 101: acquiring visible light images and polarized light images of a predetermined detection area of ​​an industrial equipment, taking a power plant boiler as an example, to obtain visible light images and polarized light images of the predetermined detection area; Step 102: extracting features from the visible light images using a multi-layer convolutional neural network and outputting them at multiple predetermined layers of the multi-layer convolutional neural network to obtain multiple visible light image features to be fused; Step 103: performing wavelet decomposition on the polarized light images and extracting multiple polarized light image features of different bands that correspond one-to-one with each visible light image feature to be fused; Step 104: fusing each visible light image feature to be fused with its corresponding polarized light image feature to be fused to obtain multiple fused image features to be identified in the predetermined detection area; and Step 105: using a defect identification neural network model to identify defects in the multiple fused image features to be identified to obtain defect information of the predetermined detection area.

[0022] The method in this embodiment can utilize a convolutional neural network to output multiple visible light image features of the boiler water-cooled wall at multiple predetermined layers, and fuse them with multiple polarized light image features of different bands obtained by polarized light wavelet decomposition. Then, the neural network model is used to identify defects in the fused image features, so that the visible light image features and the polarized light image features complement each other in different dimensions, thereby automatically acquiring more defect information of the boiler water-cooled wall, and thus timely discovering safety hazards and ensuring the safe operation of the boiler.

[0023] Step 101 describes the process of acquiring visible light images and polarized light images of a predetermined detection area of ​​the boiler water-cooled wall to obtain visible light images and polarized light images of the predetermined detection area. This process helps to extract and fuse the features of the two types of images, avoiding the difference between image information and real information caused by the objective environment, thereby avoiding the omission of boiler defect information.

[0024] Specifically, during the imaging process of polarized light images, the polarization state of electromagnetic waves changes according to the characteristics of the surface material when the target reflects and radiates electromagnetic waves. Therefore, by utilizing the polarization state information of light waves reflected or radiated by the material, it is possible to effectively distinguish different materials, different surface morphologies, and scatterers with different conductivity. It is also possible to detect birefringence, stress, surface roughness, and physical properties that cannot be detected by conventional imaging. Therefore, polarized light images can effectively supplement the information loss caused by objective reasons in visible light images.

[0025] Optionally, the above-mentioned visible light and polarized light images can be acquired using intelligent devices such as wall-climbing inspection robots or inspection drones.

[0026] In an optional specific embodiment of this application, the process of acquiring visible light images of a predetermined detection area of ​​a boiler water-cooled wall includes: performing visible light adaptive supplementary lighting on the predetermined detection area based on a preset standard light intensity and the visible light intensity of the predetermined detection area detected in real time by a light intensity sensor, so that the visible light intensity of the predetermined detection area is consistent.

[0027] In an optional embodiment of this application, a light intensity sensor measures the visible light intensity. If the measured visible light intensity is consistent with the standard light intensity, the visible light intensity is acquired. Otherwise, the visible light source intensity adjustment amount is calculated, and the visible light source is adjusted according to the visible light source intensity adjustment amount until the measured visible light intensity is consistent with the standard light intensity, and then the acquisition of a visible light image begins.

[0028] In an optional specific embodiment of this application, the process of acquiring polarized light images of a predetermined detection area of ​​a boiler water-cooled wall includes: adaptively adjusting the polarization degree and polarization angle of the polarized light based on a preset standard polarization degree and standard polarization angle, and the light intensity of the polarized light passing through a polarizer placed in the direction perpendicular to the light propagation, as detected in real time by a light intensity sensor, so that the frequency domain characteristics of the polarized light in the monitoring area are consistent.

[0029] In an optional specific embodiment of this application, a light intensity sensor measures the intensity of polarized light and calculates the polarization angle and degree of polarization based on the intensity of polarized light. If the polarization angle and degree of polarization are inconsistent with the standard polarization angle and degree of polarization, the adjusted polarization angle and degree of polarization are calculated, and the polarized light source is adjusted until the measured polarization angle and the adjusted degree of polarization are consistent with the standard polarization angle and degree of polarization.

[0030] Specifically, existing fixed and mobile supplementary lighting technologies cannot guarantee the consistency of illumination in terms of light intensity and wavelength in the shooting field of view environment, which seriously affects the accuracy of defect identification based on images in the later stage. However, the adaptive supplementary lighting mechanism can maintain the illumination consistency of the detection area through adaptive correction of polarized light source and visible light source in each frame, so that the background light intensity and polarization degree of the intelligent camera device are consistent during the shooting process, reducing the impact of background information on defect identification.

[0031] In an optional specific example of this application, the light intensity of the polarized light passing through the polarizer placed in the direction of vertical light propagation includes the light intensity behind linear polarizers placed in the direction of vertical light propagation with transmission axis directions of 0°, 90°, +45°, and -45°, respectively, and the light intensity behind right-handed and left-handed circular polarizers placed in the direction of vertical light propagation.

[0032] In an optional specific example of this application, the light intensity of the polarized light passing through the polarizer placed in the direction of vertical light propagation includes the light intensity of the polarizers placed in the direction of vertical light propagation with the light transmission axis directions being 0°, 90°, and 45° respectively, as well as the total light intensity.

[0033] In optional specific examples of this application, such as Figure 2 As shown, at the start of a round of detection, the standard polarization degree P(0), polarization angle θ0, and standard light intensity I(0) of the visible light image are first set.

[0034] In the visible light acquisition stage, the visible light source is first adaptively adjusted. After the visible light supplementary illumination is turned on, the light intensity I(a) of the detection area is obtained by the light intensity sensor, and the visible light source light intensity adjustment amount Δ(I) = I(0) - I(a) is calculated. After the light intensity adjustment, the background light intensity of the detection area is consistent, and then the visible light image is acquired.

[0035] During the polarized light image acquisition phase, the polarization light source is first adaptively adjusted. By calculating the degree of polarization and the polarization angle, the degree of polarization and light intensity of the polarized light image are kept constant during robot movement, reducing the influence of background information on defect identification.

[0036] After the polarized light source is turned on for supplemental lighting, the light intensity I of the detection area is obtained by the light intensity sensor. X I Y I +45° I -45° I τ , Where I X I Y I +45° I -45°They respectively represent the light intensities after passing through a linear polarizer placed in the vertical light propagation direction, and the transmission axis directions of the polarizer are X, Y, +45°, and -45° respectively. I τ , They respectively represent the light intensities after passing through a right-handed (τ) and a left-handed circular polarizer.

[0037] The degree of polarization P and the polarization angle θ are calculated as follows:

[0038] The degree of polarization and the polarization angle representing the polarization characteristics can be obtained from the parameters S (S0, S1, S2, S3).

[0039] The following gives the calculation process of the degree of polarization and the polarization angle expressed by light intensities:

[0040] S0 = I X + I Y

[0041] S1 = I X - I Y

[0042] S2 = I +45° - I -45°

[0043]

[0044] In the formula, S0 represents the total incident light intensity.

[0045] S1 represents the difference in light intensities between the x-component and the y-component.

[0046] S2 represents the difference in light intensities between the +45° and -45° polarization components.

[0047] S3 represents the difference in light intensities between the right-handed and left-handed circular polarization components.

[0048] Degree of polarization: The ratio of the energy of polarized light to the total light energy.

[0049]

[0050] Polarization angle: The angle θ between the major axis of the ellipse and the axis of the traditional coordinate system:

[0051]

[0052] Here, the degree of polarization P is a dimensionless number between 0 and 1. When P = 0, it means the light is unpolarized light; when P = 1, it means the light is fully polarized light; when 0 < P < 1, it means the light is partially polarized light. The polarization angle θ represents the angle between the polarization direction of the incident light and the x-axis.

[0053] Calculate the polarization degree adjustment Δ(P) = P(0) - P and the polarization angle adjustment Δ(θ) = θ0 - θ. After adjusting the polarization degree and polarization angle, the frequency domain characteristics of the polarized light in the detection area are consistent, and then a polarized light image is acquired.

[0054] Step 102 represents the process of extracting features from a visible light image using a multi-layer convolutional neural network and outputting the results in multiple predetermined layers of the multi-layer convolutional neural network to obtain multiple visible light image features to be fused. This process can obtain visible light image features of different dimensions, separate information of different dimensions from the original visible light image, form a relatively independent feature matrix, which helps to reduce information confusion, facilitates fusion with corresponding biased light image features in the later stage, and improves the accuracy of neural network recognition.

[0055] In an optional embodiment of this application, the output can be performed at two predetermined layers of the above-mentioned multi-layer convolutional neural network to obtain low-dimensional visible light image features and high-dimensional image features of the visible light image.

[0056] In an optional specific embodiment of this application, the output can be obtained from the three predetermined layers of the above-mentioned multi-layer convolutional neural network to obtain the low-dimensional visible light image features to be fused, the medium-dimensional visible light image features to be fused, and the high-dimensional visible light image features to be fused.

[0057] In an optional specific embodiment of this application, the above-mentioned multi-layer convolutional neural network is an N-layer convolutional neural network. The process of outputting multiple predetermined layers of the multi-layer convolutional neural network to obtain multiple visible light image features to be fused includes: taking the output at the |(1 / 3)N|, |(2 / 3)N|, and N-1 layers of the multi-layer convolutional neural network respectively to obtain the low-dimensional visible light image features to be fused, the medium-dimensional visible light image features to be fused, and the high-dimensional visible light image features to be fused, where N is an integer greater than 3.

[0058] Specifically, features are extracted from visible light images in three dimensions: low, medium, and high. This separates the information from the original visible image into three relatively independent feature matrices, which reduces information confusion, facilitates subsequent fusion with polarized light image features, and improves the accuracy of neural network recognition.

[0059] Step 103 represents the process of performing wavelet decomposition on the polarized light image and extracting multiple polarized light image features of different bands that correspond one-to-one with each visible light image feature to be fused. This process can obtain polarized image features of different bands, separate information of different dimensions from the original polarized light image, form a relatively independent feature matrix, which helps to reduce information confusion, facilitates subsequent fusion with the corresponding visible light image features, and improves the accuracy of neural network recognition.

[0060] In an optional specific embodiment of this application, the process of performing wavelet decomposition on the polarized light image and extracting multiple polarized light image features of different bands that correspond one-to-one with each visible light image feature to be fused includes: performing wavelet decomposition on the polarized image and extracting high-frequency polarized image features of the short-wave band corresponding to the low-dimensional visible image features to be fused, mid-frequency polarized image features of the mid-wave band corresponding to the mid-dimensional visible image features to be fused, and low-frequency polarized image features of the long-wave band corresponding to the high-dimensional visible light image features to be fused.

[0061] Specifically, wavelet decomposition is performed on the polarized light image to extract polarization images in three bands: short wave, medium wave, and long wave, corresponding to the high, medium, and low frequency features of the polarized light image, respectively. This allows for the separation of information in these three dimensions from the original polarized image, forming three relatively independent feature matrices. This reduces information clutter, facilitates subsequent fusion with polarized light image features, and improves the accuracy of neural network recognition.

[0062] Optionally, wavelet decomposition can be performed on the polarization image to extract high-frequency polarization image features in the short-wave band and low-frequency polarization image features in the long-wave band.

[0063] Optionally, the visible light image features and polarization image features mentioned above can also be: low-frequency polarization image features corresponding to the low-dimensional visible image features to be fused, mid-frequency polarization image features corresponding to the mid-dimensional visible image features to be fused, and high-frequency polarization image features corresponding to the high-dimensional visible light image features to be fused.

[0064] In an optional specific embodiment of this application, the process of performing wavelet decomposition on the polarized light image and extracting multiple polarized light image features of different bands that correspond one-to-one with each visible light image feature to be fused includes: using a convolutional pooling method to adjust the size of the high-frequency polarized image features, mid-frequency polarized image features, and low-frequency polarized image features to correspond with the size of the low-dimensional visible light image features to be fused, mid-dimensional visible light image features to be fused, and high-dimensional visible light image features to be fused, thereby obtaining the high-frequency polarized image features to be fused, the mid-frequency polarized image features to be fused, and the low-frequency polarized image features to be fused.

[0065] Specifically, convolution and pooling methods are used to adjust the image size to match the size of the visible light image features, which facilitates subsequent fusion.

[0066] In optional specific examples of this application, such as Figure 2 The model shown is constructed by concatenating N layers of convolutional neural networks to extract features from visible light images. The outputs of the layers |(1 / 3)N|, |(2 / 3)N|, and (N-1)th layers are taken as the low-dimensional, mid-dimensional, and high-dimensional features of the visible light image, respectively: Yl Y m Y h .

[0067] Wavelet decomposition was performed on the polarization image to extract polarization images for the short, medium, and long wavebands, corresponding to the high, medium, and low frequencies of the polarized light image, respectively. After resizing, the high, medium, and low frequency features of the polarized light image were obtained: X h X m X l .

[0068] Step 104 represents the process of fusing each visible light image feature to be fused with the corresponding polarized light image feature to be fused to obtain multiple fused image features to be identified in the predetermined detection area. The corresponding image features of the visible light image and the polarized light image in different dimensions are fused, so that the visible light image features and the polarized light image features complement each other, avoid the loss of defect information in the image, and facilitate the subsequent identification of defects by the neural network.

[0069] In an optional specific embodiment of this application, the process of fusing each visible light image feature to be fused with the corresponding polarized light image feature to be fused includes: channel superposition of the visible light image feature to be fused with the corresponding polarized light image to be fused.

[0070] Specifically, without changing the length and width of the features, only increasing the number of channels of the features, the polarization image features in the high, medium and low frequency ranges are horizontally superimposed with the visible light image features in the low, medium and high dimensions, respectively, to obtain the edge representation, texture representation and semantic representation of the image. Regularizing the information of the image in the three dimensions of edge, texture and semantics can help reduce information confusion and improve the accuracy of neural network recognition.

[0071] In an optional instance of this application, such as Figure 2 and Figure 3 As shown:

[0072] High-frequency polarized light image features Xh to be fused and low-dimensional visible light image features Y to be fused l The edges are fused to obtain the edge representation Z1 of the image:

[0073] Z1 = concat axis＝channels (X h Y l )

[0074] Similarly, the mid-frequency polarized light image features X to be fused m With the mid-dimensional visible light image features Y to be fused m The fusion yields texture information Z2 used to characterize the image:

[0075] Z2 = concataxis＝channels (X m Y m )

[0076] Similarly, the low-frequency polarized light image features X to be fused l With the high-dimensional visible light image features Y to be fused h The fusion yields semantic information Z3 used to characterize the image:

[0077] Z3 = concat axis＝channels (X l Y h )

[0078] Step 105 represents the process of using a defect recognition neural network model to identify defects in multiple fused image features to obtain defect information in a predetermined detection area. This process can identify fused image features in three dimensions: visible light image features, polarized light image features, etc., and ultimately automatically identify more defects, avoiding omissions, thereby timely detection of safety hazards and ensuring the safe operation of the boiler.

[0079] In an optional embodiment of this application, the above-mentioned defect recognition neural network model can be a defect recognition neural network model in the prior art.

[0080] In optional embodiments of this application, multiple defect recognition neural networks can be used to perform defect recognition on the multiple image features to be recognized, so as to ensure the efficiency and accuracy of recognition.

[0081] In an optional embodiment of this application, the method for identifying defects in large enclosed spaces further includes training a neural network model using a defect-free visible light image, a defect-free polarized light image, a defective visible light image, and a defective polarized light image of a predetermined detection area to obtain a defect identification neural network model.

[0082] In optional specific examples of this application, such as Figure 4As shown, the process of training a neural network model using a defect-free visible light image, a defect-free polarized light image, a defective visible light image, and a defective polarized light image of a predetermined detection area to obtain a defect recognition neural network model includes: Step 401, extracting features from the defect-free and defective visible light images of the predetermined detection area using a multi-layer convolutional neural network, and outputting the features at multiple predetermined layers of the corresponding multi-layer convolutional neural network to obtain multiple defect-free visible light image features to be fused and multiple defective visible light features to be fused; Step 402, performing wavelet decomposition on the defect-free and defective polarized light images of the predetermined detection area, and extracting and processing different bands and... Each defect-free visible light image feature to be fused corresponds to a defect-free polarized light image feature to be fused, and each defective visible light feature to be fused corresponds to a defective polarized light image feature to be fused; Step 403: Fuse each defect-free visible light image feature to be fused with its corresponding defect-free polarized light image feature to be fused to obtain multiple defect-free fused image features for a predetermined detection area, and fuse each defective visible light feature to be fused with its corresponding defective polarized light image feature to be fused to obtain multiple defective fused image features for a predetermined detection area; and Step 404: Train the neural network model using the defect-free fused features and the defective fused features to obtain a defect recognition neural network model.

[0083] Specifically, training a neural network model using a dataset composed of various defective and non-defective samples can yield a defect recognition neural network model with higher defect recognition efficiency and success rate.

[0084] In an optional embodiment of this application, the defect recognition neural network model is tested using the aforementioned defect-free fusion features and defective fusion features to ensure the efficiency and accuracy of the defect recognition neural network model.

[0085] In optional embodiments of this application, the aforementioned defective visible light image includes multiple defective visible light images, each defective visible light image including one or more types of defects; the defective polarized light image includes multiple defective polarized light images, each defective polarized light image including one or more types of defects.

[0086] Specifically, each visible light image or polarized light image includes a type of defect, which can help train the neural network model to distinguish and identify each type of defect.

[0087] In an optional embodiment of this application, the aforementioned defect-free visible light image includes multiple visible light images, wherein each visible light image may be captured by camera from a different angle.

[0088] In an optional specific embodiment of this application, the process of using a defect recognition neural network model to identify defects in multiple fused image features to obtain defect information of a predetermined detection area includes: using the defect recognition neural network model to identify each fused image feature to obtain multiple defect category vectors corresponding to each fused image feature to obtain multiple defect category vectors, and performing a weighted average of the multiple defect category vectors to obtain the defect category of the predetermined detection area.

[0089] Specifically, identifying defect categories can facilitate subsequent remediation of defects, thus helping to ensure the safe operation of industrial equipment in large enclosed spaces, such as boilers.

[0090] In an optional embodiment of this application, the neural network model used to train the defect recognition neural network model is a multi-branch neural network model. Optionally, each network branch is composed of several ResNet residual network modules connected together.

[0091] In an optional specific embodiment of this application, the process of training the neural network model using defect-free fusion features and defective fusion features to obtain a defect recognition neural network model includes: training each branch of the multi-branch neural network model using each defect-free fusion image feature and the defective fusion image feature obtained by fusing the same defective visible light image feature as the output layer of the defect-free visible light image feature to be fused to obtain each defect-free fusion image feature.

[0092] During the training phase, different neural network branches are trained using fused image features of different dimensions. This facilitates the use of different neural network branches to identify the fused features of corresponding dimensions during the recognition phase, ensuring both efficiency and accuracy in recognition.

[0093] In optional specific examples of this application, such as Figure 2 As shown, edge representation Z1, texture information Z, and semantic information Z3 are input into a pre-trained three-branch defect recognition neural network model to calculate the defect category vectors K1, K2, and K3 corresponding to each representation. Then, a weighted average of these defect category vectors is performed to obtain the defect category of the predetermined monitoring area. Specifically, the defect category vector represents the probability distribution of the current sample across all potential categories. For example, the defect category vector [0,0,0.5,0.25,0.2,0.05] indicates that the sample has a probability of 0.5 for category 3, 0.25 for category 4, 0.2 for category 5, and 0.05 for category 6.

[0094] Figure 5This is a structural diagram of a large confined space defect identification device provided in an embodiment of the present invention. This device is suitable for executing the large confined space defect identification method provided in this embodiment of the present invention. Figure 5 As shown, the device may specifically include:

[0095] The system comprises: an image acquisition module 501 for acquiring visible light and polarized light images of a predetermined detection area of ​​an industrial equipment, such as a power plant boiler; a first image feature extraction module 502 for extracting features from the visible light image using a multi-layer convolutional neural network and outputting the extracted features at multiple predetermined layers of the convolutional neural network to obtain multiple visible light image features to be fused; a second image feature extraction module 503 for performing wavelet decomposition on the polarized light image and extracting multiple polarized light image features corresponding one-to-one with each visible light image feature to be fused; an image feature fusion module 504 for fusing each visible light image feature to be fused with its corresponding polarized light image feature to obtain multiple fused image features to be identified in the predetermined detection area; and an identification module 505 for identifying defects in the multiple fused image features using a defect identification neural network model to obtain defect information in the predetermined detection area.

[0096] The device in this embodiment can use a convolutional neural network to output multiple visible light image features of the boiler water-cooled wall in multiple predetermined layers, and fuse them with multiple polarized light image features of different bands obtained by polarized light wavelet decomposition. Then, the neural network model is used to identify defects in the fused image features, so that the visible light image features and the polarized light image features complement each other in different dimensions, thereby automatically acquiring more defect information of the boiler water-cooled wall, and thus timely discovering safety hazards and ensuring the safe operation of the boiler.

[0097] The image acquisition module 501, which acquires visible light images and polarized light images of a predetermined detection area of ​​a boiler water-cooled wall, can extract and fuse the features of the two types of images, avoiding the difference between image information and real information caused by the objective environment, thereby avoiding the omission of surface defect information of the boiler water-cooled wall.

[0098] The first image feature extraction module 502 is used to extract features from visible light images using a multi-layer convolutional neural network and output them in multiple predetermined layers of the multi-layer convolutional neural network to obtain multiple visible light image features to be fused. It can obtain visible light image features of different dimensions, separate information of different dimensions from the original visible light image, form a relatively independent feature matrix, which helps to reduce information confusion, facilitates fusion with corresponding biased light image features in the later stage, and improves the accuracy of neural network recognition.

[0099] The second image feature extraction module 503 is used to perform wavelet decomposition on polarized light images and extract multiple polarized light image features of different bands that correspond one-to-one with the features of each visible light image to be fused. It can obtain polarized image features of different bands, separate information of different dimensions from the original polarized light image, form a relatively independent feature matrix, which helps to reduce information confusion, facilitates fusion with the corresponding visible light image features in the later stage, and improves the accuracy of neural network recognition.

[0100] The image feature fusion module 504, which fuses each visible light image feature to be fused with the corresponding polarized light image feature to be fused, to obtain multiple fused image features to be identified in a predetermined detection area, can fuse corresponding image features of different dimensions of visible light image and polarized light image, so that visible light image features and polarized light image features complement each other, avoid the loss of defect information in the image, and facilitate the subsequent identification of defects using neural networks.

[0101] The identification module 505 is used to identify defects in a predetermined detection area by using a defect identification neural network model to identify multiple fused image features. It can identify fused image features in three dimensions: visible light image features, polarized light image features, etc., and finally automatically identify more defects to avoid omissions, thereby timely detection of safety hazards and ensuring the safe operation of the boiler.

[0102] In an optional embodiment of this application, the large-scale enclosed space defect identification device of this application further includes: a model training module, used to train a neural network model using a defect-free visible light image, a defect-free polarized light image, a defective visible light image, and a defective polarized light image of a predetermined detection area, to obtain a defect identification neural network model.

[0103] The present invention provides a large enclosed space defect identification device, which can be used to perform the large enclosed space defect identification method described in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0104] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0105] This invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method for identifying defects in large enclosed spaces provided in any of the above embodiments.

[0106] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the large enclosed space defect identification method provided in any of the above embodiments.

[0107] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts.

[0108] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0110] The modules and / or units described in the embodiments of the present invention can be implemented in software or hardware. The described modules and / or units can also be housed in a processor; for example, a processor can be described as including a to-be-recognized image acquisition module, a first to-be-recognized image feature extraction module, a second to-be-recognized image feature extraction module, a to-be-recognized image feature fusion module, and a recognition module. The names of these modules do not necessarily constitute a limitation on the module itself.

[0111] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist alone and not assembled into the device.

[0112] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for identifying defects in large enclosed spaces, characterized in that, include: Visible light image acquisition and polarized light image acquisition are performed on the predetermined detection area to obtain the visible light image and polarized light image of the predetermined detection area; The visible light image is used to extract features using a multi-layer convolutional neural network, and the output is performed in multiple predetermined layers of the multi-layer convolutional neural network to obtain multiple visible light image features to be fused. Wavelet decomposition is performed on the polarized light image, and multiple polarized light image features of different bands are extracted and processed to obtain multiple polarized light image features to be fused that correspond one-to-one with each visible light image feature to be fused. Each of the visible light image features to be fused is fused with the corresponding polarized light image features to be fused to obtain multiple fused image features to be identified in the predetermined detection area; as well as Defects are identified by using a defect recognition neural network model to identify the multiple fused image features to obtain defect information of the predetermined detection area. The multi-layer convolutional neural network is an N-layer convolutional neural network. The process of outputting from multiple predetermined layers of the multi-layer convolutional neural network to obtain multiple visible light image features to be fused includes: The outputs of the layers |(1 / 3)N|, |(2 / 3)N|, and N-1 of the multi-layer convolutional neural network are taken respectively to obtain the low-dimensional visible light image features to be fused, the medium-dimensional visible light image features to be fused, and the high-dimensional visible light image features to be fused, where N is an integer greater than 3; The process of performing wavelet decomposition on the polarized light image and extracting multiple polarized light image features of different bands that correspond one-to-one with each of the visible light image features to be fused includes: Wavelet decomposition is performed on the polarized light image, and high-frequency polarized light image features in the short-wave band corresponding to the low-dimensional visible light image features to be fused, mid-frequency polarized light image features in the mid-wave band corresponding to the mid-dimensional visible light image features to be fused, and low-frequency polarized light image features in the long-wave band corresponding to the high-dimensional visible light image features to be fused are extracted respectively. The dimensions of the high-frequency polarized light image features, the mid-frequency polarized light image features, and the low-frequency polarized light image features are adjusted using the convolutional pooling method to make them correspond to the dimensions of the low-dimensional visible light image features to be fused, the mid-dimensional visible light image features to be fused, and the high-dimensional visible light image features to be fused, thus obtaining the high-frequency polarized light image features, the mid-frequency polarized light image features, and the low-frequency polarized light image features to be fused.

2. The method for identifying defects in large enclosed spaces according to claim 1, characterized in that, Also includes: The defect-free and defective visible light images of the predetermined detection area are extracted using the multi-layer convolutional neural network, and the outputs are performed in multiple predetermined layers of the corresponding multi-layer convolutional neural network to obtain multiple defect-free visible light image features to be fused and multiple defective visible light features to be fused. Wavelet decomposition is performed on the defect-free polarized light image and the defective polarized light image of the predetermined detection area, respectively, and the defective polarized light image features to be fused and the defective polarized light image features to be fused are obtained by extracting and processing different bands corresponding one-to-one with each defective visible light image feature to be fused and each defective visible light image feature to be fused. Each defect-free visible light image feature to be fused is fused with the corresponding defect-free polarized light image feature to be fused to obtain multiple defect-free fused image features of the predetermined detection area, and each defective visible light feature to be fused is fused with the corresponding defective polarized light image feature to be fused to obtain multiple defective fused image features of the predetermined detection area. as well as The neural network model is trained using the defect-free fusion features and the defective fusion features to obtain a defect recognition neural network model.

3. The method for identifying defects in large enclosed spaces according to claim 1, characterized in that, The process of acquiring visible light images of the predetermined detection area includes: performing visible light adaptive supplementary lighting on the predetermined detection area according to the preset standard light intensity and the visible light intensity of the predetermined detection area detected in real time by the light intensity sensor, so that the visible light intensity of the predetermined detection area is consistent. The process of acquiring polarized light images of the predetermined detection area includes: adaptively adjusting the polarization degree and polarization angle of the polarized light according to the preset standard polarization degree and standard polarization angle, and the light intensity of the polarized light passing through the polarizer placed in the direction perpendicular to the light propagation detected in real time by the light intensity sensor, so that the frequency domain characteristics of the polarized light in the predetermined detection area are consistent.

4. The method for identifying defects in large enclosed spaces according to claim 1, characterized in that, The process of fusing each of the visible light image features to be fused with the corresponding polarized light image features to be fused includes: The visible light image features to be fused are superimposed onto the corresponding polarized light image to be fused via channels.

5. The method for identifying defects in large enclosed spaces according to claim 1, characterized in that, The process of using a defect recognition neural network model to identify defects in the multiple fused image features to obtain defect information of the predetermined detection area includes: The defect recognition neural network model is used to identify each feature of the fused image to be identified, thereby obtaining multiple defect category vectors corresponding to each feature of the fused image to be identified. The multiple defect category vectors are then weighted and averaged to obtain the defect category of the predetermined detection area.

6. The method for identifying defects in large enclosed spaces according to claim 2, characterized in that, The neural network model is a multi-branch neural network model; The process of training the neural network model using the defect-free fusion features and the defective fusion features to obtain a defect recognition neural network model includes: Each branch of the multi-branch neural network model is trained using each defect-free fused image feature and the defective fused image feature obtained by fusing the same defective visible light image feature as the output layer of the defect-free visible light image feature to be fused.

7. A large-scale confined space defect identification device, characterized in that, include: The image acquisition module is used to acquire visible light images and polarized light images of a predetermined detection area in a large enclosed space to obtain visible light images and polarized light images of the predetermined detection area. The first image feature extraction module is used to extract features from the visible light image using a multi-layer convolutional neural network and output them in multiple predetermined layers of the multi-layer convolutional neural network to obtain multiple visible light image features to be fused. The second image feature extraction module is used to perform wavelet decomposition on the polarized light image and extract and process multiple polarized light image features of different bands that correspond one-to-one with each visible light image feature to be fused. The image feature fusion module is used to fuse each visible light image feature to be fused with the corresponding polarized light image feature to be fused, so as to obtain multiple fused image features to be fused in the predetermined detection area; as well as The identification module is used to identify defects in the multiple fused image features to be identified by using a defect identification neural network model, and to obtain defect information in the predetermined detection area. The multi-layer convolutional neural network is an N-layer convolutional neural network. The first image feature extraction module is specifically used to take the outputs of the |(1 / 3)N|, |(2 / 3)N|, and N-1 layers of the multi-layer convolutional neural network to obtain the low-dimensional visible light image features to be fused, the medium-dimensional visible light image features to be fused, and the high-dimensional visible light image features to be fused of the visible light image, where N is an integer greater than 3. The second image feature extraction module is specifically used to perform wavelet decomposition on the polarized light image and extract high-frequency polarized light image features in the short-wave band corresponding to the low-dimensional visible light image features to be fused, mid-frequency polarized light image features in the mid-wave band corresponding to the mid-dimensional visible light image features to be fused, and low-frequency polarized light image features in the long-wave band corresponding to the high-dimensional visible light image features to be fused. The dimensions of the high-frequency polarized light image features, the mid-frequency polarized light image features, and the low-frequency polarized light image features are adjusted using the convolutional pooling method to make them correspond to the dimensions of the low-dimensional visible light image features to be fused, the mid-dimensional visible light image features to be fused, and the high-dimensional visible light image features to be fused, thus obtaining the high-frequency polarized light image features, the mid-frequency polarized light image features, and the low-frequency polarized light image features to be fused.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the method for identifying defects in large enclosed spaces as described in any one of claims 1 to 6.

Images