A multispectral detection system and method for defects in industrial sites

Abstract

This invention relates to the field of intelligent detection technology, and provides a multispectral detection system and method for defects in industrial sites. The system includes: a multispectral detection device comprising a three-axis gimbal support; a housing rotatably connected to the three-axis gimbal support, the position of which is adjusted by the gimbal support; visible light lenses I and II, as well as infrared and ultraviolet lenses arranged in a cross-sectional configuration; a server or edge computing device communicatively connected to the control unit of the three-axis gimbal support and the lens control unit, respectively; and software modules deployed on the server or edge computing device; a result display platform and alarm device communicatively connected to the server or edge computing device. Through the technical solution of this invention, by combining binocular vision and multispectral detection, the visible light defect detection results and infrared or ultraviolet defect detection results for defects in industrial sites can be fused, improving the reliability and accuracy of the detection process.

Description

Technical Field

[0001] This invention relates to the field of intelligent detection technology, and more specifically, to a multispectral detection system and method for defects in industrial sites. Background Technology

[0002] Leaks and spills are common safety hazards in industrial sites' liquid pipelines and containers. How to detect the occurrence and location of leaks and spills in a timely and accurate manner is a major issue related to safe production.

[0003] Traditional leak detection relies on staff on-duty patrols, resulting in low efficiency and high labor costs. With the development of artificial intelligence, intelligent robots are increasingly being used in industrial site inspections. Current inspection systems often use visible light cameras to capture images and then use deep convolutional neural networks to detect leaks. However, due to the translucency of water, detection accuracy is limited by factors such as ambient lighting conditions, ground color, and reflectivity. Relying solely on visible light cameras for image capture is insufficient for effective identification and easily leads to missed detections, posing significant challenges to intelligent detection. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0005] Therefore, the purpose of this invention is to provide a multispectral detection system and method for defects in industrial sites, which combines binocular vision and multispectral detection to achieve the fusion of visible light defect detection results and infrared or ultraviolet defect detection results in industrial sites, thereby improving the reliability and accuracy of the detection process.

[0006] To achieve the above objectives, the first aspect of the present invention provides a multispectral defect detection system for industrial sites, comprising: a multispectral detection device, the multispectral detection device including a three-axis gimbal support; a detector housing rotatably connected to the three-axis gimbal support, the position of the detector housing being adjusted by the three-axis gimbal support; visible light lens I, visible light lens II, and infrared and ultraviolet lenses mounted on the detector housing, with the lenses of the four lenses located on the same side of the detector housing; the visible light lens I, visible light lens II, and infrared and ultraviolet lenses are arranged in a crisscross pattern.

[0007] The server or edge computing device is communicatively connected to the control unit of the three-axis gimbal bracket, the control units of the visible light lens I, the visible light lens II, the infrared lens, and the ultraviolet lens, respectively, to receive image data collected by the multispectral detection device. The server or edge computing device is equipped with software modules, including a target detection network module, a parallax calculation module, and a multispectral target detection result fusion module.

[0008] The results display platform and alarm device are connected to the server or edge computing device to display multispectral target detection results and issue alarms.

[0009] Preferably, the optical center of the visible light lens I is horizontally aligned with the optical center of the infrared lens; the optical center of the visible light lens I is vertically aligned with the optical center of the ultraviolet lens; the optical center of the visible light lens II is horizontally aligned with the optical center of the ultraviolet lens; and the optical center of the visible light lens II is vertically aligned with the optical center of the infrared lens.

[0010] Preferably, the visible light lens I, visible light lens II, infrared lens, and ultraviolet lens have the same field of view.

[0011] Preferably, an inspection door is provided on the top surface of the detector housing in a region away from the lens, and the inspection door is designed with an array of elongated heat dissipation holes;

[0012] The top surface of the detector housing is provided with a grid-type heat dissipation hole near the lens.

[0013] The second aspect of the present invention proposes a method for multispectral detection of defects in industrial sites using the multispectral detection system for industrial site defects described above, comprising the following steps:

[0014] Under the same field of view and focal length, acquire visible light images, infrared or ultraviolet images;

[0015] The acquired visible light, infrared, or ultraviolet images are input into a pre-trained target detection model to obtain visible light defect detection results and infrared or ultraviolet defect detection results. The detection results include target type labels, target bounding box corner coordinates, and classification confidence probability.

[0016] Based on Zhang Zhengyou's calibration method, dual-target calibration was performed on visible light lens 1-visible light lens 2, visible light lens 1-infrared lens, visible light lens 1-ultraviolet lens, visible light lens 2-infrared lens, and visible light lens 2-ultraviolet lens respectively, to obtain the intrinsic parameter matrices K1 and K2 and the extrinsic parameter rotation matrix R and translation vector t for each pair of lenses;

[0017] Based on the parallax algorithm, according to the intrinsic parameter matrices K1, K2 and extrinsic parameters R,t of each pair of lenses, the images captured by visible light lens 1 and visible light lens 2 are denoted as visible1 and visible2, respectively. The parallax is calculated pixel by pixel to obtain the parallax value of each corresponding pixel in visible1 and visible2. The parallax value represents the translation vector of the corresponding pixel in visible1 and visible2 in the horizontal and vertical directions, forming a parallax map D.

[0018] Define the horizontal coordinate as the x-coordinate and the vertical coordinate as the y-coordinate. Based on the stereo correction method, the coordinate transformation is performed on the infrared or ultraviolet defect detection results. The area coordinates of the infrared / ultraviolet defect detection results after correction are the same as the y-coordinates / x-coordinates of the corresponding area of ​​visible1, but have parallax in the x-coordinates / y-coordinates. The area coordinates of the infrared / ultraviolet defect detection results are the same as the y-coordinates / y-coordinates of the corresponding area of ​​visible2, but have parallax in the y-coordinates / x-coordinates.

[0019] Based on the parallax map D, determine the coordinate position of the infrared or ultraviolet defect detection result in the corresponding visible light image, and update the coordinate values ​​of the target box corner points in the infrared or ultraviolet defect detection result;

[0020] The infrared or ultraviolet defect detection results are fused with the visible light detection results to output the target detection results.

[0021] Preferably, the infrared or ultraviolet defect detection results are fused with the visible light detection results to output the target detection results, including the following steps:

[0022] Based on the corner coordinates of the target box, determine whether there is a target box in the infrared or ultraviolet defect detection results that intersects with the target box in the corresponding visible light defect detection results;

[0023] If there is an intersection, determine whether the target type label of the infrared or ultraviolet defect detection result is the same as that of the corresponding visible light defect detection result;

[0024] If the target type labels are the same, the union of the intersecting target boxes is taken and fused. The fused confidence probability is then calculated according to the confidence-weighted fusion formula based on the classification confidence probability. The confidence-weighted fusion formula is as follows:

[0025] C v =W v C v +W i C i

[0026] Among them, C o Characterized as the fusion confidence probability, W v Characterized by the visible light weighting coefficient, W iCharacterized as infrared or ultraviolet weighting coefficients, C v Characterized by the classification confidence probability in visible light defect detection results, C i Characterized as the classification confidence probability in infrared or ultraviolet defect detection results.

[0027] Preferably, the multispectral detection method for defects in industrial sites further includes the following steps:

[0028] If it is determined that the target box in the infrared or ultraviolet defect detection result does not intersect with the target box in the corresponding visible light defect detection result, the detection result will be output directly separately.

[0029] Preferably, based on the stereo correction method, the infrared defect detection result after coordinate transformation correction has the same x-coordinate as the corresponding target in visible2, thus obtaining the corresponding longitudinal clamping area in visible2;

[0030] Based on the stereo correction method, the infrared defect detection result after coordinate transformation correction has the same y coordinate as the corresponding target in visible1, thus obtaining the corresponding lateral clamping area in visible1.

[0031] In the disparity map D, determine the corresponding disparity value of the edge pixels of the corresponding vertical clamping region in visible2, denoted as D. x D y Let X and Y represent the disparity vectors in the x and y directions, respectively. As translation vectors, their coordinate vectors in the edge pixels of the corresponding vertical clamping region in visible2 are denoted as (X). v2 Y v2 );

[0032] Calculate the pixel coordinate vectors corresponding to the edge pixels in visible1 and visible2, denoted as (X... v1 Y v1 ), where X v1 =X v2 -D x Y v1 =Y v2 -D y , (X v1 Y v1 The sequence of coordinate points forms the corresponding longitudinal clamping area in visible2, which corresponds to the corresponding area in visible1. Together with the corresponding lateral clamping area in visible1, they form the corresponding coordinate position of the infrared defect detection result in the visible light image visible1.

[0033] Preferably, the target detection model is trained through the following steps:

[0034] Collect visible light, infrared, or ultraviolet image data of defects at industrial sites;

[0035] Image data is cleaned, preprocessed, and enhanced to form visible light image sets, infrared or ultraviolet image sets. Preprocessing includes spatial filtering and frequency domain filtering, while enhancement includes hue, saturation, and brightness transformations, as well as image perspective transformations and rotation transformations.

[0036] The visible light image set, infrared or ultraviolet image set were labeled respectively;

[0037] The visible light image set is used as the input to the visible light target detection network, and the infrared or ultraviolet dataset is used as the input to the infrared or ultraviolet target detection network to train the target detection model and form a pre-trained target detection model. The target detection network can be any one of the YOLO series network, SSD network, or DETR series network.

[0038] Preferably, the multispectral detection method for defects in industrial sites further includes the following steps:

[0039] Based on the output target detection results, a field leak detection report is generated. If the amount of water leaked, the amount of oil leaked, and the confidence probability are all higher than the corresponding preset thresholds, an alarm signal is issued.

[0040] The multispectral defect detection system and method for industrial sites proposed in this invention have the following beneficial technical effects:

[0041] (1) The present invention proposes a multispectral detection system and method for defects in industrial sites, which combines binocular vision and multispectral detection to achieve rapid, efficient and accurate fusion of visible light defect detection results and infrared or ultraviolet defect detection results in industrial sites, thereby improving the reliability and accuracy of the detection process.

[0042] (2) The industrial site defect multispectral detection system and detection method proposed in this invention achieves the formation of the corresponding coordinate position of infrared or ultraviolet defect detection results in the visible light image through the special design of the positions of visible light lens I, visible light lens II, infrared lens, and ultraviolet lens, as well as the method process design, etc. Then, the target box is judged and fused, making the target detection results more accurate and clearer.

[0043] (3) The multispectral detection system for defects in industrial sites proposed in this invention uses multispectral detection equipment. The optical center of the visible light lens I is horizontally aligned with the optical center of the infrared lens, the optical center of the visible light lens I is vertically aligned with the optical center of the ultraviolet lens, the optical center of the visible light lens II is horizontally aligned with the optical center of the ultraviolet lens, and the optical center of the visible light lens II is vertically aligned with the optical center of the infrared lens. Thus, the spectral information of the infrared or ultraviolet lens can be fused in by using the binocular vision of the visible light lens I and the visible light lens II. The fusion reliability is strong. After fusion, it can coordinate binocular vision and multispectral detection, which is beneficial to improving the reliability of detection.

[0044] (4) The industrial site defect multispectral detection system proposed in this invention adopts multispectral detection equipment. The three-axis gimbal bracket is rotatably connected to the outer shell of the detector, which can drive the outer shell of the detector to rotate in multiple directions and change its orientation. The four lenses are all installed on the same side of the outer shell of the detector, and can change their orientation synchronously with the outer shell of the detector, which is conducive to realizing that the four lenses can collect images under the same field of view.

[0045] Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description or may be learned by practice of the invention. Attached Figure Description

[0046] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0047] Figure 1 A structural block diagram of an industrial site defect multispectral detection system according to an embodiment of the present invention is shown;

[0048] Figure 2 A schematic diagram of the structure of a multispectral detection device in an industrial site defect multispectral detection system according to an embodiment of the present invention is shown.

[0049] Figure 3 A schematic flowchart of a multispectral detection method for defects in industrial sites according to an embodiment of the present invention is shown.

[0050] Figure 4 Another schematic flowchart of a multispectral detection method for defects in industrial sites according to an embodiment of the present invention is shown;

[0051] Figure 5 Visible light images of dripping and overflowing water were shown;

[0052] Figure 6 Infrared images of leaks and spills are shown.

[0053] Figure 7 The results of visible light defect detection for leaks and spills are shown.

[0054] Figure 8 The results of infrared defect detection for leaks and spills are shown.

[0055] Figure 9 The results of the target detection after fusion are shown;

[0056] Figure 10 A schematic diagram illustrating the principle of fusing visible light defect detection results and infrared defect detection results is shown.

[0057] Figure 11 The diagram illustrates the principle of fusing visible light defect detection results and infrared defect detection results.

[0058] in, Figures 1 to 2 The correspondence between the reference numerals and components in the attached drawings is as follows:

[0059] 102 Multispectral detection equipment, 1021 Three-axis gimbal support control unit, 1022 Lens control unit, 1023 Three-axis gimbal support, 1024 Detector housing, 1025 Visible light lens I, 1026 Visible light lens II, 1027 Infrared lens, 1028 Ultraviolet lens, 1029 Inspection door, 1030 Long strip-shaped heat dissipation hole, 1031 Fence-type heat dissipation hole, 104 Server or edge computing device, 1041 Target detection network module, 1042 Parallax calculation module, 1043 Multispectral target detection result fusion module, 106 Result display platform and alarm device. Detailed Implementation

[0060] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0061] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0062] The following is combined Figures 1 to 10 The present invention provides a detailed description of an industrial site defect multispectral detection system and detection method according to an embodiment of the present invention.

[0063] Example 1

[0064] like Figure 1 and Figure 2As shown, the industrial site defect multispectral detection system according to an embodiment of the present invention includes: a multispectral detection device 102, which includes a control unit 1021 and a lens control unit 1022 of a three-axis gimbal bracket 1023, which are respectively communicatively connected to a server or an edge computing device 104. The two can communicate with each other to realize the transmission of acquired images and the adjustment of the field of view, etc. The server or edge computing device 104 is equipped with software modules, including a target detection network module 1041, a parallax calculation module 1042, and a multispectral target detection result fusion module 1043, to realize the operation of the industrial site defect multispectral detection method. The result display platform and alarm device 106 are communicatively connected to the server or edge computing device 104, and can display the multispectral target detection results and issue alarms. The multispectral detection device 102 includes: a three-axis gimbal bracket 1023, a detector housing 1024, a visible light lens I 1025, a visible light lens II 1026, an infrared lens 1027, an ultraviolet lens 1028, etc. The detector housing 1024 is rotatably connected to the three-axis gimbal bracket 1023, enabling the detector housing 1024 to rotate in multiple directions and change its orientation. Visible light lens I 1025, visible light lens II 1026, infrared lens 1027, and ultraviolet lens 1028 are mounted on the detector housing 1024, and the lenses of the four lenses are located on the same side of the detector housing 1024. They can change their orientation synchronously with the detector housing 1024, which is beneficial for the four lenses to acquire images from the same field of view. Visible light lens I 1025, visible light lens II 1026, infrared lens 1027, and ultraviolet lens 1028 are arranged in a cross-distribution manner, which can acquire visible light images, infrared or ultraviolet images from the same field of view. Moreover, the special design of the positions of visible light lens I 1025, visible light lens II 1026, infrared lens 1027, and ultraviolet lens 1028 is conducive to realizing coordinate transformation of images, which in turn facilitates the fusion of information results from visible light detection and infrared or ultraviolet detection, thereby improving the accuracy and reliability of detection and promoting the development of intelligent detection technology.

[0065] Furthermore, the optical center of the visible light lens I 1025 is horizontally aligned with the optical center of the infrared lens 1027; the optical center of the visible light lens I 1025 is vertically aligned with the optical center of the ultraviolet lens 1028; the optical center of the visible light lens II 1026 is horizontally aligned with the optical center of the ultraviolet lens 1028; and the optical center of the visible light lens II 1026 is vertically aligned with the optical center of the infrared lens 1027.

[0066] Furthermore, the visible light lens I 1025, the visible light lens II 1026, the infrared lens 1027, and the ultraviolet lens 1028 have the same field of view.

[0067] Therefore, it is advantageous to use binocular vision such as visible light lens I 1025 and visible light lens II 1026 to fuse the spectral information of infrared lens 1027 or ultraviolet lens 1028. The fusion reliability is strong, and after fusion, it can coordinate binocular and multispectral vision, which is beneficial to improving the reliability of detection.

[0068] Furthermore, such as Figure 2 As shown, a maintenance door 1029 is provided on the top surface of the detector housing 1024 in a region away from the lens, and an array of elongated heat dissipation holes 1030 are designed on the maintenance door 1029.

[0069] A grid-type heat dissipation hole 1031 is provided on the top surface of the detector housing 1024 near the lens.

[0070] This facilitates heat dissipation and avoids excessive dust in industrial environments from affecting the lens's lifespan.

[0071] Example 2

[0072] like Figure 3 As shown, an industrial site defect multispectral detection method according to an embodiment of the present invention includes the following steps:

[0073] S202, under the same field of view and focal length, acquires visible light images, infrared or ultraviolet images;

[0074] S204. Input the collected visible light image, infrared or ultraviolet image into the pre-trained target detection model to obtain the visible light defect detection result, infrared or ultraviolet defect detection result, and the detection result includes the target type label, the target box corner coordinate value, and the classification confidence probability (0~1).

[0075] The specific training of the target detection model involves: collecting visible light image data and infrared or ultraviolet image data of defects at the industrial site; cleaning, preprocessing, and enhancing the image data to form visible light image sets and infrared or ultraviolet image sets. Preprocessing includes spatial filtering and frequency domain filtering, while enhancement includes hue, saturation, and brightness transformations, as well as image perspective and rotation transformations; labeling the visible light image sets and infrared or ultraviolet image sets respectively; using the visible light image set as input to the visible light target detection network, and using the infrared and ultraviolet datasets as input to the infrared or ultraviolet target detection network, training the target detection model to form a pre-trained target detection model. The target detection network can be any one of the YOLO series networks, SSD networks, or DETR series networks.

[0076] S206. Based on Zhang Zhengyou's calibration method, dual-target calibration is performed on visible light lens 1-visible light lens 2, visible light lens 1-infrared lens, visible light lens 1-ultraviolet lens, visible light lens 2-infrared lens, and visible light lens 2-ultraviolet lens respectively, to obtain the intrinsic parameter matrices K1 and K2 and the extrinsic parameter rotation matrix R and translation vector t for each pair of lenses.

[0077] S208, based on the parallax algorithm, according to the intrinsic parameter matrices K1, K2 and extrinsic parameters R,t of each pair of lenses, performs pixel-by-pixel parallax calculation on the images captured by visible light lens 1 and visible light lens 2, denoted as visible1 and visible2 respectively, to obtain the parallax value of each corresponding pixel in visible1 and visible2. The parallax value represents the translation vector of the corresponding pixel in visible1 and visible2 in the horizontal and vertical directions, forming a parallax map D;

[0078] S210, define the horizontal coordinate as the x-coordinate and the vertical coordinate as the y-coordinate. Based on the stereo correction method, perform coordinate transformation on the infrared or ultraviolet defect detection results. The area coordinates of the infrared / ultraviolet defect detection results after correction are the same as the y-coordinates / x-coordinates of the corresponding area of ​​visible1, but have parallax in the x-coordinates / y-coordinates. They are the same as the x-coordinates / y-coordinates of the corresponding area of ​​visible2, but have parallax in the y-coordinates / x-coordinates.

[0079] S212, Based on the disparity map D, determine the coordinate position of the infrared or ultraviolet defect detection result in the corresponding visible light image, and update the coordinate values ​​of the target box corner points in the infrared or ultraviolet defect detection result;

[0080] S214 fuses infrared or ultraviolet defect detection results with visible light detection results to output target detection results.

[0081] Specifically, based on the corner coordinates of the target box, it is determined whether there is a target box in the infrared or ultraviolet defect detection results that intersects with the target box in the corresponding visible light defect detection results;

[0082] If there is an intersection, determine whether the target type label of the infrared or ultraviolet defect detection result is the same as that of the corresponding visible light defect detection result;

[0083] If the target type labels are the same, the union of the intersecting target boxes is taken and fused. The fused confidence probability is then calculated according to the confidence-weighted fusion formula based on the classification confidence probability. The confidence-weighted fusion formula is as follows:

[0084] C o =W v C v +W i C i

[0085] Among them, C o Characterized as the fusion confidence probability, W v Characterized by the visible light weighting coefficient, W i Characterized as infrared or ultraviolet weighting coefficients, C v Characterized by the classification confidence probability in visible light defect detection results, C i Characterized as the classification confidence probability in infrared or ultraviolet defect detection results.

[0086] If it is determined that the target box in the infrared or ultraviolet defect detection result does not intersect with the target box in the corresponding visible light defect detection result, the detection result will be output directly separately.

[0087] Thus, by combining binocular vision and multispectral detection, the visible light defect detection results and infrared or ultraviolet defect detection results of defects in industrial sites are rapidly, efficiently and accurately integrated, improving the reliability and accuracy of the detection process.

[0088] Example 3

[0089] like Figure 4 As shown, an industrial site defect multispectral detection method according to an embodiment of the present invention includes the following steps:

[0090] S301, acquire visible light images, such as... Figure 5 As shown;

[0091] S302, input the visible light image into the visible light target detection network;

[0092] S303 outputs the visible light target detection results, such as Figure 7 As shown;

[0093] S304, acquires infrared images, such as... Figure 6 As shown;

[0094] S305, inputs the infrared image into the infrared target detection network;

[0095] S306 outputs the infrared target detection results, such as Figure 8 As shown;

[0096] S307, Parallax Calculation

[0097] Specifically, based on Zhang Zhengyou's calibration method, dual-target calibration was performed on visible light lens 1-visible light lens 2, visible light lens 1-infrared lens, visible light lens 1-ultraviolet lens, visible light lens 2-infrared lens, and visible light lens 2-ultraviolet lens respectively, to obtain the intrinsic parameter matrices K1, K2 and extrinsic parameters R, t for each pair of lenses;

[0098] Based on the parallax algorithm, according to the intrinsic parameter matrices K1 and K2 and the extrinsic parameter rotation matrix R and the translation vector t of each pair of lenses, the images captured by visible light lens 1 and visible light lens 2, respectively denoted as visible1 and visible2, are subjected to pixel-by-pixel parallax calculation to obtain the parallax value of each corresponding pixel in visible1 and visible2. The parallax value represents the translation vector of the corresponding pixel in visible1 and visible2 in the horizontal and vertical directions, forming a parallax map D.

[0099] S308, Coordinate transformation of infrared detection results

[0100] Specifically, the horizontal coordinate is defined as the x-coordinate, and the vertical coordinate as the y-coordinate. Based on the stereo correction method, the infrared defect detection results are transformed using coordinate transformation. The infrared defect detection results after coordinate transformation correction have the same x-coordinate as the corresponding target in visible2, thus obtaining the corresponding longitudinal clamping region in visible2. Similarly, the infrared defect detection results after coordinate transformation correction have the same y-coordinate as the corresponding target in visible1, thus obtaining the corresponding lateral clamping region in visible1. Figure 10 As shown, (a) is visible1, (b) is the infrared defect detection result, and (c) is visible2;

[0101] In the disparity map D, determine the corresponding disparity value of the edge pixels of the corresponding vertical clamping region in visible2, denoted as D. x D y Let X and Y represent the disparity vectors in the x and y directions, respectively. As translation vectors, their coordinate vectors in the edge pixels of the corresponding vertical clamping region in visible2 are denoted as (X). v2 Y v2 );

[0102] Calculate the pixel coordinate vectors corresponding to the edge pixels in visible1 and visible2, denoted as (X... v1 Y v1 ), where X v1 =X v2 -D x Y v1 =Y v2 -D y , (X v1 Y v1 The sequence of coordinate points forms the corresponding longitudinal clamping area in visible2, which corresponds to the area in visible1. Together with the corresponding lateral clamping area in visible1, they form the corresponding coordinate position of the infrared defect detection result in the visible light image visible1. Figure 11As shown, (a) is visible1, (b) is the infrared defect detection result, and (c) is visible2;

[0103] S309, Target detection results fusion output

[0104] Specifically, based on the coordinates of the corner points of the target box, it is determined whether there is a target box in the infrared defect detection result that intersects with the target box in the corresponding visible light defect detection result;

[0105] If there is an intersection, determine whether the target type label of the infrared defect detection result is the same as that of the corresponding visible light defect detection result;

[0106] If the target type labels are the same, the union of the intersecting target boxes is taken and fused. The fused confidence probability is then calculated according to the confidence-weighted fusion formula based on the classification confidence probability. The confidence-weighted fusion formula is as follows:

[0107] C o =W v C v +W i C i

[0108] Among them, C o Characterized as the fusion confidence probability, W v Characterized by the visible light weighting coefficient, W i Characterized as infrared weighting coefficient, C v Characterized by the classification confidence probability in visible light defect detection results, C i It is represented as the classification confidence probability in the infrared defect detection results.

[0109] If it is determined that the target box in the infrared defect detection result does not intersect with the target box in the corresponding visible light defect detection result, the detection result will be output directly separately.

[0110] The fused target detection results are as follows Figure 9 As shown, it is clearer and more accurate, and can clearly display the target box, compared to... Figure 7 Visible light defect detection results can detect more comprehensive defects such as leaks and spills, and are less likely to miss any.

[0111] Example 4

[0112] Based on Example 2 or Example 3, the following steps are added:

[0113] Based on the output target detection results, a field leak detection report is generated. If the amount of water leaked, the amount of oil leaked, and the confidence probability are all higher than the corresponding preset thresholds, an alarm signal is issued.

[0114] This allows for a clearer display of test results and timely warnings.

[0115] The steps in the method of this invention can be adjusted, combined, or deleted according to actual needs.

[0116] The units in the device of the present invention can be merged, divided, or deleted according to actual needs.

[0117] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0118] In this invention, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance; the term "multiple" refers to two or more unless otherwise explicitly defined. The terms "install," "connect," "link," and "fix" should be interpreted broadly. For example, "connect" can be a fixed connection, a detachable connection, or an integral connection; "link" can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0119] In the description of this invention, it should be understood that the terms "upper," "lower," "left," "right," "front," "rear," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or unit referred to must have a specific orientation or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0120] In the description of this specification, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0121] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multispectral defect detection system for industrial sites, characterized in that, include: A multispectral detection device, the multispectral detection device including a three-axis gimbal support; The detector housing is rotatably connected to the three-axis gimbal bracket, and the position of the detector housing is adjusted by the three-axis gimbal bracket. Visible light lens I, visible light lens II, infrared lens, and ultraviolet lens are mounted on the detector housing, and the lenses of all four are located on the same side of the detector housing. The visible light lens I, visible light lens II, infrared lens, and ultraviolet lens are arranged in a crisscross pattern. The optical center of the visible light lens I is horizontally aligned with the optical center of the infrared lens; the optical center of the visible light lens I is vertically aligned with the optical center of the ultraviolet lens. The optical center of the visible light lens II is horizontally aligned with the optical center of the ultraviolet lens; The optical center of the visible light lens II is vertically aligned with the optical center of the infrared lens; The server or edge computing device is communicatively connected to the control unit of the three-axis gimbal bracket, the control units of the visible light lens I, the visible light lens II, the infrared lens, and the ultraviolet lens, respectively, to receive image data collected by the multispectral detection device. The server or edge computing device is equipped with software modules, including a target detection network module, a parallax calculation module, and a multispectral target detection result fusion module. The results display platform and alarm device are connected to the server or edge computing device to display multispectral target detection results and issue alarms.

2. The industrial site defect multispectral detection system according to claim 1, characterized in that, The visible light lens I, visible light lens II, infrared lens, and ultraviolet lens have the same field of view.

3. The industrial site defect multispectral detection system according to claim 1, characterized in that, An inspection door is provided on the top surface of the detector housing, away from the lens, and an array of elongated heat dissipation holes are designed on the inspection door; The top surface of the detector housing is provided with a grid-type heat dissipation hole near the lens.

4. A method for multispectral detection of defects in industrial sites using the multispectral detection system for industrial site defects according to any one of claims 1 to 3, characterized in that, Includes the following steps: Under the same field of view and focal length, acquire visible light images, infrared or ultraviolet images; The acquired visible light, infrared, or ultraviolet images are input into a pre-trained target detection model to obtain visible light defect detection results and infrared or ultraviolet defect detection results. The detection results include target type labels, target bounding box corner coordinates, and classification confidence probability. Based on Zhang Zhengyou's calibration method, dual-target calibration was performed on visible light lens 1-visible light lens 2, visible light lens 1-infrared lens, visible light lens 1-ultraviolet lens, visible light lens 2-infrared lens, and visible light lens 2-ultraviolet lens respectively, to obtain the intrinsic parameter matrices K1 and K2 and the extrinsic parameter rotation matrix R and translation vector t for each pair of lenses; Based on the parallax algorithm, according to the intrinsic parameter matrices K1, K2 and extrinsic parameters R,t of each pair of lenses, the images captured by visible light lens 1 and visible light lens 2 are denoted as visible1 and visible2, respectively. The parallax is calculated pixel by pixel to obtain the parallax value of each corresponding pixel in visible1 and visible2. The parallax value represents the translation vector of the corresponding pixel in visible1 and visible2 in the horizontal and vertical directions, forming a parallax map D. Define the horizontal coordinate as the x-coordinate and the vertical coordinate as the y-coordinate. Based on the stereo correction method, the coordinate transformation is performed on the infrared or ultraviolet defect detection results. The region coordinates of the infrared / ultraviolet defect detection results after correction are the same as the y-coordinates / x-coordinates of the corresponding region of visible1, but have parallax in the x-coordinates / y-coordinates. The region coordinates of the infrared / ultraviolet defect detection results are the same as the y-coordinates / y-coordinates of the corresponding region of visible2, but have parallax in the y-coordinates / x-coordinates. Based on the parallax map D, determine the coordinate position of the infrared or ultraviolet defect detection result in the corresponding visible light image, and update the coordinate values ​​of the target box corner points in the infrared or ultraviolet defect detection result; The infrared or ultraviolet defect detection results are fused with the visible light detection results to output the target detection results.

5. The industrial site defect multispectral detection method of the industrial site defect multispectral detection system according to claim 4, characterized in that, The process of fusing infrared or ultraviolet defect detection results with visible light detection results to output target detection results includes the following steps: Based on the corner coordinates of the target box, determine whether there is a target box in the infrared or ultraviolet defect detection results that intersects with the target box in the corresponding visible light defect detection results; If there is an intersection, determine whether the target type label of the infrared or ultraviolet defect detection result is the same as that of the corresponding visible light defect detection result; If the target type labels are the same, the union of the intersecting target boxes is taken and fused. The fused confidence probability is then calculated according to the confidence-weighted fusion formula based on the classification confidence probability. The confidence-weighted fusion formula is as follows: C O =W v C v +W t C t Among them, C O Characterized as the fusion confidence probability, W v Characterized by the visible light weighting coefficient, W t Characterized as infrared or ultraviolet weighting coefficients, C v Characterized by the classification confidence probability in visible light defect detection results, C t Characterized as the classification confidence probability in infrared or ultraviolet defect detection results.

6. The industrial site defect multispectral detection method of the industrial site defect multispectral detection system according to claim 4, characterized in that, It also includes the following steps: If it is determined that the target box in the infrared or ultraviolet defect detection result does not intersect with the target box in the corresponding visible light defect detection result, the detection result will be output directly separately.

7. The industrial site defect multispectral detection method of the industrial site defect multispectral detection system according to claim 4, characterized in that, Based on the stereo correction method, the infrared defect detection result after coordinate transformation correction has the same x-coordinate as the corresponding target in visible2, thus obtaining the corresponding longitudinal clamping area in visible2. Based on the stereo correction method, the infrared defect detection result after coordinate transformation correction has the same y coordinate as the corresponding target in visible1, thus obtaining the corresponding lateral clamping area in visible1. In the disparity map D, determine the corresponding disparity value of the edge pixels of the corresponding vertical clamping region in visible2, denoted as D. x D y Let X and Y represent the disparity vectors in the x and y directions, respectively. As translation vectors, their coordinate vectors in the corresponding vertical clamping region of visible2 are denoted as (X). v2 ,Y v2 ); Calculate the pixel coordinate vectors corresponding to the edge pixels in visible1 and visible2, denoted as (X... v1 ,Y v1 ), where X v1 =X v2 -D x Y v1 =Y v2 -D y (X) v1 ,Y v1 The sequence of coordinate points forms the corresponding longitudinal clamping area in visible2, which corresponds to the corresponding area in visible1. Together with the corresponding lateral clamping area in visible1, they form the corresponding coordinate position of the infrared defect detection result in the visible light image visible1.

8. The industrial site defect multispectral detection method of the industrial site defect multispectral detection system according to claim 4, characterized in that, The object detection model is trained through the following steps. Collect visible light, infrared, or ultraviolet image data of defects at industrial sites; Image data is cleaned, preprocessed, and enhanced to form visible light image sets, infrared or ultraviolet image sets. Preprocessing includes spatial filtering and frequency domain filtering, while enhancement includes hue, saturation, and brightness transformations, as well as image perspective transformations and rotation transformations. The visible light image set, infrared or ultraviolet image set were labeled respectively; The visible light image set is used as the input to the visible light target detection network, and the infrared or ultraviolet dataset is used as the input to the infrared or ultraviolet target detection network to train the target detection model and form a pre-trained target detection model. The target detection network can be any one of the YOLO series network, SSD network, or DETR series network.

9. The industrial site defect multispectral detection method of the industrial site defect multispectral detection system according to claim 4, characterized in that, It also includes the following steps: Based on the output target detection results, a field leak detection report is generated. If the amount of water leaked, the amount of oil leaked, and the confidence probability are all higher than the corresponding preset thresholds, an alarm signal is issued.

Images