Industrial place defect multispectral detection system and detection method

Abstract

The invention relates to the technical field of intelligent detection, and provides an industrial site defect multispectral detection system and method, and the system comprises a multispectral detection device which comprises a three-axis holder support; the detector outer shell is rotationally connected with the three-axis holder bracket, and the position adjustment of the detector outer shell is controlled by the three-axis holder bracket; the visible light lens I, the visible light lens II, the infrared lens and the ultraviolet lens are arranged in a cross distribution mode, the server or the edge computing device is in communication connection with the three-axis holder support control unit and the lens control unit, and the server or the edge computing device is provided with a software module. And the result display platform and the alarm equipment are in communication connection with the server or the edge computing equipment. According to the technical scheme, binocular vision and multispectral detection are cooperated, fusion of visible light defect detection results and infrared or ultraviolet defect detection results of industrial site defects can be achieved, and the reliability and accuracy of the detection process are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent detection technology, and in particular to an industrial site defect multi-spectral detection system and a detection method. Background Art

[0002] Leakage is a common safety hazard in liquid pipelines and containers in industrial sites. How to detect the occurrence and location of leakage in a timely and accurate manner is a major issue related to safe production.

[0003] Traditional leakage detection is completed by staff on duty, which has low work efficiency and high labor costs. With the development of artificial intelligence technology, intelligent robots are increasingly widely used in industrial site inspection operations. Current inspection systems mostly use visible light cameras to collect on-site pictures, and detect leakage targets based on the collected pictures based on deep convolutional neural networks. However, due to the light transmittance of water, the detection accuracy is restricted by factors such as on-site light conditions, on-site ground color, and reflective properties. It is difficult to effectively identify with a single visible light camera to collect photos for detection, and it is easy to miss detection problems, which brings great difficulties to intelligent detection. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art or related art.

[0005] To this end, the purpose of the present invention is to provide a multi-spectral detection system and method for industrial site defects, which can coordinate binocular vision and multi-spectral detection to achieve the fusion of visible light defect detection results of industrial site defects with infrared or ultraviolet defect detection results, thereby improving the reliability and accuracy of the detection process.

[0006] In order to achieve the above-mentioned purpose, the technical solution of the first aspect of the present invention provides an industrial site defect multispectral detection system, including: a multispectral detection device, the multispectral detection device includes a three-axis pan-tilt bracket; a detector outer shell, which is rotatably connected to the three-axis pan-tilt bracket, and the position adjustment of the detector outer shell is controlled by the three-axis pan-tilt bracket; a visible light lens I, a visible light lens II, an infrared lens, and an ultraviolet lens are installed on the detector outer shell, and the lenses of the four are on the same side of the detector outer shell; the visible light lens I, the visible light lens II, the infrared lens, and the ultraviolet lens are arranged in a cross-distribution manner,

[0007] The server or edge computing device is respectively connected to the control unit of the three-axis gimbal bracket, the visible light lens I, the visible light lens II, the infrared lens, and the ultraviolet lens to receive the image data collected by the multispectral detection device. The server or edge computing device is equipped with software modules, including a target detection network module, a parallax calculation module, and a multispectral target detection result fusion module;

[0008] The result display platform and the alarm device are connected to the server or edge computing device in communication to display the multi-spectral target detection results and issue an alarm.

[0009] Preferably, the optical center of visible light lens I is horizontally aligned with the optical center of infrared lens; the optical center of visible light lens I is vertically aligned with the optical center of ultraviolet lens; the optical center of visible light lens II is horizontally aligned with the optical center of ultraviolet lens; the optical center of visible light lens II is vertically aligned with the optical center of infrared lens.

[0010] Preferably, the viewing angles of the visible light lens I, visible light lens II, infrared lens, and ultraviolet lens are the same.

[0011] Preferably, an inspection door is provided at a top surface of the outer shell of the detector away from the lens, and the inspection door is designed with long strip heat dissipation holes arranged in an array;

[0012] A fence-type heat dissipation hole is arranged at a region near the lens on the top surface of the outer shell of the detector.

[0013] The technical solution of the second aspect of the present invention proposes a multi-spectral detection method for industrial site defects using the multi-spectral detection system for industrial site defects in the above technical solution, comprising the following steps:

[0014] Collect visible light images, infrared or ultraviolet images at the same field of view and focal length;

[0015] Input the collected visible light images, infrared or ultraviolet images into the pre-trained target detection model to obtain the visible light defect detection results, infrared or ultraviolet defect detection results. The detection results include the target type label, the coordinate values ​​of the target frame corner points, and the classification confidence probability;

[0016] Based on Zhang Zhengyou calibration method, binocular calibration is performed on visible light lens 1-visible light lens 2, visible light lens 1-infrared lens, visible light lens 1-ultraviolet lens, visible light lens 2-infrared lens, and visible light lens 2-ultraviolet lens, respectively, and the intrinsic parameter matrices K1 and K2 and the extrinsic parameter rotation matrix R and translation vector t of each pair of lenses are obtained;

[0017] Based on the disparity algorithm, according to the intrinsic parameter matrices K1 and K2 and the extrinsic parameters R,t of each pair of lenses, the pictures taken by the visible light lens 1 and the visible light lens 2, respectively denoted as visible1 and visible2, are subjected to pixel-by-pixel disparity calculation to obtain the disparity values ​​of the corresponding pixels in visible1 and visible2, wherein the disparity values ​​represent the translation vectors of the corresponding pixels of visible1 and visible2 in the horizontal and vertical directions, thereby forming a disparity map D;

[0018] Define the horizontal coordinate as x-coordinate and the vertical coordinate as y-coordinate. Based on the stereo correction method, perform coordinate transformation on the infrared or ultraviolet defect detection result. The regional coordinates of the corrected infrared / ultraviolet defect detection result are the same as the y-direction / x-direction coordinates of the corresponding area of ​​visible1, but have parallax in the x-direction / y-direction, and the same as the x-direction / y-direction coordinates of the corresponding area of ​​visible2, but have parallax in the y-direction / x-direction.

[0019] According to the disparity map D, determine the coordinate position of the infrared or ultraviolet defect detection result in the corresponding visible light image, and update the coordinate value of the target frame corner point in the infrared or ultraviolet defect detection result;

[0020] The infrared or ultraviolet defect detection results are combined with the visible light detection results to output the target detection results.

[0021] Preferably, the infrared or ultraviolet defect detection result is merged with the visible light detection result to output the target detection result, including the following steps:

[0022] According to the coordinate values ​​of the corner points of the target frame, it is determined whether there is a target frame in the infrared or ultraviolet defect detection result that intersects with the target frame in the corresponding visible light defect detection result;

[0023] If there is an intersection, determine whether the target type label of the infrared or ultraviolet defect detection result is the same as the corresponding visible light defect detection result;

[0024] If the target type labels are the same, the intersecting target frames are fused together, and the fusion confidence probability is calculated according to the confidence weighted fusion formula based on the classification confidence probability. The confidence weighted fusion formula is:

[0025] C v =W v C v +W i C i

[0026] Among them, C o Represented as the fusion confidence probability, W v Characterized as the visible light weight coefficient, W iCharacterized as infrared or ultraviolet weight coefficient, C v Characterized by the classification confidence probability in the visible light defect detection results, C i Characterized by the classification confidence probability in infrared or ultraviolet defect detection results.

[0027] Preferably, the multi-spectral detection method for industrial site defects also includes the following steps:

[0028] If it is determined that the target frame in the infrared or ultraviolet defect detection result does not intersect with the target frame in the corresponding visible light defect detection result, the detection result is directly output separately.

[0029] Preferably, based on the stereo correction method, the infrared defect detection result after coordinate transformation correction has the same x coordinate as the corresponding target in visible2, and the corresponding longitudinal clamping area in visible2 is obtained;

[0030] Based on the stereo correction method, the infrared defect detection result after coordinate transformation correction has the same y coordinate as the corresponding target in visible1, and the corresponding lateral clamping area in visible1 is obtained;

[0031] In the disparity map D, determine the corresponding disparity value of the edge pixel of the corresponding longitudinal clamping area in visible2, denoted as D x , D y , respectively represent the disparity vectors in the x-direction and y-direction, as the translation vector, the coordinate vector of the edge pixel of the corresponding longitudinal clamping area in visible2 is recorded as (X v2 , Y v2 );

[0032] Calculate the pixel coordinate vector in visible1 corresponding to the edge pixel in visible2, denoted as (X v1 , Y v1 ), where X v1 =X v2 -D x , Y v1 =Y v2 -D y ,(X v1 , Y v1 ) The coordinate point sequence encloses the corresponding area of ​​the corresponding longitudinal clamping area in visible2 and the corresponding transverse clamping area in visible1, and together forms the corresponding coordinate position of the infrared defect detection result in the visible light image visible1.

[0033] Preferably, the target detection model is trained by the following steps:

[0034] Collect visible light image data, infrared or ultraviolet image data of industrial site defects;

[0035] Clean, preprocess and enhance the image data to form a visible light image set, infrared or ultraviolet image set, where the preprocessing includes spatial filtering and frequency domain filtering, and the enhancement includes: hue, saturation, brightness transformation, image perspective transformation, and rotation transformation;

[0036] Label the visible light image set, infrared or ultraviolet image set respectively;

[0037] Use a visible light image set as the input of a visible light target detection network, and use an infrared or ultraviolet data set as the input of an infrared or ultraviolet target detection network to train a target detection model to form a pre-trained target detection model, wherein the target detection network is any one of the YOLO series networks, SSD networks, and DETR series networks.

[0038] Preferably, the multi-spectral detection method for industrial site defects also includes the following steps:

[0039] Based on the output target detection results, an on-site leakage detection report is generated. If the amount of water leakage, the amount of oil leakage and the confidence probability are all higher than the corresponding preset thresholds, an alarm signal is issued.

[0040] The multi-spectral detection system and method for industrial site defects proposed by the present invention have the following beneficial technical effects:

[0041] (1) The present invention proposes a multi-spectral detection system and method for industrial site defects, which coordinate binocular vision and multi-spectral detection to achieve rapid, efficient and accurate fusion of visible light defect detection results and infrared or ultraviolet defect detection results of industrial site defects, thereby improving the reliability and accuracy of the detection process.

[0042] (2) The multi-spectral defect detection system and method for industrial sites proposed in the present invention realizes the formation of the corresponding coordinate position of the infrared or ultraviolet defect detection results in the visible light image through the special design of the positions of visible light lens I, visible light lens II, infrared lens, and ultraviolet lens, as well as the method process design, and then judges and fuses the target frame, making the target detection result more accurate and clearer.

[0043] (3) The multi-spectral detection system for industrial site defects proposed by the present invention adopts a multi-spectral detection device, wherein the optical center of visible light lens I is horizontally aligned with the optical center of infrared lens, the optical center of visible light lens I is vertically aligned with the optical center of ultraviolet lens, the optical center of visible light lens II is horizontally aligned with the optical center of ultraviolet lens, and the optical center of visible light lens II is vertically aligned with the optical center of infrared lens, thereby being able to utilize binocular vision of visible light lens I and visible light lens II to fuse spectral information of infrared lens or ultraviolet lens, and having strong fusion reliability. After fusion, binocular and multi-spectral can be coordinated, which is beneficial to improving detection reliability.

[0044] (4) The multi-spectral detection system for industrial defects proposed in the present invention adopts a multi-spectral detection device. The three-axis pan-tilt bracket is rotatably connected to the outer shell of the detector, which can drive the outer shell of the detector to rotate in multiple directions and change its orientation. The four lenses are installed on the same side of the outer shell of the detector and can change their orientation synchronously with the outer shell of the detector, which is conducive to the four lenses collecting pictures with the same field of view.

[0045] Additional aspects and advantages of the present invention will be given in part in the following description, and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0047] Figure 1 A structural block diagram of a multi-spectral detection system for industrial site defects according to an embodiment of the present invention is shown;

[0048] Figure 2 A schematic diagram of the structure of a multi-spectral detection device in a multi-spectral detection system for defects in industrial sites according to an embodiment of the present invention is shown;

[0049] Figure 3 A schematic diagram of a process of a multi-spectral detection method for industrial site defects according to an embodiment of the present invention is shown;

[0050] Figure 4 Another schematic diagram of the process of the multi-spectral detection method for industrial site defects according to an embodiment of the present invention is shown;

[0051] Figure 5 A visible light picture of leakage is shown;

[0052] Figure 6 An infrared picture of leakage is shown;

[0053] Figure 7 The results of visible light defect detection for leakage are shown;

[0054] Figure 8 The infrared defect detection results of leakage are shown;

[0055] Fig. 9 The fused target detection results are shown;

[0056] Fig.10 The schematic diagram of the fusion of visible light defect detection results and infrared defect detection results is shown;

[0057] Fig.11 The diagram shows the principle diagram of the fusion of visible light defect detection results and infrared defect detection results.

[0058] in, Figure 1 to Figure 2 The corresponding relationship between the reference numerals and the components is as follows:

[0059] 102 multispectral detection equipment, 1021 three-axis gimbal bracket control unit, 1022 lens control unit, 1023 three-axis gimbal bracket, 1024 detector outer shell, 1025 visible light lens I, 1026 visible light lens II, 1027 infrared lens, 1028 ultraviolet lens, 1029 maintenance door, 1030 long strip heat dissipation holes, 1031 fence-type heat dissipation holes, 104 server or edge computing equipment, 1041 target detection network module, 1042 parallax calculation module, 1043 multispectral target detection result fusion module, 106 result display platform and alarm equipment. DETAILED DESCRIPTION

[0060] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0061] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0062] Combine the following Figures 1 to 10 The industrial site defect multi-spectral detection system and detection method according to the embodiments of the present invention are described in detail.

[0063] Example 1

[0064] like Figure 1 and Figure 2As shown, the multispectral detection system for industrial site defects according to an embodiment of the present invention includes: a multispectral detection device 102, the multispectral detection device 102 includes a control unit 1021 of a three-axis pan-tilt bracket 1023, and a lens control unit 1022, which are respectively connected to the server or edge computing device 104 in communication, and the two can communicate interactively to achieve the transmission of collected pictures and the adjustment of the field of view angle. The server or edge computing device 104 is equipped with a software module, including a target detection network module 1041, a parallax calculation module 1042, and a multispectral target detection result fusion module 1043, to achieve the operation of the multispectral detection method for industrial site defects. The result display platform and alarm device 106 are connected to the server or edge computing device 104 in communication, and can display the multispectral target detection results and issue an alarm. The multispectral detection device 102 includes: a three-axis pan-tilt bracket 1023, a detector outer shell 1024, a visible light lens I 1025, a visible light lens II 1026, an infrared lens 1027, an ultraviolet lens 1028, etc. The detector housing 1024 is rotatably connected to the three-axis pan-tilt bracket 1023, which can drive the detector housing 1024 to rotate in multiple directions and change its direction. The visible light lens I 1025, the visible light lens II 1026, the infrared lens 1027, and the ultraviolet lens 1028 are installed on the detector housing 1024, and the lenses of the four are on the same side of the detector housing 1024, and can change their directions synchronously with the detector housing 1024, which is conducive to the four lenses collecting pictures under the same field of view. Visible light lens I 1025, visible light lens II 1026, infrared lens 1027 and ultraviolet lens 1028 are cross-distributed and can collect visible light images, infrared or ultraviolet images under the same field of view. Moreover, the special design of the positions of visible light lens I 1025, visible light lens II 1026, infrared lens 1027 and ultraviolet lens 1028 is conducive to realizing the coordinate transformation of the image, and then to realizing the fusion of information results of visible light detection and infrared or ultraviolet detection, which is conducive to improving the accuracy and reliability of detection and promoting the development of intelligent detection technology.

[0065] Furthermore, the optical center of visible light lens I 1025 is horizontally aligned with the optical center of infrared lens 1027; the optical center of visible light lens I 1025 is vertically aligned with the optical center of ultraviolet lens 1028; the optical center of visible light lens II 1026 is horizontally aligned with the optical center of ultraviolet lens 1028; the optical center of visible light lens II 1026 is vertically aligned with the optical center of infrared lens 1027.

[0066] Furthermore, the visible light lens I 1025, the visible light lens II 1026, the infrared lens 1027, and the ultraviolet lens 1028 have the same field of view.

[0067] Therefore, it is beneficial to utilize binocular vision such as visible light lens I 1025 and visible light lens II 1026 to integrate the spectral information of infrared lens 1027 or ultraviolet lens 1028. The fusion reliability is strong. After the fusion, it can coordinate binocular and multi-spectrum, which is beneficial to improve the reliability of detection.

[0068] Furthermore, if Figure 2 As shown, an inspection door 1029 is provided on the top surface of the detector outer shell 1024 in an area away from the lens, and an array of long strip heat dissipation holes 1030 are designed on the inspection door 1029 .

[0069] A fence-type heat dissipation hole 1031 is provided on the top surface of the detector outer shell 1024 near the lens.

[0070] This is beneficial for heat dissipation and avoids excessive dust in industrial sites that may affect the service life of the lens.

[0071] Example 2

[0072] like Figure 3 As shown, a multi-spectral detection method for industrial site defects according to an embodiment of the present invention includes the following steps:

[0073] S202, collecting visible light images, infrared or ultraviolet images at the same field of view angle and focal length;

[0074] S204, inputting the collected visible light image, infrared or ultraviolet image into a pre-trained target detection model to obtain visible light defect detection results, infrared or ultraviolet defect detection results, and the detection results include target type labels, target frame corner point coordinate values, and classification confidence probability (0-1);

[0075] The specific training of the target detection model is as follows: collecting visible light image data, infrared or ultraviolet image data of defects at industrial sites; cleaning, preprocessing and enhancing the image data to form visible light image sets and infrared or ultraviolet image sets, wherein the preprocessing includes spatial filtering and frequency domain filtering, and the enhancement includes: hue, saturation, brightness transformation, image perspective transformation, and rotation transformation; annotating the visible light image sets and infrared or ultraviolet image sets respectively; using the visible light image sets as the input of the visible light target detection network, and using the post-infrared ultraviolet data set as the input of the infrared or ultraviolet target detection network, to train the target detection model and form a pre-trained target detection model, wherein the target detection network is any one of the YOLO series networks, SSD networks, and DETR series networks.

[0076] S206, based on the Zhang Zhengyou calibration method, perform binocular calibration on visible light lens 1-visible light lens 2, visible light lens 1-infrared lens, visible light lens 1-ultraviolet lens, visible light lens 2-infrared lens, and visible light lens 2-ultraviolet lens, respectively, to obtain the intrinsic parameter matrices K1 and K2 and the extrinsic parameter rotation matrix R and the translation vector t of each pair of lenses;

[0077] S208, based on the disparity algorithm, according to the intrinsic parameter matrices K1 and K2 and the extrinsic parameters R,t of each pair of lenses, the pictures taken by the visible light lens 1 and the visible light lens 2, respectively denoted as visible1 and visible2, are calculated pixel by pixel disparity to obtain the disparity values ​​of the corresponding pixels in visible1 and visible2, the disparity values ​​representing the translation vectors of the corresponding pixels of visible1 and visible2 in the horizontal and vertical directions, and forming a disparity map D;

[0078] S210, defining the horizontal coordinate as the x coordinate and the vertical coordinate as the y coordinate, and performing coordinate transformation on the infrared or ultraviolet defect detection result based on the stereo correction method, the regional coordinates of the infrared / ultraviolet defect detection result after correction are the same as the y direction / x direction coordinates of the corresponding area of ​​visible1, but have parallax in the x direction / y direction, and are the same as the x direction / y direction coordinates of the corresponding area of ​​visible2, but have parallax in the y direction / x direction;

[0079] S212, determining the coordinate position of the infrared or ultraviolet defect detection result in the corresponding visible light image according to the disparity map D, and updating the coordinate value of the target frame corner point in the infrared or ultraviolet defect detection result;

[0080] S214, integrating the infrared or ultraviolet defect detection result with the visible light detection result, and outputting the target detection result.

[0081] Specifically, according to the coordinate values ​​of the corner points of the target frame, it is determined whether there is a target frame in the infrared or ultraviolet defect detection result that intersects with the target frame in the corresponding visible light defect detection result;

[0082] If there is an intersection, determine whether the target type label of the infrared or ultraviolet defect detection result is the same as the corresponding visible light defect detection result;

[0083] If the target type labels are the same, the intersecting target frames are fused together, and the fusion confidence probability is calculated according to the confidence weighted fusion formula based on the classification confidence probability. The confidence weighted fusion formula is:

[0084] C o =W v C v +W i C i

[0085] Among them, C o Represented as the fusion confidence probability, W v Characterized as the visible light weight coefficient, W i Characterized as infrared or ultraviolet weight coefficient, C v Characterized by the classification confidence probability in the visible light defect detection results, C i Characterized by the classification confidence probability in infrared or ultraviolet defect detection results.

[0086] If it is determined that the target frame in the infrared or ultraviolet defect detection result does not intersect with the target frame in the corresponding visible light defect detection result, the detection result is directly output separately.

[0087] Thus, by coordinating binocular vision and multi-spectral detection, the visible light defect detection results of industrial site defects and the infrared or ultraviolet defect detection results can be quickly, efficiently and accurately integrated, improving the reliability and accuracy of the detection process.

[0088] Example 3

[0089] like Figure 4 As shown, a multi-spectral detection method for industrial site defects according to an embodiment of the present invention includes the following steps:

[0090] S301, collect visible light images, the visible light images are as follows Figure 5 As shown;

[0091] S302, inputting the visible light image into a visible light target detection network;

[0092] S303, output the visible light target detection result, such as Figure 7 As shown;

[0093] S304, collecting infrared images, infrared images such as Figure 6 As shown;

[0094] S305, inputting the infrared image into an infrared target detection network;

[0095] S306, output infrared target detection results, such as Figure 8 As shown;

[0096] S307, parallax calculation,

[0097] Specifically, based on Zhang Zhengyou calibration method, binocular calibration is performed on visible light lens 1-visible light lens 2, visible light lens 1-infrared lens, visible light lens 1-ultraviolet lens, visible light lens 2-infrared lens, and visible light lens 2-ultraviolet lens, respectively, and the intrinsic parameter matrix K1, K2 and extrinsic parameter R,t of each pair of lenses are obtained;

[0098] Based on the disparity algorithm, according to the intrinsic parameter matrices K1 and K2 and the extrinsic parameter rotation matrix R of each pair of lenses, and the translation vector t, the pictures taken by the visible light lens 1 and the visible light lens 2 are respectively recorded as visible1 and visible2, and the disparity values ​​of the corresponding pixels in visible1 and visible2 are obtained. The disparity values ​​represent the translation vectors of the corresponding pixels of visible1 and visible2 in the horizontal and vertical directions, and a disparity map D is formed.

[0099] S308, infrared detection result coordinate transformation,

[0100] Specifically, the horizontal coordinate is defined as the x coordinate, and the vertical coordinate is defined as the y coordinate. Based on the stereo correction method, the infrared defect detection result is transformed into a coordinate. The infrared defect detection result after the coordinate transformation correction has the same x coordinate as the corresponding target in visible2, and the corresponding longitudinal clamping area in visible2 is obtained. The infrared defect detection result after the coordinate transformation correction has the same y coordinate as the corresponding target in visible1, and the corresponding transverse clamping area in visible1 is obtained, as shown in FIG. Fig.10 As shown, (a) is visible1, (b) is the infrared defect detection result, and (c) is visible2;

[0101] In the disparity map D, determine the corresponding disparity value of the edge pixel of the corresponding longitudinal clamping area in visible2, denoted as D x , D y , respectively represent the disparity vectors in the x-direction and y-direction, as the translation vector, the coordinate vector of the edge pixel of the corresponding longitudinal clamping area in visible2 is recorded as (X v2 , Y v2 );

[0102] Calculate the pixel coordinate vector in visible1 corresponding to the edge pixel in visible2, denoted as (X v1 , Y v1 ), where X v1 =X v2 -D x , Y v1 =Y v2 -D y ,(X v1 , Y v1 ) The coordinate point sequence encloses the corresponding longitudinal clamping area in visible2 and the corresponding transverse clamping area in visible1, and together forms the corresponding coordinate position of the infrared defect detection result in the visible light image visible1, such as Fig.11As shown, (a) is visible1, (b) is the infrared defect detection result, and (c) is visible2;

[0103] S309, target detection result fusion output,

[0104] Specifically, according to the coordinate values ​​of the corner points of the target frame, it is determined whether a target frame in the infrared defect detection result intersects with a target frame in the corresponding visible light defect detection result;

[0105] If there is an intersection, determine whether the target type labels of the infrared defect detection result and the corresponding visible light defect detection result are the same;

[0106] If the target type labels are the same, the intersecting target frames are fused together, and the fusion confidence probability is calculated according to the confidence weighted fusion formula based on the classification confidence probability. The confidence weighted fusion formula is:

[0107] C o =W v C v +W i C i

[0108] Among them, C o Represented as the fusion confidence probability, W v Characterized as the visible light weight coefficient, W i Characterized by infrared weight coefficient, C v Characterized by the classification confidence probability in the visible light defect detection results, C i Characterized by the classification confidence probability in infrared defect detection results.

[0109] If it is determined that the target frame in the infrared defect detection result does not intersect with the target frame in the corresponding visible light defect detection result, the detection result is directly output separately.

[0110] The fused target detection results are as follows: Fig. 9 As shown in the figure, it is clearer and more accurate, and can clearly display the target frame. Figure 7 The visible light defect detection results can detect leaks and other defects more comprehensively, making it less likely to miss inspections.

[0111] Example 4

[0112] On the basis of Example 2 or Example 3, the following steps are added:

[0113] Based on the output target detection results, an on-site leakage detection report is generated. If the amount of water leakage, the amount of oil leakage and the confidence probability are all higher than the corresponding preset thresholds, an alarm signal is issued.

[0114] Thus, the test results can be displayed more clearly and early warnings can be issued in a timely manner.

[0115] The steps in the method of the present invention can be adjusted in order, combined or deleted according to actual needs.

[0116] The units in the device of the present invention can be combined, divided and deleted according to actual needs.

[0117] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0118] In the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise clearly defined. The terms "installed", "connected", "connected", "fixed", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0119] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by terms such as “upper”, “lower”, “left”, “right”, “front” and “back” are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or unit referred to must have a specific direction, be constructed and operated in a specific orientation, and therefore, cannot be understood as a limitation on the present invention.

[0120] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0121] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-spectral defect detection system for industrial sites, characterized in that: include: A multi-spectral detection device, wherein the multi-spectral detection device comprises a three-axis gimbal bracket; The detector outer shell is rotatably connected to the three-axis pan-tilt bracket, and the three-axis pan-tilt bracket controls the position adjustment of the detector outer shell; the visible light lens I, the visible light lens II, the infrared lens, and the ultraviolet lens are installed on the detector outer shell, and the lenses of the four are on the same side of the detector outer shell; the visible light lens I, the visible light lens II, the infrared lens, and the ultraviolet lens are arranged in a cross distribution, The server or edge computing device is respectively connected to the control unit of the three-axis gimbal bracket, the visible light lens I, the visible light lens II, the infrared lens, and the ultraviolet lens to receive the image data collected by the multispectral detection device. The server or edge computing device is equipped with software modules, including a target detection network module, a parallax calculation module, and a multispectral target detection result fusion module; The result display platform and the alarm device are connected to the server or edge computing device in communication to display the multi-spectral target detection results and issue an alarm.

2. The industrial site defect multispectral detection system according to claim 1, characterized in that: The optical center of the visible light lens I is aligned horizontally with the optical center of the infrared lens; the optical center of the visible light lens I is aligned vertically with the optical center of the ultraviolet lens; the optical center of the visible light lens II is aligned horizontally with the optical center of the ultraviolet lens; The optical center of the visible light lens II is vertically aligned with the optical center of the infrared lens.

3. The industrial site defect multispectral detection system according to claim 2, characterized in that: The viewing angles of visible light lens I, visible light lens II, infrared lens and ultraviolet lens are the same.

4. The industrial site defect multispectral detection system according to claim 1, characterized in that: An inspection door is arranged at a position on the top surface of the outer shell of the detector away from the lens, and an array of long strip heat dissipation holes are designed on the inspection door; A fence-type heat dissipation hole is arranged at a region near the lens on the top surface of the outer shell of the detector.

5. A method for detecting industrial site defects by multi-spectral detection using the industrial site defects multi-spectral detection system according to any one of claims 1 to 4, characterized in that: The following steps are involved: Collect visible light images, infrared or ultraviolet images at the same field of view and focal length; Input the collected visible light images, infrared or ultraviolet images into the pre-trained target detection model to obtain the visible light defect detection results, infrared or ultraviolet defect detection results. The detection results include the target type label, the coordinate values ​​of the target frame corner points, and the classification confidence probability; Based on Zhang Zhengyou calibration method, binocular calibration is performed on visible light lens 1-visible light lens 2, visible light lens 1-infrared lens, visible light lens 1-ultraviolet lens, visible light lens 2-infrared lens, and visible light lens 2-ultraviolet lens, respectively, and the intrinsic parameter matrices K1 and K2 and the extrinsic parameter rotation matrix R and translation vector t of each pair of lenses are obtained; Based on the disparity algorithm, according to the intrinsic parameter matrices K1 and K2 and the extrinsic parameters R,t of each pair of lenses, the pictures taken by the visible light lens 1 and the visible light lens 2, respectively denoted as visible1 and visible2, are subjected to pixel-by-pixel disparity calculation to obtain the disparity values ​​of the corresponding pixels in visible1 and visible2, wherein the disparity values ​​represent the translation vectors of the corresponding pixels of visible1 and visible2 in the horizontal and vertical directions, thereby forming a disparity map D; Define the horizontal coordinate as x-coordinate and the vertical coordinate as y-coordinate. Based on the stereo correction method, perform coordinate transformation on the infrared or ultraviolet defect detection result. The regional coordinates of the corrected infrared / ultraviolet defect detection result are the same as the y-direction / x-direction coordinates of the corresponding area of ​​visible1, but have parallax in the x-direction / y-direction, and the same as the x-direction / y-direction coordinates of the corresponding area of ​​visible2, but have parallax in the y-direction / x-direction. According to the disparity map D, determine the coordinate position of the infrared or ultraviolet defect detection result in the corresponding visible light image, and update the coordinate value of the target frame corner point in the infrared or ultraviolet defect detection result; The infrared or ultraviolet defect detection results are combined with the visible light detection results to output the target detection results.

6. The industrial site defect multi-spectral detection method of the industrial site defect multi-spectral detection system according to claim 5, characterized in that: The infrared or ultraviolet defect detection results are integrated with the visible light detection results to output the target detection results, including the following steps: According to the coordinate values ​​of the corner points of the target frame, it is determined whether there is a target frame in the infrared or ultraviolet defect detection result that intersects with the target frame in the corresponding visible light defect detection result; If there is an intersection, determine whether the target type label of the infrared or ultraviolet defect detection result is the same as the corresponding visible light defect detection result; If the target type labels are the same, the intersecting target frames are fused together, and the fusion confidence probability is calculated according to the confidence weighted fusion formula based on the classification confidence probability. The confidence weighted fusion formula is: C0=W v C v +W t C t Among them, C o Represented as the fusion confidence probability, W v Characterized as the visible light weight coefficient, W i Characterized as infrared or ultraviolet weight coefficient, C v Characterized by the classification confidence probability in the visible light defect detection results, C i Characterized by the classification confidence probability in infrared or ultraviolet defect detection results.

7. The industrial site defect multi-spectral detection method of the industrial site defect multi-spectral detection system according to claim 5, characterized in that: The following steps are also included: If it is determined that the target frame in the infrared or ultraviolet defect detection result does not intersect with the target frame in the corresponding visible light defect detection result, the detection result is directly output separately.

8. The industrial site defect multi-spectral detection method of the industrial site defect multi-spectral detection system according to claim 5, characterized in that: Based on the stereo correction method, the infrared defect detection result after coordinate transformation correction has the same x coordinate as the corresponding target in visible2, and the corresponding longitudinal clamping area in visible2 is obtained; Based on the stereo correction method, the infrared defect detection result after coordinate transformation correction has the same y coordinate as the corresponding target in visible1, and the corresponding lateral clamping area in visible1 is obtained; In the disparity map D, determine the corresponding disparity value of the edge pixel of the corresponding longitudinal clamping area in visible2, denoted as D x , D y , respectively represent the disparity vectors in the x-direction and y-direction, as the translation vector, the coordinate vector of the edge pixel of the corresponding longitudinal clamping area in visible2 is recorded as (X v2 , Y v2 ); Calculate the pixel coordinate vector in visible1 corresponding to the edge pixel in visible2, denoted as (X v1 , Y v1 ), where X v1 =X v2 -D x , Y v1 =Y v2 -D Y ,(X v1 , Y v1 ) The coordinate point sequence encloses the corresponding area of ​​the corresponding longitudinal clamping area in visible2 and the corresponding transverse clamping area in visible1, and together forms the corresponding coordinate position of the infrared defect detection result in the visible light image visible1.

9. The industrial site defect multi-spectral detection method of the industrial site defect multi-spectral detection system according to claim 5, characterized in that: The object detection model is trained by following the steps below, Collect visible light image data, infrared or ultraviolet image data of industrial site defects; Clean, preprocess and enhance the image data to form a visible light image set, infrared or ultraviolet image set, where the preprocessing includes spatial filtering and frequency domain filtering, and the enhancement includes: hue, saturation, brightness transformation, image perspective transformation, and rotation transformation; Label the visible light image set, infrared or ultraviolet image set respectively; Use a visible light image set as the input of a visible light target detection network, and use an infrared or ultraviolet data set as the input of an infrared or ultraviolet target detection network to train a target detection model to form a pre-trained target detection model, wherein the target detection network is any one of the YOLO series networks, SSD networks, and DETR series networks.

10. The industrial site defect multi-spectral detection method of the industrial site defect multi-spectral detection system according to claim 5, characterized in that: The following steps are also included: Based on the output target detection results, an on-site leakage detection report is generated. If the amount of water leakage, the amount of oil leakage and the confidence probability are all higher than the corresponding preset thresholds, an alarm signal is issued.

Images