Dual-light fusion temperature detection method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a dual-light fusion temperature detection method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining an infrared light image and a visible light image of a specified typical part, carrying out the feature extraction of the infrared light image and the visible light image, and obtaining a feature extraction result of the infrared light image and the visible light image; the method comprises the steps of obtaining a first feature set and a second feature set corresponding to the first feature set and the second feature set, performing feature fusion on the first feature set and the second feature set to obtain a fused third feature set, performing image restoration and reconstruction based on the third feature set to obtain a target image, and detecting the temperature of the specified typical part based on the target image. The method has the beneficial effects that error factors influencing the temperature measurement precision of the typical part can be effectively removed, and the accurate real surface temperature of the typical part is obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of temperature detection with dual light fusion, and in particular to temperature detection with dual light fusion. Background Art

[0002] In practice, typical component managers often use remote sensing to directly measure the internal temperature of typical components to troubleshoot natural hazards. By measuring the temperature of each typical component at multiple points, they can find the typical component with elevated temperature and thus find the high-temperature area or spontaneous combustion hazard point. However, this detection method is not only labor-intensive, but also affected by various human factors, and has low efficiency. The temperature measurement range cannot meet management requirements. Summary of the invention

[0003] Based on this, it is necessary to propose a dual-light fusion temperature detection method to address the existing dual-light fusion temperature detection problem.

[0004] A temperature detection method using dual-light fusion, the method comprising:

[0005] Acquire infrared light images and visible light images of designated typical parts;

[0006] Extracting features from the infrared light image and the visible light image respectively to obtain corresponding first feature sets and second feature sets;

[0007] Fusing the first feature set and the second feature set to obtain a fused third feature set;

[0008] Performing image restoration and reconstruction based on the third feature set to obtain a target image;

[0009] The temperature of the designated typical component is detected based on the target image.

[0010] Furthermore, the step of acquiring the infrared light image and the visible light image of the designated typical component includes:

[0011] Modeling is performed on the surface of a designated typical component to obtain an original model;

[0012] The component model detected by the preset drone is integrated with the original model to form an equivalent digital map;

[0013] Setting temperature measurement points according to the equivalent digital map and planning a temperature change detection path for the preset drone;

[0014] An infrared light image and a visible light image of a designated typical component are acquired based on the detected temperature change path.

[0015] Furthermore, the step of planning the temperature change detection path of the preset drone includes:

[0016] Obtain the location information of each temperature measurement point;

[0017] Set the adaptivity function F mi (l i )=r 1 f angle (l i )+r 2 f step (l i )+r 3 f d (l i )+r 4 f dis (l i ), where F mi (l i ) represents the adaptive function based on the temperature measurement point, r 1 represents the weight related to security, r 2 and r 3 represents the weight related to time and detection efficiency, r 4 represents the weight associated with the preset UAV loss, f angle (l i ) represents the maximum turning angle when moving to the i-th temperature measurement point, f step (l i ) represents the minimum flight step length to move to the i-th temperature measurement point, f d (l i ) represents the minimum flight height to move to the i-th temperature measurement point, f dis (l i ) represents the maximum flight distance to the i-th temperature measurement point, l i Indicates the location information of the nth temperature measurement point;

[0018] The detected temperature change path is planned according to the adaptive function.

[0019] Furthermore, the step of performing modeling processing on the surface of a designated typical component to obtain an original model includes:

[0020] Obtaining the number of temperature measurement points on the surface of the specified typical component, as well as the height, coordinates and slope vector of each temperature measurement point;

[0021] According to the formula Carry out modeling processing to obtain the original model; among them, C 1 (x, y) represents the digital location of the original model Figure 2 dimensional matrix, k represents the number of temperature measurement points, d irepresents the height of the ith temperature measurement point, (x i ,y i ) represents the center coordinate of the i-th temperature measurement point, t i It represents the slope vector of the i-th temperature measurement point.

[0022] Furthermore, the step of performing image restoration and reconstruction based on the third feature set to obtain a target image includes:

[0023] Perform image restoration and reconstruction based on the third feature set to obtain a temporary image;

[0024] According to the formula V pi =‖WL‖ 2 Calculate the pixel difference; where V pi represents pixel difference, W represents a temporary image, L represents the infrared light image or the visible light image, ||.|| 2 Refers to the Euclidean distance between images;

[0025] According to the formula V = βV ss +V pi Calculate the loss value; where V represents the loss value, β represents the preset parameter, and V ss represents similar loss value, and V ss =1-SS(W, L), SS(W, L) represents the similarity between the infrared light image or the visible light image and the temporary image;

[0026] Determining whether the loss value is less than a preset loss value;

[0027] If it is less than the preset loss value, the temporary image is recorded as the target image.

[0028] Furthermore, the step of extracting features from the infrared light image and the visible light image respectively to obtain a corresponding first feature set and a second feature set includes:

[0029] Inputting the infrared light image and the visible light image into a preset residual dense network;

[0030] The convolutional layer in the residual dense network performs feature extraction on the infrared light image and the visible light image respectively to obtain a corresponding first feature set and a second feature set.

[0031] Furthermore, the step of performing image restoration and reconstruction based on the third feature set to obtain a target image includes:

[0032] Inputting the third feature set into a preset convolution kernel, and connecting the convolution layers in series to establish a decoder;

[0033] The decoder performs image restoration and reconstruction on the third feature set to obtain a target image.

[0034] A temperature detection device with dual light fusion, comprising:

[0035] An acquisition module, used for acquiring infrared light images and visible light images of designated typical components;

[0036] An extraction module, used to extract features from the infrared light image and the visible light image respectively to obtain corresponding first feature sets and second feature sets;

[0037] A fusion module, used for fusing the first feature set and the second feature set to obtain a fused third feature set;

[0038] A reconstruction module, used for performing image restoration and reconstruction based on the third feature set to obtain a target image;

[0039] A detection module is used to detect the temperature of the designated typical component based on the target image.

[0040] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0041] Acquire infrared light images and visible light images of designated typical parts;

[0042] Extracting features from the infrared light image and the visible light image respectively to obtain corresponding first feature sets and second feature sets;

[0043] Fusing the first feature set and the second feature set to obtain a fused third feature set;

[0044] Performing image restoration and reconstruction based on the third feature set to obtain a target image;

[0045] The temperature of the designated typical component is detected based on the target image. A computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the following steps:

[0046] Acquire infrared light images and visible light images of designated typical parts;

[0047] Extracting features from the infrared light image and the visible light image respectively to obtain corresponding first feature sets and second feature sets;

[0048] Fusing the first feature set and the second feature set to obtain a fused third feature set;

[0049] Performing image restoration and reconstruction based on the third feature set to obtain a target image;

[0050] The temperature of the designated typical component is detected based on the target image.

[0051] The beneficial effects of the present invention are as follows: by acquiring infrared light images and visible light images, performing feature extraction and fusion, and restoring and reconstructing according to the fused features, a target image is obtained, and temperature is detected based on the target image, thereby effectively removing error factors that affect the temperature measurement accuracy of typical components and obtaining a more accurate true surface temperature of typical components. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] in:

[0054] Figure 1 This is a diagram of the application environment of the temperature detection method of dual-light fusion in one embodiment;

[0055] Figure 2 A three-dimensional space mapping diagram in one embodiment;

[0056] Figure 3 is a schematic diagram of a diamond grid array in one embodiment;

[0057] Figure 4 A schematic diagram of a sensor array deployment in one embodiment

[0058] Figure 5 A flow chart of temperature detection of dual-light fusion in one embodiment;

[0059] Figure 6 It is a structural block diagram of a temperature detection device with dual light fusion in one embodiment;

[0060] Figure 7 FIG. 4 is a structural block diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] Figure 1 FIG. 1 is a diagram of a temperature detection application environment of dual-light fusion in an embodiment. Figure 1 , the dual-light fusion temperature detection method is applied to a dual-light fusion temperature detection system. The dual-light fusion temperature detection system includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal. The mobile terminal can be at least one of a mobile phone, a tablet computer, a laptop computer, etc. The server 120 can be implemented as an independent server or a server cluster composed of multiple servers. The terminal 110 is used to obtain the original image, and the server 120 is used to store the target image.

[0063] like Figure 5 As shown, in one embodiment, a dual-light fusion temperature detection method is provided. The method can be applied to both a terminal and a server. This embodiment is illustrated by applying to a terminal. The dual-light fusion temperature detection method specifically includes the following steps:

[0064] S1: Acquire infrared images and visible light images of designated typical parts;

[0065] S2: extracting features from the infrared light image and the visible light image respectively to obtain corresponding first feature sets and second feature sets;

[0066] S3: fusing the first feature set and the second feature set to obtain a fused third feature set;

[0067] S4: performing image restoration and reconstruction based on the third feature set to obtain a target image;

[0068] S5: Detecting the temperature of the designated typical component based on the target image.

[0069] As described in step S1 above, infrared images and visible light images of designated typical components are obtained. In open-pit coal mines, there may be some high-temperature areas or potential spontaneous combustion hazards. Therefore, infrared images and visible light images can be obtained through designated drones. Specifically, infrared cameras and visible light cameras can be set on designated drones to determine typical components to be photographed, such as materials, mechanical equipment, etc. Infrared temperature measurement can quickly detect the surface temperature of an object, while visible light imaging provides a clear visual image of the object. The combination of the two can improve the accuracy of temperature measurement. The fusion of the two has real-time monitoring capabilities, can quickly respond to temperature changes, and is suitable for temperature monitoring in dynamic environments. Through visible light imaging, information such as the shape and color of an object can be obtained at the same time. This information can help analyze the thermal condition of the object and improve the comprehensiveness of the measurement.

[0070] As described in step S2 above, feature extraction is performed on the infrared light image and the visible light image respectively to obtain the corresponding first feature set and second feature set. For feature extraction of infrared light images, the temperature distribution feature can be used: extracting thermal map features, such as maximum temperature, minimum temperature, mean temperature, etc. Edge detection can use the Canny or Sobel method to focus on the thermal edge of the component. Texture features can be extracted based on the texture features of the infrared image. Connectivity features of different temperature areas in the image are extracted to analyze the distribution of different temperature areas. For feature extraction of visible light images, edge detection can be used: edge features can be extracted using methods such as Canny edge detection and Sobel operator, and then features are obtained by extracting color histograms or using color space conversion (such as HSV color space). Texture features are obtained through Gray Level Co-occurrence Matrix (GLCM) or Local Binary Patterns (LBP), and finally through contour extraction and shape description, such as Fourier descriptors. Feature selection and dimensionality reduction are performed on the obtained features, which can be specifically reduced through principal component analysis (PCA), linear discriminant analysis (LDA), etc., to reduce the number of features and improve efficiency and accuracy.

[0071] The Canny edge detection algorithm is a widely used edge detection algorithm that aims to find the optimal edge in an image, ensuring good detection, localization, and minimum response. Its goal is to mark the actual edges in the image as accurately as possible, while ensuring that the detected edges are as close to the real edges as possible and that each edge point is marked only once.

[0072] The Sobel operator is an operator used in image processing, mainly for edge detection. It detects edges in an image by calculating the first-order gradient approximation of the image brightness function. Technically, the Sobel operator is a discrete difference operator that is applied at each point in the image to produce the corresponding gradient vector or normal vector.

[0073] As described in step S3 above, the first feature set and the second feature set are subjected to feature fusion to obtain a fused third feature set. The fusion method may be any one of simple connection, weighted fusion, principal component analysis, feature selection, and deep learning fusion. Effective feature fusion can help analyze the thermal condition of the object and improve the comprehensiveness of the measurement.

[0074] As described in step S4 above, image restoration and reconstruction is performed based on the third feature set to obtain a target image, wherein the reconstruction method may be any one of sparse representation, deep learning method, image interpolation method and optimization method to obtain the target image.

[0075] As described in the above step S5, the temperature of the designated typical component is detected based on the target image. After the target image of the typical component is obtained, since it contains the specific image of the typical component and the corresponding temperature, the error factors that affect the temperature measurement accuracy of the typical component can be effectively removed, and a more accurate real surface temperature of the typical component can be obtained.

[0076] In one embodiment, the step S1 of acquiring an infrared image and a visible light image of a designated typical component includes:

[0077] S101: Modeling is performed on the surface of a designated typical component to obtain an original model;

[0078] S102: Fusing the component model detected by the preset drone with the original model to form an equivalent digital map;

[0079] S103: Setting temperature measurement points according to the equivalent digital map, and planning a temperature change detection path of the preset drone;

[0080] S104: Acquire an infrared light image and a visible light image of a designated typical component based on the detected temperature change path.

[0081] As described in the above steps S101-S104, computer-aided design (CAD) software or 3D modeling tools (such as Blender, SolidWorks, etc.) are used to perform 3D modeling on the designated typical components to obtain the original geometric model. According to the flight path and detection logic of the drone, the component data (such as depth map, temperature distribution, etc.) collected by the preset drone is connected with the original model. Image processing and computer vision technology (such as point cloud processing, model registration, etc.) are used to achieve model fusion. Finally, an equivalent digital map is generated, which contains the shape of the target component and necessary environmental information. In the equivalent digital map, temperature measurement points are set according to the importance and characteristics of the component. The temperature measurement location can be determined according to the material, structure or working state of the component. A path planning algorithm (such as Dijkstra algorithm or RRT, etc.) is used to generate a path for the drone to detect temperature changes, ensuring that the path covers all key temperature measurement points and avoids obstacles. The planned path and the set temperature measurement points are displayed through visualization tools (such as MATLAB, Matplotlib in Python, etc.). Use the infrared camera and visible light camera on the drone to fly along the planned path and regularly acquire infrared images and visible light images. Set the camera parameters (such as exposure time, focal length, image resolution, etc.) according to the environment to ensure image quality.

[0082] In one embodiment, the step S103 of planning the temperature change detection path of the preset drone includes:

[0083] S1031: Obtaining location information of each temperature measurement point;

[0084] S1032: Setting the adaptability function F mi (l i )=r 1 f angle (l i )+r 2 f step (l i )+r 3 f d (l i )+r 4 f dis (l i ), where F mi (l i ) represents the adaptive function based on the temperature measurement point, r 1 represents the weight related to security, r 2 and r 3 represents the weight related to time and detection efficiency, r 4 represents the weight associated with the preset UAV loss, f angle (l i) represents the maximum turning angle when moving to the i-th temperature measurement point, f step (l i ) represents the minimum flight step length to move to the i-th temperature measurement point, f d (l i ) represents the minimum flight height to move to the i-th temperature measurement point, f dis (l i ) represents the maximum flight distance to the i-th temperature measurement point, l i Indicates the location information of the nth temperature measurement point;

[0085] S1033: Planning the detected temperature change path according to the adaptive function.

[0086] As described in the above steps S1031-S1033, the planning of the temperature change detection path is realized, and the position information of each temperature measurement point is obtained, wherein the temperature measurement point is a pre-set position, generally a raised part of a specified typical component, and then its specific position information is obtained, and a path planning algorithm (such as a genetic algorithm, an ant colony algorithm, etc.) is used to evaluate the score of the adaptive function to find the optimal path. Combine the obtained temperature measurement point positions and the adaptive function to calculate the path. Determine the starting point and the end point (the take-off position and the final position of the drone). Traverse all possible paths and calculate the value of its adaptive function. Select the path with the highest score (or the lowest cost) as the final temperature change detection path. Obtaining the position of the temperature measurement point, defining the adaptive function, and performing intelligent path planning based on the function can improve the drone's ability to obtain valid data during the temperature change detection process, thereby improving the measurement quality and efficiency.

[0087] In one embodiment, the step S101 of performing modeling processing on the surface of a designated typical component to obtain an original model includes:

[0088] S1011: Obtain the number of temperature measurement points on the surface of the designated typical component, as well as the height, coordinates and slope vector of each temperature measurement point;

[0089] S1012: According to the formula Carry out modeling processing to obtain the original model; among them, C 1 (x, y) represents the digital location of the original model Figure 2 dimensional matrix, x represents the horizontal position, y represents the vertical position, k represents the number of temperature measurement points, d i represents the height of the ith temperature measurement point, (x i ,y i ) represents the center coordinate of the i-th temperature measurement point, t i It represents the slope vector of the i-th temperature measuring point. exp represents the exponential function with the natural constant e as the base in higher mathematics.

[0090] As described in the above steps S1011-S1012, the acquisition of the original model is realized, that is, the number of temperature measurement points on the surface of the specified typical component, as well as the height, coordinates and slope vector of each temperature measurement point are obtained, wherein the slope vector represents the rate of change of the surface around the temperature measurement point in a specific direction. It can be calculated in the following ways: Coordinate difference method: The slope vector is obtained by calculating the coordinate difference of adjacent points. Normal vector: The slope of the local surface is obtained by calculating the normal vector of the point. Based on the acquired temperature measurement point data, the original model is generated using a 3D modeling tool (such as Blender, SolidWorks, MATLAB, etc.).

[0091] In one embodiment, the step S4 of performing image restoration and reconstruction based on the third feature set to obtain a target image includes:

[0092] S401: Perform image restoration and reconstruction based on the third feature set to obtain a temporary image;

[0093] S402: According to formula V pi =‖WL‖ 2 Calculate the pixel difference; where V pi represents pixel difference, W represents a temporary image, L represents the infrared light image or the visible light image, ||.|| 2 Refers to the Euclidean distance between images;

[0094] S403: According to the formula V = βV ss +V pi Calculate the loss value; where V represents the loss value, β represents the preset parameter, and V ss represents similar loss value, and V ss =1-SS(W, L), SS(W, L) represents the similarity between the infrared light image or the visible light image and the temporary image;

[0095] S404: Determine whether the loss value is less than a preset loss value;

[0096] S405: If it is less than a preset loss value, the temporary image is recorded as a target image.

[0097] As described in the above steps S401-S405, image restoration and reconstruction are performed based on the third feature set to obtain a temporary image, wherein the restoration and reconstruction method is the same as described above and will not be repeated here. Since the acquired temporary image may have a certain error, the pixel difference can be calculated, and then the loss value can be calculated based on the pixel difference. If the loss value is less than the preset loss value, it is recorded as the target image for temperature detection. In addition, since the input can be an infrared image or a visible light image, one can be arbitrarily selected for calculation, or different loss values ​​can be calculated separately. When all loss values ​​are less than the preset loss value, the temporary image can be recorded as the target image. Otherwise, it is necessary to reset the parameters of the image restoration and reconstruction until the obtained temporary image satisfies the loss value less than the preset loss value.

[0098] In one embodiment, the step S2 of extracting features from the infrared light image and the visible light image respectively to obtain the corresponding first feature set and second feature set includes:

[0099] S201: inputting the infrared light image and the visible light image into a preset residual dense network;

[0100] S202: The convolutional layer in the residual dense network performs feature extraction on the infrared light image and the visible light image respectively to obtain a corresponding first feature set and a second feature set.

[0101] As described in the above steps S201-S202, the acquisition of the first feature set and the second feature set is realized. The Residual Dense Network (RDN) is a deep learning network structure that combines residual connections and dense connections, and is mainly used for image processing tasks, especially image super-resolution (Super-Resolution) and image restoration. With its unique design, RDN can not only effectively retain features, but also reuse information to improve the performance of the model in image processing. Therefore, the infrared image and the visible light image are input into the network, and the features of the infrared image and the visible light image are respectively extracted in the convolution layer to obtain the corresponding first feature set and second feature set.

[0102] In one embodiment, the step S4 of performing image restoration and reconstruction based on the third feature set to obtain a target image includes:

[0103] S411: Input the third feature set into a preset convolution kernel, and connect the convolution layers in series to establish a decoder;

[0104] S412: Perform image restoration and reconstruction on the third feature set through the decoder to obtain a target image.

[0105] As described in the above steps S411-S412, the third feature set is input into the preset convolution kernel, and the convolution layer is connected in series to establish a decoder; since the first feature set and the second feature set are extracted through the convolution layer, on this basis, the convolution kernel and the convolution layer are connected in series to set a decoder, which can better decode the third feature set to obtain the target image.

[0106] In one embodiment, Figure 2 As shown in the figure, in order to solve the problem of monitoring the temperature of typical components in the site which is urgently needed in safety management, this paper combines UAV technology and proposes a real-time monitoring method of typical component temperature based on UAV platform based on the fusion technology of infrared and visible light temperature measurement, and designs and implements the corresponding temperature monitoring system; the correction technology of the surface temperature of typical components is studied, which effectively removes the error factors that affect the temperature measurement accuracy of typical components and obtains more accurate real surface temperature of typical components.

[0107] According to the formula Carry out modeling processing.

[0108] Among them C i (x, y) refers to the digital map of the typical component surface after modeling. Figure 2 dimensional matrix, k represents the number of surface protrusions, d i represents the height of each protrusion, (x i ,y i ) represents the center coordinate of the i-th protrusion, t i It represents the slope vector of the i-th protrusion.

[0109] The temperature change detection path refers to the optimal detection path, which can avoid the detection drone from contacting typical components and achieve the purpose of stable operation and reducing detection errors;

[0110] The fitness function of the optimal detection path is:

[0111] F mi (l t )=r 1 f angle (l i )+r 2 f step (l i )+r 3 f d (l i )+r 4 f dis (l i );

[0112] Where r is the weight, which is determined according to the task weight of the drone in the typical component inspection process.1 Biased towards safety, r 2 With r 3 Deflection time and detection efficiency, r 4 The loss of the drone itself is determined according to the actual needs of detection, where f angle Refers to the maximum turning angle of the drone, f step Refers to the minimum flight step length of the drone, f d Refers to the minimum flight altitude, f dis Refers to the maximum distance a drone can fly.

[0113] The dual-light fusion process of infrared and visible light images is:

[0114] Let L1 and L2 be recorded as the input infrared and visible light images, and perform residual dense network image fusion of L1 and L2;

[0115] After the image is input, it enters the encoding stage, using a convolution kernel of size 3x3 and a ReLU activation function to perform preliminary extraction of the original image through the basic convolution layer;

[0116] The initial extraction of the original image uses a residual dense network to extract details and deep features;

[0117] The extracted features are fused using the fusion layer, and the l-norm fusion strategy is used for image feature fusion;

[0118] The fused image features are put into three convolution kernels of size 3x3, and the decoder is established by connecting the convolution layers in series. The decoder uses the fused image features to perform image restoration and reconstruction to generate the fused image L3.

[0119] After image fusion, there will be losses, so it is necessary to optimize the loss function to realize the feature display of the image.

[0120] The image detection system includes data acquisition layer, data transmission layer, data processing layer and data display layer;

[0121] The data acquisition layer uses infrared and visible light temperature arrays to detect the surface temperature of typical components to achieve measurement of the surface temperature of typical components;

[0122] The data transmission layer transmits the visible light detection image and the infrared detection image to the data processing layer by wireless transmission;

[0123] The data display and query layer displays the fused image in the management interface and saves and records the data.

[0124] The data acquisition layer refers to the UAV platform, which includes the flight control module, temperature measurement module, GPS positioning module and data processing module;

[0125] The flight control module mainly consists of two parts: the digital map part and the track planning part;

[0126] The temperature measurement module mainly uses the STM32 minimum system as the development platform, and uses infrared sensors and visible light sensor arrays as temperature measurement units to complete the temperature measurement work of this module;

[0127] The GPS positioning module is mainly used to locate the UAV so that the system can determine the location of the high temperature measurement point;

[0128] The data processing module is implemented on the STM32MINI system and is mainly responsible for the preliminary processing of the array data and transmitting it to the monitoring center.

[0129] STM32 is a 32-bit microcontroller based on the ARM Cortex-M core. It is widely used in the embedded field and has the characteristics of high performance, low power consumption and low cost.

[0130] like Figure 3 As shown, the array of infrared sensors and visible light sensors is realized by a diamond grid array, in which the structure is 8 diamond grids connected to form a 3×3 array in the area with a diameter of 10 cm. The temperature measurement of a given single target is achieved through multiple sensors.

[0131] In a specific embodiment, infrared light and visible light fusion test is performed. In order to achieve end-to-end training, point cloud projection, and verify the influence of the array data correction model on the measurement data, we use an infrared temperature measurement module in the laboratory to detect the temperature of the surface of the heating object, so that the data measured at different temperatures of the heating object are shown in Table 1 below:

[0132] Table 1 Temperature values ​​measured by sensor array

[0133]

[0134] The above data is brought into the array data correction model for correction, and the corrected temperature data is shown in Table 2 below:

[0135] Table 2 Array data results after correction

[0136]

[0137] Set the initial error: θ = 10. , D = 2m, generate training sets and validation sets, and compare existing deep learning-based point cloud-image calibration methods: RegNet, CalibNet, and CMRNet. The corrected rotation error and translation error are shown in the table above. The core of the RegNet calibration method is to adopt a deep learning strategy to estimate the displacement vector field (DVF) directly from the image pair. CalibNet is a deep learning-based calibration method, which is mainly used to automatically calibrate the external parameters between 3D lidar and 2D camera. CMRNet is a monocular vision positioning method that can accurately locate in the LiDAR map and is independent of the camera's internal parameters. CMRNet achieves positioning capabilities independent of maps and cameras by combining deep learning and geometric techniques.

[0138] The results show that the proposed method is superior to previous methods in both rotation and translation errors, especially in translation results, where the error after calibration is significantly reduced compared to other methods.

[0139] The data collection layer includes an alarm module. After the drone measures the temperature along the predetermined route, it will alarm if the temperature data exceeds the safety value, reminding the managers of typical components to perform cooling treatment.

[0140] The temperature measurement module designed in this paper is an infrared and visible light sensor array, in which the infrared sensor contains 9 MLX90614 infrared sensors. Due to the design of the spacing between individual sensors, the measurement areas of adjacent infrared temperature sensors will have overlapping parts. Therefore, each MLX90614 infrared sensor can use the temperature measurement information of adjacent sensors as a reference for its own data correction. Since the measurement value of the infrared temperature sensor has been processed to eliminate gross errors, the consistency is relatively good. Therefore, the calibration method of the temperature measurement data of the internal sensor of the infrared temperature measurement array in this paper adopts the batch estimation theory. The algorithm using the batch estimation theory can not only obtain reliable initial measurement values, but also effectively eliminate many uncertainties in infrared temperature measurement, and improve the accuracy of the temperature measurement data of the infrared temperature measurement array. The sensor array designed in this paper is realized based on the shape of the diamond grid, and improvements have been made. The 8 diamond grids are connected to form a 3 3 array as a whole, which basically realizes the temperature measurement of the same target by multiple sensors. It is encapsulated into a fixed infrared temperature measurement device to reduce a series of errors caused by the position change inside the array to the measured temperature, making the temperature measurement more accurate and convenient. The sensor array deployment method adopted in this system is as follows Figure 4 The object distance ratio of the MLX-90614 infrared sensor selected in this paper is 12:1, that is, when the temperature measurement distance is 120cm, a single infrared sensor measures the average temperature within a circle with a diameter of 10cm, while ensuring that the overlap of the detection area of ​​a single infrared sensor is minimal and the overall measured area S is maximized. After analysis, we found that when the following occurs Figure 4 In the case shown, S is the smallest and S is the largest. At this time: Here, according to Figure 4 The temperature sensor deployment in the figure is the sensor position at the center of the sensor, and the rows and columns are arranged crosswise. The 9 infrared sensors form a diamond layout. When r = 10cm, the distance between the center of the circle, that is, the distance between the two sensors, is d = 5. The temperature measurement surface is expanded to a diamond temperature measurement surface with a side length of 10cm and a vertex angle of 60°, which expands our infrared temperature measurement module from point temperature measurement to surface temperature measurement, overcoming the defect of temperature measurement at a single infrared temperature measurement point; in addition, the 9-way sensor provides multiple sets of data for the same temperature measurement point, overcoming the drawback of low reliability of a single set of measurement data from the infrared temperature sensor, and improving the reliability of the obtained coal pile surface temperature data. The visible light detection image method is the same as the traditional one.

[0141] Reference Figure 6 The present invention also provides a dual-light fusion temperature detection device, the device comprising:

[0142] An acquisition module 10 is used to acquire an infrared image and a visible light image of a designated typical component;

[0143] An extraction module 20, configured to extract features from the infrared light image and the visible light image respectively to obtain a corresponding first feature set and a second feature set;

[0144] A fusion module 30, configured to fuse the first feature set and the second feature set to obtain a fused third feature set;

[0145] A reconstruction module 40, configured to perform image restoration and reconstruction based on the third feature set to obtain a target image;

[0146] The detection module 50 is used to detect the temperature of the designated typical component based on the target image.

[0147] It should be noted that other embodiments of the dual-light fusion temperature detection device and the dual-light fusion temperature detection method are the same as the above-mentioned dual-light fusion temperature detection method, and will not be repeated here.

[0148] Figure 7 FIG. 1 shows an internal structure diagram of a computer device in an embodiment. The computer device may be a terminal or a server. Figure 7As shown, the computer device includes a processor, a memory and a network interface connected via a system bus. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor may implement the temperature detection method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor may implement the temperature detection method. Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0149] In one embodiment, a computer device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the following steps:

[0150] Acquire infrared light images and visible light images of designated typical parts;

[0151] Extracting features from the infrared light image and the visible light image respectively to obtain corresponding first feature sets and second feature sets;

[0152] Fusing the first feature set and the second feature set to obtain a fused third feature set;

[0153] Performing image restoration and reconstruction based on the third feature set to obtain a target image;

[0154] The temperature of the designated typical component is detected based on the target image.

[0155] The error factors that affect the temperature measurement accuracy of typical components can be effectively removed, and a more accurate real surface temperature of typical components can be obtained.

[0156] In one embodiment, a computer-readable storage medium is provided, storing a computer program, wherein when the computer program is executed by a processor, the processor performs the following steps:

[0157] Acquire infrared light images and visible light images of designated typical parts;

[0158] Extracting features from the infrared light image and the visible light image respectively to obtain corresponding first feature sets and second feature sets;

[0159] Fusing the first feature set and the second feature set to obtain a fused third feature set;

[0160] Performing image restoration and reconstruction based on the third feature set to obtain a target image;

[0161] The temperature of the designated typical component is detected based on the target image.

[0162] The error factors that affect the temperature measurement accuracy of typical components can be effectively removed, and a more accurate real surface temperature of typical components can be obtained.

[0163] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0164] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A temperature detection method using dual light fusion, characterized in that: The method comprises: Acquire infrared light images and visible light images of designated typical parts; Extracting features from the infrared light image and the visible light image respectively to obtain corresponding first feature sets and second feature sets; Fusing the first feature set and the second feature set to obtain a fused third feature set; Performing image restoration and reconstruction based on the third feature set to obtain a target image; The temperature of the designated typical component is detected based on the target image.

2. The temperature detection method of dual light fusion according to claim 1, characterized in that: The step of acquiring the infrared light image and the visible light image of the designated typical component comprises: Modeling is performed on the surface of a designated typical component to obtain an original model; The component model detected by the preset drone is integrated with the original model to form an equivalent digital map; Setting temperature measurement points according to the equivalent digital map and planning a temperature change detection path for the preset drone; An infrared light image and a visible light image of a designated typical component are acquired based on the detected temperature change path.

3. The temperature detection method of dual light fusion according to claim 2 is characterized in that: The step of planning the temperature change detection path of the preset drone includes: Obtain the location information of each temperature measurement point; Set the adaptivity function F mi (l i )=r1f angle (l i )+r2f step (l i )+r3f d (l i )+r4f dis (l i ), where F mi (l i ) represents an adaptive function based on the temperature measurement point, r1 represents a weight related to safety, r2 and r3 represent weights related to time and detection efficiency, r4 represents a weight related to the preset drone loss, and f angle (l i ) represents the maximum turning angle when moving to the i-th temperature measurement point, f step (l i ) represents the minimum flight step length to move to the i-th temperature measurement point, f d (l i ) represents the minimum flight height to move to the i-th temperature measurement point, f dis (l i ) represents the maximum flight distance to the i-th temperature measurement point, l i Indicates the location information of the nth temperature measurement point; The detected temperature change path is planned according to the adaptive function.

4. The temperature detection method of dual light fusion according to claim 2, characterized in that: The step of performing modeling processing on the surface of a designated typical component to obtain an original model includes: Obtaining the number of temperature measurement points on the surface of the specified typical component, as well as the height, coordinates and slope vector of each temperature measurement point; According to the formula Modeling is performed to obtain the original model; where C1(x, y) represents the two-dimensional matrix of the digital map of the original model, k represents the number of temperature measurement points, d i represents the height of the ith temperature measurement point, (x i ,y i ) represents the center coordinate of the i-th temperature measurement point, t i It represents the slope vector of the i-th temperature measurement point.

5. The temperature detection method of dual light fusion according to claim 1, characterized in that: The step of performing image restoration and reconstruction based on the third feature set to obtain a target image comprises: Perform image restoration and reconstruction based on the third feature set to obtain a temporary image; According to the formula V pi = ‖WL‖2 calculates the pixel difference; where V pi represents pixel difference, W represents a temporary image, L represents the infrared light image or the visible light image, and ||.||2 refers to the Euclidean distance between images; According to the formula V = βV ss +V pi Calculate the loss value; where V represents the loss value, β represents the preset parameter, and V ss represents similar loss value, and V ss =1-SS(W, L), SS(W, L) represents the similarity between the infrared light image or the visible light image and the temporary image; Determining whether the loss value is less than a preset loss value; If it is less than the preset loss value, the temporary image is recorded as the target image.

6. The temperature detection method of dual light fusion according to claim 1, characterized in that: The step of respectively extracting features from the infrared light image and the visible light image to obtain corresponding first feature sets and second feature sets comprises: Inputting the infrared light image and the visible light image into a preset residual dense network; The convolutional layer in the residual dense network performs feature extraction on the infrared light image and the visible light image respectively to obtain a corresponding first feature set and a second feature set.

7. The temperature detection method of dual light fusion according to claim 6, characterized in that: The step of performing image restoration and reconstruction based on the third feature set to obtain a target image comprises: Inputting the third feature set into a preset convolution kernel, and connecting the convolution layers in series to establish a decoder; The decoder performs image restoration and reconstruction on the third feature set to obtain a target image.

8. A temperature detection device with dual light fusion, characterized in that: The device comprises: An acquisition module, used for acquiring infrared light images and visible light images of designated typical components; An extraction module, used to extract features from the infrared light image and the visible light image respectively to obtain corresponding first feature sets and second feature sets; A fusion module, used for fusing the first feature set and the second feature set to obtain a fused third feature set; A reconstruction module, used for performing image restoration and reconstruction based on the third feature set to obtain a target image; A detection module is used to detect the temperature of the designated typical component based on the target image.

9. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor executes the steps of the temperature detection method of dual-light fusion as claimed in any one of claims 1 to 7.

10. A computer device, characterized in that: The device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the temperature detection method of dual-light fusion according to any one of claims 1 to 7.

Images