Temperature monitoring method and device based on multi-modal fusion, medium and product
Through the combination of thermal infrared and visible light images, high accuracy and efficiency of temperature monitoring and identification of target areas are achieved, and the problem of low accuracy of temperature abnormality detection in the prior art is solved. It is suitable for industrial use. Real-time analysis needs in the fields of security and other fields.
Patent Information
- Application Number
- CN202411040815.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-22
AI Technical Summary
The existing thermal infrared and visible light image fusion methods have low accuracy in temperature abnormality detection and fire warning, making it difficult to meet the real-time detection and understanding and analysis needs of industrial and security fields, especially in the monitoring and identification of multiple temperatures in multiple target areas. .
Using a temperature monitoring method based on multimodal fusion, the dual-light fusion alignment process is performed by acquiring thermal infrared images and visible light images, and combining with the CLIP model, the types of high-temperature abnormal targets in the bounding box of high-temperature regions are determined, so as to achieve end-to-end semantic understanding and intelligence. .
It improves the accuracy and efficiency of temperature monitoring and target identification in the target area, has significant technical advantages and application value, and can provide efficient and accurate intelligent identification in the fields of industrial equipment hazard detection, transmission line fault diagnosis, forest fire warning, etc.
Smart Images

Figure CN119006962A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of temperature monitoring, early warning and target identification, and in particular to a temperature monitoring method, device, medium and product based on multimodal fusion. Background Art
[0002] Thermal infrared imaging technology, with its unique detection mechanism, can capture the infrared radiation emitted by objects and accurately obtain the temperature information of the target. This technology has shown a wide range of application value in many fields such as industrial inspection, medical diagnosis, and forest fire prevention. Compared with traditional single sensor warning methods such as smoke and temperature sensors, thermal infrared imaging technology has more comprehensive and detailed detection capabilities. It can not only detect high-temperature areas such as open flames and smoldering points, but also effectively identify potential risks such as equipment overheating, thereby timely obtaining key warning information such as equipment abnormalities and initial fire conditions. For safety monitoring application scenarios, the rapid response and accurate warning of thermal infrared imaging technology are crucial for emergency response.
[0003] However, thermal infrared imaging technology also faces some challenges. Due to the imaging principle and spectral characteristics of thermal infrared cameras, the resolution of their imaging is relatively limited, and it is difficult to capture the detailed texture features of objects in detail. This limits the application of thermal infrared images in target recognition and semantic understanding to a certain extent. In contrast, visible light imaging technology can capture the visual appearance information of objects with high resolution and rich colors, providing an important source of information for image understanding and computer vision. With the continuous development of large-scale data sets and deep learning technology, object detection and recognition technology based on visible light images has made significant progress. However, visible light imaging technology also has its limitations. It cannot perceive the thermal radiation information of objects, so its effect is limited in some application scenarios that require temperature detection. Therefore, in practical applications, it is necessary to comprehensively consider the advantages and disadvantages of the two technologies and select appropriate imaging technology according to the needs of specific application scenarios to achieve more comprehensive and accurate monitoring and recognition.
[0004] In view of the shortcomings of a single imaging modality, researchers at home and abroad have begun to explore methods for fusing thermal infrared and visible light images, trying to combine two complementary imaging information to achieve more comprehensive and accurate scene perception. Li et al. proposed a data enhancement method for fusing visible light and infrared images based on convolutional neural networks to solve the problem that target detectors on infrared images are difficult to handle high noise and weak visual features. Bao et al. proposed a dual-modal target detection network that combines infrared and visible light images based on the YOLO v7 (You Only Look Once v7, YOLO model v7 version) framework, which fuses the detection results of the two modalities through attention mechanisms and fusion modules, thereby improving the accuracy and robustness of infrared image target detection. The essential idea of these deep learning-based methods is to treat thermal infrared images as grayscale images, use the existing target detection algorithm framework, train a model that can directly identify specific target types on thermal infrared images through a large number of samples, and then combine the results of visible light to achieve target detection under dual-light fusion.
[0005] In specific areas such as temperature anomaly detection and fire warning, some scholars have also tried to combine thermal imaging with visible light imaging. Li Tingjian et al. disclosed a method for automatically identifying heating components of transmission lines, which detects and locates abnormal heating of transmission lines. It mainly uses Hough transform and SIFT (Scale-Invariant Feature Transform) feature recognition to match specific targets such as transmission lines and insulators, and then obtains abnormal heating information through threshold segmentation method. It does not involve automatic identification of abnormal target types. Dai Bo et al. disclosed a method and device for equipment detection based on multispectral images, which performs image recognition on visible light images and ultraviolet images in each image set to obtain temperature rise recognition results and discharge recognition results of ultraviolet images, thereby obtaining output results of whether equipment components are abnormal. This method detects visible light images based on the object recognition and positioning algorithm of YOLO, and requires the collection of a large number of image samples containing abnormal power equipment components for model training.
[0006] In summary, the existing dual-light fusion methods are still insufficient in making full use of the temperature information of thermal infrared and the visual information of visible light, especially the lack of end-to-end semantic understanding and intelligent recognition capabilities, which makes it difficult to meet the growing real-time detection and understanding analysis needs in the fields of industry, security, etc. The current application process mainly stays at the stage of "collecting abnormal samples-model fine-tuning-outputting results". Taking the inspection of power systems as an example, timely warning and rapid emergency response are crucial for dangerous situation monitoring. However, at present, thermal infrared detection mostly relies on manual duty, and abnormal conditions are discovered by observing video images or mobile handheld terminals, and then professionals analyze and interpret the collected data. This method has problems such as low efficiency, long cycle, and strong subjectivity. Most of the existing algorithms are also based on target detection frameworks such as YOLO, which are used to identify targets such as human bodies, vehicles, flames, and smoke in images. However, the difficulty in applying these specific model algorithms lies in the need to build a database containing a large number of abnormal samples, and these algorithms belong to post-detection, that is, warnings can only be issued when the transmission line has failed and significant visual features such as electric sparks, smoke, and flames appear. This makes it difficult to meet the needs of real-time monitoring, understanding and analysis of multiple temperatures in multiple target areas in the early stages of dangerous situations in industries such as industry and security.
[0007] In summary, the target area temperature monitoring methods in the prior art generally have the problem of low accuracy. Based on this, how to improve the accuracy of temperature monitoring and target identification in the target area has become a technical problem that needs to be urgently solved in this field. Summary of the invention
[0008] The purpose of this application is to provide a temperature monitoring method, device, medium and product based on multimodal fusion, which can improve the accuracy of temperature monitoring and target identification of the target area.
[0009] To achieve the above objectives, this application provides the following solutions:
[0010] In a first aspect, the present application provides a temperature monitoring method based on multimodal fusion, and the temperature monitoring method based on multimodal fusion includes:
[0011] Acquire thermal infrared and visible light images of the target area.
[0012] The thermal infrared image and the visible light image are subjected to dual-light fusion alignment processing to obtain a dual-light fusion image; the dual-light fusion image includes a thermal infrared image part and a visible light image part.
[0013] Threshold segmentation is performed on the thermal infrared image portion to obtain a plurality of high temperature region boundary boxes; the high temperature region boundary boxes are used to define the position and range of the high temperature region in the thermal infrared image portion.
[0014] According to the plurality of high temperature region boundary boxes and the visible light image parts, the CLIP model is adopted to determine the type of high temperature abnormal target in the high temperature region boundary box.
[0015] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the temperature monitoring method based on multimodal fusion described in the first aspect.
[0016] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the temperature monitoring method based on multimodal fusion described in the first aspect.
[0017] In a fourth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the temperature monitoring method based on multimodal fusion described in the first aspect.
[0018] According to the specific embodiments provided in this application, this application discloses the following technical effects:
[0019] The present application provides a temperature monitoring method, device, medium and product based on multimodal fusion, adopting a dual-light fusion method based on the combination of thermal infrared technology and visible light technology, obtaining a dual-light fusion image by acquiring a thermal infrared image and a visible light image of a target area and performing dual-light fusion alignment processing, and the dual-light fusion image includes not only the temperature information in the thermal infrared image, but also the rich texture and semantic information contained in the visible light image. Therefore, based on the dual-light fusion image, the bounding boxes of each high-temperature area in the target area and the specific types of each high-temperature abnormal target in the bounding box of the high-temperature area can be accurately determined. At the same time, the present application also introduces the CLIP model. As a visual-text pre-training model, the CLIP model can accurately identify objects in abnormal temperature areas, and combines the fusion of thermal infrared temperature information and visible light appearance semantic information to establish a complete set of "dual-light fusion alignment-thermal imaging target area positioning-visible light intelligent semantic recognition" processing flow, which effectively improves the accuracy and efficiency of abnormal high-temperature target detection and recognition, and has significant technical advantages and application value. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 A schematic diagram of a flow chart of a temperature monitoring method based on multimodal fusion provided in one embodiment of the present application;
[0022] Figure 2 A schematic diagram of the principle of a temperature monitoring method based on multimodal fusion provided in one embodiment of the present application;
[0023] Figure 3 A schematic diagram of stereoscopic correction of a binocular vision system provided in one embodiment of the present application;
[0024] Figure 4 A schematic diagram of the relationship between the disparity value and distance of a binocular vision system provided in an embodiment of the present application;
[0025] Figure 5 A schematic diagram of hierarchical segmentation of a thermal infrared image provided by an embodiment of the present application;
[0026] Figure 6 A schematic diagram of using non-maximum suppression to screen candidate region bounding boxes according to an embodiment of the present application;
[0027] Figure 7 A schematic diagram of high temperature abnormal target recognition based on the CLIP model provided in one embodiment of the present application;
[0028] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0030] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0031] like Figure 1 and Figure 2 As shown, this embodiment provides a temperature monitoring method based on multimodal fusion, which is mainly used for temperature monitoring, high temperature abnormal target recognition and high temperature danger warning in various scenarios such as industrial equipment hidden danger inspection, power transmission line fault diagnosis, forest fire warning, etc. The method specifically includes the following steps:
[0032] Step S1, obtaining a thermal infrared image and a visible light image of a target area.
[0033] Step S2: performing dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image; the dual-light fusion image includes a thermal infrared image part and a visible light image part.
[0034] Step S3, performing threshold segmentation on the thermal infrared image portion to obtain a plurality of high temperature region boundary boxes; the high temperature region boundary boxes are used to define the position and range of the high temperature region in the thermal infrared image portion.
[0035] Step S4: According to the high temperature area boundary boxes and the visible light image parts, a CLIP (Contrastive Language-Image Pre-training) model is used to determine the type of high temperature abnormal target in the high temperature area boundary box.
[0036] In an exemplary embodiment, after step S4, the following steps may also be included:
[0037] Step S5: Generate high temperature warning information according to the high temperature area boundary box and the type of the high temperature abnormal target, and send it to the user to remind the user of the high temperature abnormal situation.
[0038] In this embodiment, step S1 acquires a thermal infrared image and a visible light image of a target area, and specifically includes the following steps:
[0039] Step S11: using a binocular vision system to collect the thermal infrared image and the visible light image of the target area.
[0040] In this embodiment, the binocular vision system includes a thermal infrared camera and a visible light camera. The thermal infrared camera is used to collect the thermal infrared image, and the visible light camera is used to collect the visible light image.
[0041] In this embodiment, step S2 performs dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image, which specifically includes the following steps:
[0042] Step S21, using Zhang's calibration method to estimate the internal and external parameters of the thermal infrared camera and the visible light camera respectively; the internal and external parameters include internal parameters and external parameters, the internal parameters include focal length, principal point position and distortion coefficient, and the external parameters include rotation matrix and translation vector.
[0043] Step S22: determining the relative position relationship between the thermal infrared camera and the visible light camera according to the internal and external parameters.
[0044] Step S23: According to the relative posture relationship, a Bouguet stereo correction algorithm is used to calculate a correction rotation matrix and a projection matrix.
[0045] Step S24: performing geometric correction and epipolar alignment on the thermal infrared image and the visible light image according to the correction rotation matrix and the projection matrix, so as to align the thermal infrared image and the visible light image at the pixel level to obtain a corrected image.
[0046] Step S25: using a bilinear interpolation method, extracting original pixel values from the original thermal infrared image and the original visible light image, and filling the values into the corrected image to obtain the dual-light fusion image.
[0047] In this embodiment, step S3 performs threshold segmentation on the thermal infrared image portion to obtain a number of high temperature area boundary boxes, which specifically includes the following steps:
[0048] Step S31 , performing preprocessing of noise filtering, image enhancement and contrast enhancement on the infrared image part to obtain a preprocessed infrared image.
[0049] Step S32: Slice the preprocessed infrared image according to a preset window size to obtain a layered temperature slice image.
[0050] Step S33: adopting a threshold segmentation method to perform threshold segmentation on the layered temperature slice image to obtain a candidate region binary image.
[0051] Step S34: perform morphological processing and region contour detection processing on the candidate region binary image to obtain bounding boxes of multiple candidate regions, and simultaneously calculate a normalized weighted average temperature value in each of the bounding boxes as a score of the bounding box.
[0052] Step S35 : According to the scores of the respective bounding boxes, a non-maximum suppression algorithm is used to filter all the bounding boxes to obtain a plurality of high temperature area bounding boxes.
[0053] In this embodiment, step S4 uses the CLIP model to determine the type of high temperature abnormal target in the high temperature area boundary box according to the plurality of high temperature area boundary boxes and the visible light image portion, and specifically includes the following steps:
[0054] Step S41 : determining each image block in the visible light image portion according to the plurality of high temperature region boundary boxes, each of the image blocks including appearance and texture information of the visible light image.
[0055] Step S42: input each of the image blocks into the CLIP model to extract a feature vector corresponding to the image block.
[0056] Step S43, performing cosine similarity matching between the feature vector corresponding to each image block and the preset text feature vector to determine the type of high temperature abnormal target in each image block; the preset text feature vector is a text feature vector extracted from the object category text set based on the usage scenario using the CLIP model.
[0057] Step S44: Determine the type of the high temperature abnormal target in each of the high temperature region boundary boxes according to the type of the high temperature abnormal target in each of the image blocks.
[0058] In this embodiment, step S5 generates high temperature warning information according to the high temperature area boundary box and the type of the high temperature abnormal target, and specifically includes the following steps:
[0059] Step S51: Generate high temperature warning information according to the category hazard level and temperature hazard level of each high temperature abnormal target in the high temperature area boundary box.
[0060] In this embodiment, the category hazard level is a parameter obtained by evaluating the potential hazards of different high-temperature abnormal target categories based on prior knowledge, and is used to characterize the degree of hazard of the high-temperature abnormal target category; the temperature hazard level is a parameter calculated based on the temperature information obtained by processing the thermal infrared image, and is used to characterize the degree of hazard of the high-temperature abnormal target temperature.
[0061] Step S52: Send the high temperature warning information to the user to remind the user to pay attention to the abnormal high temperature situation in the target area, that is, inform the user of the specific location and object type of the abnormal high temperature situation.
[0062] This embodiment mainly includes three parts: dual-light fusion of thermal infrared camera and visible light camera, threshold segmentation of thermal infrared images to locate high-temperature areas, and abnormal target recognition and warning information generation. In order to facilitate understanding of the technical solution of this embodiment, the above three parts are introduced in detail below.
[0063] (1) Dual-light fusion for thermal infrared camera and visible light camera.
[0064] This embodiment builds a binocular vision system consisting of a thermal infrared camera and a visible light camera, and then uses Zhang's calibration method or other algorithms to estimate the internal and external parameters of the two cameras and solve different relative posture relationships. Based on the Bouguet stereo correction algorithm, the correction rotation matrix and projection matrix are calculated and applied to perform geometric correction and epipolar alignment on the two camera images. At a short baseline distance, the thermal infrared image is corrected by manually setting or using a sensor to adjust the offset in real time to achieve pixel-level alignment with the visible light image, where the short baseline distance refers to the distance between the optical center of the thermal infrared camera and the optical center of the visible light camera, such as Figure 3As shown. Finally, the bilinear interpolation method is used to extract the original pixel values from the original thermal infrared image and the original visible light image according to the corrected coordinates, and fill them into the corrected image to complete the fusion of the thermal infrared and visible light images, and generate a dual-light fusion image containing temperature and texture information. Compared with the traditional method of feature point detection and then alignment, the dual-light fusion method of the thermal infrared camera and the visible light camera in this embodiment does not need to repeatedly extract and match features on the real-time data stream, which greatly reduces the amount of calculation and significantly improves real-time performance and efficiency.
[0065] In this embodiment, a left-right binocular vision system consisting of a thermal infrared camera (denoted as T, representing Thermal) and a visible light camera (denoted as C, representing Color) is constructed. By adjusting the relative positions of the two cameras and selecting appropriate lenses, the three conditions of substantially parallel optical axes of the cameras of the two modes, close fields of view, and as short a baseline distance as possible are satisfied. Then, stereo calibration and stereo correction are performed. Among them, stereo calibration is used to determine the relative position and direction between the cameras, that is, to determine the internal and external parameters between the left and right thermal infrared cameras and the visible light camera. The purpose of stereo correction is to enable images of the two modes and resolutions to achieve the effect of ideally parallel optical axes and aligned scan lines. The correction schematic is shown in FIG. Figure 3 shown.
[0066] In this embodiment, during the stereo calibration operation, Zhang's calibration method or other calibration algorithms are used to simultaneously shoot the thermal calibration plate using a thermal infrared camera T and a visible light camera C, and the intrinsic parameter matrices K of the two cameras are estimated respectively. T , K C , and the distortion coefficient kc T , kc C Through the stereo calibration algorithm, the relative position relationship between the thermal infrared camera T and the visible light camera C is solved through one or more sets of associated feature points, that is, the rotation matrix R TC and the translation vector t TC .
[0067] In this embodiment, during the stereo correction operation, the intrinsic parameter matrix (K T , K C ), distortion coefficient (kc T , kc C ) and external parameters (rotation matrix R TC and the translation vector t TC ), based on the Bouguet stereo correction algorithm, calculate the correction rotation matrix (R T ,R C ), and calculate the projection matrix P T and P C , the expression is:
[0068]
[0069] Among them, O T and O C They respectively represent the center coordinates of the thermal infrared camera and the visible light camera in the common coordinate system.
[0070] Then, the corrected pixel coordinates are calculated based on the rotation matrix and the projection matrix. T (u,v), u and v represent the horizontal and vertical coordinates of the pixel in the image respectively, then the corrected pixel coordinates I′ T (u′,v′) is expressed as:
[0071]
[0072] Similarly, we can get the midpoint I of the visible light image C , its corrected pixel coordinate I′ C .
[0073] The target of epipolar correction is coplanar and the lines are aligned, so v′=v. Since the binocular setting has a parallax in the horizontal direction, the binocular setting can be used to measure the scene depth value. The relationship between the parallax d and the scene depth value Z is expressed as:
[0074]
[0075] Among them, f represents the focal length of the camera, which can usually be obtained from the camera's intrinsic parameter matrix K, b represents the baseline distance between the left and right cameras, that is, the distance between the optical centers of the two cameras, and d represents the disparity of the pixel point in the left and right images. When the baseline distance is determined, the corresponding relationship between the scene depth and the disparity value is as follows Figure 4 shown.
[0076] When the relative position of the binocular cameras is fixed and the focal length is fixed, the short baseline distance is proportional to the scene depth, that is, the shorter the baseline distance, the closer the measurable depth distance. Assuming that the baseline distance between the multimodal cameras is short enough, the binocular vision system built in this way cannot achieve depth estimation. By reversely utilizing this feature, under ideal conditions, a binocular vision system with a horizontal arrangement and a sufficiently small baseline distance can be built to achieve the multimodal image alignment effect in the horizontal direction. However, in practical applications, it is impossible to completely eliminate the baseline distance. Under the condition that the scene depth cannot be perceived, the pixel-level alignment of the multimodal images will be affected to a certain extent. However, for specific short-distance applications, such as indoor abnormality monitoring, the multimodal image alignment effect can be achieved by manually setting the maximum offset to δ.
[0077] Assuming that the offset is δ, the thermal infrared camera’s epipolar correction result is artificially offset. Assuming the offset is δ, the corrected thermal infrared image I″ T It can be expressed as:
[0078] I″ T (u′,v′)=I′ T (u+δ,v) (4).
[0079] Among them, the offset δ can be calibrated at a specific distance by an accurate calibration object (such as a high-precision target), or it can be measured in real time using a low-cost single-point ToF (Time of Flight) sensor, and the pixel-level alignment of the thermal infrared image and the visible light image can be achieved by real-time updating of the offset correction.
[0080] In this embodiment, the real-time measurement process of the single-point ToF sensor is to use a single-point ToF sensor model VL53L1X launched by ST Microelectronics. The typical working distance is 0.3-4m. This sensor has a built-in micro laser diode to emit 940nm infrared laser pulses. When the laser pulse hits the surface of the target object, it will be reflected back to the sensor. At this time, the sensor will accurately measure the flight time of the laser pulse from emission to reflection back to the sensor, and then estimate the distance value according to the formula (speed of light × flight time / 2).
[0081] Finally, bilinear interpolation or other interpolation methods are used to obtain the original pixel values from the original image according to the corrected image coordinates, and fill them into the corrected image, so as to realize the epipolar alignment and fusion of the thermal infrared camera and the visible light camera images, and obtain the dual-light fusion image with pixel-level registration at a specific distance, overcome the epipolar correction error caused by camera imaging differences, and provide good data support for subsequent target detection and recognition. This embodiment utilizes the short baseline characteristics of the binocular vision system, obtains the relative position relationship between cameras through a calibration, and realizes efficient dual-light fusion. Compared with the traditional image registration method, there is no need to repeatedly extract and match the real-time data stream, which greatly reduces the amount of calculation and significantly improves the real-time and efficiency of dual-light fusion. Only one calibration process is required when the relative position of the binocular vision system changes, and there is no need to repeatedly match the real-time data stream, which can effectively improve the efficiency of dual-light fusion.
[0082] (2) Threshold segmentation of thermal infrared images to locate high temperature areas.
[0083] In this embodiment, the collected thermal infrared image is first preprocessed, including noise filtering, image enhancement, etc. Here, gamma transform is used for contrast enhancement. Then, based on the image data range, a layered temperature slice image is established. For example, the thermal infrared image is regarded as a three-dimensional contour map, and sliced once every 10 degrees Celsius according to a preset window size. Assuming that the value range of the thermal infrared image is 0-550 degrees Celsius, it can be divided into multiple images according to the range of [0, 10], [11, 20], ..., [541, 550], such as Figure 5 As shown, a thermal infrared image is divided into multiple slices of different temperature ranges to form a layered temperature slice image. A temperature measuring thermal infrared sensor is used here to obtain an image containing the temperature value of the scene target. Using the pre-set warning temperature as the warning threshold, the threshold segmentation algorithm is used for the layered temperature slice image containing multiple temperature slices to generate a candidate region binary image. Then, the candidate region binary image is morphologically processed and the region contour is detected to obtain the bounding box of multiple candidate regions, and the normalized average temperature value in the bounding box is calculated as the score of the bounding box, which is the average temperature value in the bounding box divided by the maximum measurement value of the thermal infrared sensor. Then, the candidate region bounding boxes obtained on all slice images are filtered using the IoU (Intersection of Union) NMS (Non Maximum Suppression) algorithm to obtain the significant high temperature area in the thermal infrared image. Compared with the traditional single image threshold segmentation method, the hierarchical segmentation method of this embodiment can give priority to retaining the bounding box of the high temperature area when the bounding box IoU is large. Then, according to the bounding box of the high-temperature area, the image block containing the appearance and texture information is obtained in the aligned visible light image to facilitate the next step of area recognition.
[0084] Among them, the morphological processing uses the opening operation, which mainly includes corrosion processing and dilation processing. First, the candidate region binary image is corroded, and then the corroded image is dilated to complete the morphological processing operation. The opening operation has a denoising effect, and its purpose is to clearly divide different objects or parts that are slightly connected in the candidate region binary image into different regions.
[0085] In this embodiment, the corrected thermal infrared image I″ T Perform preprocessing, such as noise filtering, image enhancement, etc. Use gamma transformation with γ=1.5 for contrast enhancement, expressed as:
[0086]
[0087] Among them, γ is the gamma transformation parameter. When γ < 1, the temperature difference contrast of the transformed thermal infrared image is reduced; when γ > 1, the temperature difference of the transformed thermal infrared image is enhanced.
[0088] Then, based on the temperature measurement range of the thermal infrared sensor, a layered temperature slice image is established. Figure 5 As shown in the figure, the thermal infrared image is regarded as a three-dimensional contour map, and sliced according to the preset window size Δt (for example, every 10 degrees Celsius). Suppose the temperature range of the i-th slice is [t i ,t i +Δt], then the i-th slice image It can be expressed as:
[0089]
[0090] Use the pre-set warning temperature alert As a threshold, slice the image for each layer of temperature The threshold segmentation algorithm is used to generate the candidate region binary map
[0091]
[0092] For the candidate region binary map Morphological processing is performed, expressed as:
[0093]
[0094] Among them, S is the structural element; Represents the erosion operation, which is used to remove noise and separate small connected areas, ensuring that the adhered areas can be detected as multiple adjacent areas rather than a whole by the regional contour; Represents the dilation operation, which restores the original size of the binary image area and ensures that the high-temperature area with a low pixel ratio can be effectively detected for regional contours.
[0095] Then, the binary image after morphological processing Perform region contour detection to obtain the candidate region bounding box set {B i,j |j=1,2,3}, j represents the jth bounding box on the slice image. For each bounding box B i,j , with its weighted average temperature as the score, expressed as:
[0096]
[0097] Among them, g(x, y) is a two-dimensional Gaussian function, whose mean is the center of the bounding box, and the variance can be adaptively adjusted according to the size of the bounding box (that is, the number of pixels contained in the bounding box).
[0098] Then, the candidate region bounding box set {B i,j}, use the non-maximum suppression algorithm to filter and obtain the final high temperature area bounding box set like Figure 6 As shown in the figure, the purpose of using the non-maximum suppression algorithm for filtering is to prioritize the bounding boxes with higher weighted average temperatures as the input for the next region recognition when the intersection-over-union (IOU) of each bounding box is greater than the preset threshold, so as to achieve the effect of preferentially identifying high-temperature areas when there are overlapping target areas.
[0099] Finally, according to the bounding box of the high temperature area After the correction, the visible light image I' C Extract image patches containing appearance and texture information As the input for the next step of area recognition, it is expressed as:
[0100]
[0101] (3) Abnormal target identification and warning information generation.
[0102] In this embodiment, the bounding box of the high temperature area and the image block containing appearance and texture information extracted from the aligned image can provide rich discrimination basis for the generation of intelligent recognition warning information. This embodiment innovatively introduces a powerful multimodal visual pre-training model, namely the CLIP model, which makes full use of its multimodal semantic knowledge and can achieve good recognition effect without a large amount of labeled data. Specifically, the image block containing appearance and texture information is input into the visual encoder of the CLIP model to extract the feature vector, and then the cosine similarity matching is performed with the text features of the candidate category to obtain the most likely target category. Among them, the text features of the candidate category are different according to different usage scenarios, which refers to some pre-set object category texts that need to be identified, for example, "air outlet", "flame", "smoke", etc. are set. Since the usage scenario has been determined, these texts can be converted into text feature vectors using the text encoder of the CLIP model in advance, which is convenient for subsequent direct matching. Thanks to the comprehensive understanding ability of the CLIP model, the recognition process can accurately identify industrial equipment components, power line fault points, fire hazard targets, etc. without cumbersome feature engineering and model training. Finally, the recognition results of each image block are post-processed and warning information is generated. Post-processing mainly refers to the evaluation of the category hazard level and temperature hazard level of each high-temperature area. According to expert knowledge and historical data, the hazard level of the category in each candidate category text is evaluated, and corresponding hazard weights and temperature thresholds are set for different targets. The category hazard level and temperature hazard level of each high-temperature area are comprehensively considered to generate warning information, and the user is prompted to pay attention to the abnormal situation in the corresponding area through text or voice.
[0103] This embodiment uses the CLIP model to analyze image blocks containing appearance and texture information. The CLIP model is a multimodal (text and image) pre-trained model developed by OpenAI, which maps images and texts into the same semantic space to achieve alignment between images and texts. The CLIP model mainly includes a text encoder and an image encoder, where the text encoder is used to encode text data and extract text features; the image encoder is used to encode image data and extract image features. The CLIP model aims to combine natural language understanding and computer vision, so that the model can jointly understand and reason about images and texts. The processing process of abnormal target recognition based on CLIP is as follows: Figure 7 shown.
[0104] In this embodiment, firstly, a set of text descriptions related to the currently monitored scene needs to be prepared. For example, for scenarios such as industrial equipment hidden danger inspection and power transmission line fault diagnosis, the text description can be "air outlet", "power supply box", "power socket", "circuit breaker", "smoke", "flame", "human body", etc. For scenarios such as forest fire warning, the text description can be "trees", "dry wood", "wet wood", "smoke", "flame", "human body", etc. For each text description t n ,Use the text encoder of the CLIP model to extract its text feature vector, which is expressed as:
[0105] f text,n =CLIP text (t n ) (11).
[0106] It should be noted that the text description in this embodiment is not limited to physical category information, but can also be combined with specific application scenarios to pre-describe certain unclear object categories using a human-defined text attribute to improve the matching degree in specific scene applications. For example, when extracting text feature vectors, "air outlet" can be replaced with the text description "a white area with a rasterized structure".
[0107] Then, for each image block This embodiment uses the CLIP model to extract its visual feature vector, which is expressed as:
[0108]
[0109] Among them, CLIP image represents the image encoder part of the CLIP model, which maps the image into a vector representation; f image,kis the extracted visual feature vector; k∈{1,2,...,K} represents the kth image block.
[0110] Then use cosine similarity to calculate each image block With each text description n The feature vector similarity score between And get the text description with the highest similarity It is expressed as:
[0111]
[0112] Among them, ‖·‖2 represents the L2 norm of the vector. The value range of cosine similarity s(·) is [-1, 1]. The larger the value, the more similar the image block is to the text description. If Where τ is the preset similarity threshold, then the image block is considered Belongs to the category of text description Otherwise, the image patch is considered not to belong to any predefined category and is marked as “unknown”.
[0113] In this embodiment, the CLIP model is combined to identify potential abnormal high temperature targets, which can make full use of large-scale pre-trained multimodal semantic knowledge and achieve good recognition results without a large amount of labeled data. At the same time, the CLIP model has good generalization ability and can identify target categories that do not appear in the training set by adjusting the text descriptions of different text descriptions without retraining the model. This is very practical for open scenarios such as high temperature target recognition and can effectively respond to various unknown and newly emerging high temperature warning events.
[0114] Finally, the recognition results of each image block are post-processed and warning information is generated. This embodiment comprehensively considers the category hazard level and temperature hazard level of each high temperature area to generate the final warning information. The category hazard level is obtained by evaluating the potential hazard of different target categories based on prior knowledge, while the temperature hazard level is calculated based on the temperature information obtained by thermal infrared image processing. First, based on expert knowledge and historical data, each text description category t n The risk level is evaluated and corresponding risk weights W(t n ) and temperature threshold D(t n ). For example, the “flame” text description is manually assigned a higher weight due to its high potential for danger, while the “human body” category is manually assigned a lower weight due to its high temperature but relatively low danger.
[0115] Then, calculate the bounding box of each high temperature area and calculate its maximum temperature value T max,jFor each high temperature area Identify results based on their category Calculate the warning score S of the area alert,k , expressed as:
[0116]
[0117] in, and Respectively represent the text descriptions with the highest similarity The corresponding hazard weight and temperature threshold; The average temperature of the area T avg,k Mapped to the [0, 1] interval to reflect the relative high and low temperature of the area, expressed as:
[0118]
[0119] Among them, T min and T max Respectively represent the minimum and maximum temperatures on the current thermal infrared image.
[0120] In this embodiment, when generating warning information, a high temperature warning reminder threshold S is preset. th and high-risk item warning threshold w th Since the above clearly defines the corresponding risk weight W(t n ), for example, the human body's risk weight W(t n ) is set to 0.2, and the fire danger weight W(t n ) is set to 0.9, and the high-risk item warning threshold w th Set to 0.6, then if S alert,k ≥S th , indicating that high temperature is detected, and according to the recognition results For the corresponding flame description, we get If the value is higher than the high-risk item warning threshold of 0.6, a "high temperature high risk" warning will be output. alert,k ≥S th and Generate a "high temperature and high risk" warning, indicating that there are high temperature and high risk targets in the area; if S alert,k ≥S th and If the system generates an "abnormal temperature" warning, it means that the temperature in the area is abnormal, but the danger of the target is relatively low; otherwise, no warning information is generated, but for the convenience of subsequent analysis and tracking, the system will record the relevant information of the area, such as category, temperature, location, etc., for subsequent evaluation. Regardless of the output of the warning, the category information of the area will be given. So that users can quickly understand the nature and status of the warning area.
[0121] In an exemplary embodiment, in addition to generating warning information, the direct high temperature area recognition results can also be output, for example, the boundary box of the high temperature area is directly drawn on the original visible light image, and the text category to which it belongs and the highest measured temperature value are annotated. For image blocks with low recognition confidence but high weighted average temperature scores, combined with other information (such as temperature value, spatial position, etc.), they are manually reviewed, and the recognition information is output in the form of text or voice to prompt the user to pay attention to abnormal conditions in the corresponding area.
[0122] This embodiment only uses a thermal infrared camera and a visible light camera to build a binocular vision system, and achieves accurate pixel-level alignment in a scene by presetting the working distance. In other exemplary embodiments, a three-eye vision system consisting of a TOF camera (or laser radar), a thermal infrared camera and a visible light camera can also be constructed, and the depth value of the scene can be accurately measured by the TOF camera (or laser radar), to achieve the adaptive fusion process of the thermal infrared camera and the visible light camera, and to perform three-dimensional positioning of the target, so as to improve the accuracy and robustness of spatial positioning in abnormal areas.
[0123] In an exemplary embodiment, in addition to adding a TOF camera (or lidar) to accurately measure the depth value of the scene, additional training data can be collected to train a deep neural network to perform real-time depth estimation of thermal infrared images or visible light images, or to align the thermal infrared camera with the visible light camera without significant scene depth perception. However, compared with the solution provided in this embodiment, the real-time performance will be reduced and additional training data support will be required, and the generalization ability for different scenes will be weaker.
[0124] In an exemplary embodiment, in terms of abnormal target recognition, a deep neural network can be trained for a specific application scenario to specifically identify abnormal temperature areas in thermal infrared images. Through training with a large amount of labeled data, the deep neural network can learn complex features and achieve high-precision anomaly detection. However, compared with this embodiment, this type of method requires the collection of a large amount of sample data for a specific application scenario, and the actual operation is more complex.
[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image data such as thermal infrared images and visible light images, as well as temperature data, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a temperature monitoring method based on multimodal fusion is implemented.
[0126] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0127] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.
[0128] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.
[0129] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.
[0130] The embodiment proposed has the following obvious advantages:
[0131] 1) Based on the stereo calibration and correction method of the binocular vision system, real-time data stream alignment is realized, which can make full use of the complementary information between thermal infrared temperature values and visible light visual features to achieve coordinated perception of temperature and appearance features. It is more comprehensive and accurate than single modality recognition and improves the accuracy of temperature monitoring.
[0132] 2) Use hierarchical threshold segmentation and region extraction algorithms combined with the CLIP model to achieve end-to-end open object recognition, support the automation and intelligence of multiple high-temperature target detection and recognition processes in application scenarios, greatly reduce the cost of manual interpretation, and the recognition process is simple and efficient, and the algorithm responds quickly, which can meet the real-time analysis needs of industry, electricity, fire protection and other fields.
[0133] 3) Introducing an advanced multimodal visual text pre-training model, using the rich semantic knowledge learned by the CLIP model on large-scale image and text data, it does not need to be retrained for specific scenes and objects. It has superior generalization performance and a highly versatile technical solution. It is not limited to specific bands and imaging devices, and is expected to be expanded to more modal fusion recognition tasks.
[0134] This embodiment provides a complete abnormal temperature automatic identification and warning process, including multi-modal image data acquisition and fusion, abnormal area extraction and semantic recognition, warning information generation and other steps, which can not only provide a general solution framework for abnormal temperature warning, but also provide a new paradigm for the combination of computer vision and industrial scene applications. The data fusion process utilizes the short baseline characteristics of the binocular vision system, obtains the relative position relationship between cameras through offline initialization calibration, realizes efficient multi-modal real-time data stream alignment, and can make full use of the complementary information between thermal infrared temperature values and visible light visual features to achieve collaborative perception of temperature and appearance features, which is more comprehensive and accurate than single modality recognition. Compared with traditional image registration methods, there is no need to repeatedly extract and match features of real-time data streams, which greatly reduces the amount of calculation and significantly improves the real-time and matching degree of dual-light fusion. Only one calibration process is required when the internal structure of the binocular vision system changes, and there is no need to repeatedly match the real-time data stream, which can effectively improve the efficiency of dual-light fusion. At the same time, traditional target detection methods are highly dependent on building a data set containing a large number of abnormal categories. However, this embodiment extracts candidate areas through thermal infrared image processing, and then uses the CLIP model for area recognition. It does not need to rely on abnormal samples for model training, and can achieve effective fusion of temperature information and semantic information, thereby improving the versatility and reliability of target scene monitoring. The CLIP model itself does not have the ability to detect and locate targets. This embodiment cleverly applies the CLIP model to the task of abnormal temperature monitoring and early warning, and combines thermal infrared images to locate abnormal targets. This is a cross-domain innovation that explores the potential of the CLIP model in practical applications such as industrial safety and equipment monitoring.
[0135] This embodiment makes full use of visible light technology and cleverly integrates thermal infrared imaging temperature perception technology. Based on the temperature radiation information of the object, it realizes real-time monitoring and in-depth analysis and understanding of multiple temperatures in multiple target areas. Specifically, first, the precise alignment of the thermal infrared camera and the visible light camera is realized to ensure that the two modal images can be seamlessly connected at the pixel level. On this basis, the hierarchical threshold segmentation method is used to efficiently process the thermal infrared image, quickly and accurately extract different temperature radiation areas, and effectively identify abnormal temperature areas. At the same time, the rich texture and semantic information contained in the visible light image is fully utilized, combined with the visual-text pre-training model, namely the CLIP model, to accurately identify objects in abnormal temperature areas. This cross-modal information fusion method can accurately understand the object entity behind it while identifying abnormal temperatures, thereby realizing real-time, intelligent, and fine-grained analysis of high temperature or abnormal target areas. By cleverly integrating the temperature information of thermal infrared and the appearance semantic information of visible light, a complete processing flow of "dual-light fusion alignment-thermal imaging target area positioning-visible light intelligent semantic recognition" has been established. It not only improves the efficiency and accuracy of abnormally high-temperature target detection and recognition, but also demonstrates significant technical advantages and application value in practical applications. It can be widely used in the fields of industrial equipment hidden danger inspection, power transmission line fault diagnosis, forest fire warning, etc., providing related industries with efficient and accurate intelligent recognition tools, which helps to timely discover and deal with various safety hazards and improve the overall safety prevention level.
[0136] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0137] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0138] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.
Claims
1. A temperature monitoring method based on multimodal fusion, characterized in that: The temperature monitoring method based on multimodal fusion includes: Acquire thermal infrared images and visible light images of the target area; Performing dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image; the dual-light fusion image includes a thermal infrared image part and a visible light image part; Performing threshold segmentation on the thermal infrared image portion to obtain a plurality of high temperature region boundary boxes; the high temperature region boundary boxes are used to delineate the position and range of the high temperature region in the thermal infrared image portion; According to the plurality of high temperature region boundary boxes and the visible light image parts, the CLIP model is adopted to determine the type of high temperature abnormal target in the high temperature region boundary box.
2. The temperature monitoring method based on multimodal fusion according to claim 1 is characterized in that: Acquire thermal infrared images and visible light images of the target area, including: A binocular vision system is used to collect the thermal infrared image and the visible light image of the target area; wherein the binocular vision system includes a thermal infrared camera and a visible light camera, the thermal infrared camera is used to collect the thermal infrared image, and the visible light camera is used to collect the visible light image.
3. The temperature monitoring method based on multimodal fusion according to claim 2 is characterized in that: Performing dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image specifically includes: Using Zhang's calibration method, the internal and external parameters of the thermal infrared camera and the internal and external parameters of the visible light camera are estimated respectively; the internal and external parameters include internal parameters and external parameters, the internal parameters include focal length, principal point position and distortion coefficient, and the external parameters include rotation matrix and translation vector; Determining a relative position relationship between the thermal infrared camera and the visible light camera according to the internal and external parameters; According to the relative position and posture relationship, a Bouguet stereo correction algorithm is used to calculate a correction rotation matrix and a projection matrix; According to the correction rotation matrix and the projection matrix, geometric correction and epipolar alignment are performed on the thermal infrared image and the visible light image, so that the thermal infrared image and the visible light image are aligned at the pixel level to obtain a corrected image; The bilinear interpolation method is adopted to extract original pixel values from the original thermal infrared image and the original visible light image, and fill them into the corrected image to obtain the dual-light fusion image.
4. The temperature monitoring method based on multimodal fusion according to claim 1, characterized in that: Threshold segmentation is performed on the thermal infrared image portion to obtain several high temperature area boundary boxes, specifically including: Performing preprocessing of noise filtering, image enhancement and contrast enhancement on the infrared image portion to obtain a preprocessed infrared image; Slicing the preprocessed infrared image according to a preset window size to obtain a layered temperature slice image; Using a threshold segmentation method, the layered temperature slice image is subjected to threshold segmentation to obtain a candidate region binary image; Performing morphological processing and region contour detection processing on the candidate region binary image to obtain bounding boxes of multiple candidate regions, and calculating a normalized weighted average temperature value in each of the bounding boxes as a score of the bounding box; According to the scores of the respective bounding boxes, a non-maximum suppression algorithm is used to filter all the bounding boxes to obtain a plurality of high temperature area bounding boxes.
5. The temperature monitoring method based on multimodal fusion according to claim 4 is characterized in that: According to the high temperature region boundary boxes and the visible light image parts, the CLIP model is used to determine the type of high temperature abnormal target in the high temperature region boundary box, specifically including: Determining, according to the plurality of high temperature region boundary boxes, each image block in the visible light image portion, each image block including appearance and texture information of the visible light image; Input each of the image blocks into the CLIP model to extract a feature vector corresponding to the image block; Performing cosine similarity matching between the feature vector corresponding to each image block and the preset text feature vector to determine the type of high temperature abnormal target in each image block; the preset text feature vector is a text feature vector extracted from the object category text set based on the usage scenario using the CLIP model; According to the type of the abnormal high temperature target in each of the image blocks, the type of the abnormal high temperature target in each of the high temperature region boundary boxes is determined.
6. The temperature monitoring method based on multimodal fusion according to claim 5 is characterized in that: After the step of determining the type of abnormal high-temperature targets in the high-temperature region boundary box using the CLIP model according to the high-temperature region boundary boxes and the visible light image parts, the temperature monitoring method based on multimodal fusion further includes: High temperature warning information is generated according to the high temperature area boundary box and the type of the high temperature abnormal target.
7. The temperature monitoring method based on multimodal fusion according to claim 6, characterized in that: Generate high temperature warning information according to the high temperature area boundary box and the type of the high temperature abnormal target, specifically including: Generate high temperature warning information according to the category hazard level and temperature hazard level of the high temperature abnormal target in each high temperature area boundary box; wherein the category hazard level is a parameter obtained by evaluating the potential hazard of different high temperature abnormal target categories based on prior knowledge, and is used to characterize the hazard level of the high temperature abnormal target category; the temperature hazard level is a parameter calculated based on the temperature information obtained by processing the thermal infrared image, and is used to characterize the hazard level of the temperature of the high temperature abnormal target; The high temperature warning information is sent to the user to remind the user to pay attention to the abnormal high temperature situation in the target area.
8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the temperature monitoring method based on multimodal fusion as described in any one of claims 1 to 7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the temperature monitoring method based on multimodal fusion described in any one of claims 1 to 7 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the temperature monitoring method based on multimodal fusion described in any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Operation monitoring system and method of motor assembly production line
CN119472563A
Intelligent AR glasses infrared thermal image information processing method based on image processing technology
CN119649001A
Visual inertial positioning method for visible light and infrared cross-modal matching
CN119860762A
Production line abnormal condition detection method, system and terminal
CN119964094A
Intelligent gas turbine leakage inspection method based on multi-sensor data fusion
CN120369208A
