Dual-light fusion intelligent identification device and method

By adopting dual-light fusion intelligent recognition device in the industrial field, combining thermal infrared and visible light image processing technology, the problems of data integration and collaborative scheduling of fixed and portable thermal image cameras are solved, and the accuracy and efficiency of equipment temperature status monitoring are improved. .

CN119071599APending Publication Date: 2024-12-03CHINA MARITIME POLICE ACADEMY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411041553.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the existing industrial field, fixed and portable thermal cameras fail to effectively integrate data, it is difficult to achieve comprehensive monitoring and in-depth analysis of equipment temperature status, and the lack of a coordinated scheduling mechanism, resulting in low efficiency and accuracy of equipment operating status monitoring.

Method used

Using dual-light fusion intelligent recognition device, combined with thermal infrared camera unit, visible light camera unit and laser ranging unit, image sensing and data processing are performed through edge computing module, realizing dual-light fusion alignment processing of thermal infrared images and visible light images, improving Accuracy of target recognition.

Benefits of technology

Through dual-light fusion technology, the accuracy of target recognition and the efficiency of equipment temperature status monitoring are improved, effective integration and coordinated scheduling of fixed and portable thermal imager data is realized, and real-time detection and analysis of equipment operation status is enhanced. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071599A_ABST
    Figure CN119071599A_ABST
Patent Text Reader

Abstract

The invention discloses a dual-light fusion intelligent recognition device and method, and relates to the technical field of image processing and target recognition, the device comprises an image sensing module and an edge calculation module, the image sensing module comprises a thermal infrared camera unit, a visible light camera unit and a laser ranging unit, the thermal infrared camera unit, the visible light camera unit and the laser ranging unit are all connected with the edge calculation module; the thermal infrared camera unit is used for collecting a thermal infrared image of a target scene; the visible light camera unit is used for collecting a visible light image of a target scene; the laser ranging unit is used for collecting distance information from the dual-light fusion intelligent identification device to an image imaging center; and the edge calculation module is used for performing dual-light fusion alignment processing on the thermal infrared image and the visible light image, performing correction according to the distance information in the process to obtain a dual-light fusion image, and performing target identification according to the dual-light fusion image to obtain a target identification result. The accuracy of target identification can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of image processing and target recognition, and in particular to a dual-light fusion intelligent recognition device and method. Background Art

[0002] Thermal infrared imaging technology, with its unique detection mechanism, can capture the infrared radiation emitted by objects and accurately obtain the temperature information of the target. This technology has shown wide application value in many fields such as industrial inspection, medical diagnosis, and forest fire prevention. Compared with traditional single sensor warning methods such as smoke and temperature sensors, thermal infrared imaging technology has more comprehensive and detailed detection capabilities. It can measure the surface temperature distribution at a distance without direct contact with the target. It can not only detect high-temperature areas such as open flames and smoldering points, but also effectively identify potential risks such as equipment overheating, so as to obtain key warning information such as equipment abnormalities and initial fire conditions in a timely manner. For safety monitoring application scenarios, the rapid response and accurate warning of thermal infrared imaging technology are crucial for emergency response.

[0003] At present, thermal infrared products on the market are mainly divided into two types: portable handheld thermal imagers and fixed thermal imagers. Handheld thermal imagers have the advantages of small size, light weight, easy to carry and one-handed operation, which are suitable for scenes that require flexibility and mobility such as on-site detection. Handheld thermal imagers can display the thermal distribution image of the target under test in real time, helping users to quickly and intuitively find abnormal hot spots. Many handheld thermal imagers are equipped with functions such as hot spot marking, temperature alarm, image overlay, etc., which are convenient for users to perform data analysis and fault diagnosis. Fixed thermal imagers are mainly used for continuous monitoring of key equipment and areas, and can promptly detect temperature anomalies and potential fault hazards. Fixed thermal imagers transmit the collected thermal image data to the central monitoring platform in real time through wired or wireless communication to achieve remote monitoring and data integration. Compared with handheld thermal imagers, fixed thermal imagers have fixed installation positions, and temperature thresholds and regional alarm rules can be preset. When abnormal temperatures occur, sound and light alarms are automatically triggered to remind operators to deal with them in time. In addition, fixed thermal imagers can also be linked to other equipment, such as automatic fire extinguishing systems, ventilation systems, etc., to achieve automatic processing in abnormal situations.

[0004] Thermal imager products currently used in the industrial field mainly rely on manual analysis of temperature data, and the efficiency and accuracy of discovering and diagnosing abnormal working conditions are not high. On-site personnel need to manually plan the inspection routes and measurement points of portable thermal imagers, which is a lot of work and easy to miss key areas. Existing products do not fully utilize the complementarity of fixed and portable thermal imager data, making it difficult to achieve comprehensive monitoring and in-depth analysis of the temperature status of equipment. Most thermal imager products have not achieved data integration with the equipment management system, and have failed to use temperature big data to build equipment health assessment and life prediction models. In addition, there is a lack of a coordinated scheduling mechanism for fixed and portable thermal imagers, making it difficult to dynamically optimize the allocation of temperature measurement resources based on the importance and operating status of the equipment.

[0005] Wei Haoran et al. disclosed a handheld infrared thermal imager, in which the thermal imager assembly and the handpiece, as well as the thermal imager assembly and the display device connector, are all detachably connected, which is convenient for users to modify according to actual conditions and expand the application scenarios of handheld infrared thermal imagers. Kuai Yangliu et al. proposed a RGBT real-time target tracking method and device based on feature enhancement fusion, which uses a deep neural network to extract the depth features of thermal infrared and visible light images, thereby realizing the target positioning function.

[0006] At industrial sites, deploying fixed and portable thermal imagers to conduct continuous thermal infrared measurements of the same target at different times and locations can provide more comprehensive and detailed target temperature status information, which is conducive to equipment status trend analysis and predictive maintenance, thereby improving the reliability of equipment operation. However, current fixed and portable thermal imagers usually work independently and lack effective data integration and correlation analysis methods. Therefore, how to design an intelligent recognition device that can automatically identify targets, is compatible with fixed and portable thermal imager data, and has high recognition accuracy has become a technical problem that needs to be solved urgently in this field. Summary of the invention

[0007] The purpose of this application is to provide a dual-light fusion intelligent recognition device and method, which combines thermal infrared technology with visible light technology, performs target recognition based on the image fused from the thermal infrared image and the visible light image, and can automatically identify the target, thereby improving the accuracy of target recognition.

[0008] To achieve the above objectives, this application provides the following solutions:

[0009] In a first aspect, the present application provides a dual-light fusion intelligent recognition device, which includes an image sensing module and an edge computing module.

[0010] The image sensing module includes a thermal infrared camera unit, a visible light camera unit and a laser ranging unit, and the thermal infrared camera unit, the visible light camera unit and the laser ranging unit are all connected to the edge computing module; the thermal infrared camera unit is used to collect thermal infrared images of the target scene and send the thermal infrared images to the edge computing module; the visible light camera unit is used to collect visible light images of the target scene and send the visible light images to the edge computing module; the laser ranging unit is used to collect distance information from the dual-light fusion intelligent recognition device to the image imaging center, and send the distance information to the edge computing module, and the image imaging center is the center point of the thermal infrared image and the visible light image.

[0011] The edge computing module is used to perform dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image, and perform target recognition based on the dual-light fusion image to obtain a target recognition result; wherein the dual-light fusion alignment processing includes stereo calibration processing, image row alignment processing, distance correction processing and fusion image generation, and the distance correction processing is to correct the stereo calibration processing result and the image row alignment processing result according to the distance information; the target recognition result includes the bounding box, temperature information and category information of each target object in the target scene.

[0012] In a second aspect, the present application provides a dual-light fusion intelligent recognition method, which is applied to the dual-light fusion intelligent recognition device as described in the first aspect, and the dual-light fusion intelligent recognition method includes:

[0013] Acquire the thermal infrared image, visible light image and distance information from the dual-light fusion intelligent recognition device to the imaging center of the target scene; the imaging center is the center point of the thermal infrared image and the visible light image.

[0014] The thermal infrared image and the visible light image are subjected to dual-light fusion alignment processing to obtain a dual-light fusion image; the dual-light fusion alignment processing includes stereo calibration processing, image row alignment processing, distance correction processing and fusion image generation, and the distance correction processing is to correct the stereo calibration processing result and the image row alignment processing result according to the distance information.

[0015] Target recognition is performed according to the dual-light fusion image to obtain a target recognition result; the target recognition result includes a bounding box, temperature information and category information of each target object in the target scene.

[0016] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0017] The present application provides a dual-light fusion intelligent recognition device and method, which combines thermal infrared technology, visible light technology and ranging technology, collects thermal infrared images, visible light images and distance information through thermal infrared camera units, visible light camera units and laser ranging units respectively, and combines thermal infrared, visible light and distance multiple dimensions to perform target recognition, thereby improving the accuracy of target recognition. In addition, in the process of dual-light fusion alignment processing of thermal infrared images and visible light images to obtain dual-light fusion images, the stereo calibration processing results and image row alignment processing results are corrected according to the distance information to ensure that the thermal infrared image and the visible light image are accurately matched at different measurement distances. Then, target recognition based on the dual-light fusion image obtained by fusion of thermal infrared images and visible light images can effectively improve the accuracy of target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A schematic diagram of an application scenario of a dual-light fusion intelligent recognition device provided in one embodiment of the present application.

[0020] Figure 2 A schematic diagram of the structure of a dual-light fusion intelligent recognition device provided in one embodiment of the present application.

[0021] Figure 3 This is a structural explosion diagram of a dual-light fusion intelligent recognition device provided in one embodiment of the present application.

[0022] Figure 4 A schematic diagram of the functional modules of a dual-light fusion intelligent recognition device provided in one embodiment of the present application.

[0023] Figure 5 A schematic diagram of the installation of an image sensor module provided in one embodiment of the present application.

[0024] Figure 6 A schematic diagram of the structure of an external environment measurement module provided in one embodiment of the present application.

[0025] Figure 7 A schematic diagram of the structure of an edge computing module provided in one embodiment of the present application.

[0026] Figure 8 A schematic diagram of a data and power supply interface in a fixed measurement mode provided by an embodiment of the present application.

[0027] Fig. 9 This is a schematic diagram of the structure of a power supply and button module provided in one embodiment of the present application.

[0028] Fig.10 A flowchart of a dual-light fusion intelligent recognition method provided in one embodiment of the present application.

[0029] Fig.11 A schematic diagram of a dual-light fusion intelligent recognition method provided in one embodiment of the present application.

[0030] Description of reference numerals:

[0031] 1—handheld handle; 2—fixed bracket; 11—image sensor module; 12—external environment measurement module; 13—display module; 14—edge computing module; 15—battery and button module; 111—thermal infrared camera unit; 112—visible light camera unit; 113—laser ranging unit; 121—magnetic connection terminal; 122—temperature and humidity detection terminal; 131—power interface; 132—RJ45 interface; 141—core computing board; 142—bottom plate; 151—lithium battery module; 152—button. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0034] like Figure 1 As shown, the application scenarios of a dual-light fusion intelligent recognition device provided by this embodiment include handheld inspection scenarios and fixed monitoring scenarios. The device is flexibly compatible with fixed scene monitoring requirements and handheld scene inspection requirements by replacing the connecting parts at the bottom of the device body. In fixed applications, the device can be installed on a pole or other fixed structure through a fixed bracket 2 for long-term monitoring. In handheld applications, the operator can use the handheld handle 1 of the handheld device for mobile inspection. The device body integrates an embedded edge computing module 14, which can complete computing functions such as dual-light fusion, target recognition and result output. The results of fixed monitoring and handheld inspections can be shared, thereby providing a more comprehensive and detailed monitoring function for a certain scene. Through this design, not only the flexibility and adaptability of monitoring are improved, but also the real-time detection and identification capabilities of abnormal temperature areas and targets are enhanced.

[0035] like Figure 2 As shown, the dual-light fusion intelligent recognition device of the embodiment of the present application mainly includes five parts: an image sensing module 11, an external environment measurement module 12, a display module 13, an edge computing module 14, and a battery and button module 15. Figure 3 The structural explosion diagram of the dual-light fusion intelligent recognition device is shown. Figure 4 The functional module principle schematic diagram of the dual-light fusion intelligent recognition device is shown. The image sensing module 11 includes a thermal infrared camera unit 111, a visible light camera unit 112 and a laser ranging unit 113, and the thermal infrared camera unit 111, the visible light camera unit 112 and the laser ranging unit 113 are all connected to the edge computing module 14; the thermal infrared camera unit 111 is used to collect the thermal infrared image of the target scene and send the thermal infrared image to the edge computing module 14; the visible light camera unit 112 is used to collect the visible light image of the target scene and send the visible light image to the edge computing module 14; the laser ranging unit 113 is used to collect the distance information from the dual-light fusion intelligent recognition device to the image imaging center, and send the distance information to the edge computing module 14, and the image imaging center is the center point of the thermal infrared image and the visible light image.

[0036] In this embodiment, the image sensing module 11 includes a thermal infrared camera unit 111, a visible light camera unit 112 and a laser ranging unit 113, which are used to obtain multi-modal image data. Figure 5 As shown. Among them, the thermal infrared camera unit 111 can use a high-sensitivity thermal imaging sensor, for example, a small non-cooled thermal infrared detector with a USB interface, with an image resolution of 640×512 pixels, mainly used to detect physical temperature information in the scene. The visible light camera unit 112 can use a high-resolution CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) sensor, for example, a USB interface industrial camera, with an image resolution of 1920×1080 pixels, mainly used to capture visible light images of the scene. The laser ranging unit 113 can use a single-point TOF (Time Of Flight) sensor with a ranging range of 0 to 5 meters, providing distance information between the device and the center area of ​​the image imaging, and providing support for subsequent stereo calibration and data fusion.

[0037] In this embodiment, the installation requirements of each part of the image sensor module 11 are as follows: the optical axes of the thermal infrared camera unit 111, the visible light camera unit 112 and the laser ranging unit 113 are substantially parallel, and the baseline distance between the thermal infrared camera unit 111 and the visible light camera unit 112 meets the installation space requirements, and the image center distance is, Figure 5 The baseline distance shown is as short as possible to facilitate subsequent image alignment operations. The installation position of the laser distance measuring unit 113 is located on the central axis perpendicular to the baseline, which is used to balance the regional measurement distance error between the center of the thermal infrared camera unit 111 and the center of the visible light camera unit 112. In the selection of the camera FOV (Field Of View), the FOV of the visible light camera unit 112 should be slightly larger than the FOV of the thermal infrared camera unit 111 to ensure that each pixel of the thermal infrared camera unit 111 can find the corresponding color value on the visible light image through stereo matching.

[0038] In this embodiment, the edge computing module 14 is used to perform dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image, and perform target recognition based on the dual-light fusion image to obtain a target recognition result; wherein the dual-light fusion alignment processing includes stereo calibration processing, image row alignment processing, distance correction processing and fusion image generation, and the distance correction processing is to correct the stereo calibration processing result and the image row alignment processing result according to the distance information; the target recognition result includes the bounding box, temperature information and category information of each target object in the target scene.

[0039] In this embodiment, the dual-light fusion intelligent recognition device also includes an external environment measurement module 12, which is connected to the edge computing module 14. The external environment measurement module 12 is used to collect temperature and humidity information of the surrounding environment of the target scene and send the temperature and humidity information to the edge computing module 14.

[0040] In this embodiment, the edge computing module 14 is also used to perform real-time correction on the temperature measurement result in the thermal infrared image according to the temperature and humidity information to obtain a corrected thermal infrared image, and use the corrected thermal infrared image for dual-light fusion alignment processing.

[0041] In this embodiment, the external environment measurement module 12 includes a magnetic connection end 121 and a temperature and humidity detection end 122. The magnetic connection end 121 is adsorbed on the shell of the dual-light fusion intelligent identification device in a magnetic connection manner, and the temperature and humidity detection end 122 includes a hollow shell and a temperature and humidity sensor located in the hollow shell.

[0042] In this embodiment, the external environment measurement module 12 is used to monitor the temperature, humidity and other environmental factors of the current scene in real time, which will affect the temperature measurement accuracy of thermal infrared imaging. By measuring and compensating the environmental parameters online, the temperature measurement accuracy of the thermal infrared image can be improved, and the subsequent temperature analysis and abnormality detection functions can be enhanced.

[0043] In this embodiment, the external environment measurement module 12 is detachably installed and connected to the main body of the edge computing module 14. Figure 6 As shown. For high-precision temperature measurement scenarios, such as measurement scenarios with accuracy requirements of ≤±1°C, online temperature correction can be performed by adding an external environment measurement module 12. This embodiment uses a four-wire magnetic connection method for data and power transmission. The temperature and humidity sensor uses models such as SHT20 and uses a typical I2C (Inter-Integrated Circuit) method for data communication.

[0044] In this embodiment, the edge computing module 14 includes a stereo calibration processing unit, an image row alignment processing unit, a distance correction processing unit and a fused image generation unit. The stereo calibration processing unit is used to use a known calibration plate to respectively obtain the internal and external parameters of the thermal infrared camera unit 111 and the visible light camera unit 112, and establish a geometric relationship between the thermal infrared camera unit 111 and the visible light camera unit 112 according to the internal and external parameters. The image row alignment processing unit is used to transform the thermal infrared image into the coordinate system of the visible light image using a perspective transformation matrix according to the geometric relationship between the thermal infrared camera unit 111 and the visible light camera unit 112, so as to ensure that the thermal infrared image and the visible light image are aligned in the row direction. The distance correction processing unit is used to determine the depth information of the object in the central area of ​​the target scene according to the distance information, and correct the pixel alignment effect of the thermal infrared image and the visible light image in the row direction according to the depth information to obtain a corrected thermal infrared image and a corrected visible light image. The fused image generation unit is used to fuse the corrected thermal infrared image and the corrected visible light image by using a weighted average method, a pseudo color processing method, a multi-scale fusion method or an image pyramid fusion method to generate the dual-light fusion image.

[0045] In this embodiment, the edge computing module 14 also includes a candidate target area extraction unit, a visible light image block extraction and target recognition unit, and a recognition result output unit. The candidate target area extraction unit is used to set a temperature threshold for the thermal infrared image part in the dual-light fusion image and perform binarization processing on the thermal infrared image part based on the temperature threshold to obtain a binary image; and mark the pixels in the binary image that are higher than the temperature threshold as candidate target areas, and use the FloodFill algorithm to extract the bounding box of the candidate target area. The visible light image block extraction and target recognition unit is used to extract the image block corresponding to the bounding box from the visible light image part in the dual-light fusion image, use the image recognition algorithm to perform target recognition on the image block, and determine the category information corresponding to each of the image blocks. The recognition result output unit is used to output the bounding box of each candidate target area and its temperature information and the category information of each image block.

[0046] In this embodiment, the edge computing module 14 adopts an embedded processing terminal that supports GPU acceleration. The embedded processing terminal includes a core computing board 141 and a base plate 142. The base plate 142 is provided with a core computing board 141 interface, and the core computing board 141 is detachably inserted into the core computing board 141 interface.

[0047] In this embodiment, a mounting hole is provided at the bottom of the dual-light fusion intelligent recognition device, and the mounting hole is used to install a connecting component, and the connecting component includes a handheld handle 1 or a fixed bracket 2. The handheld handle 1 is applied to a handheld inspection scenario, and the fixed bracket 2 is applied to a fixed monitoring scenario.

[0048] In this embodiment, the edge computing module 14 is responsible for executing the core data processing and analysis algorithms, which specifically include the following:

[0049] (1) Dual-light fusion: Thermal infrared images and visible light images are registered, corrected, and fused to generate multimodal images with rich information, which helps to improve the accuracy and reliability of target recognition.

[0050] (2) Ambient temperature compensation: The temperature and humidity data collected by the external environment measurement module 12 are used to make real-time corrections to the temperature measurement results of the thermal infrared image to eliminate interference from environmental factors.

[0051] (3) Target recognition algorithm: Integrate target segmentation and recognition algorithms based on machine learning and deep neural networks to realize automatic recognition of abnormal areas or targets.

[0052] (4) Result output: According to the setting of mobile scene or fixed scene, the processed measurement data, recognition results and other information are output. For mobile scenes, the visualization results are output on the external display module 13 through MIPI (Mobile Industry Processor Interface) or HDMI interface; for fixed scenes, the visualization results encoded in H.264 (digital video compression format) or MJPEG (Motion Joint Photographic Experts Group, technology is motion still image or frame by frame compression technology) are pushed through RJ45 (Registered Jack 45) or wireless LAN, and the target results are output in JSON (Java Script Object Notation, JS key-value pair data) file format through Socket, providing data support for data analysis of the centralized management platform.

[0053] In this embodiment, the edge computing module 14 uses an embedded processing terminal that supports GPU acceleration. The specific implementation method includes using NVIDIA OrinNX acceleration module or Huawei Altlas 200DK acceleration module, using ONNX (OpenNeuralNetwork Exchange, open neural network exchange format) to deploy neural network models, and on the basis of completing basic data processing, realize complex functions such as target segmentation and recognition algorithms based on machine learning and deep neural networks. Figure 7 As shown, the edge computing module 14 adopts a mode in which the core computing board 141 is separated from the base board 142, and different types of computing boards can be flexibly replaced according to the computing power requirements of the deployed model.

[0054] In this embodiment, the device can be equipped with a display module 13 in the mobile inspection measurement mode, and the display module 13 provides real-time measurement results to help users intuitively understand the working status and measurement data of the device. Through the direction arrows on the screen, users can be guided to focus on temperature measurement coverage of key targets to ensure the accuracy and representativeness of the measurement. This instant feedback helps to optimize the measurement process and improve the overall measurement efficiency. Optionally, in the fixed measurement mode, such as Figure 8 As shown, the display module 13 has a power interface 131 and an RJ45 interface 132 , and the detection and identification result is output through the RJ45 interface 132 .

[0055] In this embodiment, the device can be equipped with a battery and a key module 15 in the mobile inspection measurement mode, such as Fig. 9As shown, the main purpose is to provide power supply and control requirements for the entire device during movement, and a built-in battery is used to ensure the portability and independence of the device. At the same time, the physical button 152 allows the user to perform some common operations, such as screenshot saving, parameter correction, etc., which enhances the flexibility of human-computer interaction.

[0056] In this embodiment, the device can be powered by an external power adapter through the power interface 131 of the device in the fixed monitoring measurement mode.

[0057] In this embodiment, when data processing is performed, the required data is obtained from multiple sensors and devices through the central processor of the edge computing module 14. First, the thermal infrared image data is read, which is usually captured by a thermal imaging camera to record the infrared radiation intensity of each pixel in the scene. Secondly, the visible light image data is read, which is captured by a conventional camera to provide visual information under the visible spectrum for the scene. In addition, the laser ranging values ​​are also read, which are provided by the laser ranging unit 113. The laser ranging unit 113 is a laser rangefinder, which is used to determine the distance between each object in the scene and the measuring device. Finally, the ambient temperature and humidity data are read, which are provided by the temperature and humidity sensor to record the temperature and humidity of the current environment for subsequent image data correction and analysis.

[0058] In this embodiment, the thermal infrared image data correction mainly uses the read ambient temperature and humidity data to correct the thermal infrared image measurement value. The specific steps are as follows:

[0059] (1) Reading environmental parameters: The external environment measurement module 12 is used to obtain the ambient temperature and humidity data of the current scene in real time.

[0060] (2) Calculate the atmospheric transmission rate: Calculate the atmospheric transmission rate based on the acquired ambient temperature and humidity data. The atmospheric transmission rate refers to the attenuation of infrared radiation when it propagates in the air, which is significantly affected by temperature and humidity. The calculation is performed using the radiation transmission theory, and the calculation process can refer to the relevant models in the radiation transmission theory.

[0061] (3) Pixel value correction: Use the calculated atmospheric transmission rate to correct each pixel value in the thermal infrared image. Specifically, the original radiation value of each pixel point is combined with the atmospheric transmission rate to calculate the actual target radiation rate.

[0062] (4) Conversion of emissivity to temperature: The corrected thermal infrared image is obtained by converting the corrected emissivity to a temperature value. The conversion process can be performed using the Thermimage toolkit, which provides a series of functions and algorithms for thermal imaging analysis. With this toolkit, the emissivity value can be accurately converted to a temperature value, ensuring that the corrected temperature measurement result is more accurate.

[0063] In this embodiment, during the dual-light fusion alignment process, the thermal infrared image and the visible light image are first preliminarily matched using the stereo calibration results. Stereo calibration is to obtain the internal and external parameters of the camera through a known calibration plate, and establish a geometric relationship between the thermal infrared camera unit 111 and the visible light camera unit 112. Through these parameters, the row alignment function of the image can be realized, that is, the thermal infrared image and the visible light image are aligned in the same coordinate system. Then, the matching result is finely corrected using the laser ranging value. The distance information provided by the laser ranging helps to determine the depth information of the object in the central area of ​​the scene, further adjust the alignment effect of the image, and ensure that the pixels of the measurement target in the central area are accurately aligned.

[0064] In this embodiment, dual-light fusion alignment is a key step to ensure that the thermal infrared image and the visible light image are accurately overlapped in the same coordinate system, and specifically includes the following steps:

[0065] (a) Stereo calibration: First, the thermal infrared image and the visible light image are preliminarily matched using the stereo calibration results. Stereo calibration is to obtain the internal and external parameters of the camera through a known calibration plate, and establish the geometric relationship between the thermal infrared camera unit 111 and the visible light camera unit 112. The feature points on the calibration plate are photographed by the two cameras at the same time, and then the camera's internal parameters (such as focal length, principal point position, distortion coefficient) and external parameters (such as rotation matrix and translation vector) are calculated.

[0066] (b) Image row alignment: The geometric relationship parameters obtained through stereo calibration are used to realize the image row alignment function. Row alignment refers to aligning the thermal infrared image and the visible light image in the same coordinate system so that the corresponding pixel rows of the two images have the same geometric position. Here, the perspective transformation matrix is ​​mainly used to transform the thermal infrared image into the coordinate system of the visible light image to ensure that the pixels of the two images are aligned in the row direction.

[0067] (c) Laser ranging fine correction: After the initial alignment, the matching result is finely corrected using the laser ranging value. The distance information provided by the laser ranging unit 113 helps determine the depth information of the object in the center area of ​​the scene. This depth information can be used to further adjust the alignment effect of the image, and by combining the depth information to perform parallax compensation, ensure that the pixels of the target measured in the center area are accurately aligned.

[0068] (d) Fusion image generation: After the alignment and correction process described above, the thermal infrared image and the visible light image are fused. The fusion method can use simple superposition techniques, such as weighted averaging, pseudo-color processing, etc., or can use complex fusion algorithms, such as multi-scale fusion, image pyramid fusion, etc., to generate the final dual-light fusion image. The dual-light fusion image obtained after fusion is a multimodal image that has both thermal infrared temperature information and visible light detail information, thereby effectively improving the accuracy of target recognition and analysis.

[0069] In this embodiment, during target detection and recognition, the main process of performing target detection and recognition on the fused thermal infrared image and visible light image includes the following:

[0070] (A) Candidate target region extraction: First, a temperature threshold is preset, and then the temperature threshold is used to binarize the thermal infrared image to generate a binary image. In the binary image, pixels above the set temperature threshold are marked as candidate target regions, and pixels below the temperature threshold are ignored. The binarization process can effectively highlight the heat source target, facilitating subsequent detection and identification. In the binary image, connected component analysis, such as the Flood Fill algorithm, is used to extract the bounding boxes of the high temperature area. These bounding boxes define the preliminary detection results of the candidate targets. The extraction of bounding boxes can be optimized through morphological operations (such as dilation and erosion) and ensure that the detected areas are coherent and complete.

[0071] (B) Visible light image block extraction and target recognition: Image blocks corresponding to these bounding boxes are extracted from the visible light image to ensure that the candidate target areas detected in the thermal infrared image can be matched in the visible light image. These image blocks are used for target recognition using an image recognition algorithm. The target recognition algorithm can be a deep learning model based on a convolutional neural network (CNN), such as YOLO, Faster R-CNN, ResNet, etc. These models are pre-trained and fine-tuned to classify and recognize the categories of targets and provide high-precision recognition results. In the target recognition process, the temperature information of the thermal infrared image and the visual features of the visible light image can be combined for comprehensive analysis. For example, by combining the infrared temperature features and the visible light features, the accuracy and robustness of the recognition can be improved, and post-processing steps such as removing false detections, merging overlapping areas, and time-series-based target tracking can be performed according to the needs of the application scenario.

[0072] (C) Result output: After completing the above steps, the bounding box, temperature information and category information of the candidate target area are finally output, including the bounding box (indicating the coordinates and size of the target area), the average temperature value and the maximum temperature value (the temperature information within each candidate target area, providing important thermal information), and category information (the classification result of the target recognition algorithm, indicating the category of the target).

[0073] In another possible implementation, data is collected by using multiple sensors, such as the thermal infrared camera unit 111, the visible light camera unit 112, the laser ranging unit 113, etc., and data processing can be done directly on the back-end server. Although this system can achieve multi-source data fusion and improve the accuracy of detection and recognition, it relies on network connections and back-end servers, and the real-time performance and response speed may be low, and the system complexity and cost are also high.

[0074] In another embodiment, a single spectrum monitoring system such as a single thermal infrared camera unit 111 or a single visible light camera unit 112 may be used for target detection and identification. The thermal infrared monitoring system uses the thermal infrared camera unit 111 for temperature monitoring and target detection. Although it is effective under certain conditions, it is difficult to visually identify and confirm the target due to the lack of visible light information, which may lead to false alarms. The visible light monitoring system uses a high-definition visible light camera unit 112 for conventional video monitoring and target identification. Although the image quality is high, it cannot detect temperature anomalies, especially at night or in low light environments.

[0075] The dual-light fusion intelligent recognition device in this embodiment has the following characteristics:

[0076] 1) Flexible scenario adaptability. The device realizes flexible application in fixed monitoring and mobile inspection scenarios by designing modular connection components. The device can easily replace the bottom connection components to adapt to different installation requirements. It is suitable for long-term monitoring in fixed positions and can be installed on mobile platforms for inspection. This design significantly expands the scope of application of the device, enhances the flexibility of the use scenario, meets various monitoring needs, and improves the practicality and economy of the system.

[0077] 2) Integrated edge computing capabilities. The device integrates an embedded edge computing module 14, which can complete complex computing tasks such as dual-light fusion, target recognition and result output locally. The integrated design reduces the dependence on the back-end server and realizes real-time data processing and instant response. The high-performance processing capability of the edge computing module 14 improves the response speed and processing efficiency of the system, while reducing the delay and bandwidth requirements of data transmission. Through localized processing, the system can operate efficiently in various application scenarios to ensure real-time response capabilities.

[0078] In an exemplary embodiment, Fig.10 and Fig.11 As shown, a dual-light fusion intelligent recognition method is provided, and the dual-light fusion intelligent recognition method includes the following steps:

[0079] Step S1, obtaining a thermal infrared image, a visible light image and distance information from the dual-light fusion intelligent recognition device to an imaging center of a target scene; the imaging center is the center point of the thermal infrared image and the visible light image.

[0080] Step S2, performing dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image; the dual-light fusion alignment processing includes stereo calibration processing, image row alignment processing, distance correction processing and fusion image generation, and the distance correction processing is to correct the stereo calibration processing result and the image row alignment processing result according to the distance information.

[0081] Step S3: performing target recognition according to the dual-light fusion image to obtain a target recognition result; the target recognition result includes a bounding box, temperature information, and category information of each target object in the target scene.

[0082] In this embodiment, before the step of performing dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image in step S2, the dual-light fusion intelligent recognition method further includes the following steps:

[0083] The temperature and humidity information of the surrounding environment of the target scene is collected; based on the temperature and humidity information, the temperature measurement result in the thermal infrared image is corrected in real time to obtain a corrected thermal infrared image, and the corrected thermal infrared image is used for dual-light fusion alignment processing.

[0084] In this embodiment, step S2 performs dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image, which specifically includes the following steps:

[0085] Step S21: using a known calibration plate, respectively obtain the internal and external parameters of the thermal infrared camera unit 111 and the visible light camera unit 112, and establish a geometric relationship between the thermal infrared camera unit 111 and the visible light camera unit 112 according to the internal and external parameters.

[0086] Step S22: According to the geometric relationship between the thermal infrared camera unit 111 and the visible light camera unit 112, a perspective transformation matrix is ​​used to transform the thermal infrared image into the coordinate system of the visible light image to ensure pixel alignment of the thermal infrared image and the visible light image in the row direction.

[0087] Step S23: Determine the depth information of the object in the central area of ​​the target scene according to the distance information, and correct the pixel alignment effect of the thermal infrared image and the visible light image in the row direction according to the depth information to obtain a corrected thermal infrared image and a corrected visible light image.

[0088] Step S24: using a weighted average method, a pseudo color processing method, a multi-scale fusion method or an image pyramid fusion method to fuse the corrected thermal infrared image and the corrected visible light image to generate the dual-light fusion image.

[0089] This embodiment provides a dual-light fusion intelligent recognition device and method, which integrates an embedded edge computing module 14 to achieve the fusion of visible light images and thermal infrared images, as well as the rapid and accurate detection and recognition of abnormal temperature areas in the terminal. The device can not only combine the information advantages of two spectral images, visible light images and thermal infrared images, to effectively improve the accuracy and reliability of target recognition, but also reduce dependence on back-end servers through real-time edge computing processing, thereby improving the response speed and processing efficiency of the system. At the same time, the use of modular connection components makes the device suitable for long-term monitoring at fixed locations and for mobile inspections on handheld terminals, greatly enhancing the flexibility and scope of application of the system.

[0090] A dual-light fusion intelligent recognition device provided in this embodiment is compatible with fixed and portable application requirements, and through the edge computing module 14, while outputting the measurement screen, automatically identifies the temperature measurement coverage and frequency of key targets, builds an intelligent diagnosis model for abnormal working conditions, and improves the efficiency of discovering and diagnosing equipment temperature anomalies. It can be widely used in the fields of industrial equipment hidden danger inspection, power transmission line fault diagnosis, forest fire warning, etc., providing related industries with efficient and accurate intelligent recognition tools, which helps to timely discover and deal with various safety hazards and improve the overall safety prevention level.

[0091] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A dual-light fusion intelligent recognition device, characterized in that: The dual-light fusion intelligent recognition device includes an image sensing module and an edge computing module; The image sensing module includes a thermal infrared camera unit, a visible light camera unit and a laser ranging unit, and the thermal infrared camera unit, the visible light camera unit and the laser ranging unit are all connected to the edge computing module; the thermal infrared camera unit is used to collect thermal infrared images of the target scene and send the thermal infrared images to the edge computing module; the visible light camera unit is used to collect visible light images of the target scene and send the visible light images to the edge computing module; the laser ranging unit is used to collect distance information from the dual-light fusion intelligent recognition device to the image imaging center, and send the distance information to the edge computing module, and the image imaging center is the center point of the thermal infrared image and the visible light image; The edge computing module is used to perform dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image, and perform target recognition based on the dual-light fusion image to obtain a target recognition result; wherein the dual-light fusion alignment processing includes stereo calibration processing, image row alignment processing, distance correction processing and fusion image generation, and the distance correction processing is to correct the stereo calibration processing result and the image row alignment processing result according to the distance information; the target recognition result includes the bounding box, temperature information and category information of each target object in the target scene.

2. The dual-light fusion intelligent recognition device according to claim 1, characterized in that: The dual-light fusion intelligent recognition device also includes an external environment measurement module, which is connected to the edge computing module, and is used to collect temperature and humidity information of the surrounding environment of the target scene, and send the temperature and humidity information to the edge computing module; The edge computing module is also used to perform real-time correction on the temperature measurement results in the thermal infrared image according to the temperature and humidity information to obtain a corrected thermal infrared image, and use the corrected thermal infrared image for dual-light fusion alignment processing.

3. The dual-light fusion intelligent recognition device according to claim 2, characterized in that: The external environment measurement module includes a magnetic connection end and a temperature and humidity detection end. The magnetic connection end is adsorbed on the shell of the dual-light fusion intelligent identification device in a magnetic connection manner. The temperature and humidity detection end includes a hollow shell and a temperature and humidity sensor located in the hollow shell.

4. The dual-light fusion intelligent recognition device according to claim 1, characterized in that: The edge computing module includes a stereo calibration processing unit, an image row alignment processing unit, a distance correction processing unit and a fused image generation unit; The stereo calibration processing unit is used to respectively obtain internal and external parameters of the thermal infrared camera unit and the visible light camera unit using a known calibration plate, and establish a geometric relationship between the thermal infrared camera unit and the visible light camera unit according to the internal and external parameters; The image row alignment processing unit is used to transform the thermal infrared image into the coordinate system of the visible light image by using a perspective transformation matrix according to the geometric relationship between the thermal infrared camera unit and the visible light camera unit, so as to ensure pixel alignment of the thermal infrared image and the visible light image in the row direction; The distance correction processing unit is used to determine the depth information of the object in the central area of ​​the target scene according to the distance information, and to correct the pixel alignment effect of the thermal infrared image and the visible light image in the row direction according to the depth information to obtain a corrected thermal infrared image and a corrected visible light image; The fused image generation unit is used to fuse the corrected thermal infrared image and the corrected visible light image by using a weighted average method, a pseudo color processing method, a multi-scale fusion method or an image pyramid fusion method to generate the dual-light fusion image.

5. The dual-light fusion intelligent recognition device according to claim 4, characterized in that: The edge computing module also includes a candidate target area extraction unit, a visible light image block extraction and target recognition unit, and a recognition result output unit; The candidate target region extraction unit is used to set a temperature threshold for the thermal infrared image part in the dual-light fusion image and perform binarization processing on the thermal infrared image part based on the temperature threshold to obtain a binary image; and mark pixels in the binary image that are higher than the temperature threshold as candidate target regions, and use a Flood Fill algorithm to extract a bounding box of the candidate target region; The visible light image block extraction and target recognition unit is used to extract image blocks corresponding to the bounding box from the visible light image part in the dual-light fusion image, perform target recognition on the image blocks using an image recognition algorithm, and determine category information corresponding to each of the image blocks; The recognition result output unit is used to output the bounding box of each candidate target area and its temperature information and the category information of each image block.

6. The dual-light fusion intelligent recognition device according to claim 1, characterized in that: The edge computing module adopts an embedded processing terminal that supports GPU acceleration. The embedded processing terminal includes a base plate and a core computing board. The base plate is provided with a core computing board interface, and the core computing board is detachably inserted into the core computing board interface.

7. The dual-light fusion intelligent recognition device according to claim 1, characterized in that: A mounting hole is provided at the bottom of the dual-light fusion intelligent recognition device, and the mounting hole is used to install a connecting component. The connecting component includes a handheld handle or a fixed bracket. The handheld handle is used in a handheld inspection scenario, and the fixed bracket is used in a fixed monitoring scenario.

8. A dual-light fusion intelligent recognition method, characterized in that: The dual-light fusion intelligent recognition method is applied to the dual-light fusion intelligent recognition device according to any one of claims 1 to 7, and the dual-light fusion intelligent recognition method includes: Acquire a thermal infrared image, a visible light image and distance information from the dual-light fusion intelligent recognition device to the imaging center of the target scene; the imaging center is the center point of the thermal infrared image and the visible light image; Performing dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image; the dual-light fusion alignment processing includes stereo calibration processing, image row alignment processing, distance correction processing and fusion image generation, and the distance correction processing is to correct the stereo calibration processing result and the image row alignment processing result according to the distance information; Target recognition is performed according to the dual-light fusion image to obtain a target recognition result; the target recognition result includes a bounding box, temperature information and category information of each target object in the target scene.

9. The dual-light fusion intelligent recognition method according to claim 8, characterized in that: Before the step of performing dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain the dual-light fusion image, the dual-light fusion intelligent recognition method further includes: Collecting temperature and humidity information of the surrounding environment of the target scene; According to the temperature and humidity information, the temperature measurement result in the thermal infrared image is corrected in real time to obtain a corrected thermal infrared image, and the corrected thermal infrared image is used for dual-light fusion alignment processing.

10. The dual-light fusion intelligent recognition method according to claim 8, characterized in that: Performing dual-light fusion alignment processing on the thermal infrared image and the visible light image to obtain a dual-light fusion image specifically includes: Using a known calibration plate, respectively obtaining internal and external parameters of a thermal infrared camera unit and a visible light camera unit, and establishing a geometric relationship between the thermal infrared camera unit and the visible light camera unit according to the internal and external parameters; According to the geometric relationship between the thermal infrared camera unit and the visible light camera unit, the thermal infrared image is transformed into the coordinate system of the visible light image by using a perspective transformation matrix to ensure pixel alignment of the thermal infrared image and the visible light image in a row direction; Determine depth information of an object in a central area of ​​the target scene according to the distance information, and correct pixel alignment effects of the thermal infrared image and the visible light image in a row direction according to the depth information to obtain a corrected thermal infrared image and a corrected visible light image; The corrected thermal infrared image and the corrected visible light image are fused by using a weighted average method, a pseudo color processing method, a multi-scale fusion method or an image pyramid fusion method to generate the dual-light fusion image.