Thermal infrared detection method and device based on original data

Abstract

The invention provides a thermal infrared detection method based on original data, and the method comprises the steps: a denoising network construction step: constructing a noise sensing network to carry out the preprocessing of thermal infrared original data, and generating denoised thermal infrared original data; a detection network construction step of constructing a detection task network adapted to thermal infrared data characteristics by modeling the characteristics of the de-noised thermal infrared original data; and a joint optimization and coupling step: performing end-to-end joint training optimization and coupling on the noise sensing network and the detection task network through the collected thermal infrared original data. The invention further provides a thermal infrared detection device based on the original data, a storage medium and electronic equipment. Therefore, high-efficiency and high-precision intelligent thermal infrared detection can be realized based on the thermal infrared original data.

Description

Technical Field

[0001] This invention relates to the fields of thermal infrared imaging processing and target detection technology, and in particular to a thermal infrared detection method, apparatus, storage medium and electronic device based on raw data. Background Technology

[0002] Thermal infrared imaging is an imaging technology based on the thermal radiation characteristics of objects. Objects at different temperatures emit infrared radiation of different wavelengths and intensities. Thermal infrared sensors are used to receive this infrared radiation to generate thermal infrared images. Because thermal infrared imaging does not rely on external light sources, it can still provide stable object imaging results even in low light, no light, and adverse visible light conditions such as rain and fog. Furthermore, thermal infrared images are single-channel images, do not contain color or texture information, and are unaffected by light and shadow interference, effectively highlighting target objects in a scene. Compared to visible light imaging, it is more suitable for target detection tasks in complex environments. Therefore, thermal infrared imaging is widely used in industrial safety monitoring, field target identification, and security patrols, playing a crucial role in various target detection-based monitoring scenarios.

[0003] Detection efficiency and accuracy are two crucial metrics for evaluating detection tasks. Traditional detection tasks often rely on manual operation. The raw data acquired by thermal infrared sensors needs to undergo multiple processing modules, such as non-uniformity correction and dynamic range adjustment, to improve the quality of the thermal infrared image as perceived by the human eye, thereby enhancing detection accuracy and efficiency. In recent years, with the rapid development of artificial intelligence technology, deep learning-based visual algorithms have achieved extremely high accuracy in various visual tasks, including object detection, and are gradually replacing the human eye in processing large amounts of visual data. While deep learning networks perceive visual data differently than the human eye, in intelligent detection scenarios, the raw thermal infrared data acquired by thermal infrared sensors still needs to be processed into high-quality images that conform to human visual perception before being fed into the deep learning network to perform the detection task. This image processing process consumes computational resources, prolongs processing time, and loses some original information, affecting the detection efficiency and accuracy of thermal infrared intelligent detection methods and systems.

[0004] Due to the significant advantages of thermal infrared imaging technology in detection scenarios, improving the detection efficiency and accuracy based on thermal infrared data has become a key requirement. Recently, some researchers have focused on optimizing thermal infrared imaging processing techniques and workflows, designing efficient and concise processing algorithms and corresponding hardware architectures to reduce computational complexity, improve real-time performance, and enhance image quality, enabling the human eye to better capture detected objects. One study proposed a hardware architecture suitable for different customized infrared detectors, possessing real-time performance, low power consumption, dynamic parameter configuration, high processing efficiency, and stable imaging results. This research employed Verilog HDL (Verilog Hardware Description Language) to design modules including bad pixel replacement, non-uniformity correction (NUC), adaptive histogram equalization (CLAHE), edge enhancement, and gamma correction. These modules can be adapted to different detectors through dynamic parameter configuration, significantly improving thermal infrared imaging quality for easier human detection. Another study used deep learning networks to replace traditional modules, combining physical modeling and deep learning regularization to perform efficient non-uniformity correction (NUC), denoising, and super-resolution processing without the need for training data. The core idea is to decompose raw thermal infrared data into scene-independent sensor non-uniformity and scene-dependent radiative flux, and improve image quality for detection by combining multi-frame acquisition with camera jitter. However, this research did not focus on improving detection efficiency. Furthermore, these studies primarily focused on improving infrared image quality for human visual perception, without considering the improvement of detection accuracy for deep learning networks. Therefore, some studies have incorporated the evaluation metrics of deep learning networks performing downstream visual tasks into the image processing quality evaluation criteria, jointly optimizing the image processing task and the downstream visual task. One study proposed a saliency map fusion method for object detection tasks, using lightweight neural networks to generate saliency maps and employing two strategies—channel replacement and pixel-level weighted fusion—to enhance thermal infrared imaging quality and improve the average accuracy of the detection task network on the output image. Simultaneously, this network has fewer parameters and lower computational cost, improving detection efficiency. Other studies have specifically designed deep learning visual detection networks for thermal infrared imaging, using processed infrared images to train improved deep learning visual detection networks to improve detection accuracy. Lightweight detection task networks also represent higher detection efficiency. However, these studies all used human-eye-processed thermal infrared images for quality enhancement or network training, failing to fully leverage the potential of deep learning networks to directly process raw thermal infrared data. Furthermore, the human-eye-oriented processing increases computational redundancy and loses some information from the raw thermal infrared data that could be captured by deep learning networks.The above shortcomings result in significant limitations in existing thermal infrared intelligent detection methods and systems in terms of computational efficiency, response latency, and system complexity. They also face bottlenecks in detection accuracy that are difficult to overcome, and cannot meet the requirements for efficient, high-precision, and low-latency intelligent detection.

[0005] Research on existing thermal infrared intelligent detection methods revealed that current technologies only consider how to efficiently process raw thermal infrared data into images viewable to the human eye and improve the detection accuracy and efficiency of deep learning networks on the processed images. However, they fail to consider that this image processing flow is redundant when the deep learning network performs the detection task, and it reduces the amount of information in the raw thermal infrared data, which may not be included in the system. These technologies represent a rigid mindset, failing to recognize the analytical capabilities of deep learning networks for raw thermal infrared data. This leads to additional computational resource consumption and information loss, creating a bottleneck in the efficiency and accuracy of thermal infrared intelligent detection.

[0006] In conclusion, the existing technology obviously has inconveniences and defects in practical use, so it is necessary to improve it. Summary of the Invention

[0007] To address the aforementioned shortcomings, the present invention aims to provide a thermal infrared detection method, apparatus, storage medium, and electronic device based on raw data, which can achieve high-efficiency and high-precision intelligent thermal infrared detection based on raw thermal infrared data.

[0008] To solve the above-mentioned technical problems, the present invention is implemented as follows:

[0009] In a first aspect, embodiments of the present invention provide a thermal infrared detection method based on raw data, comprising:

[0010] The steps for constructing a denoising network are as follows: a noise sensing network is constructed to preprocess the raw thermal infrared data and generate denoised raw thermal infrared data.

[0011] The detection network construction step involves modeling the characteristics of the denoised thermal infrared raw data to construct a detection task network adapted to the characteristics of the thermal infrared data.

[0012] The joint optimization and coupling step involves end-to-end joint training, optimization, and coupling of the noise perception network and the detection task network using the collected raw thermal infrared data.

[0013] According to the thermal infrared detection method based on raw data of the present invention, the denoising network construction step includes:

[0014] Noise modeling is performed on the raw thermal infrared data to generate a corresponding noise model;

[0015] Construct a targeted noise perception network based on the noise model;

[0016] The noise-sensing network is used to preprocess the raw thermal infrared data to generate the denoised raw thermal infrared data.

[0017] According to the thermal infrared detection method based on raw data of the present invention, the step of noise modeling of the raw thermal infrared data and generating a corresponding noise model includes:

[0018] First, assume that the pixel value of a single pixel in the raw thermal infrared data is represented as: I {raw}(x，y) (x, y) represents pixel coordinates, and the entire noise model is decomposed into: Where G(x, y) represents spatially dependent non-uniformity. This represents a clean pixel value that is not contaminated by noise, and 0(x, y) represents a pixel bias drift. This indicates readout circuit noise. This represents thermal noise that is affected by temperature and pixel response characteristics.

[0019] The step of constructing the targeted noise sensing network based on the noise model includes:

[0020] Based on the noise model, a targeted noise perception network E is constructed. denoise (;θ denoise The network input is the raw thermal infrared data I. raw The output is the denoised thermal infrared raw data I. raw_denoise The entire noise sensing network is represented as I. raw_denoise =E denoise (I raw ;θ denoise ).

[0021] According to the thermal infrared detection method based on raw data of the present invention, the noise sensing network includes a feature encoding unit, a multi-branch estimation unit, and a denoising and reconstruction unit.

[0022] The feature encoding unit uses an encoder f encode (;θ encoder ) used to extract shared global features F shared The network input is I raw ;

[0023] The multi-branch estimation unit f i_branch (;θ i_branch These are used respectively for regression fitting noise components. All inputs are F shared ;

[0024] The noise reduction and reconstruction unit f decoder (;θ decoder The noise-sensing results are used to reconstruct the denoised thermal infrared raw data I. raw_denoise The input is I raw and multiple fitted noise components

[0025] According to the thermal infrared detection method based on raw data of the present invention, the loss function of the noise sensing network is expressed as:

[0026] Among them I G T represents the raw thermal infrared data uncontaminated by noise.

[0027] According to the thermal infrared detection method based on raw data of the present invention, the detection network construction step includes:

[0028] Modeling the characteristics of the denoised raw thermal infrared data includes: denoised raw thermal infrared data with W pixels in the horizontal direction and H pixels in the vertical direction... Modeling of texture and multi-scale statistical properties;

[0029] Constructing the detection task network adapted to the characteristics of thermal infrared data includes: constructing a deep learning network E for the target detection task. task (;θ task Its input is the denoised raw thermal infrared data I. raw_denoise The channel concatenation result I of data characteristics M and Θ input =Concat(I raw_denoise The network represents the detection task as R = E(M, Θ), where M, Θ) is the output of the analyzed detection result R. task (I input ;θ task And an adaptive feature encoding module f is added to the detection task network. adaptive_en (;θ adaptive_en It is used to perform channel fusion on input data.

[0030] According to the thermal infrared detection method based on raw data of the present invention, the joint optimization and coupling step includes:

[0031] The raw thermal infrared data of the target area is collected in real time using a thermal infrared sensor.

[0032] The noise perception network and the detection task network are jointly trained and optimized end-to-end, including: during network training, using the raw thermal infrared data I... rawUsing the noise perception network and the detection task network as input, end-to-end joint training is performed, and optimization is performed using a predetermined joint loss function, which is: in It's a loss in the noise reduction task. It is the loss of the detection task, where α∈[0,1] is the weight parameter;

[0033] The trained noise perception network is coupled with the detection task network.

[0034] In a second aspect, embodiments of the present invention provide a thermal infrared detection device based on raw data constructed according to any one of the methods described above, the device comprising:

[0035] The denoising network construction module is used to build a noise sensing network to preprocess the raw thermal infrared data and generate denoised raw thermal infrared data.

[0036] The detection network construction module is used to construct a detection task network adapted to the characteristics of the denoised thermal infrared raw data by modeling the characteristics of the raw thermal infrared data.

[0037] The joint optimization and coupling module is used to perform end-to-end joint training optimization and coupling of the noise perception network and the detection task network using the collected raw thermal infrared data.

[0038] Thirdly, embodiments of the present invention provide a storage medium for storing a computer program for performing any of the methods described herein.

[0039] Fourthly, embodiments of the present invention provide an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor, when executing the computer program, implements any of the methods described above.

[0040] Therefore, this invention, based on raw data-driven thermal infrared detection technology, firstly introduces a noise-sensing network to preprocess the raw thermal infrared data, generating denoised raw thermal infrared data; secondly, it introduces a detection task network adapted to the characteristics of the thermal infrared data to execute the detection task; finally, it uses the raw thermal infrared data to perform end-to-end joint training, optimization, and coupling of the noise-sensing network and the detection task network, creating an end-to-end intelligent thermal infrared detection system from raw thermal infrared data input to detection result output. This invention, by modeling the noise and characteristics of the raw thermal infrared data, designs a targeted deep learning network to perform denoising and detection tasks, suppressing multi-source noise interference and improving detection robustness and accuracy. Furthermore, by performing end-to-end joint optimization and coupling of the noise-sensing network and the detection task network, this invention enables data exchange and dynamic adaptive optimization between the two networks, reducing the overall data processing complexity and achieving the goal of high-efficiency and high-precision intelligent thermal infrared detection. Thus, this invention, based on the analytical capabilities of deep learning networks for raw thermal infrared data, fully explores the processing potential of raw thermal infrared data to achieve high-efficiency and high-precision intelligent thermal infrared detection. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating the thermal infrared detection method based on raw data provided in Embodiment 1 of the present invention.

[0042] Figure 2 This is a flowchart illustrating the thermal infrared detection method based on raw data provided in Embodiment 2 of the present invention.

[0043] Figure 3 This is a schematic diagram illustrating the principle of the thermal infrared detection method based on raw data provided in Embodiment 3 of the present invention;

[0044] Figure 4 This is a schematic diagram illustrating the principle of end-to-end joint training optimization and coupling provided in Embodiment 4 of the present invention;

[0045] Figure 5 This is a schematic diagram of the structure of the thermal infrared detection device based on raw data provided in Embodiment 5 of the present invention;

[0046] Figure 6 This is a schematic diagram of the structure of the thermal infrared detection device based on raw data provided in Embodiment Six of the present invention;

[0047] Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0049] It should be noted that references to "an embodiment," "embodiment," "example embodiment," etc., in this specification refer to the described embodiment including specific features, structures, or characteristics, but not every embodiment must include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.

[0050] Furthermore, certain terms are used in the specification and subsequent claims to refer to specific components or parts. Those skilled in the art will understand that manufacturers may use different names or terms to refer to the same component or part. This specification and subsequent claims do not distinguish components or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Additionally, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections made through other means.

[0051] The thermal infrared detection method based on raw data provided by the present invention will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0052] In the field of thermal infrared intelligent detection, deep learning-based detection task networks are gradually replacing the human eye in processing massive amounts of visual data. However, existing technologies are limited by traditional image visualization approaches, still focusing on human visual effects and failing to fully explore the processing potential of detection task networks for raw thermal infrared data. This results in problems such as redundant processing flows, high computational resource consumption, and loss of raw data information. The current key issue is how to streamline the raw data processing flow for deep learning-based detection task networks, uncover the characteristics of thermal infrared data, and improve the overall performance of the detection system. Therefore, this invention proposes a thermal infrared intelligent detection method based on raw data. It abandons the traditional detection flow based on visualized images and directly constructs a noise perception network for raw thermal infrared data and a detection task network adapted to the characteristics of thermal infrared data, reducing redundant computation and improving detection accuracy and robustness. Simultaneously, by performing end-to-end joint optimization and coupling of the two networks, complementary optimization can be achieved, creating a holistic detection link based on raw thermal infrared data. Compared with existing technologies, this invention breaks through traditional bottlenecks, effectively reduces the complexity of the detection process, improves detection accuracy, and achieves a more efficient and intelligent thermal infrared detection system.

[0053] Figure 1 This is a flowchart illustrating the thermal infrared detection method based on raw data provided in Embodiment 1 of the present invention. The method includes:

[0054] Step S101, Denoising Network Construction Step: Construct a noise sensing network to preprocess the raw thermal infrared data and generate denoised raw thermal infrared data.

[0055] The raw thermal infrared data is acquired by a thermal infrared sensor and requires no processing. A noise sensing network is introduced to perform essential denoising on the raw thermal infrared data.

[0056] Preferably, step S101 further includes:

[0057] Noise modeling is performed on the raw thermal infrared data to generate the corresponding noise model;

[0058] Construct a targeted noise perception network based on the noise model;

[0059] The raw thermal infrared data is preprocessed using a noise-sensing network to generate denoised raw thermal infrared data.

[0060] Step S102, the detection network construction step, involves constructing a detection task network adapted to the characteristics of the denoised thermal infrared raw data by modeling the characteristics of the thermal infrared data. The detection task network is used to perform the detection task.

[0061] Step S103, joint optimization and coupling step, involves end-to-end joint training, optimization and coupling of the noise perception network and the detection task network using the collected raw thermal infrared data.

[0062] Preferably, step S103 further includes:

[0063] Real-time acquisition of raw thermal infrared data of the target area using a thermal infrared sensor;

[0064] The noise perception network and the detection task network are jointly trained and optimized end-to-end.

[0065] The trained noise perception network is coupled with the detection task network.

[0066] This invention provides a thermal infrared intelligent detection method based on raw data. First, a noise perception network is introduced to perform essential preprocessing and denoising of the raw thermal infrared data. Second, a detection task network adapted to the characteristics of thermal infrared data is introduced to execute the detection task. Then, the raw thermal infrared data collected by a thermal infrared sensor is used to perform end-to-end joint training and optimization of the noise perception network and the detection task network. Finally, the trained noise perception network and the detection task network are coupled to create an end-to-end thermal infrared intelligent detection system from raw thermal infrared data input to detection result output. This invention, grounded in the era of artificial intelligence, fully explores the analytical capabilities of deep learning networks for raw thermal infrared data, innovatively performing end-to-end joint optimization and coupling of raw thermal infrared data processing and detection tasks, thus constructing an efficient, real-time, and intelligent thermal infrared detection method.

[0067] Figure 2 This is a flowchart illustrating the thermal infrared detection method based on raw data provided in Embodiment 2 of the present invention. The method includes:

[0068] Step S201: Noise modeling is performed on the raw thermal infrared data to generate the corresponding noise model.

[0069] The raw thermal infrared data is collected by a thermal infrared sensor and does not require any processing.

[0070] Preferably, step S201 further includes:

[0071] First, assume that the pixel value of a single pixel in the raw thermal infrared data can be represented as: I (raw) (x, y), where (x, y) represents pixel coordinates, the entire noise model is decomposed into: Where G(x, y) represents spatially dependent non-uniformity. This represents a clean pixel value that is not contaminated by noise, and 0(x, y) represents a pixel bias drift. This indicates readout circuit noise. This represents thermal noise affected by temperature and pixel response characteristics. The noise model can, but is not limited to, using a Gaussian distribution N(μ,σ). 2 Simulations were conducted.

[0072] Step S202: Construct a targeted noise perception network based on the noise model.

[0073] Preferably, step S202 further includes:

[0074] A targeted noise perception network E is constructed based on the noise model. enolse (;θ denoise The network input is raw thermal infrared data I. raw The output is the denoised raw thermal infrared data I. raw_denoise The entire noise sensing network can be represented as I raw_denoise =E denoise (I raw ;θ denoise ).

[0075] Preferably, the noise perception network may consist of at least three parts, including a feature encoding unit, a multi-branch estimation unit, and a denoising and reconstruction unit.

[0076] The feature encoding unit uses an encoder f encoder (;θ encoder ) used to extract shared global features F shared The network input is I raw .

[0077] The multi-branch estimation unit f i_branch (;θ i_branch These are used respectively for regression fitting noise components. All inputs are F shared In this embodiment, four parallel branches f i_branch (;θ i_branch These are used respectively for regression fitting noise components. All inputs are F shared .

[0078] The noise reduction and reconstruction unit f decoder (;θ decoder The noise-sensing results are used to reconstruct the denoised raw thermal infrared data. raw_denoise The input is I raw and multiple fitted noise components

[0079] The specific implementation of the noise-aware network is not limited to convolutional neural networks, deep learning networks based on transformer architecture, or new architectures of deep learning networks that may emerge in the future.

[0080]

[0081] Raw thermal infrared data contaminated by noise.

[0082] Step S203: Preprocess the raw thermal infrared data using a noise sensing network to generate denoised raw thermal infrared data.

[0083] Step S204: Model the characteristics of the denoised thermal infrared raw data.

[0084] Preferably, step S204 further includes:

[0085] Denoising the raw thermal infrared data with W pixels in the horizontal direction and H pixels in the vertical direction Texture and multi-scale statistical property modeling are performed. Methods employed include, but are not limited to, calculating local gradients, statistical histograms, and neural network-based modeling. Taking the calculation of local gradients and feature domain transformations as an example, G... x =I raw_denoise *K x G y =I raw_denoise *K y G x G represents the gradient in the horizontal direction. y surface

[0086] The distribution Θ is represented as

[0087] Step S205: Construct a detection task network adapted to the characteristics of thermal infrared data to perform detection tasks.

[0088] Preferably, step S205 further includes:

[0089] Construct a deep learning network E for object detection tasks task (;θ task Its input is the denoised raw thermal infrared data I. raw_denoise The channel concatenation result I of data characteristics M and Θ input =Concat(I raw_denoise The network consists of M, Θ), and the output is the analyzed detection result R. The entire detection task network can be represented as R = E. task (I input ;θ taskThe basic architecture of the detection task network can be selected from classic structures such as YOLO, Faster R-CNN, or U-Net, or other structures with higher performance in the future can be chosen.

[0090] In addition, an adaptive feature encoding module f is added to the basic architecture of the detection task network. adaptive_en (;θ adaptive_en This module is used for channel fusion of input data, enabling the detection task network to better adapt to the characteristics of thermal infrared data. It can be implemented using, but is not limited to, channel attention, multi-scale convolution, and other potentially higher-performance architectures. The loss function can be, but is not limited to, cross-entropy loss.

[0091] Step S206: Real-time acquisition of raw thermal infrared data of the target area using a thermal infrared sensor.

[0092] This step involves raw thermal infrared data acquisition. Raw thermal infrared data (Iraw) is acquired in real-time from the target area using a thermal infrared sensor. The image sequence representation of the acquisition process is as follows: I raw ={I t |t=1,2,...,T},where I t This represents the raw thermal infrared data of a single frame collected at time t.

[0093] Step S207: Perform end-to-end joint training and optimization of the noise perception network and the detection task network.

[0094] Preferably, step S207 further includes:

[0095] During network training, raw thermal infrared data I raw Using the noise perception network and the detection task network as input, end-to-end joint training is performed, and optimization is performed using a predetermined joint loss function, which is: in It's a loss in the noise reduction task. This is the loss for the detection task, where α∈[0,1] is the weight parameter, controlling the degree of optimization balance between the denoising and detection tasks. Network parameters θ denoise and θ task The optimization methods are not limited to backpropagation gradient descent and genetic annealing algorithms.

[0096] Step S208: Couple the trained noise perception network with the detection task network.

[0097] The noise perception network and the detection task network are not only connected in their input and output, but also co-optimize during training, transferring gradient information to each other. This end-to-end joint training coupling makes the output of the denoising network more conducive to the detection task, while the detection task network can better analyze the characteristics of the original thermal infrared data after denoising, effectively improving the overall model's detection accuracy and efficiency. The overall principle is as follows: Figures 3-4 As shown.

[0098] The purpose of this invention is to address the problems of redundant processing links, detection efficiency, and accuracy bottlenecks in existing technologies for intelligent detection scenarios, particularly in thermal infrared raw data processing. This invention proposes an optimized method and system for intelligent thermal infrared detection based on raw data. First, a noise-sensing network is introduced to preprocess the raw thermal infrared data, efficiently eliminating noise interference. Second, a detection task network adapted to the characteristics of thermal infrared data is introduced to improve detection accuracy and adaptability. Finally, by jointly training and coupling the noise-sensing network and the detection task network, a highly efficient and accurate intelligent thermal infrared detection method based on raw thermal infrared data is achieved.

[0099] Key Point 1: A thermal infrared intelligent detection optimization method based on raw data, comprising a noise perception network for raw thermal infrared data, a detection task network adapted to the characteristics of thermal infrared data, and an end-to-end joint optimization and coupling method for the two networks; through targeted network design and joint optimization coupling method, efficient intelligent processing and analysis based on raw thermal infrared data is achieved, effectively solving the problems of high computational redundancy and bottleneck in detection accuracy in existing thermal infrared detection systems, and achieving a significant improvement in detection accuracy, detection efficiency and adaptability.

[0100] Key Point 2: Design of noise perception network and detection task network for raw thermal infrared data; By modeling the noise and characteristics of raw thermal infrared data, a targeted deep learning network is designed to perform denoising and detection tasks, suppress multi-source noise interference, and improve detection robustness and accuracy.

[0101] Key point 3: End-to-end joint optimization and coupling method for noise perception and target detection; by performing end-to-end joint optimization and coupling on the two networks, the noise perception network and the detection task network can achieve data exchange and dynamic adaptive optimization, reducing the overall data processing complexity, thereby achieving high-efficiency and high-precision thermal infrared intelligent target detection.

[0102] It should be noted that the thermal infrared detection method based on raw data provided in this embodiment of the invention can be executed by an electronic device, a apparatus, or a control module within that apparatus for executing the method. This embodiment of the invention uses an apparatus executing the method as an example to illustrate the thermal infrared detection apparatus based on raw data provided in this embodiment of the invention.

[0103] Figure 5 This is a schematic diagram of the thermal infrared detection device based on raw data provided in Embodiment 5 of the present invention. The device 100 includes a denoising network construction module 10, a detection network construction module 20, and a joint optimization and coupling module 30, wherein:

[0104] The denoising network construction module 10 is used to construct a noise perception network to preprocess the raw thermal infrared data and generate denoised raw thermal infrared data.

[0105] The detection network construction module 20 is used to construct a detection task network adapted to the characteristics of thermal infrared data by modeling the characteristics of the original denoised thermal infrared data.

[0106] The joint optimization and coupling module 30 is used to perform end-to-end joint training optimization and coupling of the noise perception network and the detection task network using the collected thermal infrared raw data.

[0107] Figure 5 This is a schematic diagram of the thermal infrared detection device based on raw data provided in Embodiment 5 of the present invention. The device 100 includes a denoising network construction module 10, a detection network construction module 20, and a joint optimization and coupling module 30, wherein:

[0108] The denoising network construction module 10 is used to construct a noise sensing network to preprocess the raw thermal infrared data and generate denoised raw thermal infrared data. Preferably, the denoising network construction module 10 further includes:

[0109] The noise modeling submodule 11 is used to model the noise of the raw thermal infrared data and generate the corresponding noise model. Preferably, it is first assumed that the pixel value of a single pixel in the raw thermal infrared data is represented as: I{ raw} (x，y) (x, y) represents the pixel coordinates. The entire noise model is decomposed into: Where G(x, y) represents spatially dependent non-uniformity. O(x, y) represents a clean pixel value that is not contaminated by noise, and O(x, y) represents the pixel bias drift. This indicates readout circuit noise. This represents thermal noise that is affected by temperature and pixel response characteristics.

[0110] The denoising and network construction submodule 12 is used to construct a targeted noise sensing network based on the noise model. Preferably, the targeted noise sensing network E is constructed based on the noise model. denoise (;θ denoise The network input is raw thermal infrared data I. raw The output is the denoised raw thermal infrared data I. raw_denoise The entire noise sensing network is represented as Iraw_denoise =Edenoise(I r aw; θdenoise).

[0111] Preferably, the noise sensing network includes a feature encoding unit, a multi-branch estimation unit, and a denoising and reconstruction unit. The feature encoding unit uses an encoder f encoder (;θ encoder ) used to extract shared global features F shared The network input is I raw Multi-branch estimation unit f i_branch (;θ i_branch These are used respectively for regression fitting noise components. All inputs are F shared Denoising and reconstruction unit f decoder (;θ decoder The noise-sensing results are used to reconstruct the denoised raw thermal infrared data. raw_denoise The input is I raw and multiple fitted noise components

[0112] Preferably, the loss function of the noise sensing network is expressed as:

[0113] Among them I G T represents the raw thermal infrared data uncontaminated by noise.

[0114] The denoising processing submodule 13 is used to preprocess the raw thermal infrared data through a noise sensing network to generate denoised raw thermal infrared data.

[0115] The detection network construction module 20 is used to construct a detection task network adapted to the characteristics of thermal infrared data by modeling the characteristics of the denoised raw thermal infrared data. Preferably, the detection network construction module 20 further includes:

[0116] The feature modeling submodule 21 is used for feature modeling of the denoised raw thermal infrared data. This includes: denoised raw thermal infrared data with W pixels in the horizontal direction and H pixels in the vertical direction. Modeling of texture and multi-scale statistical properties.

[0117] The detection network construction submodule 22 is used to construct a detection task network adapted to the characteristics of thermal infrared data. This includes: constructing a deep learning network E for the target detection task. task (;θ task Its input is the denoised raw thermal infrared data I. raw_denoise The channel concatenation result I of data characteristics M and Θ input =Concat(I raw_denoiseThe network represents the detection task as R = E, where M, Θ), and the output is the analyzed detection result R. task (I input ;θ task And an adaptive feature encoding module f is added to the detection task network. adaptive_en (;θ adaptive_en It is used to perform channel fusion on input data.

[0118] The joint optimization and coupling module 30 is used to perform end-to-end joint training optimization and coupling of the noise perception network and the detection task network using the acquired raw thermal infrared data. Preferably, the joint optimization and coupling module 30 further includes:

[0119] The data acquisition submodule 31 is used to acquire raw thermal infrared data of the target area in real time through a thermal infrared sensor.

[0120] The joint optimization submodule 32 is used to perform end-to-end joint training and optimization of the noise perception network and the detection task network, including: during network training, using raw thermal infrared data I... raw Using the noise perception network and the detection task network as input, end-to-end joint training is performed, and optimization is performed using a predetermined joint loss function, which is: in It's a loss in the noise reduction task. It is the loss of the detection task, and α∈[0,1] is the weight parameter.

[0121] The coupling submodule 33 is used to couple the trained noise perception network with the detection task network.

[0122] The present invention also provides a storage medium for storing, for example, Figures 1-4 The computer program for any of the thermal infrared detection methods based on raw data. For example, computer program instructions, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the present invention through the operation of the computer, and can achieve the same technical effect. To avoid repetition, they will not be described again here. The program instructions for invoking the methods of the present invention may be stored in a fixed or removable storage medium, and / or transmitted through a data stream in a broadcast or other signal carrying medium, and / or stored in the storage medium of a computer device operating according to the program instructions.

[0123] According to one embodiment of the present invention, the present invention also provides such a Figure 7The illustrated electronic device 400 may optionally include a storage medium 200 for storing a computer program and a processor 300 for executing the computer program. When the computer program is executed by the processor 300, it implements any of the aforementioned thermal infrared detection methods based on raw data, triggering the electronic device 400 to execute methods and / or technical solutions based on the foregoing embodiments, achieving the same technical effect. To avoid repetition, these methods will not be elaborated upon here. It should be noted that the electronic devices in this embodiment include mobile electronic devices and non-mobile electronic devices. For example, mobile electronic devices may be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, super mobile personal computers, netbooks, or personal digital assistants, etc., while non-mobile electronic devices may be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This embodiment does not specifically limit the scope of the invention.

[0124] The thermal infrared detection device based on raw data provided in this embodiment of the invention can achieve Figures 1-4 The various processes implemented in the embodiment of the thermal infrared detection method based on raw data shown are not described in detail here to avoid repetition.

[0125] The thermal infrared detection device based on raw data provided in this invention first introduces a noise perception network to preprocess the raw thermal infrared data, generating denoised raw thermal infrared data. Second, a detection task network adapted to the characteristics of the thermal infrared data is introduced to execute the detection task. Finally, the noise perception network and the detection task network are jointly trained, optimized, and coupled end-to-end using the raw thermal infrared data, creating an end-to-end intelligent thermal infrared detection system from raw thermal infrared data input to detection result output. This invention, by modeling the noise and characteristics of the raw thermal infrared data, designs a targeted deep learning network to perform denoising and detection tasks, suppressing multi-source noise interference and improving detection robustness and accuracy. Furthermore, by jointly optimizing and coupling the noise perception network and the detection task network end-to-end, the two networks achieve data exchange and dynamic adaptive optimization, reducing the overall data processing complexity and achieving the goal of high-efficiency and high-precision intelligent thermal infrared detection. Therefore, this invention, based on the analytical capabilities of deep learning networks for raw thermal infrared data, fully explores the processing potential of raw thermal infrared data to achieve high-efficiency and high-precision intelligent thermal infrared detection.

[0126] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including associated data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of the present invention can be implemented in hardware, for example, as circuitry that works with a processor to perform the various steps or functions.

[0127] This invention can be implemented on a computer as a computer-based method, or in dedicated hardware, or a combination of both. Executable code or portions thereof for the method according to the invention can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Optionally, the computer program product includes non-transitory program code components stored on a computer-readable medium so as to execute the method according to the invention when the program product is executed on a computer.

[0128] In an optional embodiment, the computer program includes computer program code components adapted to perform all the steps of the method according to the invention when the computer program is run on a computer. Optionally, the computer program is embodied on a computer-readable medium.

[0129] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0130] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A thermal infrared detection method based on raw data, characterized in that, include: The steps for constructing a denoising network are as follows: a noise sensing network is constructed to preprocess the raw thermal infrared data and generate denoised raw thermal infrared data. The detection network construction step involves modeling the characteristics of the denoised thermal infrared raw data to construct a detection task network adapted to the characteristics of the thermal infrared data. The joint optimization and coupling step involves end-to-end joint training, optimization, and coupling of the noise perception network and the detection task network using the collected raw thermal infrared data.

2. The thermal infrared detection method based on raw data according to claim 1, characterized in that, The steps for constructing the denoising network include: Noise modeling is performed on the raw thermal infrared data to generate a corresponding noise model; Construct a targeted noise perception network based on the noise model; The noise-sensing network is used to preprocess the raw thermal infrared data to generate the denoised raw thermal infrared data.

3. The thermal infrared detection method based on raw data according to claim 2, characterized in that, The step of noise modeling the raw thermal infrared data and generating a corresponding noise model includes: First, assume that the pixel value of a single pixel in the raw thermal infrared data is represented as: I {raw}(x，y) (x, y) represents pixel coordinates, and the entire noise model is decomposed into: Where G(x, y) represents spatially dependent non-uniformity. O(x, y) represents a clean pixel value that is not contaminated by noise, and O(x, y) represents the pixel bias drift. This indicates readout circuit noise. This represents thermal noise that is affected by temperature and pixel response characteristics. The step of constructing the targeted noise sensing network based on the noise model includes: Based on the noise model, a targeted noise perception network E is constructed. denoise (;θ denoise The network input is the raw thermal infrared data I. raw The output is the denoised thermal infrared raw data I. raw_denoise The entire noise sensing network is represented as I. raw_denoise =E denoise (I raw ;θ denoise ).

4. The thermal infrared detection method based on raw data according to claim 3, characterized in that, The noise perception network includes a feature encoding unit, a multi-branch estimation unit, and a denoising and reconstruction unit. The feature encoding unit uses an encoder f encoder (;θ encoder ) used to extract shared global features F shared The network input is I raw ; The multi-branch estimation unit f i_branch (;θ i_branch These are used respectively for regression fitting noise components. The input is F. shared ; The noise reduction and reconstruction unit f decoder (;θ decoder The noise-sensing results are used to reconstruct the denoised thermal infrared raw data I. raw_denoise The input is I raw and multiple fitted noise components 5. The thermal infrared detection method based on raw data according to claim 4, characterized in that, The loss function of the noise sensing network is expressed as: Among them I GT It is raw thermal infrared data that has not been contaminated by noise.

6. The thermal infrared detection method based on raw data according to claim 1, characterized in that, The detection network construction steps include: Modeling the characteristics of the denoised raw thermal infrared data includes: denoised raw thermal infrared data with W pixels in the horizontal direction and H pixels in the vertical direction... Modeling of texture and multi-scale statistical properties; Constructing the detection task network adapted to the characteristics of thermal infrared data includes: constructing a deep learning network E for the target detection task. task (;θ task Its input is the denoised raw thermal infrared data I. raw_denoise The channel concatenation result I of data characteristics M and Θ input =Concat(I raw_denoise The network represents the detection task as R = E(M, Θ), where M, Θ) is the output of the analyzed detection result R. task (I input ;θ task And an adaptive feature encoding module f is added to the detection task network. adaptive_en (;θ adaptive_en It is used to perform channel fusion on input data.

7. The thermal infrared detection method based on raw data according to claim 1, characterized in that, The joint optimization and coupling steps include: The raw thermal infrared data of the target area is collected in real time using a thermal infrared sensor. The noise perception network and the detection task network are jointly trained and optimized end-to-end, including: during network training, using the raw thermal infrared data I... raw Using the noise perception network and the detection task network as input, end-to-end joint training is performed, and optimization is performed using a predetermined joint loss function, which is: in It's a loss in the noise reduction task. It is the loss of the detection task, where α∈[0,1] is the weight parameter; The trained noise perception network is coupled with the detection task network.

8. A thermal infrared detection device based on raw data constructed according to the method described in any one of claims 1 to 7, characterized in that, The device includes: The denoising network construction module is used to build a noise sensing network to preprocess the raw thermal infrared data and generate denoised raw thermal infrared data. The detection network construction module is used to construct a detection task network adapted to the characteristics of the denoised thermal infrared raw data by modeling the characteristics of the raw thermal infrared data. The joint optimization and coupling module is used to perform end-to-end joint training optimization and coupling of the noise perception network and the detection task network using the collected raw thermal infrared data.

9. A storage medium, characterized in that, Used to store a computer program for performing the method according to any one of claims 1 to 7.

10. An electronic device comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images