Thermal infrared camera based on computational optical technology and thermal imaging method

Abstract

The invention relates to a thermal infrared camera based on a computational optical technology and a thermal imaging method. The camera comprises an infrared lens and a movement. The movement comprises an infrared detector, a first image processing module and a second image processing module; the infrared lens is connected with the infrared detector and is used for collecting a heat radiation signal emitted by a target scene; the infrared detector is connected with the first image processing module and is used for performing photoelectric conversion on the thermal radiation signal to obtain scene information; the first image processing module is connected with the second image processing module and is used for preprocessing the scene information based on an artificial intelligence calculation optical model to generate a restored image; the second image processing module is used for generating a thermal imaging image based on the restored image, and the first image processing module for operating an artificial intelligence calculation optical model is added to the machine core side, so that the imaging quality is greatly improved on the premise that the cost is low, and the problem that the cost and the quality of an existing thermal infrared camera cannot be considered at the same time is solved.

Description

Technical Field

[0001] This application relates to the field of infrared imaging technology, and particularly to a thermal infrared camera and a thermal imaging method based on computational optical technology. Background Art

[0002] Thermal infrared cameras, also known as long-wave infrared cameras, have extensive applications in fields such as temperature measurement, security, and vehicle-mounted. Conventional thermal infrared cameras consist of two parts: a lens and a core module. The core module usually includes an infrared detector and an infrared image processing chip. Scene information is collected by the detector through the infrared lens and then processed by the image algorithm in the image processing chip to generate a thermal imaging image. As the requirements for the imaging quality of thermal infrared cameras become higher and higher, the cost of thermal infrared cameras is also constantly increasing.

[0003] At present, there are mainly two types of patents for reducing the cost of thermal infrared cameras. One is to focus on reducing the cost of infrared lenses, and the other is to reduce the cost of the infrared core module, especially the cost of infrared detectors. Existing solutions for reducing the cost of infrared lenses mainly involve adding diffractive surfaces or diffractive elements, as well as new optical elements such as metasurfaces in the optical system to improve the performance indicators of the optical system, so that the optical system can use fewer lens elements. However, the introduction of such new optical elements not only does not improve the performance enough to compensate for the performance loss caused by the reduction in the number of lens elements, but also introduces problems such as chromatic aberration and processing difficulties, and does not involve using computational optical algorithms to improve the imaging quality. Existing solutions for reducing the cost of infrared detectors mainly involve using some low-cost infrared detector composition materials to replace the currently more expensive detector materials, or specifically optimizing the high-cost items such as consumables during the preparation process of infrared detectors to achieve cost reduction. However, these solutions will more or less lead to a decline in the performance of infrared detectors and can usually only be applied in low-cost scenario with low requirements for imaging quality.

[0004] Therefore, existing low-cost thermal infrared cameras are difficult to meet the requirements of high image quality and have great limitations in applications. Summary of the Invention

[0005] Based on this, it is necessary to provide a thermal infrared camera and a thermal imaging method based on computational optical technology that can both ensure image quality and reduce costs for the above technical problems.

[0006] In a first aspect, this application proposes a thermal infrared camera based on computational optical technology, the camera includes: an infrared lens and a core module; the core module includes an infrared detector, a first image processing module, and a second image processing module arranged in sequence;

[0007] The infrared lens is connected to the infrared detector; the infrared lens is used to collect the thermal radiation signal emitted by the target scene and conduct the thermal radiation signal to the infrared detector, so as to perform photoelectric conversion on the thermal radiation signal through the infrared detector to generate scene information;

[0008] The first image processing module is connected to the infrared detector; the first image processing module is used to receive the scene information and preprocess the scene information based on the artificial intelligence computational optics model to generate a restored image;

[0009] The second image processing module is linked to the first image processing module; the second image processing module is used to receive the restored image and generate a thermal imaging image based on the restored image.

[0010] In one embodiment, the infrared lens is obtained based on the design result output by the joint simulation of the optical design and the restoration model.

[0011] In one embodiment, the joint simulation of the optical design and the restoration model includes:

[0012] Based on the optical design software, the initial simulation lens is optimized to obtain the optimized design result of the simulation lens; based on the simulation of the actual use effect of the simulation lens, tolerance perturbations are added to the optimized design result to obtain the simulated design result of the simulation lens;

[0013] Based on the simulated design result, a training data set is calculated, and the restoration model is trained based on the training data set;

[0014] Based on the simulated design result, a simulated detector image is calculated; based on the restoration model, the simulated detector image is restored to obtain a simulated restored image;

[0015] The simulated restored image is evaluated; if the evaluation value reaches the simulation target, the design of the joint simulation is completed, and the current simulated design result is output; if the evaluation value does not reach the simulation target, the simulation lens is continuously optimized until the evaluation value corresponding to the optimized lens reaches the simulation target.

[0016] In one embodiment, the calculating the training data set based on the simulated design result includes:

[0017] Calculate the first point spread function of the simulated design result in different fields of view; calculate the training data set based on the sub-fields of view of the high-definition image, the first point spread function, and random noise.

[0018] In one embodiment, calculating the simulated detector image based on the simulated design result includes:

[0019] Calculating at least two second point spread functions corresponding to fluctuating individuals based on preset individual tolerance fluctuations and the simulated design result;

[0020] Calculating the simulated detector image based on the second point spread function and the high-definition checkerboard image.

[0021] In one embodiment, the first image processing module includes a neural network processor;

[0022] The neural network processor runs the artificial intelligence computational optics model;

[0023] The computing power of the neural network processor is determined based on the resolution and frame rate of the thermal infrared camera and the size of the artificial intelligence computational optics model.

[0024] In one embodiment, the artificial intelligence computational optics model is used to perform aberration correction on the input scene information to generate a restored image after the aberration correction.

[0025] In one embodiment, the artificial intelligence computational optics model is used to perform non-uniformity correction, noise reduction, and aberration correction on the input scene information to generate a restored image after the non-uniformity correction, noise reduction, and aberration correction.

[0026] In one embodiment, the training method of the artificial intelligence computational optics model includes:

[0027] Calibrating the non-uniformity correction coefficient and the noise model;

[0028] Based on the calibrated non-uniformity correction coefficient and the noise model, calibrating the third point spread function of the infrared lens;

[0029] Based on the obtained high-definition label image, the calibrated third point spread function, the non-uniformity correction coefficient, and the noise model, generating a degraded image simulating the actual shooting of the thermal infrared camera, and using the degraded image and the high-definition label image as a training data pair;

[0030] Training the artificial intelligence computational optics model based on the training data pair.

[0031] In a second aspect, the present application further provides a thermal imaging method, which is applied to the thermal infrared camera described in any one of the first aspects above; the method includes:

[0032] Collect the thermal radiation signal emitted by the target scene through the infrared lens, and conduct the thermal radiation signal to the infrared detector;

[0033] Through the infrared detector, perform optoelectronic conversion on the thermal radiation signal to obtain scene information; and transmit the scene information to the first image processing module;

[0034] Through the first image processing module, preprocess the scene information based on the artificial intelligence computational optics model to generate a restored image; send the restored image to the second image processing module;

[0035] Through the second image processing module, generate a thermal imaging image based on the restored image.

[0036] The above thermal infrared camera and thermal imaging method based on computational optics technology, through an infrared lens and a camera module; the camera module includes an infrared detector, a first image processing module, and a second image processing module; the infrared lens is connected to the infrared detector and is used to collect the thermal radiation signal emitted by the target scene; the infrared detector is connected to the first image processing module and is used to perform optoelectronic conversion on the thermal radiation signal to obtain scene information; the first image processing module is connected to the second image processing module and is used to preprocess the scene information based on the artificial intelligence computational optics model to generate a restored image; the second image processing module is used to generate a thermal imaging image based on the restored image. By adding a first image processing module that runs the artificial intelligence computational optics model on the camera module side, on the premise of relatively low cost, the imaging quality of the thermal infrared camera is greatly improved, and the problem that the existing thermal infrared cameras cannot balance cost and image quality is solved. Description of the Drawings

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings:

[0038] Figure 1 It is a schematic structural diagram of a thermal infrared camera based on computational optics technology for an embodiment;

[0039] Figure 2 It is a flowchart of the joint simulation of optical design and restoration model in an embodiment;

[0040] Figure 3 It is a flowchart of the joint simulation of optical design and restoration model in a preferred embodiment;

[0041] Figure 4 It is a flowchart of the training method of the artificial intelligence computational optics model in an embodiment:

[0042] Figure 5 Schematic diagram of the movement's infrared image processing in a preferred embodiment:

[0043] Figure 6 Comparison diagram of the imaging effects of a thermal infrared camera based on computational optical technology and other cameras in an embodiment;

[0044] Figure 7 Flowchart of a thermal imaging method in an embodiment.

[0045] Reference numerals: 10, infrared lens; 20, movement; 21, infrared detector; 22, first image processing module; 23, second image processing module. Detailed implementation manners

[0046] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments provided in the present application without creative efforts fall within the scope of protection of the present application.

[0047] Obviously, the accompanying drawings in the following description are only some examples or embodiments of the present application. For those of ordinary skill in the art, without creative efforts, the present application can also be applied to other similar scenarios based on these drawings. In addition, it can also be understood that although the efforts made in this development process may be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some design, manufacturing or production changes made based on the technical content disclosed in the present application are only conventional technical means and should not be understood as the content disclosed in the present application being insufficient.

[0048] Referring to "embodiment" in the present application means that the specific features, structures or characteristics described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears in various positions in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those of ordinary skill in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments without conflict.

[0049] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the ordinary meanings understood by those with ordinary skills in the technical field to which this application belongs. The words such as "a", "an", "one", "the" and the like involved in this application do not indicate a limitation in quantity and may represent a singular or plural number. The terms "comprise", "include", "have" and any variations thereof involved in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may further include unlisted steps or units, or may further include other steps or units inherent to these processes, methods, products or devices. The words such as "connect", "be connected", "couple" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, "A and / or B" may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after. The terms "first", "second", "third" and the like involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0050] In this embodiment, a thermal infrared camera based on computational optical technology is provided. Figure 1 As shown in the structural schematic diagram of the thermal infrared camera based on computational optical technology in an embodiment, Figure 1 as shown, the thermal infrared camera includes: an infrared lens 10 and a camera module 20; the camera module 20 includes an infrared detector 21, a first image processing module 22, and a second image processing module 23 that are sequentially arranged.

[0051] The infrared lens 10 is connected to the infrared detector 21; the infrared lens 10 is configured to collect the thermal radiation signal emitted by the target scene and conduct the thermal radiation signal to the infrared detector 21, so as to perform photoelectric conversion on the thermal radiation signal through the infrared detector 21 to generate scene information.

[0052] The first image processing module 22 is connected to the infrared detector 21; the first image processing module 22 is configured to receive the scene information and perform preprocessing on the scene information based on an artificial intelligence computational optical model to generate a restored image.

[0053] The second image processing module 23 is linked to the first image processing module 22; the second image processing module 23 is configured to receive the restored image and generate a thermal imaging image based on the restored image.

[0054] Specifically, the infrared lens 10 can adopt a conventional infrared lens in an existing thermal infrared camera. On the premise of maintaining the existing cost on the infrared lens 10 side, the first image processing module 22 can obtain a higher-quality image quality improvement with less cost. If the cost is to be further reduced, the infrared lens 10 can also adopt an existing low-cost lens with main value parameters close to those of a high-quality lens. This low-cost lens can save costs in terms of lenses or materials. Although the low-cost lens affects the image quality to a certain extent and the image quality needs to be improved by the algorithm on the movement 20 side, the cost saved on the lens side is much greater than the cost increased by the improvement on the movement 20 side, and it can more effectively reduce the cost on the premise of ensuring that the image quality does not decrease. Among them, compared with the high-quality lens, redesigning the infrared lens 10 by reducing the number of lenses can reduce the overall volume or weight of the thermal infrared camera, and can obtain a significant improvement in image quality with less cost on both the lens side and the movement 20 side.

[0055] When designing the lens or selecting a low-cost lens, consider the image quality restoration ability of the computational optics algorithm based on artificial intelligence, and conduct joint simulation end-to-end. Take whether the final imaging quality can be the same as that of the benchmark high-quality lens as the most important evaluation criterion. By optimizing or selecting the optical design in this way, an infrared lens with higher accuracy can be obtained.

[0056] The first image processing module 22 is located between the infrared detector 21 and the second image processing module 23, meeting the series connection requirements of the thermal infrared camera interface. The first image processing module 22 can adopt a lightweight SOC chip (System on Chip) with NPU (Neural network Processing Unit, neural network processor) computing power.

[0057] The second image processing module 23 is an image ISP (Image Signal Processing) processing chip. The processing units with the same functions on the second image processing module 23 can be selectively turned off according to the functions of the artificial intelligence computational optics model, so as to reduce the processing pressure on the second image processing module 23 and ensure the image quality output by the second image processing module 23.

[0058] Compared with the prior art, the thermal infrared camera and thermal imaging method based on computational optical technology provided in this embodiment pass through an infrared lens 10 and a movement 20; the movement 20 includes an infrared detector 21, a first image processing module 22, and a second image processing module 23; the infrared lens 10 is connected to the infrared detector 21 and is used to collect the thermal radiation signal emitted by the target scene; the infrared detector 21 is connected to the first image processing module 22 and is used to perform optoelectronic conversion on the thermal radiation signal to obtain scene information; the first image processing module 22 is connected to the second image processing module 23 and is used to preprocess the scene information based on an artificial intelligence computational optical model to generate a restored image; the second image processing module 23 is used to generate a thermal imaging image based on the restored image. In this application, by adding the first image processing module 22 that runs the artificial intelligence computational optical model on the movement 20 side, on the premise of paying less cost, the imaging quality of the thermal infrared camera is greatly improved, and the problem that the existing thermal infrared camera cannot balance cost and image quality is solved.

[0059] In one embodiment, the infrared lens 10 is obtained based on the design result output by the joint simulation of optical design and restoration model.

[0060] Specifically, for existing thermal infrared cameras with large volume or weight, if they want to be miniaturized or lightened without sacrificing imaging quality, the infrared lens 10 can be redesigned and optimized. The design goal is that the main parameters such as the aperture, field of view, relative illumination, and distortion of the lens do not degrade, and at least one lens is reduced, but the MTF (Modulation Transfer Function) of the inspection frequency is allowed to decrease within a preset range.

[0061] In one embodiment, Figure 2 is a flowchart of the joint simulation of optical design and restoration model in one embodiment. As Figure 2 shown, the joint simulation of optical design and restoration model includes:

[0062] Step S110, optimize the initial simulation lens based on optical design software to obtain the optimized design result of the simulation lens; based on the simulation of the actual use effect of the simulation lens, add tolerance perturbations to the optimized design result to obtain the simulated design result of the simulation lens.

[0063] Step S120, calculate the training data set based on the simulated design result, and train the restoration model based on the training data set.

[0064] Step S130, calculate the simulated detector image based on the simulated design result; restore the simulated detector image based on the restoration model to obtain the simulated restored image.

[0065] Step S140: Evaluate the simulated and restored image. If the evaluation value reaches the simulation target, complete the design of the joint simulation and output the current simulation design result. If the evaluation value does not reach the simulation target, continue to optimize the design of the simulation lens until the evaluation value corresponding to the optimized lens reaches the simulation target.

[0066] In one embodiment, based on the above step S120, calculate the training data set based on the simulation design result, which specifically includes the following steps:

[0067] Step S121: Calculate the first point spread function of the simulation design result at different fields of view.

[0068] Step S122: Calculate the training data set based on the sub-fields of view of the high-definition image, the first point spread function, and random noise.

[0069] In one embodiment, based on the above step S130, calculate the simulated detector image based on the simulation design result, which specifically includes the following steps:

[0070] Step S131: Calculate the second point spread function corresponding to at least two fluctuating individuals based on the preset individual tolerance fluctuations and the simulation design result.

[0071] Step S132: Calculate the simulated detector image based on the second point spread function and the high-definition checkerboard image.

[0072] The above joint simulation method of optical design and restoration model will be described below through a preferred embodiment. Figure 3 It is a flowchart of the joint simulation of optical design and restoration model in a preferred embodiment.

[0073] Step S201: Use optical design software for optical optimization design. By modifying the initial structure, constraint conditions, and weighing the principal value parameters, optimize and examine the MTF value of the line pairs and the MTF consistency at different fields of view. Then, add eccentricity, tilt and other tolerance perturbations to the lenses with poor sensitivity, so that the MTF value drops by an empirical value at the examined frequency of this design, which represents the MTF drop of the lens caused by actual processing and assembly. Fix this version as the current optical design result.

[0074] Step S202: Output the PSF at different fields of view of the current optical design. The fields of view are divided according to a grid, with M in each row and N in each column, for a total of M×N fields of view. Then, convolve a large number of high-definition images in sub-fields with the PSF at different fields of view, and add an appropriate amount of random noise to obtain the training data set required by the restoration model, and train the restoration model based on this training data set. Among them, overlapping (overlap) fusion processing is adopted at the junction of different fields of view to prevent obvious boundary effects.

[0075] Step S203, introduce individual tolerance fluctuations between lenses into the optical design obtained in step S201, output multiple PSF data of fluctuation individuals, and then simulate multiple PSF data of fluctuation individuals through high-definition chessboard convolution to obtain simulated detector images of multiple lens individuals. Use the restoration model obtained in step S202 to restore the simulated detector images of multiple lens individuals respectively, and evaluate the restored images.

[0076] Step S204: If all evaluation values ​​reach the simulation target, the joint simulation design is completed and the current simulation design result is output; if the evaluation value does not reach the simulation target, the process returns to step S201 to continue optimizing the simulation lens until the evaluation value corresponding to the optimized simulation lens reaches the simulation target. The simulation target is the evaluation value for the target high-quality lens.

[0077] In this preferred embodiment, by considering the image quality restoration capability of the restoration model, a joint simulation of optical design and model training is performed, and whether the final imaging quality can be consistent with the benchmark high-quality lens is used as the most important evaluation criterion, thereby achieving quality optimization of low-cost lenses.

[0078] In one embodiment, the first image processing module 22 includes a neural network processor that runs an artificial intelligence computational optical model; the computing power of the neural network processor is determined based on the resolution and frame rate of the thermal infrared camera and the size of the artificial intelligence computational optical model.

[0079] Specifically, for a thermal infrared camera with a resolution of 640×512 and a frame rate of 30fps, a lightweight SOC chip is selected, the computing power of the neural network processor NPU is 4T@int8, the size of the artificial intelligence calculation optical model is 4.5G@0.3M, and the running time is less than 5ms. Turning off the non-uniform correction and filtering module in the second image processing module 23 and using the artificial intelligence calculation optical model for non-uniform correction, noise reduction, and aberration correction can still meet the frame rate requirements and ensure real-time performance.

[0080] In one embodiment, the artificial intelligence computational optical model is used to perform aberration correction on input scene information to generate an aberration-corrected restored image.

[0081] Specifically, the main purpose of the artificial intelligence calculation optical model is to solve the aberration problem caused by the reduction in the number of lenses and to improve the clarity of the original image output by the infrared detector 21. Therefore, the artificial intelligence calculation optical model needs to have an aberration correction function. However, due to other reasons such as the infrared detector 21, the restored image after aberration correction will still have non-uniformity and noise problems. The restored image can be further processed by turning on the non-uniform correction module and the filtering module of the second image processing module 23.

[0082] In one embodiment, the artificial intelligence computational optical model is used to perform non-uniform correction, noise reduction, and aberration correction on the input scene information, and generate a restored image that has been non-uniformly corrected, noise-reduced, and aberration-corrected.

[0083] Specifically, if only aberration correction is performed on the first module side, it will increase the processing difficulty of the second image processing module 23. Therefore, in the preferred solution, the tasks of the artificial intelligence computational optical model need to include three aspects: image non-uniform correction, noise reduction, and sharpness improvement, so as to reduce the processing pressure on the second image processing module 23.

[0084] In one embodiment, referring to Figure 4 , the training method of the artificial intelligence computational optical model includes:

[0085] Step S310, calibrate the non-uniform correction coefficient and the noise model.

[0086] First, use a thermal infrared test camera to capture uniform calibration plate images at high and low temperatures, where the difference between high and low temperatures is at least 60 degrees. Collect the 14-bit Raw images directly output by the detector, and calculate the non-uniform correction coefficient according to the basic principle of the two-point method. Then, use the thermal infrared test camera to capture at least ten groups of uniform calibration plate images at different temperatures, collect the 14-bit Raw images directly output by the detector, and then cluster the pixel values of different intensities in the Raw images for noise model fitting to obtain the noise model.

[0087] Step S320, based on the calibrated non-uniform correction coefficient and the noise model, calibrate the third point spread function of the infrared lens 10.

[0088] Specifically, use a thermal infrared test camera to capture the hollow thermal infrared calibration plate as shown in Figure 5 , and take multiple shots to collect several original images directly output by the detector. First, perform non-uniform correction on the captured original images using the non-uniform correction coefficient obtained in step S310, and then perform multiple-image superposition and fusion to remove the influence of noise, obtaining a corrected image that is not affected by the non-linear response and noise of the detector. Then, learn the corrected image and the label image of the hollow thermal infrared calibration plate in different fields of view to determine several third point spread functions PSFs of the infrared lens 10 in different fields of view.

[0089] Step S330, based on the obtained high-definition label image, the calibrated third point spread function, non-uniform correction coefficient, and noise model, generate a degraded image simulating the actual shooting of a thermal infrared camera, and use the degraded image and the high-definition label image as a training data pair.

[0090] Specifically, a large number of high-definition RGB images are used to generate grayscale images, and then the images are transformed into the original image domain with high bit and low dynamic range by degrading from the RGB domain to the Raw domain, obtaining a large number of high-definition GT (Ground Truth) images. Then, the high-definition GT images are convolved with the third point spread function PSF calibrated for different fields of view, and then simulated noise is added to the convolution result based on the calibrated noise model, and combined with the non-uniform correction coefficient obtained by calibration, so as to obtain a degraded image close to the actual shot of this thermal infrared camera. The degraded image and the high-definition GT image form a training data pair.

[0091] Step S340, training an artificial intelligence computational optics model based on the training data pair.

[0092] Specifically, the artificial intelligence computational optics model can adopt a network structure and Loss constraint that are both friendly in terms of effect and deployment in the field of image processing. Exemplarily, the network structure is Res-Unet, and the Loss constraint is L1loss + TV loss. The structure of the model needs to be correspondingly simplified according to aspects such as the resolution, frame rate, NPU computing power of the thermal infrared camera, and the support degree of the NPU for operators, including but not limited to methods such as network structure lightweighting, reparameterization, distillation, and pruning quantization. Then, the model is trained on a GPU server.

[0093] Since the first image processing module 22 that runs this model can use a lightweight SOC chip, the quantization tool provided by the lightweight SOC can be used for quantization and model deployment of the artificial intelligence computational optics model, and finally the purpose of non-uniform correction, noise reduction, and clarity improvement can be achieved in the actual thermal infrared camera.

[0094] In a preferred embodiment, the movement 20 of the thermal infrared camera based on computational optics technology is as Figure 5 shown:

[0095] The infrared detector 21 performs optoelectronic conversion on the thermal radiation signal collected by the infrared lens 10 to obtain scene information.

[0096] The first image processing module 22 uses a lightweight SOC chip with NPU computing power. This lightweight SOC chip runs an artificial intelligence computational optics model, which integrates non-uniform correction, noise reduction, and aberration correction functions, and preprocesses the scene information to obtain a restored image.

[0097] The second image processing module 23 uses an image ISP processing chip. The non-uniform correction unit and the filtering unit are selectively turned off on this image ISP processing chip, and dynamic range compression and other processing are directly performed on the preprocessed restored image, and finally a thermal imaging image is generated in the image ISP processing chip.

[0098] Due to the rapid development of artificial intelligence technology, lightweight SOC chips with NPU computing power are becoming more and more widely used, and the price is also becoming more and more affordable. For a costly thermal infrared camera, the cost of adding a lightweight SOC chip with the ability to carry AI computational optical algorithms in the core 20 is much less than the cost of adding a lens at the lens end. Therefore, in this preferred embodiment, for the thermal infrared camera, based on the computational optical technology of artificial intelligence, a lightweight SOC chip with NPU computing power is added in front of the image ISP processing chip in the core 20, which reduces the cost of the infrared lens 10 while ensuring the imaging quality of the camera, thereby enhancing the overall cost advantage of the thermal infrared camera, and also providing the possibility for the miniaturization and lightweight of the thermal infrared camera.

[0099] Figure 6 It is a comparison of the imaging effects between a thermal infrared camera based on computational optical technology and other cameras in an embodiment. Among them, from left to right are the thermal imaging images generated by a conventional low-cost camera, a thermal infrared camera based on computational optical technology, and a high-cost reference camera. It can be seen that the thermal infrared camera based on computational optical technology has significantly improved image quality compared to the conventional low-cost camera and can reach the imaging level of the high-cost reference camera.

[0100] In this embodiment, a thermal imaging method is also provided. As shown in FIG. 7, it is a flowchart of the thermal imaging method in an embodiment. Figure 7 As shown, the method is applied to the thermal infrared camera in any of the above embodiments; the method includes:

[0101] Step S401, collect the thermal radiation signal emitted by the target scene through the infrared lens 10 and conduct the thermal radiation signal to the infrared detector 21.

[0102] Step S402, perform photoelectric conversion on the thermal radiation signal through the infrared detector 21 to obtain scene information; and transmit the scene information to the first image processing module 22.

[0103] Step S403, preprocess the scene information based on the artificial intelligence computational optical model through the first image processing module 22 to generate a restored image; and send the restored image to the second image processing module 23.

[0104] Step S404, generate a thermal imaging image based on the restored image through the second image processing module 23.

[0105] In the thermal imaging method of this embodiment, while preprocessing the scene information through the artificial intelligence computational optical model to ensure the imaging quality of the thermal infrared camera, the overall cost advantage of the thermal infrared camera is enhanced.

[0106] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless clearly stated herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0107] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memories can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memories can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0108] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0109] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the patent of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A thermal infrared camera based on computational optical technology, characterized in that, the camera includes: an infrared lens and a camera core; the camera core includes an infrared detector, a first image processing module, and a second image processing module arranged in sequence; the infrared lens is connected to the infrared detector; the infrared lens is used to collect the thermal radiation signal emitted by the target scene and conduct the thermal radiation signal to the infrared detector, so as to perform photoelectric conversion on the thermal radiation signal through the infrared detector to generate scene information; the first image processing module is connected to the infrared detector; the first image processing module is used to receive the scene information and preprocess the scene information based on the artificial intelligence computational optical model to generate a restored image; the second image processing module is linked to the first image processing module; the second image processing module is used to receive the restored image and generate a thermal imaging image based on the restored image.

2. The thermal infrared camera based on computational optical technology according to claim 1, characterized in that, the infrared lens is obtained based on the design result output by the joint simulation of the optical design and the restoration model.

3. The thermal infrared camera based on computational optical technology according to claim 2, characterized in that, the joint simulation of the optical design and the restoration model includes: optimizing the initial simulation lens based on optical design software to obtain the optimized design result of the simulation lens; adding tolerance perturbations to the optimized design result based on the simulation of the actual use effect of the simulation lens to obtain the simulated design result of the simulation lens; calculating a training data set based on the simulated design result, and training the restoration model based on the training data set; calculating a simulated detector image based on the simulated design result; restoring the simulated detector image based on the restoration model to obtain a simulated restored image; evaluating the simulated restored image; if the evaluation value reaches the simulation target, the design of the joint simulation is completed and the current simulated design result is output; if the evaluation value does not reach the simulation target, the simulation lens is continuously optimized until the evaluation value corresponding to the optimized lens reaches the simulation target.

4. The thermal infrared camera based on computational optical technology according to claim 3, characterized in that, the calculating the training data set based on the simulated design result includes: calculating the first point spread function of the simulated design result at different fields of view; calculating the training data set based on the sub-fields of view of the high-definition image, the first point spread function, and random noise.

5. The thermal infrared camera based on computational optical technology according to claim 3, characterized in that, the calculating the simulated detector image based on the simulated design result includes: calculating the second point spread function corresponding to at least two fluctuating individuals based on the preset individual tolerance fluctuation and the simulated design result; calculating the simulated detector image based on the second point spread function and the high-definition checkerboard image.

6. The thermal infrared camera based on computational optical technology according to claim 1, characterized in that, The first image processing module includes a neural network processor; The neural network processor runs the artificial intelligence computational optics model; The computing power of the neural network processor is determined based on the resolution and frame rate of the thermal infrared camera and the size of the artificial intelligence computational optics model.

7. The thermal infrared camera based on computational optics technology according to claim 1, wherein, The artificial intelligence computational optics model is used to perform aberration correction on the input scene information to generate a restored image after the aberration correction.

8. The thermal infrared camera based on computational optics technology according to claim 1, wherein, The artificial intelligence computational optics model is used to perform non-uniformity correction, noise reduction, and aberration correction on the input scene information to generate a restored image after the non-uniformity correction, noise reduction, and aberration correction.

9. The thermal infrared camera based on computational optics technology according to claim 8, wherein, The training method of the artificial intelligence computational optics model includes: Calibrating the non-uniformity correction coefficient and the noise model; Based on the calibrated non-uniformity correction coefficient and the noise model, calibrating the third point spread function of the infrared lens; Based on the obtained high-definition label image, the calibrated third point spread function, the non-uniformity correction coefficient, and the noise model, generating a degraded image simulating the actual shooting of the thermal infrared camera, and using the degraded image and the high-definition label image as a training data pair; Training the artificial intelligence computational optics model based on the training data pair.

10. A thermal imaging method, wherein, The method is applied to the thermal infrared camera according to any one of claims 1 to 9; the method includes: Collecting the thermal radiation signal emitted by the target scene through the infrared lens, and conducting the thermal radiation signal to the infrared detector; Performing photoelectric conversion on the thermal radiation signal through the infrared detector to obtain scene information; And transmitting the scene information to the first image processing module; Preprocessing the scene information based on the artificial intelligence computational optics model through the first image processing module to generate a restored image; sending the restored image to the second image processing module; Generating a thermal imaging image based on the restored image through the second image processing module.

Images