Image processing method, device, chip and storage medium for infrared images

Abstract

The present invention discloses an image processing method, device, system-level chip and computer-readable storage medium for infrared images. The method is applied to the system-level chip, comprising: obtaining an infrared image to be processed collected by a detector according to a determined target image pattern; determining one or more hardware processing units corresponding to the target image pattern according to hardware resources in the system-level chip, so as to split the image processing flow of the target image pattern into each hardware processing unit; utilizing one or more hardware processing units to perform image processing on the infrared image to be processed according to the image processing flow of the target image pattern, and obtaining an output image of the target image pattern; the present invention adopts a detector direct drive method and utilizes different hardware resources in the Soc to complete infrared imaging with high complexity and good processing effect, so as to realize low-cost, high-quality and high-efficiency infrared imaging based on the Soc without adopting FPGA, thereby improving user experience.

Description

Technical Field

[0001] The present invention relates to the field of infrared imaging calculation, and in particular to an infrared image processing method, device, system-level chip and computer-readable storage medium. Background Art

[0002] Infrared imaging technology boasts advantages such as strong smoke penetration and 24 / 7 operation, promising broad applications in military weaponry, industrial production, and security monitoring. However, with the rapid development of this technology, the demand for low-cost, highly sensitive systems and high-definition infrared images is becoming increasingly urgent.

[0003] The vast majority of current infrared imaging devices utilize field-programmable gate arrays (FPGAs) for image processing. These devices, however, suffer from large size, limited scalability, and high cost. Using these devices alone cannot meet the complex and ever-changing needs of intelligent analysis within the industry. Therefore, the urgent need to address the issue of developing an infrared image processing method based on a SoC (System on Chip) that leverages diverse hardware resources to achieve a highly complex and effective imaging solution while also reducing overall hardware costs through direct detector drive is a pressing issue. Summary of the Invention

[0004] The purpose of the present invention is to provide an image processing method, device, system-level chip and computer-readable storage medium for infrared images, so as to utilize different hardware resources in the Soc to complete an imaging solution with higher complexity and better processing effect, and the use of direct drive of the detector can reduce the overall hardware cost.

[0005] To solve the above technical problems, the present invention provides an infrared image processing method applied to a system-on-chip, comprising:

[0006] Determining a current target image mode, and acquiring an infrared image to be processed collected by the detector according to the target image mode;

[0007] determining, based on hardware resources in the system-on-chip, one or more hardware processing units corresponding to the target image mode, so as to split the image processing flow of the target image mode into the respective hardware processing units;

[0008] Utilizing the one or more hardware processing units, image processing is performed on the infrared image to be processed according to the image processing process of the target image mode to obtain an output image of the target image mode; wherein the image processing process includes at least one of image enhancement, non-uniformity correction and image noise reduction.

[0009] Optionally, when the image processing process includes the image enhancement, the non-uniformity correction, and the image noise reduction, and the multiple hardware processing units corresponding to the target image mode include an ARM, a DSP, and an ISP, using the one or more hardware processing units to perform image processing on the to-be-processed infrared image according to the image processing process of the target image mode to obtain an output image of the target image mode includes:

[0010] Using the ARM, performing the non-uniformity correction on the infrared image to be processed to obtain a corrected image;

[0011] Using the ISP, performing 2D image noise reduction on the corrected image to obtain a noise-reduced image;

[0012] Using the DSP, performing image enhancement corresponding to the target image mode on the noise reduction image to obtain an enhanced image;

[0013] The ISP is used to perform 3D image noise reduction on the enhanced image to obtain the output image.

[0014] Optionally, when the target image mode is a wide dynamic mode, the DSP is used to perform image enhancement corresponding to the target image mode on the noise reduction image to obtain an enhanced image, including:

[0015] Performing high-pass filtering on a preset number of denoised images to obtain the preset number of high-frequency images; wherein the denoised images are images corresponding to the infrared images to be processed at different integration times of the preset number of frames acquired by the detector;

[0016] Performing Gaussian filtering on the high-frequency image to obtain the preset number of weighted images;

[0017] Performing low-pass filtering on the noise-reduced image to obtain the preset number of base layer images;

[0018] Acquire the preset number of detail images according to the denoised image and the base image; wherein the detail image is the difference between each denoised image and its corresponding base image;

[0019] The base image is fused with the detail image according to the weight image to obtain a frame of the enhanced image.

[0020] Optionally, when the preset number is 3, fusing the base image with the detail image according to the weight image to obtain a frame of the enhanced image includes:

[0021] The enhanced image is obtained by calculating fusion=(w1*f1+w2*f2+w3*f3) / w; wherein fusion is the enhanced image, f1=base1+detail1, f2=base2+detail2, f3=base3+detail3, w=w1+w2+w3, w1, w2 and w3 are the weight images, base1, base2 and base3 are the base images, detail1, detail2 and detail3 are the detail images.

[0022] Optionally, when the non-uniformity correction using the ARM includes stripe non-uniformity correction, performing the non-uniformity correction on the infrared image to be processed to obtain a corrected image includes:

[0023] The ARM is used to perform the stripe non-uniformity correction on the infrared image to be processed to obtain the corrected image; wherein the infrared image to be processed is the image output by the detector and has undergone response consistency correction.

[0024] Optionally, when the target image mode is a wide dynamic mode, before acquiring the infrared image to be processed collected by the detector according to the target image mode, the method further includes:

[0025] The image framing parameters corresponding to the wide dynamic mode are sent to the detector; wherein the image framing parameters include the preset number of different integration times.

[0026] Optionally, before performing image processing on the infrared image to be processed according to the image processing flow of the target image mode using the one or more hardware processing units and obtaining an output image of the target image mode, the method further includes:

[0027] Obtaining image configuration parameters corresponding to the target image mode;

[0028] Sending the image configuration parameters to the respective corresponding hardware processing units;

[0029] Correspondingly, the using the one or more hardware processing units to perform image processing on the to-be-processed infrared image according to the image processing flow of the target image mode to obtain an output image of the target image mode includes:

[0030] The one or more hardware processing units are used to perform image processing of the image processing flow on the infrared image to be processed according to the image configuration parameters to obtain an output image of the target image mode.

[0031] The present invention also provides an infrared image processing device, which is applied to a system-on-chip, comprising:

[0032] An image acquisition module is used to determine a current target image mode and acquire an infrared image to be processed collected by the detector according to the target image mode;

[0033] a hardware determination module, configured to determine one or more hardware processing units corresponding to the target image mode based on hardware resources in the system-on-chip, so as to split the image processing flow of the target image mode into the respective hardware processing units;

[0034] An image processing module is used to utilize the one or more hardware processing units to perform image processing on the infrared image to be processed according to the image processing process of the target image mode, and obtain an output image of the target image mode; wherein the image processing process includes at least one of image enhancement, non-uniformity correction and image noise reduction.

[0035] The present invention also provides a system-on-chip, comprising:

[0036] memory for storing computer programs;

[0037] The processor is used to implement the steps of the infrared image processing method as described above when executing the computer program.

[0038] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image processing method for infrared images as described above are implemented.

[0039] The present invention provides an infrared image processing method, which is applied to a system-on-chip (SoC), comprising: determining a current target image mode and acquiring an infrared image to be processed, collected by a detector, according to the target image mode; determining one or more hardware processing units corresponding to the target image mode based on hardware resources in the SoC, so as to split the image processing flow of the target image mode into each hardware processing unit; and performing image processing on the infrared image to be processed according to the image processing flow of the target image mode using the one or more hardware processing units to acquire an output image of the target image mode; wherein the image processing flow includes at least one of image enhancement, non-uniformity correction, and image noise reduction.

[0040] As can be seen, the present invention utilizes one or more hardware processing units corresponding to the target image mode to process the infrared image to be processed according to the image processing flow of the target image mode, obtaining the output image of the target image mode. This allows the use of different hardware resources in the SoC to achieve highly complex imaging with good processing effects, using a detector direct drive approach. This enables low-cost, high-quality, and efficient infrared imaging based on the SoC without the use of an FPGA, thereby improving the user experience. Furthermore, the present invention also provides an infrared image processing device, system-on-chip, and computer-readable storage medium, all of which also have the aforementioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0042] Figure 1 A flowchart of an infrared image processing method provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of an image processing flow chart of a sky-ground mode of another infrared image processing method provided by an embodiment of the present invention;

[0044] Figure 3 A diagram showing an output image of a sky-ground mode of another infrared image processing method provided by an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of an image processing flow in a wide dynamic mode of another infrared image processing method provided by an embodiment of the present invention;

[0046] Figure 5 A diagram showing an output image in a wide dynamic mode of another infrared image processing method provided by an embodiment of the present invention;

[0047] Figure 6 This is a structural block diagram of an infrared image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0049] Please refer to Figure 1 , Figure 1 This is a flow chart of an infrared image processing method provided by an embodiment of the present invention. The method is applied to a system-on-chip and includes:

[0050] Step 101: Determine the current target image mode, and obtain the infrared image to be processed collected by the detector according to the target image mode.

[0051] Specifically, in this embodiment, the target image mode may be the currently used preset image mode. For example, when the number of preset image modes is 1, the target image mode may be that preset image mode. When the number of preset image modes is greater than 1, the target image mode may be any preset image mode. The preset image mode in this embodiment may be a pre-set infrared imaging image mode, i.e., an image processing method of different imaging styles used by the SoC.

[0052] It should be noted that the specific number and types of preset image modes in this embodiment can be set by the designer according to practical scenarios and user needs. For example, the number of preset image modes can be 1, for example, the preset image mode can be a sky-and-ground mode, a wide dynamic mode, or a normal mode. The number of preset image modes can also be greater than 1, for example, the preset image modes can include a sky-and-ground mode, a wide dynamic mode, and a normal mode. The preset image modes can also include an outdoor mode, so that the user can select any preset image mode (i.e., a target image mode) for image processing according to the specific application scenario and their own needs; for example, in border defense, coastal defense, or outdoor security, where the scene is mostly half sky and half ground, the sky-and-ground mode can be selected; in industrial applications where the vast majority of objects in the field of view are visible, the wide dynamic mode can be selected.

[0053] Correspondingly, the specific way in which Soc determines the current target image mode in this step can be set by the designer according to practical scenarios and user needs. For example, when the number of preset image modes is 1, Soc can directly determine the preset image mode as the target image mode. When the number of preset image modes is greater than 1, Soc can determine the target image mode from the preset image modes based on the acquired image mode selection instruction, that is, determine the preset image mode corresponding to the image mode selection instruction as the target image mode; that is, the user can select the preset image mode required by triggering the image mode selection instruction; Soc can also perform scene recognition on the infrared image to be processed collected by the detector, and determine the preset image mode corresponding to the recognized scene as the target image mode. For example, when the recognized scene is a half-sky and half-earth sky and earth scene, the sky and earth mode can be determined as the target image mode. This embodiment does not impose any restrictions on this.

[0054] Among them, the detector in this step can be an infrared module for collecting infrared images, such as an uncooled infrared detector; the infrared image to be processed in this step can be an infrared image collected by the detector and output to the Soc (system-on-chip), that is, the original infrared image that needs to be processed by the Soc to complete infrared imaging.

[0055] Correspondingly, this embodiment does not limit the specific type of infrared image to be processed. For example, when the detector is an uncooled infrared detector (such as an uncooled infrared focal plane detector), the infrared image to be processed can specifically be an infrared image captured by the uncooled infrared detector.

[0056] Similarly, the specific method in which Soc obtains the infrared image to be processed collected by the detector according to the target image mode in this step can be set by the designer according to the practical scenario and user needs. For example, when the number of preset image modes in this embodiment is 1, Soc can directly obtain the infrared image to be processed collected by the detector, that is, the detector can directly output the collected infrared image (that is, the infrared image to be processed) to Soc according to the pre-configured image frame parameters corresponding to the preset image mode. When the number of preset image modes is greater than 1, Soc can first send the image framing parameters corresponding to the target image mode to the detector to complete the detector configuration, and then receive the infrared image to be processed collected by the detector according to the received image framing parameters; that is, the detector outputs the collected infrared image (that is, the infrared image to be processed) to Soc according to the image framing parameters corresponding to the received target image mode; for example, when the target image mode is a wide dynamic mode, before this step, Soc can obtain the image framing parameters corresponding to the wide dynamic mode (that is, the target image mode); and send the image framing parameters corresponding to the wide dynamic mode to the detector; wherein, the image framing parameters corresponding to the wide dynamic mode may include a preset number (that is, the number of frames) of different integration times, so that Soc can perform image processing on the infrared images of different integration times of a preset number of frames (such as 3 frames) collected by the detector in the wide dynamic mode to obtain 1 frame of output image.

[0057] Accordingly, the preset number can be greater than or equal to 2, so that when the target image mode is wide dynamic range mode, a multi-frame input of the preset number of frames is used, and infrared images to be processed in different states are obtained by configuring the detector's integration time. Since, for the same target, the integration time continuously increases, the output target is continuously enhanced, making the target gradually clearer. When the integration time is too long, oversaturation occurs. Taking all factors into consideration, the preset number in this embodiment can be specifically 3, so that three frames of input are obtained using three different detector integration times. Among the three different integration times, the short integration time can ensure the display of details of objects in high temperature areas, the medium integration time can ensure the normal display of the entire image, and the long integration time can ensure the display of details of objects in low temperature areas. This embodiment does not impose any restrictions on the configuration of different integration times for different detectors based on their respective noise-equivalent temperature differences (NETD).

[0058] Specifically, in this embodiment, Soc may include multiple functional modules such as detector configuration, data stitching and shutter correction to realize detector driving and obtain the infrared image to be processed collected by the detector.

[0059] Step 102: Determine one or more hardware processing units corresponding to the target image mode based on hardware resources in the system-on-chip, so as to split the image processing flow of the target image mode into each hardware processing unit.

[0060] It can be understood that the one or more hardware processing units corresponding to the target image mode in this step can be the hardware processing resources for completing the image processing corresponding to the target image mode in Soc; in this embodiment, the image processing process corresponding to the target image mode is split into each hardware processing unit, so as to utilize the hardware processing unit to complete the entire image processing process of the target image mode, thereby realizing infrared imaging based on Soc.

[0061] Specifically, the specific number and type of hardware processing units corresponding to the target image mode in this step can be set by the designer according to the practical scenario and user needs. For example, the hardware processing units corresponding to the target image mode can include at least one of ARM (Advanced RISC Machines, a microprocessor), DSP (Digital Signal Processing, digital signal processor), GPU (Graphics Processing Unit, graphics processor) ISP (Image Signal Processing, image signal processing unit), and NPU (Neural Processing Unit, neural network processor). For example, if only ARM resources are available in the Soc, the entire image processing process can be run in the ARM. If the hardware processing unit corresponding to the target image mode only includes ARM, the ARM can use multi-threading combined with the NEON (an ARM architecture processor extension structure) structure to perform image processing on the infrared image to be processed according to the image processing process to obtain the output image of the target image mode; if ARM+DSP resources are available in the Soc, the entire image processing process can be split into ARM and DSP; if ARM+DSP+NPU resources are available in the Soc, the entire image processing process can be split into ARM, DSP and NPU, and a neural network model can be used to replace part of the image processing process to complete the entire image processing process more efficiently. This embodiment does not impose any restrictions on this.

[0062] It should be noted that the specific method of determining one or more hardware processing units corresponding to the target image mode based on the hardware resources in the system-level chip in this step can be set by the designer according to the practical scenario and user needs. When the hardware processing units corresponding to each preset image mode are pre-set, the Soc can determine the hardware processing unit corresponding to the preset image mode determined as the target image mode as the hardware processing unit corresponding to the target image mode; the Soc can also detect its own hardware resources and determine the target resources (such as ARM, DSP, NPU and ISP, etc.) in the hardware resources as the hardware processing unit corresponding to the target image mode.

[0063] Step 103: Using one or more hardware processing units, perform image processing on the infrared image to be processed according to the image processing process of the target image mode to obtain an output image of the target image mode; wherein the image processing process includes at least one of image enhancement, non-uniformity correction and image noise reduction.

[0064] It can be understood that in this step, Soc can use the hardware processing unit corresponding to the target image mode to perform image processing on the infrared image to be processed according to the image processing flow corresponding to the target image mode, obtain the output image of the target image mode, and realize infrared imaging based on Soc.

[0065] Correspondingly, the image processing flow in this step can be the infrared imaging processing flow completed by the hardware processing unit. The specific process configuration of the image processing flow in this step can be set by the designer according to the practical scenario and user needs. For example, the image processing flow can include image enhancement to ensure infrared imaging of the image style of the target image mode; in order to improve the imaging effect of infrared imaging, the image processing flow can also include non-uniformity correction and / or image noise reduction (such as image 2D noise reduction and / or image 2D noise reduction). This embodiment does not impose any restrictions on this.

[0066] Specifically, in this step, Soc uses one or more hardware processing units corresponding to the target image mode to perform image processing on the infrared image to be processed according to the image processing process, and the specific method of obtaining the output image of the target image mode can be set by the designer. For example, when the image processing process includes image enhancement, non-uniformity correction and image noise reduction, Soc can perform non-uniformity correction on the infrared image to be processed in this step to obtain a corrected image; perform image 2D noise reduction on the corrected image to obtain a noise-reduced image; perform image enhancement corresponding to the target image mode on the noise-reduced image to obtain an enhanced image; perform image 3D noise reduction on the enhanced image to obtain an output image; wherein, image noise reduction includes image 2D noise reduction and image 3D noise reduction; that is, Soc can use the hardware processing unit to complete the image processing process including non-uniformity correction, image 2D noise reduction, image enhancement and image 3D noise reduction to obtain the output image of the target image mode. For example, Soc can use ARM to perform non-uniformity correction on the infrared image to be processed to obtain a corrected image; use ISP to perform image 2D noise reduction on the corrected image to obtain a noise-reduced image; use DSP to perform image enhancement corresponding to the target image mode on the noise-reduced image to obtain an enhanced image; use ISP to perform image 3D noise reduction on the enhanced image to obtain an output image.

[0067] It should be noted that the process of performing non-uniformity correction on the infrared image to be processed and obtaining a corrected image can be set by the designer. For example, the non-uniformity correction can include response consistency correction and stripe non-uniformity correction. That is, the SoC can perform response consistency correction on the infrared image to be processed to obtain a consistency-corrected image; and perform stripe non-uniformity correction on the consistency-corrected image to obtain a corrected image. When the detector performs response consistency correction on the infrared image to be processed, the non-uniformity correction in this embodiment can also include only stripe non-uniformity correction. That is, the SoC can perform stripe non-uniformity correction on the infrared image to be processed to obtain a corrected image. The infrared image to be processed is the image output by the detector that has undergone response consistency correction. For example, the SoC can use ARM to perform non-uniformity correction on the infrared image to be processed to obtain a corrected image.

[0068] Specifically, the above-mentioned response consistency correction process can be implemented using the response consistency correction method calibrated in the prior art, and this embodiment does not impose any restrictions on this. Since uncooled infrared focal plane detectors usually do not have independent amplifiers for each pixel in order to save production costs, the detection units in the same column or row share the same readout circuit; taking the column-integrated uncooled focal plane array as an example for analysis, the integrated current of the detection units in the same column corresponds to the same bias voltage. Due to the existence of amplifier bias voltage noise, the voltage obtained by the gate is different when the left and right columns are integrated. This results in that even if the same radiation value is obtained, the integrated current flowing through each detection unit in the same column is different; at the same time, the amplification factors of the two amplifiers in different columns will not be exactly the same, but there will be a small difference, so the output values will also have a small difference; therefore, for the stripes existing in the uncooled focal plane array, the stripe correction in the image is completed in this embodiment through the above-mentioned stripe non-uniformity correction process.

[0069] Correspondingly, the process of stripe non-uniformity correction in this embodiment may include: using the hardware processing unit (ARM or DSP) corresponding to the stripe non-uniformity correction to calculate the horizontal gradient map of the image to be corrected for stripe non-uniformity (such as the infrared image to be processed), and statistically calculating the gradient difference histogram. The absolute value of the difference can be limited to 0-255, such as using histogram[n]=MIN(abs(src[i+1]-src[i]),255) to obtain the gradient difference histogram, where n can be 0-255 statistical data, src is the input image (such as the horizontal gradient map), and i is the image traversal coordinate point; using the gradient threshold determination parameter in the image configuration parameter issued to obtain the gradient threshold. For example, when the gradient threshold determination parameter is 0.85, it can be used Obtain a gradient threshold, where width and height are the width and height of the image resolution, respectively. If the cumulative value is greater than the threshold (0.85*width*height) corresponding to the gradient threshold determination parameter, output n as the gradient threshold. Obtain the image stripe intensity value based on the stripe intensity determination parameter in the issued image configuration parameters. Perform stripe non-uniformity correction on the image to be corrected based on the image stripe intensity value and the gradient threshold to obtain a corrected image.

[0070] In this embodiment, when the image processing process includes image denoising, the image denoising may include 2D image denoising and 3D image denoising. The 2D image denoising process in this embodiment may include: utilizing a hardware processing unit (such as an ARM, DSP, or ISP) corresponding to the 2D image denoising, performing corresponding processing according to different noise types to complete spatial noise reduction (i.e., image 2D denoising), such as performing Poisson denoising on the 2D denoised image (such as the corrected image described above) to obtain a Poisson denoised image, thereby removing Poisson noise caused by photon signals in the 2D denoised image; performing glitch denoising on the Poisson denoised image to obtain a glitch denoised image, thereby removing glitch noise caused by abnormal edge mutations; and performing Gaussian denoising on the glitch denoised image to obtain a denoised image, thereby removing Gaussian noise. The 3D image denoising process in this embodiment may be implemented using a hardware processing unit (such as an ARM, DSP, or ISP) corresponding to the 3D image denoising, using a method identical or similar to the prior art 3D image denoising (i.e., time-domain denoising) method, and this embodiment does not impose any limitations on this.

[0071] Specifically, in this embodiment, the various preset image modes have great differences in the image enhancement process. The hardware processing units (such as ARM, DSP or GPU) corresponding to each preset image mode can perform their own corresponding image enhancement processes according to the different image enhancement parameters in the image configuration parameters issued. For example, when the target image mode is the normal mode, the hardware processing unit (such as ARM, DSP or ISP) corresponding to the normal mode can be used to first divide the image to be enhanced (such as the above-mentioned denoised image) into m*m blocks to obtain a histogram, and the size of each block is width / m, height / m; then the maximum pixel value maxValue and the minimum pixel value minValue of each block are counted; then, linear mapping is performed on the alignment to obtain a mapping result dst = (src-minValue) / (maxValue-minValue), where src is the input image (such as a histogram); after mapping The mapping table is q*q mean filtered to obtain a filtered image to prevent the block from having excessive halo phenomenon. For example, the mapping value corresponding to each pixel value of the first block is determined by taking the average of the mapping tables of the q blocks in its neighborhood; the filtered image is threshold segmented to obtain the foreground area and the background area to retain the foreground area of the highlight area and only enhance the significant area of the image that the human eye pays attention to; the foreground area in the histogram corresponding to the image to be enhanced is equalized to obtain an enhanced image, and the grayscale is reset according to the foreground mapping table. The background is not specially processed to achieve the effect of a very dark background and a very bright foreground, thereby completing the block contrast enhancement of the image to be enhanced.

[0072] When the target image mode is the sky and earth mode, the hardware processing unit corresponding to the sky and earth mode (such as ARM, or ARM and DSP, or ARM and GPU) can be used to first perform histogram statistics on the image to be enhanced (such as the denoised image mentioned above), and calculate the image vertical gradient difference histogram. The histogram corresponding to the image to be enhanced obtained by statistics and the calculated image vertical gradient difference histogram are combined to determine the cutting threshold for cutting the sky area and the ground area; the image to be enhanced is cut using the cutting threshold to determine the sky area and the ground area in the image to be enhanced; the preset fixed grayscale distribution range (such as 10-20 grayscales) is used to redistribute the grayscale outside the preset fixed grayscale distribution range in the sky area of the image to be enhanced, and the enhanced image is obtained after being fused with the ground area, thereby obtaining the image as shown below. Figure 3 Output image of the sky-ground mode is shown.

[0073] When the target image mode is the wide dynamic mode, the hardware processing unit corresponding to the wide dynamic mode (such as ARM, DSP or GPU) can be used to perform high-pass filtering on a preset number of images to be enhanced (such as the above-mentioned noise reduction images) to obtain a preset number of high-frequency images; Gaussian filtering is performed on the high-frequency images to obtain a preset number of weighted images; low-pass filtering is performed on the noise reduction images to obtain a preset number of base images; based on the noise reduction images and the base images, a preset number of detail images are obtained, and the detail images are the differences between each noise reduction image and its corresponding base image; based on the weighted images, the base images and the detail images are fused to obtain a frame of enhanced image, so as to fuse images with different integration times, thereby obtaining the following: Figure 5 The output image of the wide dynamic mode is shown; wherein the image to be enhanced corresponds to the image corresponding to the infrared image to be processed with different integration times and a preset number of frames acquired by the detector. If the preset number is 3, that is, when the detector acquires and outputs 3 frames of infrared images to be processed with different integration times to the Soc, the above-mentioned fusion of the base image and the detail image according to the weight image to obtain a frame of enhanced image can include: calculating the enhanced image by fusion = (w1*f1+w2*f2+w3*f3) / w; wherein fusion is the enhanced image, that is, the output result of the image enhancement in the wide dynamic mode; f1 = base1+detail1, f2 = base2+detail2, f3 = base3+detail3, w = w1+w2+w3, w1, w2, and w3 are weight images, base1, base2, and base3 are base images, and detail1, detail2, and detail3 are detail images.

[0074] Further, such as Figure 2As shown, when the target image mode is the sky-ground mode, Soc can perform sky-ground scene analysis on the image to be enhanced before using the hardware processing unit corresponding to the sky-ground mode to perform image enhancement on the image to be enhanced, and detect whether the image to be enhanced is in the sky-ground scene. If so, the hardware processing unit corresponding to the sky-ground mode is used to perform image enhancement on the image to be enhanced; if not, the hardware processing unit corresponding to the normal mode is used to perform image enhancement on the image to be enhanced.

[0075] Specifically, in this embodiment, the Soc can obtain image configuration parameters corresponding to the target image mode before step 103, and send the image configuration parameters to the corresponding hardware processing units, so that in step 103, the hardware processing units can be used to perform image processing of the image processing flow on the infrared image to be processed according to the received image configuration parameters, thereby obtaining the output image of the target image mode. For example, when the target image mode is the normal mode, the ARM processor in the Soc can obtain and send image configuration parameters corresponding to the normal mode (such as default parameters) to the corresponding hardware processing units. For example, in the normal mode, the Soc can use a single frame input and use the hardware processing units corresponding to the normal mode to perform image processing of the image processing flow on the infrared image to be processed with the single frame input according to the default parameters, thereby obtaining the output image of the normal mode. When the target image mode is the sky and earth mode, the ARM processor in the Soc can obtain and send image configuration parameters corresponding to the sky and earth mode to the corresponding hardware processing units, such as modifying the data statistics part of the image enhancement process and the parameters of the image 3D noise reduction process in the default parameters. For example, in the sky and earth mode, the Soc can use a single frame input and use the hardware processing units corresponding to the sky and earth mode to perform image processing of the image processing flow on the infrared image to be processed with the single frame input according to the partially modified default parameters, thereby obtaining the output image of the sky and earth mode.

[0076] Specifically, the ARM processor in this embodiment can take into account the scheduling of different tasks, and can dynamically call hardware processing resources such as DSP, NUP or GPU to run according to the hardware processing resources and parameter configuration of Soc, so that the algorithm operation can be sufficiently efficient. By adopting algorithm process optimization, compiler optimization, SIMD (single instruction stream multiple data stream) instruction set optimization, cevedsp instruction set and NEON multi-threading and other optimization methods, the image processing process requirements of real-time processing of the entire infrared image are met.

[0077] In this embodiment, the embodiment of the present invention utilizes one or more hardware processing units corresponding to the target image mode to perform image processing on the infrared image to be processed according to the image processing flow of the target image mode to obtain the output image of the target image mode. It can adopt the detector direct drive method and utilize different hardware resources in the Soc to complete imaging with higher complexity and better processing effect. In this way, without using FPGA, low-cost, high-quality and high-efficiency infrared imaging is achieved based on the Soc, thereby improving the user experience.

[0078] Corresponding to the above method embodiment, an embodiment of the present invention further provides an image processing device for infrared images. The image processing device for infrared images described below and the image processing method for infrared images described above can refer to each other.

[0079] Please refer to Figure 6 , Figure 6 This is a block diagram of the structure of an infrared image processing device provided by an embodiment of the present invention. The device is applied to a system-on-chip and may include:

[0080] The image acquisition module 10 is used to determine the current target image mode and acquire the infrared image to be processed collected by the detector according to the target image mode;

[0081] The hardware determination module 20 is configured to determine one or more hardware processing units corresponding to the target image mode based on the hardware resources in the system-on-chip, so as to split the image processing flow of the target image mode into the respective hardware processing units;

[0082] The image processing module 30 is used to use one or more hardware processing units to perform image processing on the infrared image to be processed according to the image processing process of the target image mode to obtain an output image of the target image mode; wherein the image processing process includes at least one of image enhancement, non-uniformity correction and image noise reduction.

[0083] Optionally, when the image processing process includes image enhancement, non-uniformity correction, and image noise reduction, and the multiple hardware processing units corresponding to the target image mode include an ARM, a DSP, and an ISP, the image processing module 30 may include:

[0084] The correction submodule is used to use ARM to perform non-uniformity correction on the infrared image to be processed and obtain a corrected image;

[0085] A 2D denoising submodule is used to perform 2D denoising on the corrected image using the ISP to obtain a denoised image;

[0086] The image enhancement submodule is used to perform image enhancement corresponding to the target image pattern on the noise reduction image using DSP to obtain an enhanced image;

[0087] The 3D denoising submodule is used to perform 3D denoising on the enhanced image using the ISP to obtain an output image.

[0088] Optionally, when the target image mode is a wide dynamic mode, the image enhancement submodule may include:

[0089] A high-pass filtering unit is used to perform high-pass filtering on a preset number of denoised images to obtain a preset number of high-frequency images; wherein the denoised images are images corresponding to the infrared images to be processed with different integration times of a preset number of frames acquired by the detector;

[0090] A Gaussian filter unit is used to perform Gaussian filtering on the high-frequency image to obtain a preset number of weighted images;

[0091] A low-pass filtering unit, configured to perform low-pass filtering on the noise-reduced image to obtain a preset number of base-level images;

[0092] A detail acquisition unit, configured to acquire a preset number of detail images based on the denoised image and the base image; wherein the detail image is the difference between each denoised image and its corresponding base image;

[0093] The fusion unit is used to fuse the base image with the detail image according to the weight image to obtain a frame of enhanced image.

[0094] Optionally, when the preset number is 3, the fusion unit can be specifically used to calculate an enhanced image through fusion = (w1*f1+w2*f2+w3*f3) / w; wherein, fusion is the enhanced image, f1 = base1+detail1, f2 = base2+detail2, f3 = base3+detail3, w = w1+w2+w3, w1, w2 and w3 are weight images, base1, base2 and base3 are base images, detail1, detail2 and detail3 are detail images.

[0095] Optionally, when the non-uniformity correction includes stripe non-uniformity correction, the correction submodule may include:

[0096] The fringe non-uniformity correction unit is used to use ARM to perform fringe non-uniformity correction on the infrared image to be processed to obtain a corrected image; wherein the infrared image to be processed is an image output by the detector that has been corrected for response consistency.

[0097] Optionally, when the target image mode is a wide dynamic mode, the apparatus may further include:

[0098] The detector configuration module is used to send image framing parameters corresponding to the wide dynamic mode to the detector; wherein the image framing parameters include a preset number of different integration times.

[0099] Optionally, the device may further include:

[0100] A parameter acquisition module is used to obtain image configuration parameters corresponding to the target image mode;

[0101] A parameter configuration module, used to send image configuration parameters to the corresponding hardware processing units;

[0102] Correspondingly, the image processing module 30 may be specifically configured to utilize one or more hardware processing units to perform image processing on the infrared image to be processed according to the image configuration parameters, and obtain an output image.

[0103] In this embodiment, the embodiment of the present invention utilizes one or more hardware processing units corresponding to the target image mode through the image processing module 30 to perform image processing on the infrared image to be processed according to the image processing flow of the target image mode to obtain the output image of the target image mode. It can adopt the detector direct drive method and utilize different hardware resources in the Soc to complete imaging with higher complexity and better processing effect, thereby achieving low-cost, high-quality and high-efficiency infrared imaging based on the Soc without adopting FPGA, thereby improving user experience.

[0104] Corresponding to the above method embodiment, an embodiment of the present invention further provides a system-on-chip. The system-on-chip described below and the infrared image processing method described above can refer to each other.

[0105] A system-on-chip comprises: a memory for storing a computer program; and a processor for implementing the steps of the infrared image processing method provided in the above embodiment when executing the computer program.

[0106] The system-on-chip provided in this embodiment may include a hardware processing unit, and the hardware processing unit may include at least one of ARM, DSP, GPU and NPU.

[0107] Corresponding to the above method embodiment, an embodiment of the present invention further provides a computer-readable storage medium. The computer-readable storage medium described below and the infrared image processing method described above can refer to each other.

[0108] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the infrared image processing method provided by the above method embodiment.

[0109] The computer-readable storage medium may be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, which may store program codes.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. References to the common and similar parts between the various embodiments are sufficient. The devices, system-on-chips, and computer-readable storage media disclosed in the embodiments correspond to the methods disclosed in the embodiments, so their descriptions are relatively simple. For relevant details, refer to the descriptions of the methods.

[0111] The above describes in detail the infrared image processing method, device, system-on-chip, and computer-readable storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is intended only to facilitate understanding of the method and core concepts of the present invention. It should be noted that those skilled in the art may make various improvements and modifications to the present invention without departing from the principles of the present invention, and such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. An image processing method for infrared images, characterized in that: Applications in system-on-chips include: Determining a current target image mode, and acquiring an infrared image to be processed collected by the detector according to the target image mode; determining, based on hardware resources in the system-on-chip, one or more hardware processing units corresponding to the target image mode, so as to split the image processing flow of the target image mode into the respective hardware processing units; Using the one or more hardware processing units, perform image processing on the infrared image to be processed according to the image processing process of the target image mode to obtain an output image of the target image mode; wherein the image processing process includes at least one of image enhancement, non-uniformity correction, and image noise reduction; When the image processing process includes the image enhancement, the non-uniformity correction, and the image noise reduction, and the multiple hardware processing units corresponding to the target image mode include an ARM, a DSP, and an ISP, performing image processing on the to-be-processed infrared image according to the image processing process of the target image mode by using the one or more hardware processing units to obtain an output image of the target image mode includes: Using the ARM, performing the non-uniformity correction on the infrared image to be processed to obtain a corrected image; Using the ISP, performing 2D image noise reduction on the corrected image to obtain a noise-reduced image; Using the DSP, performing image enhancement corresponding to the target image mode on the noise reduction image to obtain an enhanced image; The ISP is used to perform 3D image noise reduction on the enhanced image to obtain the output image.

2. The image processing method of infrared images according to claim 1, characterized in that: When the target image mode is a wide dynamic mode, performing image enhancement corresponding to the target image mode on the noise reduction image by using the DSP to obtain an enhanced image includes: Performing high-pass filtering on a preset number of denoised images to obtain the preset number of high-frequency images; wherein the denoised images are images corresponding to the infrared images to be processed of the preset number of frames with different integration times acquired by the detector; Performing Gaussian filtering on the high-frequency image to obtain the preset number of weighted images; Performing low-pass filtering on the noise-reduced image to obtain the preset number of base layer images; Acquire the preset number of detail images according to the denoised image and the base image; wherein the detail image is the difference between each denoised image and its corresponding base image; The base image is fused with the detail image according to the weight image to obtain a frame of the enhanced image.

3. The image processing method of infrared images according to claim 2, characterized in that: When the preset number is 3, fusing the base image with the detail image according to the weight image to obtain a frame of the enhanced image includes: The enhanced image is obtained by calculating fusion=(w1*f1+w2*f2+w3*f3) / w; wherein fusion is the enhanced image, f1=base1+detail1, f2=base2+detail2, f3=base3+detail3, w=w1+w2+w3, w1, w2 and w3 are the weight images, base1, base2 and base3 are the base images, detail1, detail2 and detail3 are the detail images.

4. The image processing method of infrared images according to claim 1, characterized in that: When the non-uniformity correction includes stripe non-uniformity correction, the step of using the ARM to perform the non-uniformity correction on the infrared image to be processed to obtain a corrected image includes: The ARM is used to perform the stripe non-uniformity correction on the infrared image to be processed to obtain the corrected image; wherein the infrared image to be processed is the image output by the detector and has undergone response consistency correction.

5. The image processing method of infrared images according to claim 1, characterized in that: When the target image mode is a wide dynamic mode, before acquiring the infrared image to be processed collected by the detector according to the target image mode, the method further includes: The image framing parameters corresponding to the wide dynamic mode are sent to the detector; wherein the image framing parameters include a preset number of different integration times.

6. The infrared image processing method according to any one of claims 1 to 5, characterized in that: Before performing image processing on the infrared image to be processed according to the image processing flow of the target image mode by using the one or more hardware processing units and obtaining an output image of the target image mode, the method further includes: Obtaining image configuration parameters corresponding to the target image mode; Sending the image configuration parameters to the respective corresponding hardware processing units; Correspondingly, the using the one or more hardware processing units to perform image processing on the to-be-processed infrared image according to the image processing flow of the target image mode to obtain an output image of the target image mode includes: The one or more hardware processing units are used to perform image processing of the image processing flow on the infrared image to be processed according to the image configuration parameters to obtain an output image of the target image mode.

7. An image processing device for infrared images, characterized in that: Applications in system-on-chips include: An image acquisition module is used to determine a current target image mode and acquire an infrared image to be processed collected by the detector according to the target image mode; a hardware determination module, configured to determine one or more hardware processing units corresponding to the target image mode based on hardware resources in the system-on-chip, so as to split the image processing flow of the target image mode into the respective hardware processing units; an image processing module, configured to perform image processing on the infrared image to be processed according to the image processing process of the target image mode using the one or more hardware processing units, and obtain an output image of the target image mode; wherein the image processing process includes at least one of image enhancement, non-uniformity correction, and image noise reduction; When the image processing flow includes the image enhancement, the non-uniformity correction, and the image noise reduction, and the multiple hardware processing units corresponding to the target image mode include an ARM, a DSP, and an ISP, the image processing module includes: A correction submodule, configured to use the ARM to perform non-uniformity correction on the infrared image to be processed to obtain a corrected image; a 2D denoising submodule, configured to perform 2D denoising on the corrected image using the ISP to obtain a denoised image; An image enhancement submodule, configured to utilize the DSP to perform image enhancement corresponding to the target image pattern on the noise reduction image to obtain an enhanced image; The 3D noise reduction submodule is configured to perform 3D noise reduction on the enhanced image using the ISP to obtain the output image.

8. A system-on-chip, characterized in that: include: Memory for storing computer programs; A processor is configured to implement the steps of the infrared image processing method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the infrared image processing method according to any one of claims 1 to 6 are implemented.

Images