Display methods, devices, equipment, and media for wide-angle low-light infrared image fusion

Abstract

This application provides a display method, apparatus, device, and medium for wide-angle low-light infrared image fusion, relating to the field of image processing technology. It involves acquiring infrared images through a wide-angle lens; acquiring low-light images through the same lens; fusing the infrared and low-light images to obtain a target fused image; merging the target fused image to obtain a fused video; cropping the fused video to obtain candidate display video images; performing distortion correction on the candidate display video images to obtain the target display video image, which is then displayed. This application improves the clarity of the displayed video image through image fusion of infrared and low-light images. Acquiring infrared and low-light images with a wide-angle lens increases the viewing angle, and distortion correction further enhances the clarity of the displayed video image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a display method, apparatus, device and medium for wide-angle low-light infrared image fusion. Background Technology

[0002] Currently, to meet the need for 24 / 7 real-time monitoring of the surrounding environment, images are typically captured using infrared or low-light cameras alone. However, the resulting displayed images are generally affected by environmental factors, leading to low image clarity. Alternatively, display methods using infrared and low-light image fusion are available, which are largely unaffected by environmental factors, but still suffer from poor image quality, such as low clarity. Therefore, improving the clarity of displayed images has become an urgent technical problem to be solved. Summary of the Invention

[0003] The main objective of this application is to provide a display method, apparatus, device, and medium for wide-angle low-light infrared image fusion, which can improve the clarity of the displayed image.

[0004] To achieve the above objectives, a first aspect of this application proposes a display method for wide-angle low-light infrared image fusion, the method comprising:

[0005] Infrared images are captured using a wide-angle lens;

[0006] Capture low-light images using a wide-angle lens;

[0007] The infrared image and the low-light image are fused together to obtain the target fused image.

[0008] The target fused images are merged to obtain a fused video;

[0009] The fused video is cropped to obtain candidate display video images;

[0010] The candidate display video image is subjected to distortion correction processing to obtain the target display video image, and the target display video image is then displayed.

[0011] In some embodiments, the step of performing image fusion processing on the infrared image and the low-light image to obtain a target fused image includes:

[0012] Each infrared image is decomposed into two scales to obtain an infrared basic feature image and an infrared detail feature image.

[0013] Each of the aforementioned low-light images is decomposed into two scales to obtain a low-light basic feature image and a low-light detail feature image;

[0014] The infrared basic feature image and the low-light basic feature image are summed and averaged to obtain a fused basic feature image;

[0015] The infrared detail image and the low-light detail image are weighted and summed to obtain a fused detail image;

[0016] The fused basic feature image and the fused detailed feature image are subjected to feature fusion processing to obtain the target fused image.

[0017] In some embodiments, the target fused image is a grayscale image. After performing image fusion processing on the infrared image and the low-light image to obtain the target fused image, the method further includes:

[0018] Obtain the pixel values ​​of the target fused image to get the original grayscale values;

[0019] The corrected red value is obtained by searching for the red value in a preset red curve lookup table based on the original gray value; wherein, the red curve lookup table includes matching information between the original gray value and the corrected red value;

[0020] The corrected green value is obtained by searching for a green value in a preset green curve lookup table based on the original gray value; wherein, the green curve lookup table includes matching information between the original gray value and the corrected green value;

[0021] The corrected blue value is obtained by searching for the blue value in a preset blue curve lookup table based on the original gray value; wherein, the blue curve lookup table includes matching information between the original gray value and the corrected blue value;

[0022] Pixels are synthesized based on the corrected red value, the corrected green value, and the corrected blue value to obtain the target color value;

[0023] The original grayscale value in the target fused image is replaced according to the target color value to obtain the updated target fused image.

[0024] In some embodiments, before performing image fusion processing on the infrared image and the low-light image to obtain the target fused image, the method further includes:

[0025] Obtain the pixel values ​​of the infrared image to get the original infrared pixel values;

[0026] The corrected infrared pixel value is obtained by searching the pixel value in a preset pixel correction lookup table based on the original infrared pixel value; wherein, the pixel correction lookup table includes matching information between the original pixel value and the corrected pixel value;

[0027] The original infrared pixel values ​​in the infrared image are replaced according to the corrected infrared pixel values ​​to obtain the updated infrared image;

[0028] Obtain the pixel values ​​of the low-light image to get the original low-light pixel values;

[0029] Based on the original low-light pixel value, the pixel value is looked up in the pixel correction lookup table to obtain the corrected low-light pixel value;

[0030] The original low-light pixel values ​​in the low-light image are replaced according to the corrected low-light pixel values ​​to obtain the updated low-light image.

[0031] In some embodiments, before performing a pixel value lookup in a preset pixel correction lookup table based on the original infrared pixel value to obtain the corrected infrared pixel value, the method further includes:

[0032] Creating the pixel correction lookup table specifically includes:

[0033] Obtain the original pixel values; wherein any two original pixel values ​​are different from each other;

[0034] Each of the original pixel values ​​is corrected to obtain a corrected pixel value; the correction calculation includes normalization calculation, pre-compensation calculation, and inverse normalization calculation.

[0035] The pixel correction lookup table is obtained based on the matching information between the original pixel value and the corrected pixel value.

[0036] In some embodiments, the step of cropping the fused video to obtain candidate display video images includes:

[0037] Acquire attitude information, which includes the current angle parameter;

[0038] Determine the position offset based on the current angle parameter;

[0039] Using the position offset as the cropping center, the fused video is cropped at a preset cropping angle to obtain candidate display video images.

[0040] In some embodiments, the distortion correction processing of the candidate display video image to obtain the target display video image includes:

[0041] The pixel value is searched in a preset distortion correction lookup table based on the position offset to obtain the target distortion correction pixel value.

[0042] The pixel values ​​of the candidate display video image are replaced according to the target distortion correction pixel values ​​to obtain the target display video image;

[0043] Specifically, creating the distortion correction lookup table includes:

[0044] Obtain the resolution of the fused video to get the original resolution;

[0045] Obtain the original field of view of the wide-angle lens;

[0046] The target resolution is obtained by calculating the resolution based on the original resolution, the cropping angle, and the original angle; wherein the target resolution is smaller than the original resolution.

[0047] The offset is calculated based on the original resolution, the cropping viewpoint, and the current angle parameter to obtain the target offset.

[0048] Using the target offset as the dividing center, the fused video is divided into images according to the target resolution to obtain divided images; wherein, the resolution of the divided images is the target resolution;

[0049] Obtain the pixel values ​​of the segmented image to obtain the initial segmentation pixel values;

[0050] Distortion correction is performed on the initial segmented pixel values ​​to obtain the target distortion-corrected pixel values ​​of the segmented image;

[0051] The distortion correction lookup table is obtained based on the matching information between the target offset and the target distortion correction pixel value.

[0052] To achieve the above objectives, a second aspect of this application provides a display device for wide-angle low-light infrared image fusion, the device comprising:

[0053] The system includes a processor, an infrared camera with a fisheye lens, a low-light camera with a fisheye lens, an attitude sensor, and a display.

[0054] An infrared camera with a fisheye lens is used to acquire infrared images;

[0055] A fisheye lens low-light camera used to capture low-light images;

[0056] An attitude sensor is used to acquire attitude information, which includes current angle parameters.

[0057] The processor is configured to perform image fusion processing on the infrared image and the low-light image to obtain a target fused image; merge the target fused image to obtain a fused video; determine a position offset based on the current angle parameter; use the position offset as the cropping center to crop the fused video at a preset cropping angle to obtain a candidate display video image; and perform distortion correction processing on the candidate display video image to obtain a target display video image.

[0058] A display for displaying the target video image.

[0059] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory, a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0060] To achieve the above objectives, a fourth aspect of the present application provides a storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0061] The wide-angle low-light infrared image fusion display method, apparatus, device, and medium proposed in this application improve the clarity of displayed video images by fusing infrared and low-light images. By acquiring infrared and low-light images through a wide-angle lens, more information about the object can be obtained due to the large viewing angle. Then, distortion correction processing restores the true shape of the object, further improving the clarity of the displayed video image. Attached Figure Description

[0062] Figure 1 This is a flowchart of a display method for wide-angle low-light infrared image fusion provided in one embodiment of this application;

[0063] Figure 2 This is a flowchart of a display method for wide-angle low-light infrared image fusion provided in another embodiment of this application;

[0064] Figure 3 yes Figure 1 The flowchart of step S103 in the process;

[0065] Figure 4 This is a schematic diagram illustrating the mapping between image grayscale values ​​and color values ​​according to an embodiment of this application;

[0066] Figure 5 This is a flowchart of a display method for wide-angle low-light infrared image fusion provided in another embodiment of this application;

[0067] Figure 6This is a target plate image provided in one embodiment of this application;

[0068] Figure 7 This is a schematic diagram of a fisheye imaging model provided in one embodiment of this application;

[0069] Figure 8 This is a schematic diagram of a small field of view in the original image provided in one embodiment of this application;

[0070] Figure 9 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0073] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0074] First, let's analyze some of the terms used in this application:

[0075] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0076] Encoder-decoder: This is a common model framework in deep learning. Many common applications are designed using the encoder-decoder framework. The encoder and decoder parts can be any text, speech, image, and video data, etc. Various models can be designed based on the encoder-decoder.

[0077] Image fusion refers to the process of combining image data of the same target acquired from multiple sources using image processing and computer technology. This process maximizes the extraction of useful information from each source and synthesizes the resulting high-quality image. This improves the utilization of image information, enhances the accuracy and reliability of computer interpretation, and increases the spatial and spectral resolution of the original image, thus facilitating monitoring. Image fusion involves preprocessing multiple images, such as denoising and registration, and then combining them according to certain fusion rules to create a single image. The fused image provides a clearer and more accurate description of the target and is more suitable for subsequent image processing. (Multi-sensor image fusion (visible light and infrared image fusion), single-sensor multi-focus image fusion).

[0078] Three basic principles that image fusion must follow:

[0079] 1) The fused image should contain prominent information from all the source images;

[0080] 2) No human information should be added to the merged image;

[0081] 3) For information that is not of interest in the source image, such as noise, try to suppress it as much as possible from appearing in the fused image.

[0082] Based on the principle of information extraction levels from low to high, they can be divided into three categories: pixel-level image fusion, feature-level image fusion, and decision-level image fusion.

[0083] Pixel-level fusion is a process that directly fuses pixel-based features of the source image according to certain fusion rules to generate a fused image. It retains the most original information of the source image and has the highest fusion accuracy. However, this type of method also has disadvantages such as requiring the most information, having high requirements for hardware and registration, long computation time, and poor real-time processing.

[0084] Currently, to meet the need for all-weather, real-time monitoring of the surrounding environment, both day and night, images are typically captured using infrared cameras or low-light cameras alone. However, this approach has the following drawbacks: I. Disadvantages of traditional infrared camera imaging systems: 1. Only objects radiating heat can be imaged, resulting in loss of detail. 2. Low image contrast and poor detail resolution. Since infrared thermal imagers rely on temperature differences for imaging, and the temperature difference of most targets is small, the low contrast of infrared thermal images reduces the ability to resolve details. 3. Poor penetration; targets cannot be seen through transparent obstacles, such as window glass. Because infrared thermal imagers rely on temperature differences for imaging, transparent obstacles like window glass cannot be detected by the infrared thermal imager due to the temperature difference of objects behind them, thus preventing clear visibility. II. Disadvantages of traditional low-light camera imaging systems: 1. In darkness, the specific outlines of objects with weak light reflection cannot be clearly seen, leading to the loss of some features and the inability to determine the specific outlines of objects. Therefore, using infrared cameras or low-light cameras alone results in images that are less than ideal due to the influence of various environmental factors. Infrared low-light image fusion systems can also be used, but the fusion algorithms employed are not ideal, are slow, and distortion correction relies solely on simple feature matching, resulting in low matching accuracy. Furthermore, the algorithms are slow and lack real-time performance. Therefore, improving the clarity of the displayed image has become a pressing technical problem.

[0085] Based on this, embodiments of this application provide a display method, apparatus, device, and medium for wide-angle low-light infrared image fusion, aiming to improve the clarity of the displayed image through image fusion of infrared and low-light images. The image is acquired using a wide-angle lens and distortion correction is performed, providing a large viewing angle and restoring the true shape of objects, further improving the clarity of the displayed image. Additionally, video is cropped before distortion correction, reducing the resolution of the distortion correction process and improving its efficiency.

[0086] The wide-angle low-light infrared image fusion display method provided in this application embodiment can be applied to a server, or it can be software running on a terminal or server. The server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or it can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the wide-angle low-light infrared image fusion display method, etc., but is not limited to the above forms.

[0087] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include server computers, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0088] This application provides a display method, display device, electronic device, and storage medium for wide-angle low-light infrared image fusion, which are specifically described through the following embodiments. First, the display method for wide-angle low-light infrared image fusion in this application embodiment is described.

[0089] Figure 1 This is an optional flowchart of the wide-angle low-light infrared image fusion display method provided in the embodiments of this application, which may include, but is not limited to, steps S101 to S106.

[0090] Step S101: Acquire infrared images using a wide-angle lens;

[0091] Step S102: Acquire low-light images using a wide-angle lens;

[0092] Step S103: Perform image fusion processing on the infrared image and the low-light image to obtain the target fused image;

[0093] Step S104: Merge the target images to obtain the fused video;

[0094] Step S105: Perform cropping processing on the fused video to obtain candidate display video images;

[0095] Step S106: Perform distortion correction processing on the candidate display video image to obtain the target display video image, and then display the target display video image.

[0096] Steps S101 to S106, as illustrated in this embodiment, improve the clarity of the displayed video image through image fusion of infrared and low-light images. By acquiring infrared and low-light images using a wide-angle lens, more information about the object can be obtained due to the large viewing angle. Then, distortion correction processing restores the true shape of the object, further improving the clarity of the displayed video image. Additionally, the video is cropped before distortion correction, reducing the resolution of the distortion correction process and improving its efficiency.

[0097] In step S101 of some embodiments, a wide-angle lens is a photographic lens with a shorter focal length and a wider angle of view than a standard lens. Wide-angle lenses are further divided into ordinary wide-angle lenses and ultra-wide-angle lenses. Ordinary wide-angle lenses typically have a focal length of 38-24 mm and an angle of view of 60-84 degrees; ultra-wide-angle lenses have a focal length of 20-13 mm and an angle of view of 94-118 degrees. In one example, the wide-angle lens has an observation angle of 110 degrees × 80 degrees.

[0098] In one embodiment, an infrared image is acquired via an infrared acquisition module. This infrared acquisition module includes an infrared camera, which includes a wide-angle lens. Specifically, the wide-angle lens may be a fisheye lens. More specifically, the infrared acquisition module acquires the analog signal from the infrared camera and converts it into a digital signal via an AD chip, which is then input to an ARM chip.

[0099] In step S102 of some embodiments, a low-light image is acquired by a low-light acquisition module. This low-light acquisition module includes a low-light camera, which includes a wide-angle lens. Specifically, the wide-angle lens may also be a fisheye lens. More specifically, the acquisition module acquires the analog signal from the low-light camera and converts it into a digital signal via an AD chip, which is then input to an ARM chip.

[0100] Before image fusion in step S103, the display method for wide-angle low-light infrared image fusion also includes image enhancement processing of the infrared image and the low-light image respectively.

[0101] Traditional histogram equalization algorithms can be used for image enhancement, but these algorithms often fail to effectively enhance dark areas and can over-brighten bright areas, resulting in less than ideal image enhancement. This application employs a gamma algorithm for image enhancement, achieving better results.

[0102] The gamma algorithm works as follows: 1. Normalization: Converts pixel values ​​to real numbers between 0 and 1. Assuming the pixel value is i, then f = (i + 0.5) / 256. 2. Pre-compensation: Pre-compensates the normalized f value, F = (1 / gamma) * f. 3. Inverse normalization: Inversely transforms the pre-compensated real value to an integer value between 0 and 255, P = F * 256 - 0.5. When the gamma coefficient is 1, there is a linear relationship; it does not brighten dark areas or darken bright areas. When the gamma coefficient is 0 or 6, the brightening is more noticeable in dark areas, gradually decreasing from dark to bright areas. When the gamma coefficient is 0.1, the enhancement in dark areas is too pronounced compared to a gamma coefficient of 0.6. When the gamma coefficients are 2.0 and 5.0, the brightness gradually decreases from dark to bright areas, resulting in an overall darkening effect. If the environment is too bright, a value greater than 1 can be used for adjustment.

[0103] As mentioned above, if programming directly according to the formula, assuming the image resolution is 800*600, performing gamma correction on it would require performing 480,000 floating-point multiplications, divisions, and exponentiations, which is too inefficient and cannot achieve real-time results at all.

[0104] To address the above situation, this application proposes a fast calculation algorithm. Since any pixel value in an image can only be one of 256 integers from 0 to 255, and the gamma value is chosen from 0 to 10, the specific value is adjusted according to the enhancement intensity requirements. Assuming the gamma value is known, and the pixel value is any integer between 0 and 255, after the operations of "normalization calculation, pre-compensation calculation, and denormalization calculation," the corresponding result is unique and also falls within the range of 0 to 255. For example, if the gamma value is known to be 2.2 and the original pixel value is 200, the value after gamma correction can be calculated to be 228. Based on the above principle, it is only necessary to perform normalization calculation, pre-compensation calculation, and denormalization calculation for each integer between 0 and 255, and store the corresponding value in a pre-established gamma correction lookup table, ultimately obtaining the pixel correction lookup table.

[0105] In one embodiment, creating a pixel correction lookup table specifically includes the following steps:

[0106] Obtain the original pixel values; where any two original pixel values ​​are distinct.

[0107] For each original pixel value, a correction calculation is performed to obtain the corrected pixel value; the correction calculation includes normalization calculation, pre-compensation calculation, and inverse normalization calculation;

[0108] A pixel correction lookup table is obtained based on the matching information between the original pixel values ​​and the corrected pixel values.

[0109] Additionally, a pixel correction lookup table is used to correct the image, resulting in an enhanced image. Of course, looking up a table is a time-consuming process. In one example, the hardware acceleration engine IVE function provided by the HIS chip, along with the map library functions, can be used to quickly calculate the lookup table, completing the lookup and replacement of an image to obtain the enhanced image. This eliminates the need for additional FPGA hardware, reducing development difficulty and cost while still meeting real-time requirements.

[0110] See Figure 2 In one embodiment, prior to step S103, the wide-angle low-light infrared image fusion display method of this embodiment further includes, but is not limited to, the following steps:

[0111] Step S201: Obtain the pixel values ​​of the infrared image to get the original infrared pixel values;

[0112] Step S202: The pixel value is searched in a preset pixel correction lookup table according to the original infrared pixel value to obtain the corrected infrared pixel value; wherein, the pixel correction lookup table includes matching information between the original pixel value and the corrected pixel value.

[0113] Step S203: Replace the original infrared pixel values ​​in the infrared image according to the corrected infrared pixel values ​​to obtain the updated infrared image;

[0114] Step S204: Obtain the pixel values ​​of the low-light image to get the original low-light pixel values;

[0115] Step S205: Based on the original low-light pixel values, perform a pixel value lookup in the pixel correction lookup table to obtain the corrected low-light pixel values;

[0116] Step S206: Replace the original low-light pixel values ​​in the low-light image according to the corrected low-light pixel values ​​to obtain the updated low-light image.

[0117] Steps S201 to S206, as illustrated in the embodiments of this application, enhance the image by replacing pixel values, improve the discernibility of dark area images, and help improve the clarity of subsequently displayed images.

[0118] Specifically, considering that performing correction calculations for every single original infrared pixel value in an infrared image would consume a significant amount of computation time, this embodiment pre-configures a pixel correction lookup table. Based on matching information, each original pixel value has a corresponding corrected pixel value in the pixel correction lookup table. Thus, the original infrared pixel value is used as the original pixel value, and the corresponding corrected pixel value is used as the corrected infrared pixel value. This pixel lookup method using the pixel correction lookup table improves processing efficiency compared to the previous method that required correction calculations for every single original pixel value.

[0119] After image enhancement, distortion correction and calibration can be performed on the low-light image and infrared image separately. For example, using OpenCV for image distortion correction does not yield ideal results. For instance, it suffers from slow processing speed, failing to meet real-time requirements. Therefore, this application proposes an alternative solution: first, fuse the low-light image and infrared image, then extract a short video clip for distortion correction. The aim is to further improve distortion processing efficiency while ensuring distortion correction effectiveness.

[0120] In step S103 of some embodiments, the infrared image and the low-light image are subjected to image fusion processing to obtain the target fused image.

[0121] See Figure 3 In one embodiment, step S103 specifically includes, but is not limited to, the following steps:

[0122] Step S301: Perform two-scale decomposition on each infrared image to obtain the infrared basic feature image and the infrared detail feature image.

[0123] Step S302: Perform two-scale decomposition on each low-light image to obtain a low-light basic feature image and a low-light detail feature image;

[0124] Step S303: The infrared basic feature image and the low-light basic feature image are averaged and summed to obtain the fused basic feature image;

[0125] Step S304: Perform weighted summation on the infrared detail feature image and the low-light detail feature image to obtain the fused detail feature image;

[0126] Step S305: Perform image fusion processing on the fused basic feature image and the fused detail feature image to obtain the target fused image.

[0127] Steps S301 to S305, as illustrated in this embodiment, improve the image fusion effect by employing a multi-scale fusion method. Specifically, after image enhancement of each acquired low-light image and infrared image, a two-scale decomposition is performed, separating it into basic features (low-frequency components of the image) and detail features (high-frequency components of the image). A summation-averaging method is used to fuse the basic image features, and a weighted summation method is used to fuse the detail features. Finally, the fused images from each layer are reconstructed to obtain the target fused image.

[0128] In one example, a low-light image is represented as I. E Infrared images are represented as I R The infrared image and the low-light image are respectively input into a low-pass filter (LP). Obtain infrared basic feature image and low-light basic feature images Subtracting the infrared image from the infrared basic feature image yields the infrared detail feature image. Subtracting the low-light image from the low-light basic feature image yields the low-light detail feature image. Image fusion is performed using the average addition rule to obtain the fused basic feature image. Image fusion is performed using a weighted summation rule to obtain a fused detail feature image. and These are all weighting coefficients, which can be set according to requirements. The final target is the fused image I. f By fusing basic feature images With fused detail feature images The result is obtained by adding and reconstructing.

[0129] It should be noted that the embodiments of this application use the hardware acceleration engine IVE function provided by the Hisi chip for image fusion. The Hisi chip provides an interface for calculating the add function of multi-degree fusion algorithm, which has a fast calculation speed and meets the real-time requirements. The weight ratio is sent to the host computer in real time, and the image fusion effect can be dynamically adjusted in real time.

[0130] After obtaining the target fused image, it is initially a grayscale image. A color conversion can be performed on the target fused image to obtain an updated target fused image. Generally, the human eye can distinguish fewer than 20 grayscale levels, but is very sensitive to color, able to distinguish thousands of hues and saturations. The infrared camera used in this embodiment acquires 8-bit grayscale images (0-255). To detect minute grayscale changes in the image, each grayscale level is matched to a point in the color space (RGB), thus mapping a monochrome image into a color (pseudo-color) image.

[0131] One approach proposed in this application is to independently map the three color channels R, G, and B, establish a color curve lookup table for the three channels based on the three mapping curves, and use the IVE acceleration engine and map function provided by Hisi to look up the table to obtain the synthesized RGB color values, thus obtaining a color image.

[0132] See Figure 4 After step S103, the wide-angle low-light infrared image fusion display method of this embodiment also includes, but is not limited to, the following steps:

[0133] Obtain the pixel values ​​of the target fused image to get the original grayscale values;

[0134] The corrected red value is obtained by searching for the red value in the preset red curve lookup table based on the original gray value; the red curve lookup table includes matching information between the original gray value and the corrected red value.

[0135] The corrected green value is obtained by searching for the green value in the preset green curve lookup table based on the original gray value; the green curve lookup table includes matching information between the original gray value and the corrected green value.

[0136] The corrected blue value is obtained by searching for the blue value in a preset blue curve lookup table based on the original gray value; the blue curve lookup table includes matching information between the original gray value and the corrected blue value.

[0137] The target color value is obtained by combining pixels based on the corrected red value, corrected green value, and corrected blue value.

[0138] The original grayscale values ​​in the target fusion image are replaced based on the target color values ​​to obtain the updated target fusion image.

[0139] Specifically, by color mapping the original grayscale values ​​to their three color channels, corrected red, corrected green, and corrected blue values ​​are obtained. Pixel synthesis is then performed, followed by replacement based on the target color values, ultimately resulting in an updated target fused image. The updated target fused image is a color image.

[0140] In step S105 of some embodiments, the fused video is cropped to obtain candidate display video images. The fused video includes multiple target fused images, each with a resolution of 800*600. Directly correcting distortion in the target fused images would require significant computation time. Therefore, in this embodiment, cropping the fused video is equivalent to cropping the target fused images to obtain candidate display video images. The resolution of the candidate display video images is smaller than that of the target fused images. For example, the resolution of the candidate display video images could be 290*226.

[0141] In one embodiment, step S105 specifically includes, but is not limited to, the following steps:

[0142] Acquire attitude information, which includes the current angle parameters;

[0143] Determine the position offset based on the current angle parameters;

[0144] Using the position offset as the cropping center, the fused video is cropped at a preset cropping angle to obtain candidate display video images.

[0145] Specifically, the position offset is determined using the current angle parameters of the pose information, and the cropping center of the fused video is determined based on the position offset. Using the cropping center as a reference and the cropping angle as the cropping size, the fused video is cropped to obtain candidate display video images.

[0146] It should be noted that attitude information is used to represent the attitude of the object being held by the camera. For example, if the camera (including infrared and low-light cameras) is mounted on the head of the object being held, the attitude information specifically refers to head attitude information, which can be acquired through an attitude sensor. For instance, the attitude sensor acquires head attitude information in real time, including the current angle parameters. Each time the head moves, a different current angle parameter is obtained. The position offset is determined based on the current angle parameter, and the cropping center is updated according to the position offset to match the head movement, thus updating the candidate display video images in real time. This can also be done if the camera is mounted on the hand or other parts of the body.

[0147] In one embodiment, step S106 specifically includes, but is not limited to, the following steps:

[0148] The pixel value is searched in the preset distortion correction lookup table based on the position offset to obtain the target distortion correction pixel value;

[0149] The pixel values ​​of the candidate display video image are replaced based on the target distortion correction pixel values ​​to obtain the target display video image.

[0150] See Figure 5 In one embodiment, creating a distortion correction lookup table specifically includes, but is not limited to, the following steps:

[0151] Step S401: Obtain the resolution of the merged video to get the original resolution;

[0152] Step S402: Obtain the original field of view of the wide-angle lens;

[0153] Step S403: Calculate the target resolution based on the original resolution, the cropping angle, and the original angle; wherein the target resolution is smaller than the original resolution.

[0154] Step S404: Calculate the offset based on the original resolution, the captured viewpoint, and the current angle parameters to obtain the target offset;

[0155] Step S405: Using the target offset as the division center, the fused video is divided into images according to the target resolution to obtain the divided images; wherein, the resolution of the divided images is the target resolution.

[0156] Step S406: Perform distortion correction on the pixel values ​​of the segmented image to obtain the target distortion-corrected pixel values ​​of the segmented image;

[0157] Step S407: Obtain a distortion correction lookup table based on the matching information between the target offset and the target distortion correction pixel value.

[0158] In the embodiments of this application, steps S401 to S407 involve pre-calculating a distortion correction lookup table to quickly complete distortion correction.

[0159] It should be noted that fisheye distortion center calibration is a prerequisite for image distortion correction. The distortion of a fisheye image increases with the field of view and is globally symmetrical about the distortion center. This application utilizes the global symmetry of fisheye image distortion for center calibration. A fisheye lens using equidistant projection satisfies the following imaging relationship:

[0160] y'=fω (1)

[0161] In the formula: y' is the imaging height, f is the focal length of the fisheye lens, and ω is the object-side half field of view.

[0162] According to Gaussian optics theory, the radial and axial magnifications of equidistant projection imaging are respectively:

[0163]

[0164]

[0165] In the formula: r is the object distance. From the above two formulas, it can be seen that the circle in object space will be compressed on the fisheye imaging plane, forming a compressed image. To quantitatively analyze the shape of the tiny circle in object space as imaged by a fisheye camera, a method is specifically designed as follows: Figure 6 The target plate shown is used as the objective. The target plate has a diameter of d = 10 cm and a center distance of d. c It consists of a group of circles, each 15cm long.

[0166] Establish a system with the center of the target plate as the origin. Figure 6 The coordinate system shown is used, and the circles are numbered. The circle in the i-th row and j-th column is denoted as C. ij Let d = 1, 2, ..., 6, j = 1, 2, ..., 8.b =d c -d, then circle C ij The coordinates of the center of the circle can be expressed as:

[0167]

[0168]

[0169] Represent circle C using parametric equations. ij for:

[0170]

[0171] With the target plate placed perpendicular to the optical axis of the fisheye camera, and the two separated by a distance R, then by formulas (1) and (6), we can deduce that the circle C ij The image shape in an equidistant fisheye camera can be represented as:

[0172]

[0173] Based on equations (1) to (7), the target plate and its fisheye image were plotted. After repeated attempts, the ellipse equation was found to fit the circular image of the target plate very well.

[0174] As the incident angle increases, the elliptical distortion compression becomes more severe, and the ellipse area becomes smaller. Moreover, this distortion is globally symmetrical about the center of the fisheye image. The method of "ellipse fitting of a small circle in object space for fisheye imaging + Gaussian surface fitting of the ellipse area to determine the vertex" can accurately calibrate the fisheye distortion center. Based on the above method, the distortion center of the fisheye camera is calibrated. First, the actual target plate fisheye image is obtained. Then, ellipse fitting and Gaussian surface fitting are used to process the target plate fisheye image. Finally, the coordinates of the Gaussian surface vertex are obtained, and this vertex is the distortion center of the fisheye image.

[0175] Fisheye Imaging Law Calibration: The large distortion imaging characteristics of fisheye lenses render traditional linear imaging models inapplicable. Furthermore, due to factors such as optical design and manufacturing errors, the actual imaging of fisheye lenses does not strictly conform to the object-image relationship in formula (1). To obtain a more accurate fisheye imaging relationship, the fisheye imaging law is calibrated based on actual test data from the fisheye imaging system. This section uses a Peltier plate as the experimental target. By changing the moving Peltier, the imaging position on the fisheye focal plane at different incident angles of the Peltier plate is recorded, thus completing the imaging relationship test of the fisheye camera.

[0176] During the calibration process, firstly, the fisheye camera is placed facing a large, vertical plane wall of the laboratory, ensuring the principal optical axis of the fisheye lens is perpendicular to the wall. Then, based on the calibrated fisheye distortion center o', the object point O corresponding to the image distortion center is marked on the plane wall, and a Peltier movement path is planned through point O. Thirdly, world coordinate systems, camera coordinate systems, and image coordinate systems are established, with point O, camera center C, and fisheye image distortion center o' as origins, respectively. Fourthly, the Peltier is powered on, and by moving the Peltier, the target image is captured at different positions using the fisheye camera.

[0177] Assuming the vertical distance between the wall and the camera is R, then when the target moves to P(X) t ,Y t Z t When point ), its distance from the origin O is Since the distance from the fisheye lens to the focal plane is much smaller than the distance R between the camera and the wall, the angle of incidence of point P relative to the fisheye camera can be approximated as:

[0178]

[0179] At this time, the image point p'(x) corresponding to object point P t ,y t ,z t The distance between it and its origin o' is

[0180]

[0181] Therefore, by experimentally testing the imaging position of the target at different angles, data characterizing the relationship between the incident angle and the target imaging position can be obtained. Because the imaging area of ​​the fisheye camera used in the experiment is larger than the photosensitive area of ​​the detector, the image is fragmented. However, the camera's field of view can reach 180° in the diagonal direction of the FPA. Therefore, during the test, by controlling the relative position of the target and the camera, the target is moved along the diagonal direction, and multiple sets of incident angle and target imaging position data are obtained.

[0182] Through repeated trials, the imaging characteristics of this fisheye camera can be well described by a fourth-order fitting polynomial, with a fitting accuracy better than 0.2%. The imaging relationship satisfies:

[0183] y'=k0+k1ω+k2ω 2 +k3ω 3 +k4ω 4 (9)

[0184] In the formula: ω is the object-side incident angle, and k0, k1, k2, k4 are fisheye imaging parameters.

[0185] The principle of distortion correction for fisheye lenses: For a fisheye lens, its object space can be considered as a hemisphere. Figure 7 A simplified fisheye imaging model is given. Substituting ω=π / 2 into equation (9) yields the result. Figure 7 The radius R of the hemisphere is assumed to be along the OZ direction of the fisheye camera's principal optical axis, and the imaging plane is the OXY plane. In the simplified model, light rays from the object side directed towards the origin O intersect the hemisphere. It can be considered that the 3D spatial hemisphere's field of view information is first mapped onto the hemisphere and then projected onto the 2D imaging plane. Image correction is the inverse process of imaging, that is: the 2D fisheye image is projected back onto the hemisphere, and then projected onto the sectional plane ABCD using perspective projection.

[0186] Assume that the incident angle and azimuth angle of a 3D spatial point P relative to a fisheye camera are ω and θ, respectively, corresponding to an image point p, and OP intersects the sphere at point P'. Assume point P is the center of a target of interest in space. A virtual camera is created here, with a maximum half-field of view of ω' in both the horizontal and vertical directions. This virtual camera is placed at the origin and directly faces point P, capturing a distortion-free image of the target. Thus, as long as the center of the target image to be corrected is determined, a virtual camera can always be found to acquire an image of the target.

[0187] The following is a mathematical derivation of the principle of fisheye image correction based on virtual cameras:

[0188] According to Euclidean geometry, the coordinates of point P' on the hemisphere are P'(Rsinωcosθ, Rsinωsinθ, Rcosθ), where ω and θ are the angle of incidence and the azimuth, respectively. Let X'0 = Rsinωcosθ, Y'0 = Rsinωsinθ, and Z'0 = Rcosθ, then the tangent plane ABCD satisfies the following equation:

[0189] X'0X+Y'0Y+Z'0Z-R 2 =0 (10)

[0190] Assuming the optical axis of the virtual camera is along the OP direction, the image "captured" by the camera is actually the projection of the sphere containing curve A'B' onto its tangent plane. The size of the distortion-free image "captured" by the virtual camera can be obtained by the following formula:

[0191] l AB =l CD =2Rtanω' (11)

[0192] The corrected image is located in coordinate system PX c Y c On a plane, PX c The direction of the tangent to the hemispherical surface is perpendicular to OP, with upward being positive; PY c Let P(P') be the spherical tangent parallel to OXY, with its positive direction starting from PX. c×OP determines the size. Assuming the corrected image size is H×V, then the size of a single pixel in the virtual camera image is:

[0193]

[0194] For any point S to be corrected, if it lies in coordinate system P'X c Y c If the coordinates in the coordinate system are S(i,j) (i∈[-H / 2,H / 2],j∈[-V / 2,V / 2]), then its coordinates in the physical coordinate system are S(iH). pixel ,jV pixel Combining coordinate transformation, the coordinates of point S in the OXY coordinate system are:

[0195]

[0196] The angle of incidence and azimuth of point S relative to the center O are expressed as:

[0197]

[0198] Therefore, the coordinates of the spherical coordinate point P' corresponding to point S are P'(Rsinωcosθ,Rsinωsinθ,Rcosθ), and the coordinates of the corresponding fisheye image point s are s(r S cosθ S ,r S sinθ S ), where r s The distance from image point s to the distortion center O can be obtained using the imaging formula (9).

[0199] According to formulas (10) to (14), the correspondence between the corrected plane points, spherical points and image plane points can be obtained. In this way, by setting the camera half field of view ω' and the corrected image size H and V, and using the above formulas to match all the fisheye image points to be corrected with the corrected image points, and then combining data interpolation, a distortion-free image centered on the target can be obtained.

[0200] In a specific application example, suppose the target fused image has a resolution of 800 pixels * 600 pixels, the camera's field of view is 110 degrees * 80 degrees, and the cropped image has a field of view of 40 degrees * 30 degrees, cropped in a proportional manner. According to the calculation formulas: H = (40 / 110) * 800 pixels, V = (30 / 80) * 600 pixels, the calculated resolution is 290 pixels * 226 pixels. Figure 8 As shown, a small-viewpoint image (candidate display video image) is cropped from the target fused image. This small-viewpoint image is then offset within the target fused image by a certain amount to obtain a new small-viewpoint image, as shown below. Figure 8The small images are numbered 1, 2, and 3. Assuming the helmet moves horizontally by one degree, the small-view image shifts horizontally by 40 pixels. Similarly, the 800-pixel image is divided into 20 equal parts horizontally to obtain 20 small-view images. Likewise, in the vertical direction, assuming the helmet moves one degree, the small-view image shifts vertically by 30 pixels, using this as the offset to obtain 30 images. The entire 800*600 resolution image is divided horizontally and vertically to obtain 20*30, or 600 images. The correction center of each image is obtained based on the division offset and is a known quantity. According to formulas (10) to (14), the new pixel value corresponding to each pixel value in each image after correction can be calculated, thus obtaining the corrected image. The new pixel value of each image is stored in a distortion correction lookup table of size 290*226 resolution. When the helmet is moved, the new pixel value of the corrected image can be obtained by looking up the corresponding position in the table based on the position offset. The new image can be enlarged to a resolution of 800*600 pixels using the resize function algorithm of the Hisi processor and output to the helmet display for human observation.

[0201] It is understood that the embodiments of this application can perform real-time image enhancement by establishing a pixel correction lookup table (gamma curve mapping table). Real-time color conversion of the image can be performed by establishing a color mapping table (including a red curve lookup table, a green curve lookup table, and a blue curve lookup table). Real-time distortion correction of the image can be performed by establishing a distortion correction lookup table (also known as a small-viewpoint correction lookup table).

[0202] This application embodiment also provides a display device for wide-angle low-light infrared image fusion, which can realize the above-mentioned wide-angle low-light infrared image fusion display method. The display device includes a processor, a fisheye lens infrared camera, a fisheye lens low-light camera, an attitude sensor, and a display. The fisheye lens infrared camera is used to acquire infrared images; the fisheye lens low-light camera is used to acquire low-light images; the attitude sensor is used to acquire attitude information; the attitude information includes current angle parameters; the processor is used to perform image fusion processing on the infrared image and the low-light image to obtain a target fused image; the target fused image is merged to obtain a fused video; the position offset is determined according to the current angle parameters; the fused video is cropped at a preset cropping angle with the position offset as the cropping center to obtain a candidate display video image; the candidate display video image is subjected to distortion correction processing to obtain a target display video image; the display is used to display the target display video image.

[0203] The processor may refer to a Hisilicon processor, also known as a Hisilicon chip. In one example, the Hisilicon 3559av100 chip is used as the processor. Compared to FPGA processors, it has advantages such as low power consumption and rich hardware interfaces. It can connect to Ethernet to transmit video to a PC for real-time display. It can also connect to a helmet-mounted display via a digital interface to observe low-light, infrared, fused, and small field-of-view video images in real time. The overall structure is small, lightweight, easy to carry, and convenient to use. The chip procurement cost is low, it runs on a Linux system, has rich library functions and easily portable mature algorithm libraries, can implement complex image processing functions, has low development cost, fast running speed, and good real-time performance.

[0204] The embodiments of this application use a fisheye camera, which has a wide viewing angle and has undergone distortion correction, resulting in realistic images.

[0205] In one example, the display is specifically a multi-channel display. Taking a four-channel display as an example, users can switch between infrared, low-light, grayscale blended, color blended, and small field-of-view images in real time via buttons on the device. Specifically, the H.265 compression algorithm is used, accelerated by a coprocessor hardware, to compress and encode each image channel, which is then sent to a PC via the network port for playback. The PC plays back the reproduced video, which consists of four channels: the original low-light camera video, the original infrared video, the blended video, and the small field-of-view video.

[0206] It should be noted that the specific implementation of the wide-angle low-light infrared image fusion display device is basically the same as the specific implementation of the wide-angle low-light infrared image fusion display method described above, and will not be repeated here.

[0207] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned wide-angle low-light infrared image fusion display method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0208] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0209] The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0210] The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 to execute the wide-angle low-light infrared image fusion display method of the embodiments of this application.

[0211] The input / output interface 503 is used to implement information input and output;

[0212] The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0213] Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);

[0214] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0215] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to realize the above-described wide-angle low-light infrared image fusion display method.

[0216] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0217] The wide-angle low-light infrared image fusion display method, display device, electronic device, and storage medium provided in this application improve the clarity of displayed video images through image fusion of infrared and low-light images. By acquiring infrared and low-light images through a wide-angle lens, more information about the object can be obtained due to the large viewing angle. Then, distortion correction processing restores the true shape of the object, further improving the clarity of the displayed video image. In addition, the video is cropped before distortion correction, reducing the resolution of the distortion correction processing and improving the processing efficiency.

[0218] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0219] It will be understood by those skilled in the art that Figure 1-3 and Figure 5 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0220] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0221] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0222] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0223] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0224] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0225] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0226] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0227] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0228] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A display method for wide-angle low-light infrared image fusion, characterized in that, The method includes: Infrared images are captured using the wide-angle lens of an infrared camera; Low-light images are captured using the wide-angle lens of a low-light camera; The infrared image and the low-light image are fused together to obtain the target fused image. The target fused images are merged to obtain a fused video; Acquire attitude information, which includes current angle parameters; wherein the attitude information is used to represent the attitude of the camera holding object of the infrared camera and the low-light camera; Determine the position offset based on the current angle parameter; Using the position offset as the cropping center, the fused video is cropped at a preset cropping angle to obtain candidate display video images; The pixel value is searched in a preset distortion correction lookup table based on the position offset to obtain the target distortion correction pixel value. The pixel values ​​of the candidate display video image are replaced according to the target distortion correction pixel values ​​to obtain the target display video image, and the target display video image is then displayed. Specifically, creating the distortion correction lookup table includes: Obtain the resolution of the fused video to get the original resolution; Obtain the original field of view of the wide-angle lens; The target resolution is obtained by calculating the resolution based on the original resolution, the cropping angle, and the original angle; wherein the target resolution is smaller than the original resolution. The offset is calculated based on the original resolution, the cropping viewpoint, and the current angle parameter to obtain the target offset. Using the target offset as the dividing center, the fused video is divided into images according to the target resolution to obtain divided images; wherein, the resolution of the divided images is the target resolution; Obtain the pixel values ​​of the segmented image to obtain the initial segmentation pixel values; Distortion correction is performed on the initial segmented pixel values ​​to obtain the target distortion-corrected pixel values ​​of the segmented image; The distortion correction lookup table is obtained based on the matching information between the target offset and the target distortion correction pixel value.

2. The method according to claim 1, characterized in that, The step of performing image fusion processing on the infrared image and the low-light image to obtain the target fused image includes: Each infrared image is decomposed into two scales to obtain an infrared basic feature image and an infrared detail feature image. Each of the aforementioned low-light images is decomposed into two scales to obtain a low-light basic feature image and a low-light detail feature image; The infrared basic feature image and the low-light basic feature image are summed and averaged to obtain a fused basic feature image; The infrared detail image and the low-light detail image are weighted and summed to obtain a fused detail image; The fused basic feature image and the fused detailed feature image are subjected to feature fusion processing to obtain the target fused image.

3. The method according to claim 1, characterized in that, The target fused image is a grayscale image. After performing image fusion processing on the infrared image and the low-light image to obtain the target fused image, the method further includes: Obtain the pixel values ​​of the target fused image to get the original grayscale values; The corrected red value is obtained by searching for the red value in a preset red curve lookup table based on the original gray value; wherein, the red curve lookup table includes matching information between the original gray value and the corrected red value; The corrected green value is obtained by searching for a green value in a preset green curve lookup table based on the original gray value; wherein, the green curve lookup table includes matching information between the original gray value and the corrected green value; The corrected blue value is obtained by searching for the blue value in a preset blue curve lookup table based on the original gray value; wherein, the blue curve lookup table includes matching information between the original gray value and the corrected blue value; Pixels are synthesized based on the corrected red value, the corrected green value, and the corrected blue value to obtain the target color value; The original grayscale value in the target fused image is replaced according to the target color value to obtain the updated target fused image.

4. The method according to claim 1, characterized in that, Before performing image fusion processing on the infrared image and the low-light image to obtain the target fused image, the method further includes: Obtain the pixel values ​​of the infrared image to get the original infrared pixel values; The corrected infrared pixel value is obtained by searching the pixel value in a preset pixel correction lookup table based on the original infrared pixel value; wherein, the pixel correction lookup table includes matching information between the original pixel value and the corrected pixel value; The original infrared pixel values ​​in the infrared image are replaced according to the corrected infrared pixel values ​​to obtain the updated infrared image; Obtain the pixel values ​​of the low-light image to get the original low-light pixel values; Based on the original low-light pixel value, the pixel value is looked up in the pixel correction lookup table to obtain the corrected low-light pixel value; The original low-light pixel values ​​in the low-light image are replaced according to the corrected low-light pixel values ​​to obtain the updated low-light image.

5. The method according to claim 4, characterized in that, Before performing a pixel value lookup in a preset pixel correction lookup table based on the original infrared pixel value to obtain the corrected infrared pixel value, the method further includes: Creating the pixel correction lookup table specifically includes: Obtain the original pixel values; wherein any two original pixel values ​​are different from each other; Each of the original pixel values ​​is corrected to obtain a corrected pixel value; the correction calculation includes normalization calculation, pre-compensation calculation, and inverse normalization calculation. The pixel correction lookup table is obtained based on the matching information between the original pixel value and the corrected pixel value.

6. A display device for wide-angle low-light infrared image fusion, characterized in that, The device includes: The system includes a processor, an infrared camera with a fisheye lens, a low-light camera with a fisheye lens, an attitude sensor, and a display. An infrared camera with a fisheye lens is used to acquire infrared images; A fisheye lens low-light camera used to capture low-light images; An attitude sensor is used to acquire attitude information, which includes current angle parameters. The attitude information is used to represent the attitude of the camera-holding object of the infrared camera and the low-light camera. The processor is configured to perform image fusion processing on the infrared image and the low-light image to obtain a target fused image; merge the target fused image to obtain a fused video; determine a position offset based on the current angle parameter; crop the fused video with the position offset as the cropping center and a preset cropping angle to obtain a candidate display video image; search for pixel values ​​in a preset distortion correction lookup table based on the position offset to obtain a target distortion correction pixel value; replace the pixel values ​​of the candidate display video image with the target distortion correction pixel value to obtain the target display video image; wherein, creating the distortion correction lookup table specifically includes: obtaining the resolution of the fused video to obtain the original resolution; obtaining the original angle of view of the wide-angle lens; The target resolution is calculated based on the original resolution, the cropping angle, and the original angle; wherein the target resolution is less than the original resolution. An offset is calculated based on the original resolution, the cropping angle, and the current angle parameter to obtain a target offset. Using the target offset as the dividing center, the fused video is divided into image segments according to the target resolution to obtain segmented images; wherein the resolution of the segmented images is the target resolution. Pixel values ​​of the segmented images are obtained to obtain initial segmented pixel values. Distortion correction is performed on the initial segmented pixel values ​​to obtain target distortion-corrected pixel values ​​for the segmented images. A distortion correction lookup table is obtained based on the matching information between the target offset and the target distortion-corrected pixel values. A display for displaying the target video image.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method according to any one of claims 1 to 5.

8. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images