An infrared image processing method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose an infrared image processing method, device and equipment and a storage medium. The technical scheme provided by the embodiments of the present application finds a temperature mutation point in an original image set, determines a to-be-compensated image and a thermal stable image from the original image set based on the temperature mutation point, determines reference statistical information according to temperature information of the thermal stable image, and compensates the temperature of each to-be-compensated image based on the reference statistical information and the temperature information of each to-be-compensated image, thereby improving the temperature measurement effect of the original image in the original image set, and effectively improving the image acquisition and image splicing efficiency without waiting for the thermal stable time of an infrared camera before shooting the original image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an infrared image processing method, apparatus, device, and storage medium. Background Technology

[0002] In the reconstruction process of 3D scenes, image fusion and stitching are often performed on multiple frames of infrared images captured continuously by an infrared camera to obtain a stitched image. For example, an infrared camera mounted on a drone can be used to capture multiple frames of infrared images by controlling the drone to move over the target scene. Then, the pose information between the multiple frames of infrared images is used to construct 3D scene information. An orthophoto image set is obtained from the 3D scene information and stitched together to form a 2D orthophoto image.

[0003] Because infrared cameras have a thermal stabilization time during operation, and the temperature measurement is unstable when the infrared camera captures images during this time, the temperature measurement of the output infrared image is inaccurate. Summary of the Invention

[0004] This application provides an infrared image processing method, apparatus, device, and storage medium to solve the technical problem of inaccurate temperature measurement of output infrared images in the prior art, and to ensure good image temperature measurement effect.

[0005] In a first aspect, embodiments of this application provide an infrared image processing method, comprising:

[0006] Based on temperature abrupt change points in the original image set, the image to be compensated and the thermally stable image are determined from the original image set;

[0007] Reference statistical information is determined based on the temperature information of the thermally stable image;

[0008] Temperature compensation is performed on the image to be compensated based on the reference statistical information and the temperature information of the image to be compensated.

[0009] In a second aspect, embodiments of this application provide an infrared image processing apparatus, including an image segmentation module, a reference determination module, and an image compensation module, wherein:

[0010] The image segmentation module is used to determine the image to be compensated and the thermally stable image from the original image set based on the temperature abrupt change points in the original image set;

[0011] The reference determination module is used to determine reference statistical information based on the temperature information of the thermally stable image;

[0012] The image compensation module is used to perform temperature compensation on the image to be compensated based on the reference statistical information and the temperature information of the image to be compensated.

[0013] In a third aspect, embodiments of this application provide an infrared image processing device, including: a memory and one or more processors;

[0014] The memory is used to store one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the infrared image processing method as described in the first aspect.

[0016] In a fourth aspect, embodiments of this application provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the infrared image processing method as described in the first aspect.

[0017] This application embodiment finds temperature abrupt change points in the original image set, determines the image to be compensated and the thermally stable image from the original image set based on the temperature abrupt change points, and determines reference statistical information based on the temperature information of the thermally stable image. Based on the reference statistical information and the temperature information of each image to be compensated, temperature compensation is performed on each image to be compensated, improving the temperature measurement effect of the original images in the original image set. Subsequently, the target stitched image can be obtained by image stitching based on the original image set and the temperature-compensated images to be compensated. This reduces the situation where the display effect of the stitched image is not ideal due to the unstable temperature measurement of the original images captured by the infrared camera during the thermal stabilization time. By performing temperature compensation on the images to be compensated with unstable temperature in the stitched and fused image content, the image stitching effect is improved. It is not necessary to wait for the thermal stabilization time of the infrared camera before capturing the original image, effectively improving the efficiency of image acquisition and image stitching. Attached Figure Description

[0018] Figure 1 This is a flowchart of an infrared image processing method provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram illustrating the display effect of a stitched image without temperature compensation, provided in an embodiment of this application.

[0020] Figure 3 This is a schematic diagram illustrating the display effect of a target stitched image provided in an embodiment of this application;

[0021] Figure 4 This is a flowchart of another infrared image processing method provided in the embodiments of this application;

[0022] Figure 5This is a flowchart of another infrared image processing method provided in the embodiments of this application;

[0023] Figure 6 This is a schematic diagram of a mask display provided in an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the structure of an infrared image processing device provided in an embodiment of this application;

[0025] Figure 8 This is a schematic diagram of the structure of an infrared image processing device provided in an embodiment of this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0027] Figure 1 A flowchart of an infrared image processing method provided in an embodiment of this application is given. The infrared image processing method provided in this application can be executed by an infrared image processing device, which can be implemented by hardware and / or software and integrated into an infrared image processing equipment.

[0028] The following description uses an infrared image processing device to perform an infrared image processing method as an example. (Reference) Figure 1 The infrared image processing method includes:

[0029] S101: Based on the temperature abrupt change points in the original image set, determine the image to be compensated and the thermally stable image from the original image set.

[0030] The original image set comprises multiple original images, which can be captured by infrared cameras mounted on mobile devices (such as drones, unmanned vehicles, etc.). The original images captured by the infrared cameras also record corresponding temperature information and the camera's pose information. The original image set can be obtained by continuously capturing multiple frames of the scene using an infrared camera mounted on a drone. This original image set can be used to reconstruct the 3D data of the entire scene, constructing a stitched image of orthorectified infrared thermal imaging data for a large-scale scene, and obtaining global scene image information. For example, the pose information between multiple frames of original images can be used to construct 3D scene information. An orthorectified projection image set can then be obtained from the 3D scene information and stitched together to form a 2D orthorectified stitched image.

[0031] In related technologies, infrared cameras have a thermal stabilization time during operation. During this time, the temperature measurement during image capture is unstable. If the camera is captured before the thermal stabilization time has ended, the original image captured during the thermal stabilization time and the original image captured after the thermal stabilization time will have different display effects due to the difference in temperature measurement stability. Consequently, in the final stitched image, obvious discontinuities will occur due to temperature jumps, resulting in poor display quality of the stitched image.

[0032] like Figure 2 The provided illustration shows a display effect of a stitched image without temperature compensation. It assumes that a drone equipped with an infrared camera acquires the original image of the target scene, where the drone's flight path is from... Figure 2 Starting from the bottom right corner of the image, the drone flies from right to left, acquiring raw images at set time intervals or flight distances. Upon reaching the far left of the target scene, it turns and collects raw images from left to right. Each time the drone turns, it records its trajectory... Figure 2 The image is shifted upwards until the original image sequence of each location in the target scene is acquired, resulting in the original image set. Generally, adjacent original images (including original images adjacent in multiple directions such as top, bottom, left, and right) have overlapping areas.

[0033] Assuming the infrared camera begins acquiring raw images without waiting for thermal stabilization time, the stitched image obtained from stitching together the raw image set will exhibit significant temperature anomalies in corresponding areas due to the unstable temperature acquisition of the raw images captured during the thermal stabilization period. For example... Figure 2As shown in the darker area at the bottom of the stitched image, the display effect of the stitched image is not ideal. In related technologies, to ensure the display effect of the stitched image, it is necessary to wait for the thermal stabilization time of the infrared camera to end before acquiring the original image, resulting in low work efficiency. This solution uses the overlapping area information of multiple frames of original images and utilizes the temperature consistency between adjacent original images to perform temperature compensation on the original images frame by frame. This eliminates the difference in display effect caused by the temperature measurement difference due to the thermal stabilization time of the infrared camera, thus ensuring the display effect of the final target stitched image.

[0034] For example, before performing temperature compensation on the original images captured during the thermal stabilization period, it is necessary to first determine the original images (i.e., the images to be compensated) from the set of captured original images. In one embodiment, since the temperature measurement stability of the infrared camera changes significantly before and after the thermal stabilization period, correspondingly, there will also be temperature abrupt changes between the original images captured before and after the thermal stabilization period. Based on this, according to the temperature abrupt change points in each original image in the original image set, and using the original images corresponding to the temperature abrupt change points as boundaries, the original images in the original image set are divided into images to be compensated and thermally stabilized images. Optionally, the temperature abrupt change points can be determined based on the extreme values ​​of the temperature gain of the average temperature between adjacent frames or the extreme values ​​of the ratio of the average temperatures between adjacent frames in the original image set.

[0035] In one embodiment, a set of images to be compensated is composed of individual images to be compensated, and a set of thermally stable images is composed of individual thermally stable images. For example, each original image in the original image set is marked by an image number, and the earlier the original image was captured, the smaller the corresponding image number. It can be understood that the thermal stabilization time is in the early part of the entire shooting process. The image number of the original image corresponding to the temperature change point can be used as the index value. The original images with image numbers at or below the index value are taken as images to be compensated, and the original images with image numbers above the index value are determined as thermally stable images.

[0036] Here, the image to be compensated can be understood as the original image that needs temperature compensation and was taken within the thermal stabilization time, while the thermally stabilized image can be understood as the original image that does not need temperature compensation and was taken after the thermal stabilization time. When stitching images together, the image to be compensated after temperature compensation and the original thermally stabilized image are used for stitching.

[0037] S102: Determine reference statistical information based on the temperature information of the thermally stable image.

[0038] For example, after determining the image to be compensated and the thermally stable image from the original image set, the reference statistical information used to perform temperature compensation on the image to be compensated can be determined based on the temperature information (temperature value) of the thermally stable image.

[0039] It is understandable that a thermally stabilized image is the original image acquired by an infrared camera after the thermal stabilization time. Its temperature measurement is in a stable state. The temperature distribution during the stable operation of the infrared camera can be reflected by the statistical information of the temperature information of the thermally stabilized image. Since the image to be compensated and the thermally stabilized image are captured in the same target scene, their temperature measurement distribution should be the same or similar. The statistical information of the temperature information of the thermally stabilized image can be used as reference statistical information to adjust the temperature value of the image to be compensated, so as to adjust the temperature distribution of the image to be compensated to the same or similar range, thereby achieving temperature compensation of the image to be compensated.

[0040] S103: Based on the reference statistical information and the temperature information of the image to be compensated, perform temperature compensation on the image to be compensated.

[0041] For example, after determining the reference statistics, temperature compensation can be performed on the image to be compensated using the reference statistics and the temperature information (temperature values) of the image to be compensated. For instance, based on the temperature distribution reflected in the reference statistics (e.g., mean, variance, standard deviation, etc.), the temperature value of each pixel in each image to be compensated is adjusted so that the distribution of the adjusted temperature values ​​is closer to the temperature distribution reflected in the reference statistics. That is, by using the temperature distribution of the original image before and after thermal stabilization, temperature compensation of the image to be compensated is achieved through temperature image distribution migration.

[0042] In one possible embodiment, after performing temperature compensation on the image to be compensated based on the reference statistical information and the temperature information of the image to be compensated, the method further includes: performing image stitching based on the thermally stable image and the temperature-compensated image to obtain a target stitched image.

[0043] For example, after temperature compensation is completed for each image to be compensated, image stitching is performed based on the original image set (thermally stable images in the original image set) and the temperature-compensated images to be compensated to obtain the target stitched image. The image stitching of the thermally stable images and the temperature-compensated images to be compensated can be performed using a feathering fusion algorithm.

[0044] For example, after obtaining the original image set, each original image in the original image set is first aligned to determine the image position of each original image in the stitched image. Then, the image to be compensated and the thermally stable image are determined from the original image set, and reference statistical information is determined based on the temperature information of the thermally stable image. Based on the reference statistical information and the temperature information of the image to be compensated, temperature compensation is performed on each image to be compensated. Then, based on the image positions of the temperature-compensated image to be compensated and the thermally stable image on the stitched image, the image to be compensated and the thermally stable image are stitched together using a fusion feathering algorithm to obtain the target stitched image.

[0045] like Figure 3 The provided illustration shows the display effect of a target stitched image. If image stitching is performed directly based on the original image set, after aligning and blending the original images, the result is... Figure 2 The original stitched image shown shows that, within the display area corresponding to the original image captured during the thermal stabilization period, the image is noticeably darker, and the stitched image exhibits obvious tonal discontinuity. In contrast, the image stitching method provided in this solution, after aligning, temperature compensating, and fusing the original image set, yields the image shown... Figure 3 As shown in the target stitched image, it can be seen that the display area corresponding to the original image captured during the thermal stabilization time is basically consistent with the display area corresponding to the original image captured after the thermal stabilization time. There is no dark local display effect or temperature discontinuity as in the original stitched image. Furthermore, it does not require waiting for the infrared camera to reach thermal stabilization time before capturing the original image, resulting in higher efficiency in both original image acquisition and image stitching.

[0046] The above describes a process where temperature abrupt changes are identified in the original image set. Based on these abrupt changes, images to be compensated and thermally stable images are determined from the original image set. Reference statistical information is then determined based on the temperature information of the thermally stable images. Based on the reference statistical information and the temperature information of each image to be compensated, temperature compensation is performed on each image to be compensated, improving the temperature measurement effect of the original images in the original image set. Subsequently, image stitching can be performed based on the original image set and the temperature-compensated images to be compensated to obtain the target stitched image. This reduces the situation where the display effect of the stitched image is not ideal due to the unstable temperature measurement of the original images captured by the infrared camera during the thermal stabilization time. By performing temperature compensation on the images to be compensated that have unstable temperature in the stitched and fused image content, the image stitching effect is improved. It is not necessary to wait for the thermal stabilization time of the infrared camera before capturing the original image, effectively improving the efficiency of image acquisition and image stitching.

[0047] Based on the above embodiments, Figure 4A flowchart of another infrared image processing method provided in an embodiment of this application is given, which is a concretization of the above-described infrared image processing method. (Reference) Figure 4 The infrared image processing method includes:

[0048] S201: Determine a set of temperature mean ratios based on the temperature mean of each original image in the original image set, wherein the temperature mean ratios in the set are used to indicate the ratio of the temperature mean of adjacent original images.

[0049] This embodiment describes the determination of temperature abrupt change points based on the ratio of average temperatures between adjacent frames in the original image set. For example, the average temperature of each original image (the average temperature of each pixel) is calculated, and a set of average temperature ratios is created based on the average temperature of each original image. The set of average temperature ratios records the ratio of the average temperature of each original image to that of its adjacent original images, where the average temperature ratio indicates the ratio of the average temperature of adjacent original images.

[0050] Understandably, the ratio of temperature mean values ​​reflects the degree of change in the average temperature between adjacent original images. When the ratio of temperature mean values ​​is the ratio of the average temperature of the later original image to that of the earlier original image, the larger the ratio, the greater the degree of change in the average temperature between adjacent original images. Conversely, when the ratio of temperature mean values ​​is the ratio of the average temperature of the earlier original image to that of the later original image, the larger the ratio, the smaller the degree of change in the average temperature between adjacent original images.

[0051] In one embodiment, the ratio of the average temperature values ​​of adjacent original images can be calculated in reverse order. That is, the average temperature ratio is the ratio of the average temperature value of the original image with the later image number to the average temperature value of the original image with the earlier image number. In this case, the greater the degree of change in the average temperature value between adjacent original images, the larger the corresponding average temperature ratio.

[0052] S202: Determine the image index value from the set of temperature mean ratios, and determine the image to be compensated and the thermally stable image from the original image set based on the image index value.

[0053] For example, after obtaining the set of temperature mean ratios, image index values ​​are determined from the set of temperature mean ratios, and the images to be compensated and thermally stable images are determined from the original image set based on the image index values. The image index values ​​reflect temperature abrupt change points in the original image set, and are represented by the image sequence numbers of the original images.

[0054] The image to be compensated can be determined from the original image set based on its image index value, according to the calculation method of the temperature mean ratio. For example, when the temperature mean ratio is the ratio of the average temperature of the later original image to the average temperature of the earlier original image, the original images with image indices at or below the image index value are identified as images to be compensated, while the original images with image indices above the image index value are identified as thermally stable images. Conversely, when the temperature mean ratio is the ratio of the average temperature of the earlier original image to the average temperature of the later original image, the original images with image indices at or above the image index value are identified as images to be compensated, while the original images with image indices below the image index value are identified as thermally stable images.

[0055] In one embodiment, the temperature mean ratio is the ratio of the temperature mean of the original image with the later image number to the temperature mean of the original image with the earlier image number. Correspondingly, the image index is the maximum value among all temperature mean ratios in the set of temperature mean ratios. In this case, the image number of the image to be compensated is less than or equal to the image index value, and the image number of the thermally stable image is greater than the image index value.

[0056] For example, suppose the number of original images in the original image set is N, and the width and height of the final stitched image are W and H respectively. The set of position information (image position) of each original image in the final stitched image is P = {(px1, py1), (px2, py2), ..., (px... N ,py N Obtain the original image set S0 = {I1, I2, ..., I...}. N} Calculate the mean temperature m and standard deviation temperature s of all original images in the original image set to obtain the statistical information set S5 = {(m1,s1), (m2,s2), ..., (m... N ,s N ), calculate adjacent original images I in reverse order i and I i+1 ratio of temperature average Obtain the set of temperature mean ratios The maximum value in the set of temperature mean ratios is taken as the image index value M, resulting in the set of images to be compensated, S1 = {I1, I2, ..., I...} M}, correspondingly, the thermally stable image set S2={I M+1 I M+2 , ......, I N}

[0057] S203: Determine reference statistical information based on the temperature information of the thermally stable image.

[0058] In one embodiment, the original image corresponding to the median of the temperature mean of each thermally stable image can be used as a reference for temperature compensation. Based on this, when determining the reference statistical information based on the temperature information of the thermally stable images, this solution specifically includes: determining the median corresponding to the temperature mean of each thermally stable image, and determining the thermally stable image corresponding to the median as the reference image; obtaining the reference statistical information by using the temperature mean of the reference image as the reference mean and the temperature standard deviation of the reference image as the reference standard deviation.

[0059] For example, a temperature information set S5 = {(m1, s1), (m2, s2), ..., (m...} is formed based on the mean temperature m and the standard deviation temperature s of all original images in the original image set. N ,s N The median of these temperature means is determined, and the thermal stability image corresponding to this median is designated as the reference image R. Further, the temperature mean corresponding to the reference image R is obtained as the reference mean, and the temperature standard deviation of the reference image is used as the reference standard deviation to obtain reference statistical information. That is, the statistical information {m, s} of the reference image R is used as the reference statistical information {m}. r ,s r}

[0060] In one embodiment, reference statistical information can also be determined based on the mean value corresponding to the statistical information of each thermally stable image. Therefore, when determining reference statistical information based on the temperature information of the thermally stable images, this solution specifically involves: using the mean value corresponding to the temperature mean value of each thermally stable image as the reference mean value, and using the mean value of the temperature standard deviation of each thermally stable image as the reference standard deviation to obtain the reference statistical information.

[0061] For example, a temperature information set S5 = {(m1, s1), (m2, s2), ..., (m...} is formed based on the mean temperature m and the standard deviation temperature s of all original images in the original image set. N ,s N )}, calculate the mean m_mean of the temperature mean m of all original images, and the mean s_mean of the temperature standard deviation s of all original images. Then, use the mean of the temperature mean of each thermally stabilized image as the reference mean (m_mean), and the standard deviation of the temperature mean of each thermally stabilized image as the reference standard deviation to obtain reference statistical information {m r ,s r}

[0062] S204: Based on the reference statistical information and the temperature information of the image to be compensated, perform temperature compensation on the image to be compensated.

[0063] For example, after determining the reference statistics, the temperature information (temperature values ​​of each pixel) of the image to be compensated is adjusted according to the reference statistics, thereby performing temperature compensation on the image to be compensated. The direction of adjustment for the temperature values ​​of each pixel is such that the adjusted statistics of the image to be compensated are consistent with or approach the reference statistics.

[0064] In one embodiment, temperature compensation can be performed on the image to be compensated based on the following formula:

[0065]

[0066] Among them, I′ i,j,k I represents the temperature value after temperature compensation. i,j,k The temperature value before temperature compensation, m r s is the reference mean in the reference statistics. r m is the reference standard deviation in the reference statistics. i s represents the mean temperature value in the temperature information of the image to be compensated. i Let be the temperature standard deviation in the temperature information of the image to be compensated, i be the image index of the image to be compensated, and (j,k) be the pixel coordinates on the image to be compensated. The temperature value of the image to be compensated is adjusted using the above formula. Utilizing the temperature distribution of the original image before and after thermal stabilization, a temperature image distribution migration method is used to adjust the temperature distribution of the image to be compensated to the same or similar range, thus achieving temperature compensation for the image to be compensated.

[0067] In one possible embodiment, after temperature compensation is performed on the image to be compensated based on the reference statistical information and the temperature information of the image to be compensated, the target stitched image can be obtained by image stitching based on the thermally stable image and the temperature-compensated image to be compensated.

[0068] The above describes a process where temperature abrupt changes are identified in the original image set. Based on these abrupt changes, images to be compensated and thermally stable images are determined from the original image set. Reference statistical information is then determined based on the temperature information of the thermally stable images. Based on the reference statistical information and the temperature information of each image to be compensated, temperature compensation is performed on each image to be compensated, improving the temperature measurement effect of the original images in the original image set. Subsequently, image stitching can be performed based on the original image set and the temperature-compensated images to be compensated to obtain the target stitched image. This reduces the situation where the display effect of the stitched image is not ideal due to the unstable temperature measurement of the original images captured by the infrared camera during the thermal stabilization time. By performing temperature compensation on the images to be compensated that have unstable temperature in the stitched and fused image content, the image stitching effect is improved. It is not necessary to wait for the thermal stabilization time of the infrared camera before capturing the original image, effectively improving the efficiency of image acquisition and image stitching. Simultaneously, temperature abrupt change points are determined based on the extreme values ​​of the ratios of average temperatures between adjacent frames in the original image set. This accurately identifies the images to be compensated captured during the thermal stabilization period and the thermally stabilized images captured after the thermal stabilization period, precisely locating the range for temperature compensation, improving temperature compensation efficiency, and more accurately determining the range for obtaining reference statistical information to ensure the accuracy of image temperature compensation. Furthermore, reference statistical information is determined based on the statistical information of the thermally stabilized image corresponding to the median of the temperature mean of each thermally stabilized image, or the mean of the temperature mean and the mean of the temperature standard deviation of each thermally stabilized image. This precise determination serves as the basis for adjusting image temperature compensation, ensuring the effectiveness of temperature compensation.

[0069] Based on the above embodiments, Figure 5 A flowchart of another infrared image processing method provided in an embodiment of this application is given, which is a concretization of the above-described infrared image processing method. (Reference) Figure 5 The infrared image processing method includes:

[0070] S301: Generate a set of sample pairs based on the mask images corresponding to adjacent original images in the original image set, wherein the sample pairs in the set are used to indicate the temperature value of the adjacent original images within the mask image.

[0071] This embodiment describes the determination of temperature abrupt change points based on the temperature gain of the average temperature between adjacent frames in the original image set. The mask image is the image that falls on the mask after registration and alignment of two adjacent original images in the original image set. The mask image is the image on the overlapping area of ​​the two adjacent original images. The two adjacent original images are two original images whose frames or shooting areas overlap (or reach a set overlap requirement). They can be two original images with consecutive image numbers, or two original images whose shooting positions or image positions in the stitched image are close or within a set distance range. Figure 6As shown in the provided schematic diagram of a mask display, assuming Figure 6 The images on the left and in the middle are two consecutive frames of original images (with adjacent image numbers). The image on the right is a mask (white area in the figure) determined by aligning and registering the two original images. The image of the original image that falls within the mask is the mask image.

[0072] For example, the mask images corresponding to adjacent original images are determined, and a set of sample pairs is generated on the overlapping region (mask image) of the two adjacent original images based on the temperature information (temperature values) of the mask images. The sample pair set includes multiple sample pairs, which indicate the temperature values ​​of the adjacent original images within their corresponding mask images. Each sample pair includes the temperature values ​​of the mask images corresponding to the two adjacent original images (including two temperature values ​​on each of the two original images). Specifically, for all or part of the pixels (sampling points) in the mask images corresponding to adjacent original images, the number of pixels (sampling points) used to construct the sample pairs can be set as needed. Sample pairs are constructed based on the temperature values ​​corresponding to these pixels in the two original images, and these sample pairs constitute the set of sample pairs for the mask images corresponding to the two adjacent original images. In one embodiment, each original image corresponds to one sample pair with the next adjacent original image, and the last original image constructs its own sample pair as its adjacent original image. That is, the number of sample pairs in the sample pair set is the same as the number of original images.

[0073] S302: Determine a temperature gain set based on the set of sample pairs, wherein the temperature gain in the temperature gain set is used to indicate the temperature gain value between adjacent original images.

[0074] The temperature gain in the temperature gain set is used to indicate the temperature gain value between adjacent original images in the overlapping region. For example, after obtaining the sample pair set, the temperature gain value between each pair of adjacent original images is calculated based on each sample pair in the sample pair set, and the temperature gain set is obtained based on these temperature gain values.

[0075] The temperature gain value reflects the degree of change in the average temperature of adjacent original images in the overlapping area. When the temperature gain value is the ratio of the average temperature of the later original image to that of the earlier original image in the overlapping area, the larger the temperature gain value, the greater the degree of change in the average temperature between adjacent original images. Conversely, when the temperature gain value is the ratio of the average temperature of the earlier original image to that of the later original image in the overlapping area, the larger the temperature gain value, the smaller the degree of change in the average temperature between adjacent original images.

[0076] In one embodiment, the temperature gain value of adjacent original images in the overlapping region is determined based on the ratio of the average temperature of the two original images in the sample pair within the overlapping region. In this case, the greater the change in the average temperature between adjacent original images, the greater the corresponding temperature gain value.

[0077] S303: Determine the image index value from the temperature gain set, and determine the image to be compensated and the thermally stable image from the original image set based on the image index value.

[0078] The image index value reflects temperature abrupt changes in the original image set, and is represented by the image number of the original image. For example, the image index value is determined from the temperature gain set, and the image to be compensated and the thermally stable image are determined from the original image set based on the image index value.

[0079] The image to be compensated can be determined from the original image set based on its image index value, according to the calculation method of the temperature gain value. For example, when the temperature gain value is the ratio of the average temperature of the subsequent original image to the average temperature of the preceding original image, the original images with image indices at or below the image index value are identified as images to be compensated, while the original images with image indices above the image index value are identified as thermally stable images. Conversely, when the temperature gain value is the ratio of the average temperature of the preceding original image to the average temperature of the subsequent original image, the original images with image indices at or above the image index value are identified as images to be compensated, while the original images with image indices below the image index value are identified as thermally stable images.

[0080] In one embodiment, the temperature gain value between adjacent original images is determined based on the ratio of the average temperature of the two original images in the overlapping area of ​​the sample pair. That is, the temperature gain value is the ratio of the average temperature of the latter original image to that of the former original image. Correspondingly, the image index value is the maximum value among all temperature gain values ​​in the temperature gain set, and the image number of the image to be compensated is less than or equal to the image index value.

[0081] For example, suppose the number of original images in the original image set is N, and the width and height of the final stitched image are W and H respectively. The set of position information of each original image in the final stitched image is P = {(px1, py1), (px2, py2), ..., (px... N ,py N Obtain the original image set S0 = {I1, I2, ..., I...}. N}, based on two consecutive original images I i and I i+1 Generate a set of sample pairs S3 = {(I} in the mask image (overlapping region) corresponding to the sample pairs. i，j Ii+1，j )}, where i is the image index of the original image, i = 1, 2, ..., N, and j is the index of the adjacent original image pixel (sample point), j = 1, 2, ..., K. The temperature gain set S4 = {gain} can be calculated based on the sample pair set. i,i+1 gain i+1,i+2 ,......,gain N-1,N gain N,N}, where any adjacent original images (I i and I i+1 The calculation method for the temperature gain value between ) is as follows:

[0082]

[0083] The average() function calculates the average of K temperature values.

[0084] Furthermore, the maximum value M in the temperature gain set S4 is determined, and this maximum value M is used as the image index value to obtain the image set to be compensated S1 = {I1, I2, ..., I...} M}, correspondingly, the thermally stable image set S2={I M+1 I M+2 , ......, I N}

[0085] S304: Determine reference statistical information based on the temperature information of the thermally stable image.

[0086] S305: Based on the reference statistical information and the temperature information of the image to be compensated, perform temperature compensation on the image to be compensated.

[0087] In one possible embodiment, after temperature compensation is performed on the image to be compensated based on the reference statistical information and the temperature information of the image to be compensated, the target stitched image can be obtained by image stitching based on the thermally stable image and the temperature-compensated image to be compensated.

[0088] As described above, by identifying temperature abrupt change points in the original image set and determining the images to be compensated and thermally stable images from the original image set based on these points, and by determining reference statistical information based on the temperature information of the thermally stable images, temperature compensation is performed on each image to be compensated based on the reference statistical information and the temperature information of each image to be compensated. This improves the temperature measurement effect of the original images in the original image set. Subsequently, image stitching can be performed based on the original image set and the temperature-compensated images to be compensated to obtain the target stitched image. This reduces the situation where the display effect of the stitched image is not ideal due to the unstable temperature measurement of the original images captured by the infrared camera within the thermal stabilization time. By performing temperature compensation on the images to be compensated with unstable temperatures in the stitched and fused image content, the image stitching effect is improved, and it is not necessary to wait for the thermal stabilization time of the infrared camera before capturing the original image, effectively improving the efficiency of image acquisition and image stitching. At the same time, by determining the extreme values ​​of the temperature gain in the average temperature between adjacent frames in the original image set, temperature abrupt change points are determined, accurately identifying the images to be compensated captured within the thermal stabilization time in the original image set, accurately locating the range for temperature compensation, and improving the efficiency of temperature compensation.

[0089] Figure 7 A schematic diagram of an infrared image processing apparatus provided in an embodiment of this application is given. (Reference) Figure 7 The infrared image processing device includes an image segmentation module 31, a reference determination module 32, an image compensation module 33, and an image stitching module 34.

[0090] The image segmentation module 31 is used to determine the image to be compensated and the thermally stable image from the original image set based on the temperature abrupt change points in the original image set; the reference determination module 32 is used to determine reference statistical information based on the temperature information of the thermally stable image; and the image compensation module 33 is used to perform temperature compensation on the image to be compensated based on the reference statistical information and the temperature information of the image to be compensated.

[0091] The above describes a process where temperature abrupt changes are identified in the original image set. Based on these abrupt changes, images to be compensated and thermally stable images are determined from the original image set. Reference statistical information is then determined based on the temperature information of the thermally stable images. Based on the reference statistical information and the temperature information of each image to be compensated, temperature compensation is performed on each image to be compensated, improving the temperature measurement effect of the original images in the original image set. Subsequently, image stitching can be performed based on the original image set and the temperature-compensated images to be compensated to obtain the target stitched image. This reduces the situation where the display effect of the stitched image is not ideal due to the unstable temperature measurement of the original images captured by the infrared camera during the thermal stabilization time. By performing temperature compensation on the images to be compensated that have unstable temperature in the stitched and fused image content, the image stitching effect is improved. It is not necessary to wait for the thermal stabilization time of the infrared camera before capturing the original image, effectively improving the efficiency of image acquisition and image stitching.

[0092] In one possible embodiment, the image segmentation module 31 is specifically used for:

[0093] A set of sample pairs is generated based on the mask images corresponding to adjacent original images in the original image set. The sample pairs in the set are used to indicate the temperature value of the adjacent original images within the mask image.

[0094] A temperature gain set is determined based on the set of sample pairs, and the temperature gain in the temperature gain set is used to indicate the temperature gain value between adjacent original images;

[0095] Image index values ​​are determined from the temperature gain set, and images to be compensated and thermally stable images are determined from the original image set based on the image index values, wherein the image index values ​​are used to reflect temperature abrupt changes in the original image set.

[0096] In one possible embodiment, the temperature gain value between adjacent original images is determined based on the ratio of the average temperature of the two original images in the overlapping region of the sample pair.

[0097] In one possible embodiment, the image index value is the maximum value among the temperature gain values, the image number of the image to be compensated is less than or equal to the image index value, and the image number of the thermally stable image is greater than the image index value.

[0098] In one possible embodiment, the image segmentation module 31 is specifically used for:

[0099] A set of temperature mean ratios is determined based on the temperature mean of each original image in the original image set. The temperature mean ratios in the set are used to indicate the ratio of the temperature mean of adjacent original images.

[0100] Image index values ​​are determined from the set of temperature mean ratios, and images to be compensated and thermally stable images are determined from the original image set based on the image index values, wherein the image index values ​​are used to reflect temperature abrupt change points in the original image set.

[0101] In one possible embodiment, the temperature mean ratio is the ratio of the temperature mean of the original image with the later image number to the temperature mean of the original image with the earlier image number.

[0102] In one possible embodiment, the image number of the image to be compensated is less than or equal to the image index value, and the image number of the thermally stabilized image is greater than the image index value.

[0103] In one possible embodiment, the reference determination module 32 is specifically used for:

[0104] Determine the median value corresponding to the mean temperature of each of the thermally stable images, and determine the thermally stable image corresponding to the median value as the reference image;

[0105] Reference statistical information is obtained by using the average temperature of the reference image as the reference mean and the standard deviation of the temperature of the reference image as the reference standard deviation.

[0106] In one possible embodiment, the reference determination module 32 is specifically used for:

[0107] Reference statistical information is obtained by taking the mean value of the mean temperature of each of the thermally stable images as the reference mean value and the mean value of the standard deviation of the temperature of each of the thermally stable images as the reference standard deviation.

[0108] In one possible embodiment, temperature compensation is performed on the image to be compensated based on the following formula:

[0109]

[0110] Among them, I′ i,j,k I represents the temperature value after temperature compensation. i,j,k The temperature value before temperature compensation, m r s is the reference mean in the reference statistics. r m is the reference standard deviation in the reference statistics. i s represents the mean temperature value in the temperature information of the image to be compensated. i Let be the temperature standard deviation in the temperature information of the image to be compensated, i be the image number of the image to be compensated, and (j,k) be the pixel coordinates on the image to be compensated.

[0111] In one possible embodiment, the apparatus further includes an image stitching module for stitching images based on the thermally stable image and the temperature-compensated image to be compensated to obtain a target stitched image.

[0112] It is worth noting that in the above-described embodiments of the image stitching device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0113] This application also provides an infrared image processing device, which can integrate the infrared image processing apparatus provided in this application. Figure 8 This is a schematic diagram of the structure of an infrared image processing device provided in an embodiment of this application. (Reference) Figure 8 The infrared image processing device includes: an input device 43, an output device 44, a memory 42, and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the infrared image processing method provided in the above embodiments. The input device 43, output device 44, memory 42, and processors 41 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0114] The memory 42, as a computing device readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the infrared image processing method described in any embodiment of this application (e.g., image segmentation module 31, reference determination module 32, image compensation module 33, and image stitching module 34 in an infrared image processing device). The memory 42 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 42 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 42 may further include memory remotely located relative to the processor 41, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0115] Input device 43 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 44 may include display devices such as a display screen.

[0116] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 42, thereby realizing the infrared image processing method described above.

[0117] The infrared image processing apparatus, device, and computer provided above can be used to execute the infrared image processing method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0118] This application embodiment also provides a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform an infrared image processing method as provided in the above embodiments. The infrared image processing method includes: determining an image to be compensated and a thermally stable image from the original image set based on temperature abrupt change points in the original image set; determining reference statistical information based on the temperature information of the thermally stable image; and performing temperature compensation on the image to be compensated based on the reference statistical information and the temperature information of the image to be compensated.

[0119] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. Furthermore, storage media may reside in a first computer system in which a program is executed, or may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage media may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0120] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the infrared image processing method described above, but can also perform related operations in the infrared image processing method provided in any embodiment of this application.

[0121] The infrared image processing apparatus, device, and storage medium provided in the above embodiments can execute the infrared image processing method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the infrared image processing method provided in any embodiment of this application.

[0122] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. An infrared image processing method, characterized by, The method comprises the following steps: Based on the temperature mutation point in the original image set, the image to be compensated and the thermal stable image are determined from the original image set, comprising: generating a sample pair set based on the mask images corresponding to the adjacent original images in the original image set, the sample pair in the sample pair set is used to indicate the temperature value of the adjacent original images in the mask image; determining a temperature gain set according to the sample pair set, the temperature gain in the temperature gain set is used to indicate the temperature gain value between the adjacent original images; determining an image index value from the temperature gain set, and determining the image to be compensated and the thermal stable image from the original image set based on the image index value, the image index value is used to reflect the temperature mutation point in the original image set; Determine the reference statistical information according to the temperature information of the thermal stable image; Based on the reference statistical information and the temperature information of the image to be compensated, the temperature compensation is carried out on the image to be compensated; Based on the thermal stable image and the image to be compensated after temperature compensation, the image stitching is carried out to obtain the target stitching image.

2. The method of processing an infrared image according to claim 1, wherein, The temperature gain value between the adjacent original images is determined based on the ratio of the temperature average value of the two frames of original images in the overlapping area in the sample pair.

3. The method of claim 1, wherein, The image index value is the maximum value in the temperature gain value, the image serial number of the image to be compensated is less than or equal to the image index value, and the image serial number of the thermal stable image is greater than the image index value.

4. The method of claim 1, wherein, Based on the temperature mutation point in the original image set, the image to be compensated and the thermal stable image are determined from the original image set, further comprising: Determine a temperature mean ratio set based on the temperature mean of each original image in the original image set, the temperature mean ratio in the temperature mean ratio set is used to indicate the ratio of the temperature mean of the adjacent original images; Determine an image index value from the temperature mean ratio set, and determine the image to be compensated and the thermal stable image from the original image set based on the image index value, the image index value is used to reflect the temperature mutation point in the original image set.

5. The method of processing an infrared image according to claim 4, wherein, The temperature mean ratio is the ratio of the temperature mean of the original image with the image serial number behind to the temperature mean of the original image with the image serial number in front.

6. The method of claim 4, wherein, The image serial number of the image to be compensated is less than or equal to the image index value, and the image serial number of the thermal stable image is greater than the image index value.

7. The method of processing an infrared image according to any one of claims 1 to 6, characterized in that, The reference statistical information is determined according to the temperature information of the thermal stable image, comprising: Determine the median value corresponding to the temperature mean of each thermal stable image, and determine the thermal stable image corresponding to the median value as the reference image; According to the reference mean value as the reference mean value of the reference image, and the reference standard deviation as the reference standard deviation of the reference image, the reference statistical information is obtained.

8. The method of processing an infrared image according to any one of claims 1 to 6, characterized in that, The reference statistical information is determined according to the temperature information of the thermal stable image, comprising: The mean value corresponding to the temperature mean of each thermal stable image is determined as the reference mean value, and the mean value of the temperature standard deviation of each thermal stable image is determined as the reference standard deviation to obtain the reference statistical information.

9. The method of processing an infrared image according to any one of claims 1 to 6, characterized in that, The temperature compensation is carried out on the image to be compensated based on the following formula: wherein, is a temperature value after temperature compensation, is a temperature value before temperature compensation, is a reference mean value in the reference statistical information, is a reference standard deviation in the reference statistical information, is a temperature mean value in the temperature information of the image to be compensated, is a temperature standard deviation in the temperature information of the image to be compensated, i is an image sequence number of the image to be compensated, and (j, k) is a pixel coordinate on the image to be compensated.

10. An infrared image processing apparatus characterized by comprising: The infrared image processing method comprises an image division module, a reference determination module, an image compensation module and an image splicing module. The image division module is configured to determine a to-be-compensated image and a thermal stable image from the original image set based on a temperature mutation point in the original image set, and comprises: generating a sample pair set based on mask images corresponding to adjacent original images in the original image set, wherein a sample pair in the sample pair set is used to indicate a temperature value of the adjacent original images in the mask images; determining a temperature gain set according to the sample pair set, wherein a temperature gain in the temperature gain set is used to indicate a temperature gain value between the adjacent original images; determining an image index value from the temperature gain set, and determining the to-be-compensated image and the thermal stable image from the original image set based on the image index value, wherein the image index value is used to reflect the temperature mutation point in the original image set; The reference determination module is configured to determine reference statistical information according to temperature information of the thermal stable image; The image compensation module is configured to perform temperature compensation on the to-be-compensated image based on the reference statistical information and temperature information of the to-be-compensated image; The image splicing module is configured to perform image splicing on the thermal stable image and the to-be-compensated image after temperature compensation to obtain a target spliced image.

11. An infrared image processing device, characterized by The infrared image processing method comprises an image division module, a reference determination module, an image compensation module and an image splicing module. The infrared image processing method comprises an image division module, a reference determination module, an image compensation module and an image splicing module. The computer executable instructions, when executed by the computer processor, are configured to perform the infrared image processing method. The computer executable instructions, when executed by the computer processor, are configured to perform the infrared image processing method.

12. A storage medium storing computer-executable instructions, wherein: ​

Images