An image processing method, apparatus, device and storage medium

Abstract

This application discloses an image processing method, apparatus, device, and storage medium. The technical solution provided by this application identifies temperature abrupt change points in an original image set, determines the image to be compensated from the original image set based on these temperature abrupt change points, and determines a reference image for each image to be compensated based on the position information of the image to be compensated and each original image in the original image set. Temperature compensation is then performed on each image to be compensated based on a first mask image of the image to be compensated and the corresponding reference image. This improves the temperature measurement effect of the original images in the original image set, eliminates the need to wait for the infrared camera's thermal stabilization time before capturing the original image, and effectively improves image acquisition and image stitching efficiency.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] In the reconstruction process of 3D scenes, image fusion and stitching are often performed on multiple frames of infrared images captured continuously by an infrared camera to obtain a stitched image. For example, an infrared camera mounted on a drone can be used to capture multiple frames of infrared images by controlling the drone to move over the target scene. Then, the pose information between the multiple frames of infrared images is used to construct 3D scene information. An orthophoto image set is obtained from the 3D scene information and stitched together to form a 2D orthophoto image.

[0003] Because infrared cameras have a thermal stabilization time during operation, and the temperature measurement is unstable when the infrared camera captures images during this time, the temperature measurement of the output infrared image is inaccurate. Summary of the Invention

[0004] This application provides an image processing method, apparatus, device, and storage medium to solve the technical problem of inaccurate temperature measurement of output infrared images in the prior art, and to ensure good image temperature measurement results.

[0005] In a first aspect, embodiments of this application provide an image processing method, comprising:

[0006] Based on the temperature abrupt change points in the original image set, the image to be compensated is determined from the original image set;

[0007] Based on the location information of the image to be compensated and the original image set, a reference image is determined from the original image set for each image to be compensated;

[0008] Temperature compensation is performed on the image to be compensated based on a first mask image of the image to be compensated and the reference image.

[0009] In a second aspect, embodiments of this application provide an image processing apparatus, including a compensation image determination module, a reference image determination module, and an image compensation module, wherein:

[0010] The compensation image determination module is used to determine the image to be compensated from the original image set based on the temperature abrupt change points in the original image set;

[0011] The reference image determination module is used to determine a reference image for each image to be compensated from the original image set based on the position information of the image to be compensated and the original image set.

[0012] The image compensation module is used to perform temperature compensation on the image to be compensated based on a first mask image of the image to be compensated and the reference image.

[0013] In a third aspect, embodiments of this application provide an image processing device, including: a memory and one or more processors;

[0014] The memory is used to store one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in the first aspect.

[0016] In a fourth aspect, embodiments of this application provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the image processing method as described in the first aspect.

[0017] This application embodiment finds temperature abrupt change points in the original image set, determines the image to be compensated from the original image set based on the temperature abrupt change points, and determines a reference image for each image to be compensated based on the position information of the image to be compensated and each original image in the original image set. Temperature compensation is then performed on each image to be compensated based on the first mask image of the image to be compensated and the corresponding reference image, improving the temperature measurement effect of the original images in the original image set. Subsequently, image stitching can be performed based on the original image set and the temperature-compensated image to be compensated to obtain the target stitched image. This reduces the situation where the temperature measurement of the original images captured by the infrared camera during the thermal stabilization time is unstable, resulting in an unsatisfactory display effect of the stitched image. By performing temperature compensation on the image to be compensated with unstable temperature in the stitched and fused image content, the image stitching effect is improved, eliminating the need to wait for the infrared camera's thermal stabilization time before capturing the original image, effectively improving the efficiency of image acquisition and image stitching. Attached Figure Description

[0018] Figure 1 This is a flowchart of an image processing method provided in an embodiment of this application;

[0019] Figure 2 This is a schematic diagram illustrating the display effect of a stitched image without temperature compensation.

[0020] Figure 3 This is a schematic diagram illustrating the display effect of a target stitched image provided in an embodiment of this application;

[0021] Figure 4 This is a flowchart of another image processing method provided in the embodiments of this application;

[0022] Figure 5 This is a schematic diagram of a process for determining an image to be compensated, provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of a mask display provided in an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of another process for determining the image to be compensated provided in an embodiment of this application;

[0025] Figure 8 This is a schematic diagram of a reference route direction provided in an embodiment of this application;

[0026] Figure 9 This is a schematic diagram of a reference image selection process provided in an embodiment of this application;

[0027] Figure 10 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0028] Figure 11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely for explaining this application and not for limiting it. It should also be noted that, for ease of description, only the parts relevant to this application are shown in the drawings, not all of them. Before discussing exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe operations (or steps) as sequential processes, many of these operations can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but may also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0030] Figure 1 A flowchart of an image processing method provided in an embodiment of this application is given. The image processing method provided in this application embodiment can be executed by an image processing device, which can be implemented by hardware and / or software and integrated into an image processing equipment.

[0031] The following description uses an image processing apparatus performing an image processing method as an example. (Reference) Figure 1 The image processing method includes:

[0032] S101: Based on the temperature abrupt change points in the original image set, determine the image to be compensated from the original image set.

[0033] The original image set comprises multiple original images, which can be captured by infrared cameras mounted on mobile devices (such as drones, unmanned vehicles, etc.). The original images captured by the infrared cameras also record corresponding temperature information and the camera's pose information. The original image set can be obtained by continuously capturing multiple frames of the scene using an infrared camera mounted on a drone. This original image set can be used to reconstruct the 3D data of the entire scene, constructing a stitched image of orthorectified infrared thermal imaging data for a large-scale scene, and obtaining global scene image information. For example, the pose information between multiple frames of original images can be used to construct 3D scene information. An orthorectified projection image set can then be obtained from the 3D scene information and stitched together to form a 2D orthorectified stitched image.

[0034] In related technologies, infrared cameras have a thermal stabilization time during operation. During this time, the temperature measurement during image capture is unstable. If the camera is captured before the thermal stabilization time has ended, the original image captured during the thermal stabilization time and the original image captured after the thermal stabilization time will have different display effects due to the difference in temperature measurement stability. Consequently, in the final stitched image, obvious discontinuities will occur due to temperature jumps, resulting in poor display quality of the stitched image.

[0035] like Figure 2 The provided illustration shows a display effect of a stitched image without temperature compensation. It assumes that a drone equipped with an infrared camera acquires the original image of the target scene, where the drone's flight path is from... Figure 2 Starting from the bottom right corner of the image, the drone flies from right to left, acquiring raw images at set time intervals or flight distances. Upon reaching the far left of the target scene, it turns and collects raw images from left to right. Each time the drone turns, it records its trajectory... Figure 2 The image is shifted upwards until the original image sequence of each location in the target scene is acquired, resulting in the original image set. Generally, adjacent original images (including original images adjacent in multiple directions such as top, bottom, left, and right) have overlapping areas.

[0036] Assuming the infrared camera begins acquiring raw images without waiting for thermal stabilization time, the stitched image obtained from stitching together the raw image set will exhibit significant temperature anomalies in corresponding areas due to the unstable temperature acquisition of the raw images captured during the thermal stabilization period. For example... Figure 2As shown in the darker area at the bottom of the stitched image, the display effect of the stitched image is not ideal. In related technologies, to ensure the display effect of the stitched image, it is necessary to wait for the thermal stabilization time of the infrared camera to end before acquiring the original image, resulting in low work efficiency. This solution uses the overlapping area information of multiple frames of original images and utilizes the temperature consistency between adjacent original images to perform temperature compensation on the original images frame by frame. This eliminates the difference in display effect caused by the temperature measurement difference due to the thermal stabilization time of the infrared camera, thus ensuring the display effect of the final target stitched image.

[0037] For example, before performing temperature compensation on the original images captured during the thermal stabilization period, it is necessary to first determine the original images (i.e., the images to be compensated) from the set of captured original images. In one embodiment, since the temperature measurement stability of the infrared camera changes significantly before and after the thermal stabilization period, correspondingly, there will also be a temperature abrupt change between the original images captured before and after the thermal stabilization period. Based on this, according to the temperature abrupt change points in each original image in the original image set, and using the original images corresponding to the temperature abrupt change points as boundaries, the original images in the original image set are divided into images to be compensated and thermally stabilized images. Optionally, the sets of images to be compensated can be composed of the images to be compensated, and the sets of thermally stabilized images can be composed of the thermally stabilized images.

[0038] For example, each original image in the original image set is marked by an image number, and the earlier the original image was captured, the smaller the corresponding image number. It can be understood that the thermal stabilization time is in the early part of the entire shooting process. The image number of the original image corresponding to the temperature change point can be used as the index value. The original images with image numbers at or below the index value are regarded as images to be compensated. Correspondingly, the original images with image numbers above the index value are determined as thermally stable images.

[0039] Here, the image to be compensated can be understood as the original image that needs temperature compensation and was taken within the thermal stabilization time, while the thermally stabilized image can be understood as the original image that does not need temperature compensation and was taken after the thermal stabilization time. When stitching images together, the image to be compensated after temperature compensation and the original thermally stabilized image are used for stitching.

[0040] S102: Based on the position information of the image to be compensated and the original image set, determine a reference image for each image to be compensated from the original image set.

[0041] For example, after determining the image to be compensated from the original image set, a reference image for each image to be compensated is determined from the original image set based on the position information of the image to be compensated and each original image in the original image set. The position information can be the shooting position of the original image or the image position in the stitched image. The reference image can be a thermally stable image, or it can be either a thermally stable image or the image to be compensated. Generally, the reference image and the image to be compensated are close in shooting position or image position and have overlapping shooting areas. Furthermore, when the reference image of the image to be compensated is the image to be compensated, the image to be compensated, which serves as the reference image, has already undergone temperature compensation. That is, the temperature-compensated image to be compensated is used as a reference to perform temperature compensation on the uncompensated image to be compensated.

[0042] The reference image can be understood as the original image used when performing temperature compensation on the image to be compensated. The reference image can be used to adjust the temperature level of the corresponding image to be compensated to the same or similar temperature level as the reference image, reducing the situation where the temperature difference between the original images at adjacent locations is large due to the different temperature measurement stability.

[0043] S103: Perform temperature compensation on the image to be compensated based on the first mask image of the image to be compensated and the reference image.

[0044] The mask image is the image of the overlapping region between two images. For example, by registering (aligning) the two images and solving for the overlapping region, the overlapping region becomes the mask image of the two images. In this embodiment, the mask image of the image to be compensated and the reference image is the first mask image.

[0045] For example, after determining a reference image for each image to be compensated, a first mask image corresponding to the reference image is determined for each image to be compensated. Temperature compensation is then performed on the image to be compensated based on the mask image to obtain a temperature-compensated image. For instance, a compensation coefficient is determined based on the temperature difference between the image to be compensated and the reference image on the mask image, and temperature compensation is then performed on each pixel of the image to be compensated based on this compensation coefficient to obtain a temperature-compensated image.

[0046] In one embodiment, to improve the accuracy of temperature compensation, a reverse-order frame-by-frame compensation method can be used to perform temperature compensation on the set of images to be compensated. Based on this, when performing temperature compensation on the images to be compensated according to the first mask images of the images to be compensated and the reference images, specifically: temperature compensation is performed on the images to be compensated according to the reverse order of the images to be compensated, based on the first mask images of the images to be compensated and the reference images. That is, temperature compensation is performed on the images to be compensated in descending order of image number in the set of images to be compensated. At this time, the reference images corresponding to the first one or more images to be compensated are thermally stable images, and the reference images corresponding to the subsequent images to be compensated are images that have already undergone temperature compensation, ensuring that the images referenced for temperature compensation are thermally stable images or temperature-compensated images, thus guaranteeing the accuracy of temperature compensation.

[0047] In one possible embodiment, after performing temperature compensation on the image to be compensated based on a first mask image of the image to be compensated and the reference image, the method further includes: performing image stitching based on the original image set and the temperature-compensated image to be compensated to obtain a target stitched image.

[0048] For example, after temperature compensation is completed for each image to be compensated, image stitching is performed based on the original image set (thermally stable images in the original image set) and the temperature-compensated images to be compensated to obtain the target stitched image. The image stitching of the thermally stable images and the temperature-compensated images to be compensated can be performed using a feathering fusion algorithm.

[0049] For example, after obtaining the original image set, each original image in the original image set is first aligned to determine the image position of each original image in the stitched image. Then, the image to be compensated is determined from the original image set, and a reference image for each image to be compensated is determined. Based on the overlapping area information of the reference image and the image to be compensated (temperature information of the first mask image), temperature compensation is performed on each image to be compensated frame by frame in reverse order using temperature consistency. Then, based on the image positions of the temperature-compensated image to be compensated and the thermally stable image on the stitched image, the image to be compensated and the thermally stable image are stitched together using a fusion feathering algorithm to obtain the target stitched image.

[0050] like Figure 3 The provided illustration shows the display effect of a target stitched image. If image stitching is performed directly based on the original image set, after aligning and blending the original images, the result is... Figure 2The original stitched image shown shows that, within the display area corresponding to the original image captured during the thermal stabilization period, the image is noticeably darker, and the stitched image exhibits obvious tonal discontinuity. In contrast, the image processing method provided in this solution, after aligning, temperature compensating, and fusing the original image set, yields the image as shown... Figure 3 As shown in the target stitched image, it can be seen that the display area corresponding to the original image captured during the thermal stabilization time is basically consistent with the display area corresponding to the original image captured after the thermal stabilization time. There is no dark local display effect or temperature discontinuity as in the original stitched image. Furthermore, it does not require waiting for the infrared camera to reach thermal stabilization time before capturing the original image, resulting in higher efficiency in both original image acquisition and image stitching.

[0051] The above describes a process where temperature abrupt changes are identified in the original image set, and images to be compensated are determined from the original image set based on these abrupt changes. Reference images for each image to be compensated are determined based on the positional information of the images to be compensated and the corresponding original images in the original image set. Temperature compensation is then applied to each image to be compensated using a first mask image of the image to be compensated and the corresponding reference image, improving the temperature measurement effect of the original images in the original image set. Subsequently, image stitching can be performed based on the original image set and the temperature-compensated images to be compensated to obtain the target stitched image. This reduces the likelihood of unsatisfactory stitched image display due to unstable temperature measurement of the original images captured by the infrared camera during its thermal stabilization period. By applying temperature compensation to the temperature-unstable images in the stitched and fused image content, the image stitching effect is improved, eliminating the need to wait for the infrared camera's thermal stabilization period before capturing the original images, effectively improving image acquisition and stitching efficiency.

[0052] Based on the above embodiments, Figure 4 A flowchart of another image processing method provided in an embodiment of this application is given, which is a concretization of the above-described image processing method. (Reference) Figure 4 The image processing method includes:

[0053] S201: Based on the temperature abrupt change points in the original image set, determine the image to be compensated from the original image set.

[0054] Temperature abrupt changes can be determined based on extreme values ​​of the temperature gain or the ratio of average temperatures between adjacent frames in the original image set. In one embodiment, such as... Figure 5A schematic diagram of the process for determining the image to be compensated is provided. For determining temperature abrupt change points based on the temperature gain of the average temperature between adjacent frames in the original image set, this scheme includes steps S2011-S2013 when determining the image to be compensated from the original image set based on the temperature abrupt change points in the original image set:

[0055] S2011: Generate a set of sample pairs based on the second mask images corresponding to adjacent original images in the original image set, wherein the sample pairs in the set are used to indicate the temperature values ​​of adjacent original images within the second mask images.

[0056] The second mask image is the image that falls onto the mask after registration and alignment of two adjacent original images from the original image set. The second mask image is the image on the overlapping area of ​​the two adjacent original images. The two adjacent original images are two original images whose image frame or shooting area overlaps (or reaches a set overlap requirement). These can be two original images with consecutive image numbers, or two original images whose shooting positions or image positions in the stitched image are close or within a set distance range. Figure 6 As shown in the provided schematic diagram of a mask display, assuming Figure 6 The images on the left and in the middle are two consecutive frames of original images (with adjacent image numbers). The image on the right is a mask (white area in the figure) determined by aligning and registering the two original images. The image of the original image that falls within the mask is the second mask image.

[0057] For example, a second mask image corresponding to adjacent original images is determined, and a set of sample pairs is generated on the overlapping region (second mask image) of two adjacent original images based on the temperature information (temperature value) of the second mask image. The sample pair set includes multiple sample pairs, which indicate the temperature value of the adjacent original image within the corresponding second mask image. Each sample pair data includes the temperature values ​​of the mask images corresponding to the two adjacent original images (including two temperature values ​​on each of the two original images). Specifically, for all or part of the pixels (sampling points) in the second mask image corresponding to adjacent original images, the number of pixels (sampling points) used to construct the sample pair can be set as needed. Sample pairs are constructed based on the temperature values ​​corresponding to these pixels in the two original images, and these sample pairs constitute the sample pair set of the second mask image corresponding to the two adjacent original images. In one embodiment, each original image corresponds to one sample pair with the next adjacent original image, and the last original image constructs a sample pair with itself as an adjacent original image. That is, the number of sample pairs in the sample pair set is consistent with the number of original images.

[0058] S2012: Determine a temperature gain set based on the set of sample pairs, wherein the temperature gain in the temperature gain set is used to indicate the temperature gain value between adjacent original images.

[0059] The temperature gain in the temperature gain set is used to indicate the temperature gain value between adjacent original images in the overlapping region. For example, after obtaining the sample pair set, the temperature gain value between each pair of adjacent original images is calculated based on each sample pair in the sample pair set, and the temperature gain set is obtained based on these temperature gain values.

[0060] The temperature gain value reflects the degree of change in the average temperature of adjacent original images in the overlapping area. When the temperature gain value is the ratio of the average temperature of the later original image to that of the earlier original image in the overlapping area, the larger the temperature gain value, the greater the degree of change in the average temperature between adjacent original images. Conversely, when the temperature gain value is the ratio of the average temperature of the earlier original image to that of the later original image in the overlapping area, the larger the temperature gain value, the smaller the degree of change in the average temperature between adjacent original images.

[0061] In one embodiment, the temperature gain value of adjacent original images in the overlapping region is determined based on the ratio of the average temperature of the two original images in the sample pair within the overlapping region. In this case, the greater the change in the average temperature between adjacent original images, the greater the corresponding temperature gain value.

[0062] S2013: Determine the image index value from the temperature gain set, and determine the image to be compensated from the original image set based on the image index value.

[0063] The image index value reflects temperature abrupt changes in the original image set, and is represented by the image number of the original image. For example, the image index value is determined from the temperature gain set, and the image to be compensated is determined from the original image set based on the image index value.

[0064] The image to be compensated can be determined from the original image set based on its image index value, according to the calculation method of the temperature gain value. For example, when the temperature gain value is the ratio of the average temperature of the later original image to the average temperature of the earlier original image, the original images with image indices below the image index value can be identified as the images to be compensated. Conversely, when the temperature gain value is the ratio of the average temperature of the earlier original image to the average temperature of the later original image, the original images with image indices above the image index value can be identified as the images to be compensated.

[0065] In one embodiment, the temperature gain value between adjacent original images is determined based on the ratio of the average temperature of the two original images in the overlapping region of the sample pair. The temperature gain value is the ratio of the average temperature of the latter original image to that of the former original image. Correspondingly, the image index value is the maximum value among all temperature gain values ​​in the temperature gain set, and the image number of the image to be compensated is less than or equal to the image index value.

[0066] For example, suppose the number of original images in the original image set is N, and the width and height of the final stitched image are W and H respectively. The set of position information of each original image in the final stitched image is P = {(px1, py1), (px2, py2), ..., (px...} N py N Obtain the original image set S0 = {I1, I2, ..., I...}. N}, based on two consecutive original images I i and I i+1 Generate a set of sample pairs S3 = {(I} in the mask image (overlapping region) corresponding to the sample pairs. i，j I i+1，j )}, where i is the image index of the original image, i = 1, 2, ..., N, and j is the index of the adjacent original image pixel (sample point), j = 1, 2, ..., K. The temperature gain set S4 = {gain} can be calculated based on the sample pair set. i，i+1 gain i+1，i+2 , ......, gain N-1，N gain N，N}, where any adjacent original images (I i and I i+1 The calculation method for the temperature gain value between ) is as follows:

[0067]

[0068] The average() function calculates the average of K temperature values.

[0069] Furthermore, the maximum value M in the temperature gain set S4 is determined, and this maximum value M is used as the image index value to obtain the image set to be compensated S1 = {I1, I2, ..., I...} M}, correspondingly, the thermally stable image set S2={I M+1 I M+2 , ......, I N}

[0070] In one embodiment, such as Figure 7Another schematic diagram of the process for determining the image to be compensated is provided. For determining temperature abrupt change points based on the ratio of average temperatures between adjacent frames in the original image set, this scheme includes steps S2014-S2015 when determining the image to be compensated from the original image set based on the temperature abrupt change points in the original image set:

[0071] S2014: Determine a set of temperature mean ratios based on the temperature mean of each original image in the original image set, wherein the temperature mean ratios in the set are used to indicate the ratio of the temperature mean of adjacent original images.

[0072] For example, the average temperature of each original image (the average temperature of each pixel) is calculated, and a set of temperature mean ratios is created based on the average temperature of each original image. The set of temperature mean ratios records the ratio of the average temperature of each original image to that of its neighboring original images, where the temperature mean ratios indicate the ratio of the average temperature of neighboring original images.

[0073] Understandably, the ratio of temperature mean values ​​reflects the degree of change in the average temperature between adjacent original images. When the ratio of temperature mean values ​​is the ratio of the average temperature of the later original image to that of the earlier original image, the larger the ratio, the greater the degree of change in the average temperature between adjacent original images. Conversely, when the ratio of temperature mean values ​​is the ratio of the average temperature of the earlier original image to that of the later original image, the larger the ratio, the smaller the degree of change in the average temperature between adjacent original images.

[0074] In one embodiment, the ratio of the average temperature values ​​of adjacent original images can be calculated in reverse order. That is, the average temperature ratio is the ratio of the average temperature value of the original image with the later image number to the average temperature value of the original image with the earlier image number. In this case, the greater the degree of change in the average temperature value between adjacent original images, the larger the corresponding average temperature ratio.

[0075] S2015: Determine the image index value from the set of average temperature ratios, and determine the image to be compensated from the original image set based on the image index value.

[0076] For example, after obtaining the set of temperature mean ratios, image index values ​​are determined from the set of temperature mean ratios, and the images to be compensated are determined from the original image set based on the image index values. The image index values ​​reflect temperature abrupt changes in the original image set, and are represented by the image sequence numbers of the original images.

[0077] The image to be compensated can be determined from the original image set based on its image index value, according to the calculation method of the temperature mean ratio. For example, when the temperature mean ratio is the ratio of the average temperature of the later original image to the average temperature of the earlier original image, the original images with image indices below the image index value can be identified as the images to be compensated. Conversely, when the temperature mean ratio is the ratio of the average temperature of the earlier original image to the average temperature of the later original image, the original images with image indices above the image index value can be identified as the images to be compensated.

[0078] In one embodiment, the temperature mean ratio is the ratio of the temperature mean of the original image with the later image number to the temperature mean of the original image with the earlier image number. Correspondingly, the image index is the maximum value among all temperature mean ratios in the set of temperature mean ratios. In this case, the image number of the image to be compensated is less than or equal to the image index value.

[0079] For example, suppose the number of original images in the original image set is N, and the width and height of the final stitched image are W and H respectively. The set of position information (image position) of each original image in the final stitched image is P = {(px1, py1), (px2, py2), ..., (px... N py N Obtain the original image set S0 = {I1, I2, ..., I...}. N} Calculate the mean temperature m of all original images in the original image set to obtain the temperature information set S5 = {m1, m2, ..., m} N}. Calculate adjacent original images I in reverse order. i and I i+1 ratio of temperature average Obtain the set of temperature mean ratios The maximum value in the set of temperature mean ratios is taken as the image index value M, resulting in the set of images to be compensated, S1 = {I1, I2, ..., I...} M}, correspondingly, the thermally stable image set S2={I M+1 I M+2 , ......, I N}

[0080] S202: Based on the position information of the image to be compensated and the original image set, determine a candidate image set for each image to be compensated from the original image set.

[0081] For example, for each image to be compensated, the image distance between the image to be compensated and each original image in the original image set (including other images to be compensated) is calculated based on the position information of the image to be compensated and each original image in the original image set. A candidate image set for the image to be compensated is then determined from the original image set based on the image distance. For example, original images whose image distance from the image to be compensated is less than a preset threshold can be determined from the original image set, and these original images are used as the candidate image set for the compensated image. The position information can be the shooting location or the image location; correspondingly, the overlapping area is the shooting overlap area or the image overlap area. This embodiment uses the image location as the position information as an example for description. In one embodiment, the candidate images in the candidate image set have an overlapping area with the image to be compensated.

[0082] In one embodiment, the image distance between a candidate image in the candidate image set and the corresponding image to be compensated is less than the image size (image height and / or image width) of the image to be compensated. That is, for each image to be compensated, the original set of all images whose image distance from the current image to be compensated is less than the image height and / or image width of the current image to be compensated is obtained as the candidate image set.

[0083] For example, for the image I to be compensated i Assume its image width and image height are w respectively. i and h i The corresponding location information P i =(px i py i For each image to be compensated, calculate the position information of the image to be compensated compared with that of the original image (P). k =(px j py j Image distance and will satisfy dis i，j <w i and dis i，j <h i The original image is used as the reference image for the corresponding image to be compensated, and then a set of candidate images corresponding to each image to be compensated is obtained.

[0084] S203: Based on the position information of the image to be compensated and the candidate images, determine a reference image for each image to be compensated from the set of candidate images.

[0085] For example, for each image to be compensated, a reference image for temperature compensation is determined from the candidate image set based on its position information relative to each candidate image in the candidate image set. In one possible embodiment, temperature compensation is performed on each image to be compensated in reverse order, and the reference image is a candidate image from the candidate image set in the direction perpendicular to the reference flight path direction of the corresponding image to be compensated. In one embodiment, the image number of the candidate image is greater than the image number of the corresponding image to be compensated.

[0086] like Figure 8 A schematic diagram of a reference flight path is provided, where the white arrow L1 points to the reference flight path direction of the current image to be compensated, and the white arrow L2 points to the direction perpendicular to the reference flight path direction. It can be understood that since the thermal stabilization time of the infrared camera is a short period after it is turned on, only the first few segments of the flight path during the flight mission will be affected by the temperature measurement stability. By utilizing the principle of temperature consistency between adjacent areas, candidate images in the direction perpendicular to the reference flight path direction are selected as reference images, which can more quickly select the optimal reference image. Similarly, in order to improve the accuracy of subsequent compensation, temperature compensation can be performed frame by frame in reverse order, which can ensure that the selected reference image is the compensated image.

[0087] In one embodiment, such as Figure 9 A schematic diagram of a reference image selection process is provided. When determining the reference image for each image to be compensated from the candidate image set based on the position information of the image to be compensated and the candidate images, the process specifically includes steps S2031-S2032:

[0088] S2031: Based on the position information of the image to be compensated and the candidate images in the stitched image, determine the reference flight path direction of the image to be compensated and the candidate image direction of each candidate image relative to the image to be compensated.

[0089] S2032: Determine the vector inner product corresponding to the reference direction and each candidate image direction, and determine the candidate image corresponding to the smallest vector inner product as the reference image of the image to be compensated.

[0090] For example, for each image to be compensated, based on the position information of the image to be compensated and each candidate image in the corresponding candidate image set within the stitched image, the reference flight path direction of the current image to be compensated, and the candidate image direction of each candidate image relative to the current image to be compensated, are determined. Further, the dot product between the reference direction and each candidate image direction is calculated, the smallest dot product among these dot products is determined, and the candidate image corresponding to the smallest dot product is determined as the reference image of the current image to be compensated. It can be understood that the smaller the dot product, the closer the corresponding reference direction and candidate image direction are to a right angle, and the closer the candidate image direction is to the direction perpendicular to the reference flight path direction.

[0091] For example, the image to be compensated I i The base route direction is baseDir = {(px i+1 -px i Py i+1 -pyi), candidate image I j Relative to the image to be compensated I i The candidate graph orientation is candiDirj = (pxj - pxi, pyj - pyi). Calculate the vector inner product AngleDistbase,j = dot Product(baseDir, candiDir) of the baseline orientation and the candidate graph orientation. j ), and take AngleDist base，j The candidate image corresponding to the minimum value in the image is used as the reference image R of the corresponding image to be compensated. i This yields a set of reference images S7 = {R1, R2, ..., R...} for all images to be compensated. M}

[0092] S204: Determine the set of temperature-corresponding points based on the first mask image of the image to be compensated and the reference image, and determine the compensation coefficient based on the set of temperature-corresponding points corresponding to each image to be compensated.

[0093] For example, for each image to be compensated, a set of temperature-corresponding points within the first mask image region is constructed based on the image to be compensated and the first mask image of the reference image. This set of temperature-corresponding points records multiple temperature-corresponding points, each recording the temperature corresponding to multiple pixels in the corresponding region of the image to be compensated and the reference image within the first mask image. For instance, assuming n (all pixels or a pre-defined number) temperature-corresponding points are taken from the first mask image, then the set of temperature-corresponding points for the image to be compensated and the reference image on the first mask image is {(x1, y1), ..., (x... n y n )}.

[0094] After obtaining the set of temperature-corresponding points for the image to be compensated and the reference image on the first mask image, compensation coefficients are determined based on the set of temperature-corresponding points for the image to be compensated. These compensation coefficients reflect the transformation relationship of temperature values ​​from the image to be compensated to the reference image in the first mask image. The corresponding compensation coefficients are calculated based on the set of temperature-corresponding points for each image to be compensated, resulting in a set of compensation coefficients for temperature compensation of the set of images to be compensated.

[0095] In one embodiment, the compensation coefficient can be solved by constructing an overdetermined equation. Based on this, the present solution determines the compensation coefficient based on the set of temperature corresponding points for each image to be compensated, including: constructing an overdetermined equation set based on the set of temperature corresponding points for each image to be compensated, and solving the overdetermined equation set to obtain the compensation coefficient for each image to be compensated.

[0096] For example, a linear model y = ax + b is used to describe the temperature of adjacent images in the mask image (overlapping region), where the compensation coefficients include a first compensation coefficient and a second compensation coefficient, a and b are the first compensation coefficient and the second compensation coefficient, respectively, x is the temperature value before temperature compensation, and y is the temperature value after temperature compensation. Based on the image to be compensated I i The set of points corresponding to the temperature is {(x1, y1), ..., (x... n y n Construct an overdetermined system of equations:

[0097]

[0098] The above system of equations can be solved using the normal equations or the gradient descent method to obtain the compensation coefficients (first compensation coefficient a and second compensation coefficient b) for the corresponding image to be compensated, and finally obtain the set of compensation coefficients C. M ={(a1, b1), (a2, b2), ..., (a M b M )}.

[0099] S205: Perform temperature compensation on the image to be compensated based on the compensation coefficient corresponding to each image to be compensated.

[0100] For example, the compensation coefficients obtained above are used to perform temperature compensation on the image to be compensated. For instance, in the reverse direction of the image to be compensated, for each pixel of each image to be compensated, based on the linear model y = ax + b, its temperature value is taken as x, the corresponding first compensation coefficient and second compensation coefficient are taken as a and b, and the result y is taken as the temperature value after temperature compensation. The temperature value of each pixel is compensated to obtain the temperature-compensated image to be compensated.

[0101] In one possible embodiment, after temperature compensation is performed on the image to be compensated based on a first mask image of the image to be compensated and the reference image, the target stitched image can be obtained by image stitching based on the original image set and the temperature-compensated image to be compensated.

[0102] The above describes a process where temperature abrupt changes are identified in the original image set, and images to be compensated are determined from the set based on these abrupt changes. Reference images for each image to be compensated are determined based on the positional information of the images to be compensated and the corresponding original images in the original image set. Temperature compensation is then applied to each image to be compensated using a first mask image of the image to be compensated and its corresponding reference image. This improves the temperature measurement accuracy of the original images in the original image set. Subsequently, image stitching can be performed based on the original image set and the temperature-compensated images to be compensated to obtain the target stitched image. This reduces the likelihood of unsatisfactory stitched image display due to unstable temperature measurements of the original images captured by the infrared camera during its thermal stabilization period. By compensating for temperature-unstable images in the stitched and merged image content, the image stitching effect is improved, eliminating the need to wait for the infrared camera's thermal stabilization period before capturing the original images, thus effectively improving image acquisition and stitching efficiency. Furthermore, temperature abrupt changes are determined based on the extreme values ​​of the temperature gain or the ratio of the average temperatures between adjacent frames in the original image set. This accurately identifies the images to be compensated captured during the thermal stabilization period in the original image set, precisely locating the range for temperature compensation and improving temperature compensation efficiency. By utilizing the principle of temperature consistency between adjacent areas, candidate images in the vertical direction of the baseline flight path are selected as reference images. This allows for faster selection of the optimal reference image and ensures that the reference image selected for temperature compensation is the compensated image, thus guaranteeing the temperature compensation effect.

[0103] Figure 10 A schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application is given. (Reference) Figure 10 The image processing device includes a compensation image determination module 31, a reference image determination module 32, and an image compensation module 33.

[0104] The compensation image determination module 31 is used to determine the image to be compensated from the original image set based on the temperature abrupt change points in the original image set; the reference image determination module 32 is used to determine a reference image for each image to be compensated from the original image set according to the position information of the image to be compensated and the original image set; and the image compensation module 33 is used to perform temperature compensation on the image to be compensated based on a first mask image of the image to be compensated and the reference image.

[0105] The above describes a process where temperature abrupt changes are identified in the original image set, and images to be compensated are determined from the original image set based on these abrupt changes. Reference images for each image to be compensated are determined based on the positional information of the images to be compensated and the corresponding original images in the original image set. Temperature compensation is then applied to each image to be compensated using a first mask image of the image to be compensated and the corresponding reference image, improving the temperature measurement effect of the original images in the original image set. Subsequently, image stitching can be performed based on the original image set and the temperature-compensated images to be compensated to obtain the target stitched image. This reduces the likelihood of unsatisfactory stitched image display due to unstable temperature measurement of the original images captured by the infrared camera during its thermal stabilization period. By applying temperature compensation to the temperature-unstable images in the stitched and fused image content, the image stitching effect is improved, eliminating the need to wait for the infrared camera's thermal stabilization period before capturing the original images, effectively improving image acquisition and stitching efficiency.

[0106] In one possible embodiment, the compensated image determination module 31 is specifically used for:

[0107] A set of sample pairs is generated based on the second mask images corresponding to adjacent original images in the original image set. The sample pairs in the set are used to indicate the temperature values ​​of adjacent original images within the second mask image.

[0108] A temperature gain set is determined based on the set of sample pairs, and the temperature gain in the temperature gain set is used to indicate the temperature gain value between adjacent original images;

[0109] An image index value is determined from the temperature gain set, and an image to be compensated is determined from the original image set based on the image index value, the image index value being used to reflect temperature abrupt changes in the original image set.

[0110] In one possible embodiment, the temperature gain value between adjacent original images is determined based on the ratio of the average temperature of the two original images in the overlapping region of the sample pair.

[0111] In one possible embodiment, the image index value is the maximum value among the temperature gain values, and the image number of the image to be compensated is less than or equal to the image index value.

[0112] In one possible embodiment, the compensated image determination module 31 is specifically used for:

[0113] A set of temperature mean ratios is determined based on the temperature mean of each original image in the original image set. The temperature mean ratios in the set are used to indicate the ratio of the temperature mean of adjacent original images.

[0114] An image index value is determined from the set of average temperature ratios, and an image to be compensated is determined from the original image set based on the image index value, wherein the image index value is used to reflect temperature abrupt changes in the original image set.

[0115] In one possible embodiment, the temperature mean ratio is the ratio of the temperature mean of the original image with the later image number to the temperature mean of the original image with the earlier image number.

[0116] In one possible embodiment, the image number of the image to be compensated is less than or equal to the image index value.

[0117] In one possible embodiment, the reference image determination module 32 is specifically used for:

[0118] Based on the location information of the image to be compensated and the original image set, a candidate image set for each image to be compensated is determined from the original image set;

[0119] Based on the position information of the image to be compensated and the candidate images, a reference image is determined from the set of candidate images for each image to be compensated.

[0120] In one possible embodiment, the image distance between a candidate image in the candidate image set and its corresponding image to be compensated is less than the image height and / or image width of the image to be compensated.

[0121] In one possible embodiment, the reference image is a candidate image of the candidate image set in the direction perpendicular to the reference flight path direction of the corresponding image to be compensated.

[0122] In one possible embodiment, when the reference image determination module 32 determines a reference image for each image to be compensated from the candidate image set based on the position information of the image to be compensated and the candidate images, it specifically includes:

[0123] Based on the position information of the image to be compensated and the candidate images in the stitched image, the reference flight path direction of the image to be compensated and the candidate image direction of each candidate image relative to the image to be compensated are determined.

[0124] Determine the inner product of the vectors corresponding to the reference direction and each candidate image direction, and determine the candidate image corresponding to the smallest inner product as the reference image of the image to be compensated.

[0125] In one possible embodiment, the image compensation module 33 is specifically used for:

[0126] A set of temperature-corresponding points is determined based on the first mask image of the image to be compensated and the reference image, and a compensation coefficient is determined based on the set of temperature-corresponding points corresponding to each image to be compensated.

[0127] Temperature compensation is performed on the image to be compensated based on the compensation coefficient corresponding to each image to be compensated.

[0128] In one possible embodiment, when the image compensation module 33 determines the compensation coefficient based on the set of temperature-corresponding points corresponding to each image to be compensated, it specifically includes:

[0129] An overdetermined set of equations is constructed based on the set of temperature-corresponding points for each image to be compensated, and the compensation coefficients for each image to be compensated are obtained by solving the overdetermined set of equations.

[0130] In one possible embodiment, the image compensation module 33 is specifically used to: perform temperature compensation on the image to be compensated according to a first mask image of the image to be compensated and the reference image, in the reverse order of the image to be compensated.

[0131] In one possible embodiment, the device further includes an image stitching module for stitching images based on the original image set and the temperature-compensated image to be compensated to obtain a target stitched image.

[0132] It is worth noting that in the above-described embodiments of the image processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0133] This application also provides an image processing device that can integrate the image processing apparatus provided in this application. Figure 11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. (Reference) Figure 11 The image processing device includes an input device 43, an output device 44, a memory 42, and one or more processors 41. The memory 42 stores one or more programs. When the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the image processing method provided in the above embodiments. The input device 43, output device 44, memory 42, and processors 41 can be connected via a bus or other means. Figure 11 Taking the example of a connection between China and Israel via a bus.

[0134] The memory 42, as a computing device readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the image processing methods described in any embodiment of this application (e.g., the compensated image determination module 31, reference image determination module 32, image compensation module 33, and image stitching module 34 in the image processing apparatus). The memory 42 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory 42 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 42 may further include memory remotely located relative to the processor 41, and these remote memories can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0135] Input device 43 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Output device 44 may include display devices such as a display screen.

[0136] The processor 41 executes various functional applications and data processing of the device by running software programs, instructions and modules stored in the memory 42, thereby realizing the above-mentioned image processing method.

[0137] The image processing apparatus, device, and computer provided above can be used to execute the image processing method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0138] This application embodiment also provides a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform an image processing method as provided in the above embodiments. The image processing method includes: determining an image to be compensated from the original image set based on temperature abrupt change points in the original image set; determining a reference image for each image to be compensated from the original image set according to the image to be compensated and the position information of the original image set; and performing temperature compensation on the image to be compensated according to a first mask image of the image to be compensated and the reference image.

[0139] Storage medium – any type of memory device or storage device. The term “storage medium” is intended to include: mounting media, such as CD-ROMs, floppy disks, or magnetic tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (e.g., hard disks or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. Furthermore, storage media may reside in a first computer system in which a program is executed, or may reside in a different second computer system connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term “storage medium” can include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). Storage media may store program instructions (e.g., specifically implemented as a computer program) executable by one or more processors.

[0140] Of course, the computer-executable instructions provided in the embodiments of this application are not limited to the image processing method described above, but can also perform related operations in the image processing method provided in any embodiment of this application.

[0141] The image processing apparatus, device, and storage medium provided in the above embodiments can execute the image processing method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the image processing method provided in any embodiment of this application.

[0142] The above description is merely a preferred embodiment and the technical principles employed in this application. This application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions that can be made by those skilled in the art will not depart from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Determining the image to be compensated from the original image set based on temperature abrupt changes in the original image set includes: generating a set of sample pairs based on second mask images corresponding to adjacent original images in the original image set, wherein the sample pairs in the set are used to indicate the temperature values ​​of adjacent original images within the second mask images; determining a set of temperature gains based on the set of sample pairs, wherein the temperature gains in the set of temperature gains are used to indicate the temperature gain values ​​between adjacent original images; determining an image index value from the set of temperature gains, and determining the image to be compensated from the original image set based on the image index value, wherein the image index value reflects the temperature abrupt changes in the original image set; Based on the position information of the image to be compensated and the original image set, a candidate image set for each image to be compensated is determined from the original image set; based on the position information of the image to be compensated and the candidate images, a reference image for each image to be compensated is determined from the candidate image set. Temperature compensation is performed on the image to be compensated based on a first mask image of the image to be compensated and the reference image.

2. The image processing method according to claim 1, characterized in that, The temperature gain value between adjacent original images is determined based on the ratio of the average temperature of the two original images in the overlapping region of the sample pair.

3. The image processing method according to claim 1, characterized in that, The image index value is the maximum value among the temperature gain values, and the image number of the image to be compensated is less than or equal to the image index value.

4. The image processing method according to claim 1, characterized in that, The step of determining the image to be compensated from the original image set based on temperature abrupt change points in the original image set further includes: A set of temperature mean ratios is determined based on the temperature mean of each original image in the original image set. The temperature mean ratios in the set are used to indicate the ratio of the temperature mean of adjacent original images. An image index value is determined from the set of average temperature ratios, and an image to be compensated is determined from the original image set based on the image index value, wherein the image index value is used to reflect temperature abrupt changes in the original image set.

5. The image processing method according to claim 4, characterized in that, The temperature mean ratio is the ratio of the temperature mean of the original image with the later image number to the temperature mean of the original image with the earlier image number.

6. The image processing method according to claim 4, characterized in that, The image number of the image to be compensated is less than or equal to the image index value.

7. The image processing method according to claim 1, characterized in that, The image distance between a candidate image in the candidate image set and its corresponding image to be compensated is less than the image height and / or image width of the image to be compensated.

8. The image processing method according to claim 1, characterized in that, The reference image is a candidate image of the candidate image set in the direction perpendicular to the reference flight path direction of the corresponding image to be compensated.

9. The image processing method according to claim 1, characterized in that, The step of determining a reference image for each image to be compensated from the candidate image set based on the position information of the image to be compensated and the candidate images includes: Based on the position information of the image to be compensated and the candidate images in the stitched image, the reference flight path direction of the image to be compensated and the candidate image direction of each candidate image relative to the image to be compensated are determined. Determine the inner product of the vectors corresponding to the reference route direction and each candidate image direction, and determine the candidate image corresponding to the smallest inner product as the reference image of the image to be compensated.

10. The image processing method according to any one of claims 1-6, characterized in that, The step of performing temperature compensation on the image to be compensated based on a first mask image of the image to be compensated and the reference image includes: A set of temperature-corresponding points is determined based on the first mask image of the image to be compensated and the reference image, and a compensation coefficient is determined based on the set of temperature-corresponding points corresponding to each image to be compensated. Temperature compensation is performed on the image to be compensated based on the compensation coefficient corresponding to each image to be compensated.

11. The image processing method according to claim 10, characterized in that, The step of determining the compensation coefficient based on the set of temperature-corresponding points corresponding to each of the images to be compensated includes: An overdetermined set of equations is constructed based on the set of temperature-corresponding points for each image to be compensated, and the compensation coefficients for each image to be compensated are obtained by solving the overdetermined set of equations.

12. The image processing method according to any one of claims 1-6, characterized in that, The step of performing temperature compensation on the image to be compensated based on a first mask image of the image to be compensated and the reference image includes: Temperature compensation is performed on the image to be compensated based on the first mask image of the image to be compensated and the reference image, following the reverse order of the image to be compensated.

13. The image processing method according to any one of claims 1-6, characterized in that, After performing temperature compensation on the image to be compensated based on the first mask image of the image to be compensated and the reference image, the method further includes: The target stitched image is obtained by stitching together the original image set and the temperature-compensated image to be compensated.

14. An image processing apparatus, characterized in that, It includes a compensation image determination module, a reference image determination module, and an image compensation module, wherein: The compensation image determination module is used to determine the image to be compensated from the original image set based on temperature abrupt change points in the original image set, including: generating a sample pair set based on second mask images corresponding to adjacent original images in the original image set, wherein the sample pairs in the sample pair set are used to indicate the temperature values ​​of adjacent original images within the second mask images; determining a temperature gain set based on the sample pair set, wherein the temperature gains in the temperature gain set are used to indicate the temperature gain values ​​between adjacent original images; determining an image index value from the temperature gain set, and determining the image to be compensated from the original image set based on the image index value, wherein the image index value is used to reflect the temperature abrupt change points in the original image set; The reference image determination module is used to determine a candidate image set for each image to be compensated from the original image set based on the position information of the image to be compensated and the original image set; and to determine a reference image for each image to be compensated from the candidate image set based on the position information of the image to be compensated and the candidate images. The image compensation module is used to perform temperature compensation on the image to be compensated based on a first mask image of the image to be compensated and the reference image.

15. An image processing device, characterized in that, include: Memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1-12.

16. A storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the image processing method as described in any one of claims 1-12.

Images