Aerial filter array multispectral image strip gray scale adjusting method

Abstract

The application discloses an aerial filter array multispectral image strip gray scale adjusting method, and belongs to the technical field of aerial image processing.The application aims to simultaneously adjust the gray scale of each wave band of the whole image according to the average value of the average value of the gray scale mean value of the overlapped area of each wave band image, and the gray scale of the same ground object of the adjusted spliced image is consistent, so that the aerial filter array multispectral image strip gray scale adjusting method reduces the influence on feature extraction and matching in the later image registration process.The application firstly determines the overlapped area of each strip image, then obtains the gray scale mean value of the overlapped area of the image, and finally adjusts the gray scale of the image.The gray scale adjusting method of the filter array multispectral image of the application has better gray scale processing effect than the current method, better solves the problem of inconsistent gray scale of the filter array multispectral image, the overall gray scale of the adjusted single wave band image is consistent, and the ground object spectral information can be well maintained, so that the application can meet the later processing and application requirements of the image.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of aerial image processing. BACKGROUND

[0002] In recent years, spectral cameras have become important optoelectronic remote sensing payloads, among which filter array multispectral cameras are widely used in unmanned aerial vehicle multispectral remote sensing due to their low platform requirements. Filter array multispectral cameras can obtain multiple band strip image data at the same time, and then perform cropping and splicing processing on the multi-band strip image data obtained at different times to obtain single-band images. Due to the difference in exposure amount of the camera at different times, there is a significant gray difference between the strips at different times in the single-band image after cropping and splicing, which affects subsequent image processing and application requirements. In addition, the gray difference of ground objects makes it difficult to extract image spectral features and increases the mismatch rate during feature matching [2] . In order to meet the application requirements of filter array multispectral data, gray processing must be performed.

[0003] At present, image gray processing is mainly used in the field of image splicing, and the more commonly used method is to perform dodging processing on the image, such as the MASK algorithm, homomorphic filtering algorithm, Retinex algorithm ] , Wallis algorithm, histogram matching method, etc. However, the above algorithms are not ideal when dealing with the gray difference of filter array multispectral images. In view of the gray difference of filter array multispectral images, Fang Xiuxiu et al. proposed an image gray linear transformation method to adjust the gray of each strip image through an image gray linear transformation model. However, the gray adjustment effect of this method is too dependent on the number and distribution of matching points in the overlapping area, and when the matching points extracted in the overlapping area are few, there are no matching points or the distribution is concentrated, it is difficult to accurately establish the gray transformation relationship between images, or the transformation relationship has a large deviation.

[0004] In addition, this method only considers the gray difference between single-band images, but ignores the gray variation relationship between single-band images, and the spectral features of ground objects in the multispectral image after gray adjustment are prone to deviation, which cannot accurately reflect the characteristics of different bands of ground objects. SUMMARY

[0005] The purpose of the present application is to adjust the gray of each band of the entire image according to the average value of the gray mean value ratio of the overlapping area of each band image, so that the gray of the same ground object in the spliced image is consistent, and the influence of feature extraction and matching in the later image registration process is reduced.

[0006] The steps of the present application are:

[0007] S1, determining the overlapping area of each strip image

[0008] Determine the four vertex coordinates of each band strip effective area, and construct a strip image effective area template. According to the translation relationship between images, respectively expand template 1 and template 2, then multiply the expanded template 1 and template 2 to obtain the overlapping area template, and then split the expanded template into the first and second band original template 1. Then, template 1 is multiplied with the first band image in the two multi-spectral images to obtain the overlapping area image.

[0009] S2, image overlapping area gray mean value

[0010] Based on the single-band image overlapping area, the image gray mean value and the gray coefficient relative to the adjacent image can be calculated. First, obtain the proportional coefficient of the gray mean value of each band overlapping area between adjacent images, and then average the proportional coefficient of each band to obtain the gray adjustment coefficient of the whole image.

[0011] Adjust the i-th and j-th adjacent images:

[0012] First, obtain the average gray values a of the first, second, third, fourth, fifth, sixth, seventh and eighth band overlapping areas in the i-th and j-th images i1 j1 i2 j2 i3 j3 i4 j4 i5 j5 i6 j6 i7 j7 i8 j8 Then obtain the gray ratios c1, c2, c3, c4, c5, c6, c7 and c8 of the eight band overlapping areas, and the gray adjustment coefficient g ij is:

[0013]

[0014] g ij is the gray adjustment coefficient of the j-th image. The gray of the adjusted image overlapping area is consistent with the gray of the i-th image.

[0015] S3, image gray adjustment

[0016] Obtain the image gray adjustment coefficient g ij ​​​​​​​​​​​​​​​After that, the image is adjusted in gray scale, the 5th image in the middle is selected as the reference, and each gray scale adjustment coefficient is multiplied by the gray scale of each pixel in the corresponding sequence image in turn, and the calculation formula of each gray scale adjustment coefficient is as follows:

[0017] The gray scale adjustment method of the filter array multispectral image of the application has better gray scale processing effect than the current method, better solves the problem of inconsistent gray scale of the filter array multispectral image, the overall gray scale of the adjusted single-band image is consistent, and the ground feature spectral information can be well maintained, and the application requirements of the image post-processing and application can be met. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a multispectral image;

[0019] Figure 2 is an image overlap area;

[0020] Figure 3 is a 1st waveband overlap area image;

[0021] Figure 4 is an overlap area image gray scale ratio;

[0022] Figure 5 is a sequence image translation and overlap area average gray scale schematic diagram;

[0023] Figure 6 is a multispectral image splicing;

[0024] Figure 7 is a matching point of adjacent images;

[0025] Figure 8 is a filter array multispectral image splicing;

[0026] Figure 9 is an experimental image;

[0027] Figure 10 is a gray scale adjustment comparison diagram;

[0028] Figure 11 is a gray scale adjustment effect comparison;

[0029] Figure 12 is a gray scale adjustment local magnification comparison diagram;

[0030] Figure 13 is an overlap area gray scale mean value difference comparison. DETAILED DESCRIPTION

[0031] The filter array multispectral image gray scale adjustment technical process is basically the same as the splicing process of other images, mainly including three steps of spatial position determination, gray scale adjustment and strip image cutting and splicing.

[0032] 1. Determining the spatial location of the image

[0033] Determining the spatial location of an image mainly includes two parts: determining the projection transformation and translation of the strip image.

[0034] Image projection transformation

[0035] In addition to recording image data, multispectral imaging using a filter array also records platform state parameters during imaging, such as platform position information (longitude L, latitude B, altitude H) and attitude information (pitch angle). (Roll angle ω, heading angle κ, etc.). Therefore, select t1, t2, ..., t n Given n multi-strip images at time points, perform a projection transformation on the images at each time point according to the following two formulas.

[0036]

[0037]

[0038] In the formula, X and Y are the coordinates of the projected image points in pixels; H is the flight altitude; x and y are the coordinates of the image points on the strip image in millimeters; f is the camera focal length in millimeters. a1, a2, a3, b1, b2, b3, c1, c2, and c3 are calculated from the second formula. In the second formula, ω and κ represent the camera's pitch, roll, and yaw angles, respectively.

[0039] Determine the translation amount

[0040] Generally, after projection transformation, images only exhibit translation relationships. Considering that latitude and longitude sampling errors affect the accuracy of translation calculation, this paper uses the SIFT algorithm to determine the translation relationship between adjacent images. First, the SIFT algorithm is used to sequentially extract the t-th image from adjacent sequences. i Image and t j Feature points of the image; then, using the matching point pairs coordinates (r ik ,c ik ) and (r jk ,c jk ) difference mean, calculate t i and t j The number of rows and columns parallel in the time-strip image (R) ij C ij ), as shown in the following formula

[0041]

[0042] For example, local SIFT matching points between two adjacent multispectral strip images, such as Figure 7 As shown. The calculated row and column parallelism of the strip image is as follows.Figure 8 R in 12 C 12 R 23 C 23 R 34 C 34 ..., thus determining the spatial positional relationships of the striped image.

[0043] 2. Grayscale adjustment of each band image

[0044] When stitching together multiple images, the intermediate image is generally chosen as the reference to reduce cumulative errors. Since multispectral images of a filter array are also created by adjusting the grayscale of multiple images and stitching them together, the intermediate sequence of multispectral images is selected as the reference.

[0045] Image grayscale adjustment primarily utilizes the average grayscale value of the overlapping regions in striped images. This is achieved by calculating the average grayscale value of the overlapping regions in each band of adjacent multispectral images, and then averaging the proportions of these average grayscale values ​​for each band to obtain the multispectral image grayscale adjustment coefficient. For non-adjacent images, grayscale adjustment is performed sequentially through grayscale transfer between adjacent images. The key technologies involved mainly include three aspects: determining the overlapping regions of each striped image, determining the image grayscale adjustment coefficient, and image grayscale adjustment itself.

[0046] Taking an 8-band filter array multispectral camera as an example, the overlapping regions of each band image are determined as follows: Figure 1 As shown in (a). The transition area of ​​the band image is approximately 50 rows, see... Figure 1 (a) The region within the frame. Therefore, it is necessary to calculate the coordinates of the four vertices of the effective region for each band and construct a template for the effective region of the band image, such as... Figure 1 As shown in (b).

[0047] Taking the first band image from two adjacent multispectral images as an example, template 1 and template 2 are expanded according to the translation relationship between the images, as follows: Figure 2 As shown in (a) and (b). Then multiply the expanded template 1 and template 2 together, as follows: Figure 2 As shown in (c); the overlapping region template is obtained as follows: Figure 2 As shown in (d); the expanded template is then split into the first and second images of the original template 1 for band 1, as shown in (d). Figure 2 As shown in (e) and (f).

[0048] Then, template 1 is multiplied by the first band image from each of the two multispectral images to obtain the overlapping region image, as shown below. Figure 3 As shown. Figure 3 The two images above are the first and second band images, respectively. Figure 3 The two images below are the overlapping areas of the first band in the first and second images, respectively.

[0049] Calculate the average gray value of the image overlap area

[0050] Based on the single-band image overlap area, the average gray value of the image and the gray coefficient relative to the adjacent image can be calculated. However, using the single-band image gray coefficient to adjust the image gray value is easy to damage the spectral characteristics of the ground object under different bands. Therefore, the proportional coefficient of the average gray value of each band in the overlap area between adjacent images is calculated first, and then the proportional coefficients of each band are averaged to obtain the gray adjustment coefficient of the entire image (each band).

[0051] Take the adjacent i-th and j-th images in the 8-band filter array multispectral sequence image as an example for adjustment, as shown in Figure 4 .

[0052] First, calculate the average gray values a i1 , a j1 , a i2 , a j2 , a i3 , a j3 , a i4 , a j4 , a i5 , a j5 , a i6 , a j6 , a i7 , a j7 , a i8 , a j8 of the 1st, 2nd, 3rd, 4th, 5th, 6th, 7th, and 8th band overlap areas in the i-th and j-th images, and then calculate the gray ratios c1, c2, c3, c4, c5, c6, c7, and c8 of the 8-band overlap areas. The gray adjustment coefficient g ij is:

[0053]

[0054] g ij is the gray adjustment coefficient of the j-th image. The gray value of the adjusted image overlap area is basically consistent with that of the i-th image.

[0055] Image gray value adjustment

[0056] After the image gray value adjustment coefficient g ij is obtained, the image can be adjusted. For example, taking 9 multispectral strip images as an example, the image translation relationship and the overlap area gray value diagram are shown in Figure 5 .

[0057] Select the 5th image in the middle as the reference, multiply each gray adjustment coefficient by the gray value of each pixel in the corresponding sequence image in turn, and the calculation formula of each gray adjustment coefficient is as follows:

[0058]

[0059] Image cropping and splicing

[0060] By cropping each single-band image of the gray-adjusted sequence images and splicing according to the translation relationship between images, a filter array multi-spectral single-band image with consistent gray is obtained, as shown in Figure 6 .

[0061] Experiment and analysis

[0062] In order to verify the feasibility of the algorithm proposed in the present application, multi-spectral image data of a certain region taken by a filter array multi-spectral camera of a certain aerial unmanned aerial vehicle was used for verification experiment. Among them, the first 6 sequence multi-spectral images are as shown in Figure 9 .

[0063] The cropping and splicing of the first band image after gray adjustment in the present paper is as shown in Figure 10 . It can be clearly seen from Figure 10 (a) that the single-band image directly spliced from the multi-spectral image data has uneven overall gray, and there is a large gray difference for the same ground object. Figure 10 The image gray shown in (b) is relatively uniform, which shows that the method in the present paper solves the problem of uneven strip gray.

[0064] In order to better illustrate the performance of the algorithm in the present paper, the gray consistency correction algorithm of filter type multi-spectral data (literature 12, Fang Xiuxiu, Huang Min, Wang Dezhi, et al. Multi-spectral image preprocessing algorithm based on elevation and ground object spectrum constraint [J]. Semiconductor Optoelectronics, 2020, 41(02): 264-267+272.) and the gray adjustment algorithm proposed in the present application were selected for comparison experiment. The single-band images processed by literature 12 and the algorithm in the present paper are as shown in Figure 11 . The comparison images are 1, 3, 6, 7 and 8 band images, among which Figure 11 the left column, the middle column and the right column are images without gray adjustment, processed by literature 12 and the algorithm in the present paper respectively.

[0065] As can be seen from Figure 11 , the 1st band and 8th band images processed by literature 12 better solve the problem of uneven image gray, but the overlapping area of the same band in adjacent time strip images is small, the number of extracted same-named pixels is small, which easily causes the inaccuracy of the gray adjustment coefficient, so that there is still a problem of uneven ground object gray in the 3rd, 6th, 7th band images, and the gray difference of part of the strips in the image is more obvious than that of the unadjusted image. The local enlarged view is as shown in Figure 12 , wherein Figure 12The leftmost column, Figure 12 The middle column and Figure 12 The middle and rear columns show the images of bands 3, 6, and 7 after grayscale adjustment, processing according to reference 12, and processing according to the algorithm of this invention.

[0066] Depend on Figure 12 It is evident that the grayscale inconsistency is more pronounced in the images processed in Reference 12. Furthermore, grayscale adjustment cannot be performed on bands 4 and 5 because no corresponding image points were extracted in the overlapping region. Therefore, when the number of matching points in the overlapping region is small, the grayscale linear transformation model calculated based on the grayscale of the matching points has a significant deviation, increasing the grayscale difference between adjacent strip images within a single band. It can even lead to a situation where grayscale adjustment is impossible due to the inability to extract corresponding image points.

[0067] Therefore, when the overlap rate of adjacent strip images is low, the number of matching points in the overlapping area is small, and the distribution of these points is uneven, the algorithm in Reference 12 cannot solve the problem of inconsistent gray levels in large images, and may even increase the gray level differences of ground objects in the same area of ​​the image. However, the algorithm in this paper, after adjustment, achieves a uniform gray level distribution in each single-band image, thus effectively solving the problem of inconsistent gray levels in multispectral single-band images.

[0068] To further illustrate the superiority of the algorithm presented in this paper, the difference in the average grayscale value of the overlapping region of each single-band strip image after adjustment is calculated. Figure 13 As shown (only images of bands 1, 3, and 7 are displayed).

[0069] Figure 13 The vertical axis represents the average grayscale difference of the overlapping area, and the horizontal axis represents the image sequence number. For example... Figure 13 When the first image on the left is numbered 1, the grayscale difference in this algorithm is 0.0017, which is obtained by calculating the average grayscale difference in the overlapping area of ​​the first band in the first and second images. The method is the same for other bands. Figure 13 It can be seen that the difference in the mean gray level of the overlapping region of the single-band image after processing in Reference 12 changes significantly overall, indicating that there are still large gray level differences in some areas of the image after adjustment. In contrast, in the algorithm of this paper, the difference in the mean gray level of the overlapping region is small, approximately a straight line, indicating that the gray level changes of adjacent strip images are small, and the image gray levels are approximately consistent.

[0070] Therefore, the adjustment coefficient calculated based on the average gray value of the overlapping area of ​​adjacent images in this paper makes full use of the gray value information of the overlapping area image. After adjustment, the gray values ​​of each single-band image are uniform, which effectively solves the problem of inconsistent gray values ​​in single-band images.

Claims

1. An aerial filter array multispectral image strip gray scale adjustment method, characterized in that: The steps are: S1, determining the overlapping area of each band image Determine the four vertex coordinates of the effective area of each band strip, and construct a band image effective area template. According to the translation relationship between the images, template 1 and template 2 are expanded respectively, then the expanded template 1 and template 2 are multiplied to obtain the overlapping area template, and then the expanded template is split into the first and second band original templates 1. Then, template 1 is multiplied with the first band images in the two multi-spectral images respectively to obtain the overlapping area images; S2, average gray value of image overlapping area Based on the single-band image overlapping area, the average gray value of the image and the gray coefficient relative to the adjacent image can be calculated. First, the proportional coefficient of the average gray value of each band overlapping area between adjacent images is obtained, and then the proportional coefficients of each band are averaged to obtain the gray adjustment coefficient of the whole image The i-th and j-th adjacent images are adjusted: First, the average gray value a of the 1st, 2nd, 3rd, 4th, 5th, 6th, 7th, 8th band overlapping area in the i, j image is obtained i1 , a j1 , a i2 , a j2 , a i3 , a j3 , a i4 , a j4 , a i5 , a j5 , a i6 , a j6 , a i7 , a j7 , a i8 , a j8 Then the gray ratio c1, c2, c3, c4, c5, c6, c7, c8 of the 8 band overlapping areas is obtained, and the gray adjustment coefficient g ij is: g ij As the gray scale adjustment factor of the jth image, the gray scale of the overlapped region of the adjusted image is consistent with the gray scale of the ith image. S3, image gray adjustment The image gray scale adjustment coefficient g is obtained ij After that, the image gray scale is adjusted, the fifth image in the middle is selected as the reference, each gray scale adjustment coefficient is multiplied by the gray scale of each pixel in the corresponding sequence image in turn, and the calculation formula of each gray scale adjustment coefficient is as follows:

Images