A same-named point-based strip image fusion method and device and electronic equipment

Abstract

The application belongs to the technical field of image processing, and discloses a flight strip image fusion method and device based on homonym points and electronic equipment. A first flight strip image and a second flight strip image are acquired. The first flight strip image and the second flight strip image are subjected to feature matching to obtain a plurality of feature point matching pairs as initial homonym point pairs. The initial homonym point pairs are screened based on the attributes of a plurality of homonym point connecting lines to obtain first homonym point pairs, and second homonym point pairs are obtained in response to a screening operation of a target user on the first homonym point pairs. The first flight strip image and the first flight strip image are subjected to blocking to obtain first and second blocked images. The second blocked image is subjected to resampling. The second blocked image and the first blocked image are subjected to image fusion based on the horizontal offset of each row of pixels. The accurate homonym point pairs are obtained through secondary screening, and the image fusion method is realized without relying on the quality and content of the flight strip image through resampling.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a flight strip image fusion method and device based on homonym points and electronic equipment. BACKGROUND

[0002] Image splicing technology refers to a technology of splicing two or more images containing overlapping parts into a high-resolution, wide-view image containing information of each image through image preprocessing, image registration and image fusion technology.

[0003] The flight strip image is a sequence of images obtained by aerial photography in a certain direction, which are mutually overlapped in front and back, and the sequence of images has important application in surveying and mapping and geographic information collection. In related technologies, when the flight strip image is fused, the image quality and image content of the flight strip image are usually relied on. If the image quality of the flight strip image is low or the feature points are missing in the flight strip image, the image fusion will fail. SUMMARY

[0004] The purpose of the present application is to obtain accurate homonym point pairs in adjacent flight strip images by using a secondary screening method, and to cope with the possible squeezing or stretching in the flight strip image by resampling, thereby realizing an image fusion method independent of the quality and content of the flight strip image.

[0005] In a first aspect, an embodiment of the present application provides a flight strip image fusion method based on homonym points, which comprises:

[0006] Obtaining a first flight strip image and a second flight strip image, the first flight strip image and the second flight strip image being images corresponding to adjacent flight strips in a bow-tie flight process of a UAV;

[0007] Performing feature matching on the first flight strip image and the second flight strip image to obtain a plurality of feature point matching pairs as initial homonym point pairs, and connecting two homonym points included in each initial homonym point pair to obtain a plurality of homonym point connecting lines;

[0008] Screening the initial homonym point pairs based on the attributes of the plurality of homonym point connecting lines to obtain first homonym point pairs, and in response to a screening operation of a target user on the first homonym point pairs, adding and / or deleting the first homonym point pairs to obtain second homonym point pairs;

[0009] For each second homonym point pair, performing block division on the first flight strip image based on the row where the first homonym point included in the second homonym point pair is located to obtain a plurality of first block images, and performing block division on the second flight strip image based on the row where the second homonym point included in the second homonym point pair is located to obtain a plurality of second block images;

[0010] Using each first block image as a reference, the second block image corresponding to the first block image is resampled to obtain the resampled second block image, wherein the resampled second block image has the same size as the first block image;

[0011] For each second block image, based on the second horizontal coordinate of the second corresponding point in the second block image and the first horizontal coordinate of the first corresponding point in the first block image corresponding to the second block image, the horizontal offset of each row of pixels in the second block image is calculated, and the second block image and the first block image are fused based on the horizontal offset of each row of pixels.

[0012] Optionally, the step of filtering the initial pairs of corresponding points based on the attributes of the obtained multiple lines connecting corresponding points to obtain the first pair of corresponding points includes:

[0013] Obtain the vertical mapping line segments and the horizontal mapping line segments of the multiple lines connecting the points with the same name, respectively.

[0014] The vertically mapped line segment will be compared with the preset segment length. If the vertically mapped line segment is greater than the preset segment length, the corresponding initial pair of points with the same name will be filtered out.

[0015] The horizontally mapped line segment is compared with the image width of a preset ratio. If the horizontally mapped line segment is smaller than the image width of the preset ratio, the corresponding initial pair of identical points is filtered out.

[0016] The length of the lines connecting the multiple points with the same name is compared with the image width. If the length of the line connecting the points with the same name is greater than the image width, the corresponding initial pairs of points with the same name are filtered out. The image width is the first width of the first flight strip image or the second width of the second flight strip image, and the first width and the second width are the same.

[0017] Optionally, the method further includes:

[0018] Determine whether there are multiple first-named points within a preset number of pixels along the vertical direction of the first flight strip image. If multiple first-named points exist, retain only one target first-named point and delete the other first-named points.

[0019] Delete the second corresponding point in the second flight strip image that matches the first corresponding point of the target.

[0020] Optionally, the method further includes:

[0021] The first flight strip image, the second flight strip image, and the first corresponding point pair are displayed in the visualization display interface;

[0022] Accordingly, in response to the target user's filtering operation on the first pair of identically named points, the first pair of identically named points is added and / or deleted to obtain a second pair of identically named points, including:

[0023] The system receives filtering operations input by the target user through a visual interface. The filtering operations include deleting incorrect homonyms in the first homonym pair and / or adding correct homonyms to the first homonym pair.

[0024] If the filtering operation is to delete the incorrect same-name point pairs in the first same-name point pair, respond to the filtering operation and delete the incorrect same-name point pairs in the first same-name point pair;

[0025] If the filtering operation is to add the correct corresponding point pair to the first corresponding point pair, then in response to the filtering operation, add the correct corresponding point pair to the first corresponding point pair.

[0026] Optionally, for each second segmented image, calculating the lateral offset of each row of pixels in the second segmented image based on the second abscissa of the second corresponding point in the second segmented image and the first abscissa of the first corresponding point in the first segmented image corresponding to the second segmented image includes:

[0027] For each second block image, based on the second abscissa of each second corresponding point in the second block image, the first abscissa of the first corresponding point in the first block image corresponding to each second corresponding point, and the image width, the horizontal offset of the row pixel of each second corresponding point is calculated.

[0028] For any row of pixels between two vertically adjacent second-named points in the second block image, the horizontal offset of the row of pixels is calculated based on the horizontal offset of the row of pixels in the row of the two vertically adjacent second-named points, the row number of the row of pixels, and the height of the image between the two vertically adjacent second-named points.

[0029] Optionally, the image fusion of the second block image and the first block image based on the horizontal offset of each row of pixels includes:

[0030] Based on the lateral offset of each row of pixels, in the first block image, a first pixel that overlaps with each second pixel in the second block image is determined;

[0031] For each second pixel in the second segmented image, the pixel value of the second pixel is linearly weighted and summed with the pixel value of the first pixel that overlaps with the second pixel.

[0032] Secondly, embodiments of the present invention provide a flight strip image fusion device based on corresponding points, the device comprising:

[0033] The flight strip image acquisition module is used to acquire a first flight strip image and a second flight strip image, wherein the first flight strip image and the second flight strip image are images corresponding to adjacent flight strips during the bow-shaped flight of the UAV;

[0034] The corresponding point pair determination module is used to perform feature matching on the first flight strip image and the second flight strip image to obtain multiple feature point matching pairs as initial corresponding point pairs, and to connect the two corresponding points included in each initial corresponding point pair to obtain multiple corresponding point connections;

[0035] The same-name point pair filtering module is used to filter the initial same-name point pairs based on the attributes of the obtained multiple same-name point connection lines to obtain the first same-name point pair, and respond to the target user's filtering operation on the first same-name point pair by adding and / or deleting the first same-name point pair to obtain the second same-name point pair.

[0036] The flight strip image segmentation module is used to segment the first flight strip image into multiple first segmented images based on the row containing the first corresponding points included in the second corresponding point pair for each second corresponding point pair, and to segment the second flight strip image into multiple second segmented images based on the row containing the second corresponding points included in the second corresponding point pair.

[0037] The image block resampling module is used to resample the second block image corresponding to each first block image as a reference to obtain the resampled second block image, wherein the resampled second block image has the same size as the first block image.

[0038] The flight strip image fusion module is used to calculate the lateral offset of each row of pixels in the second block image based on the second abscissa of the second corresponding point in the second block image and the first abscissa of the first corresponding point in the first block image corresponding to the second block image, and to perform image fusion of the second block image and the first block image based on the lateral offset of each row of pixels.

[0039] Thirdly, embodiments of the present invention provide an electronic device, comprising:

[0040] At least one processor;

[0041] Memory for storing the at least one processor-executable instruction;

[0042] The at least one processor is configured to execute the instructions to implement the method described in the first aspect.

[0043] Fourthly, embodiments of the present invention provide a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the first aspect.

[0044] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0045] The technical solution provided by the embodiments of the present invention filters the initial homonymous point pairs through a secondary screening method, thereby accurately obtaining homonymous point pairs of the flight strip image to be stitched. Furthermore, by resampling the image to be stitched, it can address possible compression or stretching in the flight strip image, thus achieving flight strip image fusion that does not depend on the quality and content of the flight strip image. Attached Figure Description

[0046] Figure 1 A flowchart illustrating a flight strip image fusion method based on corresponding points provided in an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of a flight strip image fusion device based on corresponding points provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] The present invention will be described in detail below through embodiments.

[0050] Image stitching technology refers to the technique of stitching together two or more images with overlapping parts into a single high-resolution, wide-view image that contains information from each image through image preprocessing, image registration, and image fusion.

[0051] Flight strip images are sequences of overlapping images acquired during aerial photography along a specific direction. These image sequences have important applications in surveying and geographic information acquisition. In related technologies, the fusion of flight strip images typically relies on their image quality and content. If the image quality of the flight strip images is low or they lack feature points, the image fusion will fail.

[0052] To address the aforementioned technical problems, embodiments of the present invention provide a general image fusion method that is independent of the quality and content of flight strip images. The flight strip image fusion method provided by these embodiments can be applied to UAV bow-shaped flight scenarios, where the flight strip images are linear array pushbroom imaging data.

[0053] The overall technical solution of this invention may include the following steps:

[0054] 1. Acquire adjacent flight strip images, and use segmented feature detection and matching methods between adjacent flight strip images to obtain multiple feature point matching pairs as initial corresponding point pairs, wherein the width of adjacent flight strip images is the same.

[0055] 2. For each initial pair of corresponding points, connect the two corresponding points included in the initial pair to obtain the corresponding point connection line. Perform an initial screening of the initial pair of corresponding points based on the position, length, angle and other attributes of the corresponding point connection line to obtain the first pair of corresponding points after the initial screening. The adjacent flight strip images and the corresponding point connection line of the first pair of corresponding points after the initial screening can be displayed in the visualization display interface.

[0056] 3. Users can input filtering operations for the first pair of identical points in the visual display interface to obtain the second pair of identical points after secondary filtering; the filtering operations include deleting incorrect identical points in the first pair of identical points and adding correct identical points in the first pair of identical points.

[0057] 4. After obtaining the second pair of corresponding points after secondary filtering, the corresponding points included in the first flight strip image can be called the first corresponding points, and the corresponding points included in the second flight strip image can be called the second corresponding points. The first flight strip image can be divided into blocks based on the rows containing the first corresponding points, resulting in multiple first image blocks. Similarly, the second flight strip image can be divided into blocks based on the rows containing the second corresponding points, resulting in multiple second image blocks.

[0058] It is understandable that, since there is a one-to-one correspondence between the first and second corresponding points, there is a one-to-one correspondence between the first image block and the second image block.

[0059] 5. Using each first image block as a reference, resample the second image block corresponding to the first image block so that the size of the resampled second image block is the same as the size of the first image block, thus aligning the resampled second image block with the first image block.

[0060] 6. After aligning each second image block with the corresponding first image block through resampling, the horizontal offset of each row of pixels in each second image block is calculated, and image fusion is performed on the overlapping parts of the second image block and the first image block based on the calculated horizontal offset of each row of pixels.

[0061] As can be seen, the present invention uses a secondary screening method to obtain accurate pairs of corresponding points in adjacent flight strip images, and uses resampling to deal with possible compression or stretching in the flight strip images, thus realizing an image fusion method that does not depend on the quality and content of the flight strip images.

[0062] The following will describe in detail a flight strip image fusion method based on corresponding points provided by an embodiment of the present invention.

[0063] like Figure 1 As shown in the figure, an embodiment of the present invention provides a flight strip image fusion method based on corresponding points, which may include the following steps:

[0064] S110, acquire the first flight strip image and the second flight strip image.

[0065] Among them, the first flight strip image and the second flight strip image are images corresponding to adjacent flight strips during the UAV's bow-shaped flight process.

[0066] Specifically, the start and end row numbers of each flight strip can be obtained manually or through GPS trajectory analysis, and the flight strip images can then be extracted. The raw image data collected by the UAV linear array pushbroom consists of rows of images arranged vertically. Therefore, when collecting data in a bow-shaped flight pattern, it is necessary to extract the flight strip images based on the start and end row numbers of each flight strip. The start and end row numbers can be manually specified, or, if real-time GPS information is available during the flight strip image acquisition process, the start and end row numbers can be obtained by analyzing the flight strip trajectory.

[0067] In practical applications, multiple flight strip images are typically captured. When fusing flight strip images, adjacent flight strip images are usually fused step by step. For example, assuming there are 3 flight strip images, the first flight strip image can be fused with the second flight strip image to obtain a fused flight strip image, and then the fused flight strip image can be fused with the third flight strip image in the same way.

[0068] The embodiments of the present invention can be applied to the application scenario of UAVs flying in a bow shape. By acquiring the first flight strip image and the second flight strip image corresponding to any adjacent flight strips during the UAV's bow-shaped flight, the first flight strip image and the second flight strip image are fused together.

[0069] S120, perform feature matching on the first flight strip image and the second flight strip image to obtain multiple feature point matching pairs, which are used as initial homonymous point pairs. Then, connect the two homonymous points included in each initial homonymous point pair to obtain multiple homonymous point lines.

[0070] Specifically, after acquiring the first flight strip image and the second flight strip image, feature matching is performed on the first flight strip image and the second flight strip image to obtain multiple feature point matching pairs, which serve as initial corresponding point pairs.

[0071] Since there may be erroneous feature point matching pairs among multiple feature point matching pairs, there may also be erroneous homonymous point pairs among the initial homonymous point pairs. Therefore, it is necessary to connect the two homonymous points included in each initial homonymous point pair to obtain multiple homonymous point connections. In the following steps, the initial homonymous point pairs can be filtered by the attributes of multiple homonymous point connections.

[0072] S130: Based on the attributes of the obtained multiple lines connecting the same points, filter the initial pairs of points to obtain the first pair of points, and respond to the target user's filtering operation on the first pair of points by adding and / or deleting the first pair of points to obtain the second pair of points.

[0073] Specifically, the attributes of lines connecting corresponding points can include the position, length, and angle of the line. Since in practice, multiple lines connecting corresponding points are generally of similar length and angle, pairs of corresponding points connected by lines of similar length and angle will not be filtered out. Pairs of corresponding points whose length and angle are significantly different from other lines will be filtered out.

[0074] Furthermore, pairs of corresponding points can be filtered based on the mapping line segments of the lines connecting them in the longitudinal and transverse directions, respectively. They can also be filtered based on the density of corresponding points in the longitudinal direction of the flight strip image. For clarity, this will be explained in detail in the following embodiments.

[0075] To improve the accuracy of identical point pairs, the initial identical point pairs are filtered based on the attributes of the lines connecting the identical points. After obtaining the first identical point pair, a second filtering can be performed on the first identical point pair.

[0076] As one embodiment of the present invention, the method may further include: displaying the first flight strip image, the second flight strip image, and the first corresponding point pair in a visualization display interface.

[0077] Accordingly, in response to the target user's filtering operation on the first pair of identically named points, adding and / or deleting the first pair of identically named points to obtain the second pair of identically named points may include the following steps:

[0078] The system receives filtering operations input by the target user through a visual interface. The filtering operations include deleting incorrect homonyms in the first homonym pair and / or adding correct homonyms to the first homonym pair.

[0079] If the filtering operation is to delete incorrect pairs of points in the first pair of points with the same name, respond to the filtering operation and delete the incorrect pairs of points with the same name in the first pair of points with the same name.

[0080] If the filtering operation is to add the correct corresponding point pair to the first pair of corresponding points, then respond to the filtering operation by adding the correct corresponding point pair to the first pair of corresponding points.

[0081] Specifically, the first flight strip image, the second flight strip image, and the first pair of corresponding points can be displayed in a visualization interface. From this interface, it can be seen that correct corresponding point connections have similar lengths, angles, and other attributes, clearly identifying incorrect corresponding point pairs. Users can input the operation to delete incorrect corresponding point pairs in the visualization interface, thus achieving the deletion of erroneous corresponding point pairs. Furthermore, for sparse areas of corresponding point pairs, users can manually specify correct corresponding point pairs and add new ones. The corresponding point pairs after this secondary filtering are referred to as the second pair of corresponding points. By performing a secondary filtering on the initial corresponding point pairs, the accuracy of the final pair of corresponding points can be guaranteed.

[0082] S140, for each second pair of corresponding points, the first flight strip image is divided into blocks based on the row containing the first corresponding points included in the second pair of corresponding points to obtain multiple first block images, and the second flight strip image is divided into blocks based on the row containing the second corresponding points included in the second pair of corresponding points to obtain multiple second block images.

[0083] Specifically, after obtaining the final pairs of corresponding points, each pair includes two corresponding points. The corresponding point located in the first flight strip image is called the first corresponding point, and the corresponding point located in the second flight strip image is called the second corresponding point. Assume the ordinates of the first corresponding points are Ay1, Ay2, Ay3, ..., AyN, where N is the total number of corresponding points in the first flight strip image, and N is a positive integer. Assume the ordinates of the second corresponding points are By1, By2, By3, ..., ByN. After obtaining the positions of the N first corresponding points in the first flight strip image, the first flight strip image can be divided into blocks according to the row where the ordinates of the first corresponding points are located, resulting in multiple first block images. After obtaining the positions of the N second corresponding points in the second flight strip image, the second flight strip image can be divided into blocks according to the row where the ordinates of the second corresponding points are located, resulting in multiple second block images.

[0084] It is understandable that the first segmented image and the second segmented image have a one-to-one correspondence. For example, the first segmented image obtained by dividing the first flight zone image by the row where Ay1 is located and the row where Ay2 is located corresponds to the second segmented image obtained by dividing the second flight zone image by the row where By1 is located and the row where By2 is located.

[0085] S150, using each first block image as a reference, resample the second block image corresponding to the first block image to obtain the resampled second block image.

[0086] The second segment image after resampling has the same size as the first segment image.

[0087] Because linear pushbroom imaging may involve compression and stretching, the differences between [Ay1, Ay2, Ay3, ... AyN] and [By1, By2, By3, ... ByN] are not entirely equal. Therefore, the first block image of the first flight zone image can be used as a reference to resample the second block image corresponding to the second flight zone image, ensuring that the resampled second block image is the same size as the first block image. For example, assuming the first block image has 100 pixels vertically and the second block image has 90 pixels vertically, resampling can be used to make the number of pixels vertically in the second block image also 100, thus ensuring that the resampled second block image is the same size as the first block image. It is understandable that the ordinates of each second corresponding point in the resampled second block image also change.

[0088] S160, for each second block image, based on the second abscissa of the second corresponding point in the second block image and the first abscissa of the first corresponding point in the first block image corresponding to the second block image, calculate the lateral offset of each row of pixels in the second block image, and perform image fusion between the second block image and the first block image based on the lateral offset of each row of pixels.

[0089] As one implementation of this invention, in step S160, for each second block image, based on the second abscissa of the second corresponding point in the second block image and the first abscissa of the first corresponding point in the first block image corresponding to the second block image, the horizontal offset of each row of pixels in the second block image can be calculated, which may include the following two steps:

[0090] The first step is to calculate the horizontal offset of the row pixels of each second corresponding point in the second block image based on the second horizontal coordinate of each second corresponding point in the second block image, the first horizontal coordinate of the first corresponding point in the first block image corresponding to each second corresponding point, and the image width.

[0091] The second step is to calculate the horizontal offset of a row of pixels between two vertically adjacent second-named points in the second block image, based on the horizontal offset of the row of pixels in the row of the two vertically adjacent second-named points, the row number of the row of pixels, and the height of the image between the two vertically adjacent second-named points.

[0092] Specifically, the x-coordinates of the second corresponding points in the resampled second block image are Bx1', Bx2', Bx3', ..., BxN', corresponding to the x-coordinates of the first corresponding points in the first block image are Ax1, Ax2, Ax3, ... . Assuming the stitching direction of the resampled second block image is the right side of the first block image, for the row pixels of the second block image containing Bx1', the horizontal offset for alignment with the first block image is w + Bx1' - Ax1, denoted as offset_1, where w is the width of the first or second flight strip image. For the row pixels of the Bx2' row, the horizontal offset is w + Bx2' - Ax2, denoted as offset_2. Similarly, offset_1 and offset_2 are not equal when the flight strip image quality is generally poor. Therefore, for the row pixels between Bx1' and Bx2', the formula for calculating the horizontal offset_i is:

[0093] offset_i=round(offset_1+(offset_2-offset_1) / h*i).

[0094] i is the row index between Bx1' and Bx2', h is the image height between Bx1' and Bx2', and round() is for rounding to the nearest integer.

[0095] After calculating the lateral offset of the row pixels in each second block image, image fusion can be performed on the overlapping parts of each second block image and its corresponding first block image based on the lateral offset of the row pixels. By fusing adjacent flight strip images for all flight strips, the fusion of all flight strip images can be achieved.

[0096] As one implementation of this invention, image fusion of the second block image and the first block image based on the horizontal offset of each row of pixels may include the following steps:

[0097] Based on the lateral offset of each row of pixels, the first pixel point that overlaps with each second pixel point in the second block image is determined in the first block image.

[0098] For each second pixel in the second block image, the pixel value of the second pixel is linearly weighted and summed with the pixel value of the first pixel that overlaps with the second pixel.

[0099] Specifically, after determining the horizontal and vertical offsets of each row of pixels in the second flight strip image, a linear weighted fusion is performed on the overlapping portion of the first and second flight strip images. The weight of the overlapping portion decreases linearly from 1 to 0 from near the row center to the edge. Of course, the weight of the overlapping portion can also be determined according to the actual situation, and this embodiment of the invention does not impose specific limitations on it.

[0100] The technical solution provided by the embodiments of the present invention filters the initial homonymous point pairs through a secondary screening method, thereby accurately obtaining homonymous point pairs of the flight strip image to be stitched. Furthermore, by resampling the image to be stitched, it can address possible compression or stretching in the flight strip image, thus achieving flight strip image fusion that does not depend on the quality and content of the flight strip image.

[0101] To ensure clarity, the process of selecting initial points with the same name will be explained in detail below.

[0102] As one embodiment of the present invention, filtering the initial pairs of corresponding points based on the attributes of the obtained multiple lines connecting corresponding points to obtain the first pair of corresponding points may include the following steps:

[0103] Obtain the vertical and horizontal mapping segments of multiple lines connecting points with the same name.

[0104] The vertically mapped line segment will be compared with the preset segment length. If the vertically mapped line segment is greater than the preset segment length, the corresponding initial pair of points with the same name will be filtered out.

[0105] The horizontally mapped line segment will be compared with the image width of the preset ratio. If the horizontally mapped line segment is smaller than the image width of the preset ratio, the corresponding initial pair of points with the same name will be filtered out.

[0106] The length of the line connecting multiple points with the same name is compared with the image width. If the length of the line connecting points with the same name is greater than the image width, the corresponding initial pair of points with the same name is filtered out.

[0107] The image width is either the width of the first flight strip image or the second width of the second flight strip image, with the first width being the same as the second width.

[0108] Specifically, after obtaining the line connecting the corresponding points, the vertical mapping line segment of the line connecting the corresponding points can be obtained, and the vertical mapping line segment is compared with the preset segment length. If the vertical mapping line segment is greater than the preset segment length, it means that the probability of it being an incorrect pair of corresponding points is high. Therefore, the incorrect pairs of corresponding points will be filtered out. The preset segment length can be determined according to the actual situation and is not specifically limited here.

[0109] After obtaining the lines connecting the corresponding points, the horizontal mapping segment of these lines can be obtained. This horizontal mapping segment is then compared to the image width at a preset ratio, which can be 1 / 10. If the horizontal mapping segment is smaller than the preset ratio of the image width, it indicates a high probability of an incorrect pair of corresponding points. Therefore, incorrect pairs of corresponding points are filtered out. Thus, this implementation method can remove incorrect pairs of corresponding points, resulting in more accurate pairs of corresponding points.

[0110] Based on the above embodiments, the method may further include the following steps:

[0111] Determine whether there are multiple first-named points within a preset number of pixels along the vertical direction of the first flight strip image. If multiple first-named points exist, retain only one target first-named point and delete the others.

[0112] Delete the second corresponding point in the second flight strip image that matches the first corresponding point of the target.

[0113] Specifically, to prevent the number of identical point pairs in the flight strip image from being too dense, identical point pairs can be filtered based on the density of identical points in the vertical direction of the flight strip image. For example, only one pair of identical points can be retained every 10 pixels in the vertical direction. This filtering method is to prevent the detected identical point pairs from being too dense.

[0114] To ensure clarity, the technical solution of this invention will be described in detail below with reference to a specific example.

[0115] This solution is suitable for stitching and fusing hyperspectral flight strip images acquired by UAVs in a bow-shaped flight pattern. A specific example is as follows:

[0116] 1. Based on the time record of the original image and the corresponding GPS trajectory, crop out the flight strip image.

[0117] 2. Extract the images of the i-th and i+1-th flight strips, with widths and heights of [w, h1] and [w, h2] respectively. Divide the image of the i+1-th flight strip into image blocks with widths and heights of w (blocks with heights less than w are merged with the previous block). Perform the following operations on each image block:

[0118] First, detect the feature matching point with the i-th flight strip and use it as the initial corresponding point.

[0119] Second, filter pairs of points based on the positional attributes, length attributes, angle attributes, and vertical position attributes of the lines connecting the points, obtain the initial filtered pairs of points, and record the coordinates of the pairs of points.

[0120] 3. Manually screen the pairs of points with the same name after the initial screening. If there are incorrect pairs of points with the same name, delete them and add correct pairs of points with the same name in areas where they are sparse to ensure the image fusion and stitching effect. Finally, sort the pairs of points with the same name after the second screening in ascending order of the vertical coordinate.

[0121] 4. Traverse the ordinates of the corresponding points. Let the difference between the ordinates of the j-th and j+1-th corresponding points in the (i+1)-th flight strip be the height h21. Let the height of the corresponding image block in the i-th flight strip be h11. The image enclosed by the j-th and j+1-th corresponding points in the (i+1)-th flight strip has a width and height of [w, h21]. Resample it to a width and height of [w, h11] and update the ordinates of the corresponding points in the (i+1)-th flight strip.

[0122] 5. Traverse the x-coordinates of the points with the same name, and calculate the offset of each row of pixels in the (i+1)th flight strip image according to the offset calculation formula.

[0123] 6. Adjust the position of the (i+1)th flight strip image row by row according to the calculated offset, and perform linear weighted fusion on the overlapping parts of the flight strip images;

[0124] 7. Repeat steps 2-6 for the remaining flight strips to complete the stitching and fusion of all flight strip images.

[0125] Secondly, embodiments of the present invention provide a flight strip image fusion device 20 based on corresponding points, such as... Figure 2 As shown, the device includes:

[0126] The flight strip image acquisition module 210 is used to acquire a first flight strip image and a second flight strip image, wherein the first flight strip image and the second flight strip image are images corresponding to adjacent flight strips during the bow-shaped flight of the UAV;

[0127] The corresponding point pair determination module 220 is used to perform feature matching on the first flight strip image and the second flight strip image to obtain multiple feature point matching pairs as initial corresponding point pairs, and to connect the two corresponding points included in each initial corresponding point pair to obtain multiple corresponding point connections.

[0128] The same-name point pair filtering module 230 is used to filter the initial same-name point pairs based on the attributes of the obtained multiple same-name point connection lines to obtain the first same-name point pair, and to add and / or delete the first same-name point pair in response to the target user's filtering operation on the first same-name point pair to obtain the second same-name point pair.

[0129] The flight strip image segmentation module 240 is used to segment the first flight strip image into multiple first segmented images based on the row of the first corresponding point included in the second corresponding point pair for each second corresponding point pair, and to segment the second flight strip image into multiple second segmented images based on the row of the second corresponding point included in the second corresponding point pair.

[0130] The image block resampling module 250 is used to resample the second block image corresponding to each first block image based on the first block image to obtain the resampled second block image, wherein the resampled second block image has the same size as the first block image.

[0131] The flight strip image fusion module 260 is used to calculate the lateral offset of each row of pixels in the second block image based on the second abscissa of the second corresponding point in the second block image and the first abscissa of the first corresponding point in the first block image corresponding to the second block image, and to perform image fusion of the second block image and the first block image based on the lateral offset of each row of pixels.

[0132] The technical solution provided by the embodiments of the present invention filters the initial homonymous point pairs through a secondary screening method, thereby accurately obtaining homonymous point pairs of the flight strip image to be stitched. Furthermore, by resampling the image to be stitched, it can address possible compression or stretching in the flight strip image, thus achieving flight strip image fusion that does not depend on the quality and content of the flight strip image.

[0133] Thirdly, embodiments of the present invention provide an electronic device 300, such as... Figure 3 As shown, it includes:

[0134] At least one processor 301;

[0135] Memory 302 for storing the at least one processor-executable instruction;

[0136] The at least one processor is configured to execute the instructions to implement the method described in the embodiments of the present invention.

[0137] The technical solution provided by the embodiments of the present invention filters the initial homonymous point pairs through a secondary screening method, thereby accurately obtaining homonymous point pairs of the flight strip image to be stitched. Furthermore, by resampling the image to be stitched, it can address possible compression or stretching in the flight strip image, thus achieving flight strip image fusion that does not depend on the quality and content of the flight strip image.

[0138] Fourthly, embodiments of the present invention provide a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the methods described in the embodiments of the present invention.

[0139] The technical solution provided by the embodiments of the present invention filters the initial homonymous point pairs through a secondary screening method, thereby accurately obtaining homonymous point pairs of the flight strip image to be stitched. Furthermore, by resampling the image to be stitched, it can address possible compression or stretching in the flight strip image, thus achieving flight strip image fusion that does not depend on the quality and content of the flight strip image.

[0140] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the embodiments of the present invention.

[0141] The technical solution provided by the embodiments of the present invention filters the initial homonymous point pairs through a secondary screening method, thereby accurately obtaining homonymous point pairs of the flight strip image to be stitched. Furthermore, by resampling the image to be stitched, it can address possible compression or stretching in the flight strip image, thus achieving flight strip image fusion that does not depend on the quality and content of the flight strip image.

[0142] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A flight strip image fusion method based on corresponding points, characterized in that, The method includes: Acquire a first flight strip image and a second flight strip image, wherein the first flight strip image and the second flight strip image are images corresponding to adjacent flight strips during the UAV's bow-shaped flight process; The first flight strip image and the second flight strip image are subjected to feature matching to obtain multiple feature point matching pairs, which are used as initial homonymous point pairs. The two homonymous points included in each initial homonymous point pair are connected to obtain multiple homonymous point lines. Based on the attributes of the multiple lines connecting the same points, the initial pairs of points are filtered to obtain the first pair of points. In response to the target user's filtering operation on the first pair of points, the first pair of points is added and / or deleted to obtain the second pair of points. For each second pair of corresponding points, the first flight strip image is divided into blocks based on the row containing the first corresponding points included in the second pair of corresponding points to obtain multiple first block images, and the second flight strip image is divided into blocks based on the row containing the second corresponding points included in the second pair of corresponding points to obtain multiple second block images. Using each first block image as a reference, the second block image corresponding to the first block image is resampled to obtain the resampled second block image, wherein the resampled second block image has the same size as the first block image; For each second block image, based on the second horizontal coordinate of the second corresponding point in the second block image and the first horizontal coordinate of the first corresponding point in the first block image corresponding to the second block image, the horizontal offset of each row of pixels in the second block image is calculated, and the second block image and the first block image are fused based on the horizontal offset of each row of pixels.

2. The method according to claim 1, characterized in that, The process of filtering the initial pairs of corresponding points based on the attributes of the obtained multiple lines connecting corresponding points to obtain the first pair of corresponding points includes: Obtain the vertical mapping line segments and the horizontal mapping line segments of the multiple lines connecting the points with the same name, respectively. The vertically mapped line segment will be compared with the preset segment length. If the vertically mapped line segment is greater than the preset segment length, the corresponding initial pair of points with the same name will be filtered out. The horizontally mapped line segment is compared with the image width of a preset ratio. If the horizontally mapped line segment is smaller than the image width of the preset ratio, the corresponding initial pair of identical points is filtered out. The length of the lines connecting the multiple points with the same name is compared with the image width. If the length of the line connecting the points with the same name is greater than the image width, the corresponding initial pairs of points with the same name are filtered out. The image width is the first width of the first flight strip image or the second width of the second flight strip image, and the first width and the second width are the same.

3. The method according to claim 1, characterized in that, The method further includes: Determine whether there are multiple first-named points within a preset number of pixels along the vertical direction of the first flight strip image. If multiple first-named points exist, retain only one target first-named point and delete the other first-named points. Delete the second corresponding point in the second flight strip image that matches the first corresponding point of the target.

4. The method according to claim 1, characterized in that, The method further includes: The first flight strip image, the second flight strip image, and the first corresponding point pair are displayed in the visualization display interface; Accordingly, in response to the target user's filtering operation on the first pair of identically named points, the first pair of identically named points is added and / or deleted to obtain a second pair of identically named points, including: The system receives filtering operations input by the target user through a visual interface. The filtering operations include deleting incorrect homonyms in the first homonym pair and / or adding correct homonyms to the first homonym pair. If the filtering operation is to delete the incorrect same-name point pairs in the first same-name point pair, respond to the filtering operation and delete the incorrect same-name point pairs in the first same-name point pair; If the filtering operation is to add the correct corresponding point pair to the first corresponding point pair, then in response to the filtering operation, add the correct corresponding point pair to the first corresponding point pair.

5. The method according to any one of claims 1 to 4, characterized in that, For each second image block, based on the second abscissa of the second corresponding point in the second image block and the first abscissa of the first corresponding point in the corresponding first image block, the horizontal offset of each row of pixels in the second image block is calculated, including: For each second block image, based on the second abscissa of each second corresponding point in the second block image, the first abscissa of the first corresponding point in the first block image corresponding to each second corresponding point, and the image width, the horizontal offset of the row pixel of each second corresponding point is calculated. For any row of pixels between two vertically adjacent second-named points in the second block image, the horizontal offset of the row of pixels is calculated based on the horizontal offset of the row of pixels in the row of the two vertically adjacent second-named points, the row number of the row of pixels, and the height of the image between the two vertically adjacent second-named points.

6. The method according to any one of claims 1 to 4, characterized in that, The image fusion of the second block image and the first block image based on the horizontal offset of each row of pixels includes: Based on the lateral offset of each row of pixels, in the first block image, a first pixel that overlaps with each second pixel in the second block image is determined; For each second pixel in the second segmented image, the pixel value of the second pixel is linearly weighted and summed with the pixel value of the first pixel that overlaps with the second pixel.

7. A flight strip image fusion device based on corresponding points, characterized in that, The device includes: The flight strip image acquisition module is used to acquire a first flight strip image and a second flight strip image, wherein the first flight strip image and the second flight strip image are images corresponding to adjacent flight strips during the bow-shaped flight of the UAV; The corresponding point pair determination module is used to perform feature matching on the first flight strip image and the second flight strip image to obtain multiple feature point matching pairs as initial corresponding point pairs, and to connect the two corresponding points included in each initial corresponding point pair to obtain multiple corresponding point connections; The same-name point pair filtering module is used to filter the initial same-name point pairs based on the attributes of the obtained multiple same-name point connection lines to obtain the first same-name point pair, and respond to the target user's filtering operation on the first same-name point pair by adding and / or deleting the first same-name point pair to obtain the second same-name point pair. The flight strip image segmentation module is used to segment the first flight strip image into multiple first segmented images based on the row containing the first corresponding points included in the second corresponding point pair for each second corresponding point pair, and to segment the second flight strip image into multiple second segmented images based on the row containing the second corresponding points included in the second corresponding point pair. The image block resampling module is used to resample the second block image corresponding to each first block image as a reference to obtain the resampled second block image, wherein the resampled second block image has the same size as the first block image. The flight strip image fusion module is used to calculate the lateral offset of each row of pixels in the second block image based on the second abscissa of the second corresponding point in the second block image and the first abscissa of the first corresponding point in the first block image corresponding to the second block image, and to perform image fusion of the second block image and the first block image based on the lateral offset of each row of pixels.

8. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images