Image registration method and device, panoramic stitching method and device, storage medium and electronic equipment

Abstract

The invention discloses an image registration method and device, a panoramic stitching method and device, a storage medium and electronic equipment, and relates to the field of computer vision. The method comprises the following steps: acquiring sub-images shot by an unmanned aerial vehicle, wherein the sub-images are from different shooting directions; a preset image adjacency relation table is inquired, adjacent images corresponding to all the sub-images are determined, and the adjacency relation between all the sub-images is recorded in the preset image adjacency relation table; determining a plurality of adjacent image pairs to be matched according to the adjacent images corresponding to the sub-images; performing feature matching on the plurality of adjacent image pairs to obtain feature matching results of the plurality of adjacent image pairs; and performing image registration on the plurality of sub-images according to the plurality of feature matching results. According to the method, the feature matching process of image registration can be optimized, so that the calculation amount can be reduced, and the image registration efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision, and in particular to an image registration, panoramic stitching method, device, storage medium and electronic device. Background Art

[0002] In recent years, with the development of digital image processing technology, panoramic stitching algorithms have been widely used in various terminal devices. For example, multiple images are taken by handheld camera devices, and the panorama is stitched together by a computer, or a surrounding panoramic image is processed by the acceleration module embedded in the mobile device to create a complete panoramic view. As a device for collecting data at high altitude, drones naturally have the function of taking panoramic photos. Compared with the panoramic stitching algorithm of mobile devices, the panoramic stitching algorithm of drones usually needs to stitch images taken from different directions into a 360° image, which is convenient for comprehensive and accurate data analysis. Therefore, the panoramic stitching algorithm of drones has been widely used in map making, security monitoring and other fields.

[0003] At present, traditional image registration and panoramic stitching methods usually require the captured images to be matched in pairs. However, drones need to capture sub-images in multiple directions, and the number of images is large. If the traditional panoramic stitching method is directly used, the image matching calculation amount will be too large, and the efficiency of image registration and panoramic stitching will be too low, which cannot meet the application requirements. Summary of the invention

[0004] In view of this, the present application provides an image registration, panoramic stitching method, device, storage medium and electronic device, the main purpose of which is to optimize the image matching process of image registration and drone panoramic stitching, thereby reducing the amount of calculation and improving the efficiency of image registration and drone panoramic stitching.

[0005] According to a first aspect of the present application, there is provided an image registration method, the method comprising:

[0006] Acquire each sub-image taken by the drone, wherein each sub-image comes from a different shooting position;

[0007] Querying a preset image adjacency relationship table to determine adjacent images corresponding to each of the sub-images, wherein the preset image adjacency relationship table records the adjacency relationship between the sub-images;

[0008] Determining a plurality of adjacent image pairs to be matched according to the adjacent images corresponding to the respective sub-images;

[0009] Performing feature matching on the plurality of adjacent image pairs respectively to obtain feature matching results of the plurality of adjacent image pairs;

[0010] Image registration is performed on the multiple sub-images according to the multiple feature matching results.

[0011] According to a second aspect of the present application, a panoramic stitching method is provided, the method comprising:

[0012] Acquire a plurality of registered sub-images, wherein the plurality of registered sub-images are determined by an image registration method;

[0013] Determining seams of the registered plurality of sub-images;

[0014] Based on the calculated seams, the registered multiple sub-images are fused to obtain a panoramic image.

[0015] According to a third aspect of the present application, an image registration device is provided, the device comprising:

[0016] An acquisition unit, used to acquire each sub-image taken by the drone, wherein each sub-image comes from a different shooting direction;

[0017] a determining unit, configured to query a preset image adjacency relationship table to determine adjacent images corresponding to each of the sub-images, wherein the preset image adjacency relationship table records the adjacency relationship between the sub-images;

[0018] The determining unit is further configured to determine a plurality of adjacent image pairs to be matched according to the adjacent images corresponding to the respective sub-images;

[0019] A matching unit, used to perform feature matching on the plurality of adjacent image pairs respectively to obtain feature matching results of the plurality of adjacent image pairs;

[0020] A registration unit is used to perform image registration on the multiple sub-images according to the multiple feature matching results.

[0021] According to a fourth aspect of the present application, a panoramic stitching device is provided, the device comprising:

[0022] An acquisition unit, used for acquiring a plurality of registered sub-images, wherein the plurality of registered sub-images are determined by an image registration method;

[0023] A calculation unit, used to determine the seams of the plurality of sub-images after the registration;

[0024] A fusion unit is used to fuse the registered multiple sub-images based on the calculated seams to obtain a panoramic image.

[0025] According to a fifth aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned image registration method is implemented.

[0026] According to a sixth aspect of the present application, an electronic device is provided, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned image registration method when executing the program.

[0027] According to a seventh aspect of the present application, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above-mentioned panoramic stitching method is implemented.

[0028] According to an eighth aspect of the present application, an electronic device is provided, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the above-mentioned panoramic stitching method when executing the program.

[0029] By means of the above technical solution, the present application provides an image registration, panoramic stitching method, device, storage medium and electronic device. Compared with the current prior art, the present application determines the adjacent images corresponding to each sub-image by querying the preset image adjacency table, and can obtain multiple adjacent image pairs to be matched. Then, by performing feature matching on the multiple adjacent image pairs respectively, the registration and panoramic stitching of multiple sub-images can be achieved. Since the present application only performs feature matching on sub-image pairs with adjacency relationships, instead of performing feature matching on all image pairs like the prior art, it can optimize the image matching process of image registration and drone panoramic stitching, simplify the complexity of feature matching, thereby reducing the amount of calculation, improving the efficiency of image registration and drone panoramic stitching, and thus meeting application requirements.

[0030] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0032] Figure 1 A schematic diagram of a process of an image registration method provided in an embodiment of the present application is shown;

[0033] Figure 2 A schematic diagram showing the angle between any two adjacent images provided in an embodiment of the present application is shown;

[0034] Figure 3 A schematic diagram showing the relationship between adjacent sub-images provided in an embodiment of the present application is shown;

[0035] Figure 4 A schematic diagram of a process flow of another image registration method provided in an embodiment of the present application is shown;

[0036] Figure 5 A schematic diagram of a flow chart of a panoramic stitching method provided in an embodiment of the present application is shown;

[0037] Figure 6 A schematic diagram of the structure of an image registration device provided in an embodiment of the present application is shown;

[0038] Figure 7 A schematic diagram showing the structure of another image registration device provided in an embodiment of the present application is shown;

[0039] Figure 8 A schematic structural diagram of a panoramic stitching device provided in an embodiment of the present application is shown;

[0040] Fig. 9 A schematic structural diagram of another panoramic stitching device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0041] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0042] Directly using traditional image registration and panoramic stitching methods will result in excessive image matching calculations, and the efficiency of image registration and panoramic stitching will be too low to meet application requirements.

[0043] In order to solve the above technical problems. This embodiment provides an image registration method, such as Figure 1 As shown, the method includes:

[0044] Step 101: Acquire each sub-image taken by the drone.

[0045] Each sub-image is taken by a camera of a drone, which may be a camera, a video camera, etc. Each sub-image is taken from different shooting positions, each shooting position includes multiple camera positions, and different sub-images correspond to different shooting positions or camera positions.

[0046] The embodiment of the present invention is mainly applicable to the image registration scenario. The execution subject of the embodiment of the present invention is a device or equipment capable of performing image registration.

[0047] For the embodiment of the present invention, during the shooting process, the drone is in a hovering state, and the program controls the posture of the drone body or the gimbal posture so that the camera lens shoots in different surrounding directions, and each direction includes a different camera position. Assuming that the drone shoots images in N surrounding directions, each direction includes three camera positions: upper, middle, and lower, then a total of 3*N sub-images are shot. For example, the drone shoots in 8 horizontal directions, and each direction includes three camera positions: upper, middle, and lower, then a total of 8*3=24 sub-images are shot. When shooting specifically, the shooting can be performed in a predetermined image shooting order, for example, shooting is performed in 8 directions in a horizontal clockwise direction, and the position order when shooting in each direction is upper, middle, and lower. Afterwards, each sub-image shot by the drone is input into a device or equipment for image registration in the shooting order, so that each sub-image can be obtained.

[0048] It should be noted that the number and specific directions of the positions and camera positions in the embodiments of the present invention are not limited to the above examples and can be set according to actual business needs. In addition, in order to facilitate hardware implementation and reduce shooting time, the image shooting order should follow the principle of shooting adjacent images to avoid excessive rotation angles.

[0049] The panoramic stitching algorithm requires that adjacent images have a portion of overlapping viewing angles, so N should not be too large. If the pinhole camera model is used, the angle between two adjacent sub-images is:

[0050]

[0051] Among them, the above angle It needs to be smaller than FOV to ensure that there is a partial overlap of viewing angles between adjacent sub-images, such as Figure 2 As shown, where θ is the field of view FOV.

[0052] Step 102: query a preset image adjacency table to determine the adjacent images corresponding to each sub-image.

[0053] The preset image adjacency relationship table records the adjacency relationship between the sub-images.

[0054] For the embodiments of the present invention, Figure 3As shown, there must be overlapping perspectives between the sub-images taken by the drone. If the two images completely overlap or overlap more, the confidence of their feature matching is usually high; if the two images overlap less or do not overlap at all, the confidence of their feature matching is low. The sub-images with overlapping perspectives are usually sub-images that are adjacent in the shooting order. With the image shooting order as a priori, the embodiment of the present invention can use the first-order neighborhood method to determine which sub-image pairs have overlapping perspectives and which sub-image pairs do not have overlapping perspectives. In the image matching process, the sub-image pairs without overlapping perspectives can be ignored to speed up the image matching process and reduce the amount of calculation.

[0055] Assume that the number of surround positions N=6, and each position includes three pitch positions: upper, middle, and lower. Figure 3 As shown, for sub-image S1, its first-order neighbors are the upper, lower, left, right, upper left, upper right, lower left, and lower right of sub-image S1 itself; for sub-image S2, its first-order neighbors are the lower, left, right, lower left, and lower right of sub-image S2 itself. Thus, the adjacency relationship between each sub-image can be determined according to the predetermined image shooting sequence. In order to facilitate the direct determination of the adjacent images of each sub-image during subsequent feature matching, a preset image adjacency relationship table can be pre-constructed based on the adjacency relationship between each sub-image. The preset image adjacency relationship table records the identification information of each sub-image and its corresponding adjacent image, wherein the identification information of each sub-image can specifically be the number of each sub-image.

[0056] When constructing the preset image adjacency relationship table, each sub-image can be numbered according to a predetermined image shooting order, and the adjacent images corresponding to each sub-image can be determined according to the image shooting order. Finally, the preset image adjacency relationship table can be constructed according to the numbers and adjacent images corresponding to each sub-image. The preset image adjacency relationship table can reflect the adjacency relationship between each sub-image.

[0057] The embodiments of the present invention can improve the search efficiency of adjacent images corresponding to each sub-image by constructing a preset image adjacency relationship table.

[0058] When performing image registration, the preset image adjacency table can be directly queried according to the identification information (such as the serial number) corresponding to each sub-image to determine the adjacent image corresponding to each sub-image.

[0059] For example, for sub-image S1, by querying the preset image adjacency table, it is determined that sub-image S1 has 8 adjacent sub-images; for sub-image S, by querying the preset image adjacency table, it is determined that sub-image S2 has 5 adjacent sub-images.

[0060] Step 103: Determine a plurality of adjacent image pairs to be matched according to the adjacent images corresponding to the respective sub-images.

[0061] In the embodiment of the present invention, after the adjacent images corresponding to each sub-image are determined, each sub-image and its corresponding adjacent image are respectively combined into a plurality of adjacent image pairs. Figure 3 For example, for sub-image S1, 8 adjacent image pairs can be formed with its adjacent images; for sub-image S2, 5 adjacent image pairs can be formed with its adjacent images. Thus, the total number of adjacent image pairs can be calculated as:

[0062] c=5*N*2+8*N=18*N

[0063] Wherein, c is the number of adjacent image pairs that need to be matched in the embodiment of the present invention, and N is the number of surrounding directions.

[0064] If the first-order neighborhood method of the embodiment of the present invention is not used, the number of image pairs that need to be matched is:

[0065] s=N*3*N*3=9*N*N

[0066] Wherein, s is the number of image pairs that need to be matched in the prior art, and N is the number of surrounding orientations.

[0067] It can be seen that the first-order neighborhood method of the embodiment of the present invention can reduce the computational complexity of feature matching and reduce the amount of calculation, and the reduction ratio is:

[0068]

[0069] Among them, ρ1 is the ratio of image pairs that need to be matched that can be reduced by the embodiment of the present invention, c is the number of adjacent image pairs that need to be matched in the embodiment of the present invention, s is the number of image pairs that need to be matched in the prior art, and N is the number of surrounding directions.

[0070] When the number of surround orientations N=8, the reduction ratio can reach 75%. Since the above ratio calculation formula is an inverse proportional function, it means that when N>2, the larger N is, the higher the reduction ratio is.

[0071] Step 104: Perform feature matching on the plurality of adjacent image pairs respectively to obtain feature matching results of the plurality of adjacent image pairs.

[0072] The feature matching result includes the matching feature point pairs in the two sub-images of each adjacent image pair.

[0073] For the embodiment of the present invention, after determining multiple adjacent image pairs to be matched, feature matching is performed on each adjacent image pair respectively to obtain a feature matching result for each adjacent image pair. For this process, step 104 specifically includes: extracting feature points of two sub-images of each adjacent image pair in the multiple adjacent image pairs respectively, and calculating feature descriptors corresponding to the feature points; based on the feature descriptors, matching the feature points in the two sub-images to obtain matching feature point pairs in each adjacent image pair; and determining the feature matching result of each adjacent image pair based on the matching feature point pairs in each adjacent image pair.

[0074] The feature points can be key points such as corner points, edge points and spots. The feature descriptor corresponding to the feature point is a numerical vector used to describe the local image area where a key point in the sub-image is located. It is usually composed of pixels or pixel differences around the feature point, describing the image information around the feature point as much as possible, while retaining the local invariance of the feature point as much as possible.

[0075] Specifically, for the two images of each adjacent image pair, SIFT, SURF, ORB and other algorithms can be used to detect feature points. For each detected feature point, a feature description is required. Specifically, the feature descriptor corresponding to each feature point can be determined by calculating the direction, scale and gradient of the pixels around the feature point. For example, a 128-bit vector is generated using the SIFT descriptor to describe each feature point. After the feature point detection and description of the two images are completed, feature point matching is required. Specifically, feature point matching can be performed based on the feature descriptors corresponding to the feature points through distance measurement methods (such as Euclidean distance and cosine similarity). In this way, the matching feature point pairs in each adjacent image pair can be determined, so that the feature matching results of each adjacent image pair can be determined.

[0076] It should be noted that the embodiment of the present invention may also first extract feature points of all sub-images, then query a preset image adjacency table to determine multiple adjacent image pairs, and then perform feature matching calculations only for the multiple adjacent image pairs.

[0077] Since the embodiment of the present invention only performs feature matching on adjacent image pairs instead of all image pairs as in the prior art, it is possible to optimize the feature matching process of image registration and simplify the complexity of feature matching, thereby reducing the amount of calculation and improving the efficiency of drone panoramic stitching.

[0078] Step 105: perform image registration on the multiple sub-images according to the multiple feature matching results.

[0079] For an embodiment of the present invention, after determining the feature matching results of each adjacent image pair, the correspondence between the matching feature point pairs in each adjacent image pair is converted into a homography matrix, and then the position and posture of the drone camera device is estimated based on the homography matrix corresponding to each adjacent image pair, thereby completing the image registration of multiple sub-images.

[0080] An image registration method provided by an embodiment of the present invention can obtain multiple adjacent image pairs to be matched by querying a preset image adjacency table to determine the adjacent images corresponding to each sub-image, and then perform feature matching on the multiple adjacent image pairs respectively to achieve image registration of the multiple sub-images. Since the embodiment of the present invention only performs feature matching on sub-image pairs with an adjacency relationship, instead of performing feature matching on all image pairs like the prior art, it is possible to optimize the feature matching process of image registration, simplify the complexity of feature matching, thereby reducing the amount of calculation, improving the efficiency of image registration, and thus meeting application requirements.

[0081] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to fully illustrate the implementation of this embodiment, this embodiment also provides another image registration method, such as Figure 4 As shown, the method includes:

[0082] Step 201: Acquire each sub-image taken by the drone.

[0083] The sub-images are from different shooting positions, and each shooting position includes multiple camera positions.

[0084] For the embodiment of the present invention, the images may be shot in a predetermined image shooting order, and each sub-image may be numbered in the image shooting order. For example, the images may be shot in 6 directions in a horizontal clockwise direction, and the order of the directions in each direction is top, middle, and bottom, and a total of 18 sub-images are shot, and each sub-image is numbered 0, 1, ..., 17. Then, each sub-image shot by the drone is input into a device or apparatus for image registration, so that each sub-image can be obtained.

[0085] Step 202: Determine the input order of the sub-images according to the identification information corresponding to the sub-images.

[0086] The identification information corresponding to each sub-image may specifically be the serial number of the sub-image.

[0087] Specifically, according to the numbers carried by each sub-image, such as 0, 1, ..., 17, the input order of each sub-image can be determined.

[0088] Step 203a: If the input sequence is inconsistent with the image shooting sequence of the drone, the image registration is terminated.

[0089] For example, according to the image shooting order of the drone, the input order of normal sub-images should be 0, 1, ..., 17. If the order of the input sub-images is 1, 2, 0, 3, 4, ..., 17, it means that the input order is inconsistent with the image shooting order of the drone, and the image registration is terminated.

[0090] The embodiments of the present invention can significantly reduce the algorithm complexity from the quadratic level to the linear level. However, the premise of all these analyses is to adopt the correct sub-image input order. If the sub-image input order is not the preset order, that is, it is inconsistent with the image shooting order of the drone, the first-order neighborhood method cannot guarantee that the sub-image pairs have overlapping perspectives. Therefore, the embodiments of the present invention can ensure that the input sub-image sequence is in line with expectations through this abnormality detection method.

[0091] Step 203b: If the input sequence is consistent with the image shooting sequence of the drone, query the preset image adjacency table to determine the adjacent images corresponding to each sub-image, and determine multiple adjacent image pairs to be matched based on the adjacent images corresponding to each sub-image.

[0092] The preset image adjacency relationship table records the adjacency relationship between the sub-images.

[0093] For the embodiment of the present invention, if the input order of each sub-image is consistent with the image shooting order of the drone, the feature matching optimization algorithm of the embodiment of the present invention can continue to be executed to determine multiple adjacent image pairs to be matched. The specific process is the same as steps 102 and 103 and will not be repeated here.

[0094] Step 204: Perform feature matching on the plurality of adjacent image pairs respectively to obtain feature matching results of the plurality of adjacent image pairs.

[0095] For the embodiment of the present invention, since the calculation of each sub-image is independent, parallel processing can be performed when extracting the feature points of the sub-images. In addition, since each adjacent image pair is independent of each other, multi-threading can be used to accelerate the matching process.

[0096] Step 205: Calculate homography matrices corresponding to the plurality of adjacent image pairs respectively according to the plurality of feature matching results.

[0097] For the embodiments of the present invention, the RANSAC algorithm can be used to calculate the homography matrix. During the specific calculation, for each adjacent image pair, 4 pairs of matching feature point pairs are randomly selected from the matching feature point pairs, and the corresponding initial homography matrix is ​​calculated. For the remaining matching feature point pairs, their predicted coordinates are calculated according to the initial homography matrix, and the error between the predicted coordinates and the actual coordinates is calculated. If the error is less than a certain threshold, the feature point is added to the inlier set as an inlier, and the above process is repeated to select the initial homography matrix with the most inliers as the homography matrix corresponding to each adjacent image pair.

[0098] Step 206: determining valid adjacent image pairs according to the plurality of feature matching results, and determining whether the number of the valid adjacent image pairs is greater than a preset number of image pairs;

[0099] For the embodiment of the present invention, the matching stage of adjacent image pairs can obtain feature point matching information, homography matrix and matching confidence, but it may not be possible to successfully stitch the panoramic image, such as inclement weather, lens contamination, lens occlusion, etc. In the case of the above situation, it is often not known that the stitching cannot be done until halfway, wasting a lot of user time. Therefore, the embodiment of the present invention provides an abnormality detection method, that is, by determining whether the number of valid adjacent image pairs is greater than the preset number of image pairs, to determine whether the input sub-images can be stitched into a panoramic image.

[0100] When determining the number of preset image pairs, determine the supine sub-images among the sub-images; determine the number of supine image pairs according to the number of supine sub-images; and subtract the number of adjacent image pairs from the number of supine image pairs to obtain the preset number of image pairs.

[0101] Specifically, most sub-images that do not meet the requirements are those that have almost no distinguishable feature points, which is common in bad weather such as haze. Through experiments, it is found that in scenes shot on drones, such images are most likely to appear at the top of the image, that is, the sub-image on the camera device that is tilted upward.

[0102] Furthermore, the specific calculation formula for the number of upward-facing image pairs is:

[0103] c u =5*M

[0104] Where M is the number of upward-facing sub-images, c u is the number of upward image pairs.

[0105] If a complete panoramic image can be assembled, the number of valid adjacent image pairs must be greater than the preset number of image pairs c. v , where the number of preset image pairs c vThe specific calculation formula is as follows:

[0106] c v =cc u =13*M

[0107] Furthermore, when determining a valid adjacent image pair, the number of matching feature point pairs in the two sub-images of any one of the plurality of adjacent image pairs is determined based on the feature matching result of the adjacent image pair; the number of inliers of the homography matrix corresponding to the adjacent image pair is determined; and it is determined whether the number of matching feature point pairs in the adjacent image pair is greater than or equal to a first preset number threshold, and whether the number of inliers is greater than or equal to a second preset number threshold. If the number of matching feature point pairs in the adjacent image pair is greater than or equal to the first preset number threshold, and the number of inliers is greater than or equal to the second preset number threshold, then the adjacent image pair is determined to be valid. If the number of matching feature point pairs in the adjacent image pair is less than the first preset number threshold, or the number of inliers is less than the second preset number threshold, then the adjacent image pair is determined to be invalid.

[0108] The first preset quantity threshold and the second preset quantity threshold may be obtained based on experiments or set based on actual business requirements, and the embodiment of the present invention does not specifically limit this.

[0109] Specifically, after the adjacent image pairs complete feature matching, each adjacent image pair is traversed, and the valid adjacent image pairs are screened out according to the number of matching feature point pairs in the two sub-images of any adjacent image pair and the number of internal points corresponding to the homography matrix, and the number of valid adjacent image pairs is counted.

[0110] The embodiment of the present invention can effectively detect whether each input sub-image meets the requirements. When it is found that each input sub-image does not meet the requirements, the stitching process is terminated in time to avoid invalid shooting and stitching and avoid wasting user time.

[0111] Step 207a: If the number of the valid adjacent image pairs is less than or equal to the preset number of image pairs, it is prompted that the registration fails.

[0112] For the embodiment of the present invention, if the number of valid adjacent image pairs is less than or equal to the preset number of image pairs c v , it is determined that each sub-image cannot be registered and the registration failure is reported.

[0113] Step 207b: If the number of the valid adjacent image pairs is greater than the preset relative number, the position and posture of the drone camera device is estimated according to the homography matrix to complete the image registration of the plurality of adjacent image pairs.

[0114] For the embodiment of the present invention, if the number of valid adjacent image pairs is greater than the preset number of image pairs c v , the subsequent process can be continued, that is, the pose of the drone camera device is estimated according to the homography matrix, thereby completing the pose-based image registration.

[0115] The embodiment of the present invention can detect in advance whether a panoramic image can be completely stitched by determining the number of valid adjacent image pairs, and can know whether the stitching is successful without entering the stage of estimating the posture, thereby avoiding subsequent invalid execution and shortening the user's waiting time.

[0116] Another image method provided by an embodiment of the present invention can obtain multiple adjacent image pairs to be matched by querying a preset image adjacency table to determine the adjacent images corresponding to each sub-image, and then realize image registration of multiple sub-images by performing feature matching on the multiple adjacent image pairs respectively. Since the embodiment of the present invention only performs feature matching on sub-image pairs with adjacency relationships, instead of performing feature matching on all image pairs like the prior art, it is possible to optimize the feature matching process of image registration and simplify the complexity of feature matching, thereby reducing the amount of calculation, improving the efficiency of image registration, and thus meeting application requirements. In addition, the embodiment of the present invention can also perform anomaly detection, so that the algorithm can not only run effectively on drones, but also improve the robustness of the algorithm.

[0117] Furthermore, this embodiment also provides an image registration method, such as Figure 5 As shown, the method includes:

[0118] Step 301: Acquire multiple registered sub-images.

[0119] The registered multiple sub-images are determined by the registration method of the first or second embodiment.

[0120] Step 302: Determine the seams of the registered sub-images.

[0121] According to the embodiment of the present invention, image registration results of multiple adjacent image pairs can be obtained through image registration, and then the image overlapping area of ​​two sub-images of each adjacent image pair is determined according to the image registration results, and then a seam is selected in the middle position of the image overlapping area.

[0122] Step 303: Based on the calculated seams, perform image fusion on the registered multiple sub-images to obtain a panoramic image.

[0123] For the embodiment of the present invention, when stitching the panorama, it is also necessary to perform gain compensation on the multiple registered sub-images, that is, to eliminate the brightness difference in the panoramic image by adjusting the gain value of each sub-image. Then, based on the calculated seams, the compensated multiple sub-images are image fused to obtain a fused image. Finally, the fused image is cropped to obtain the panoramic image.

[0124] In some embodiments, since the embodiments of the present invention only use a small number of adjacent image pairs for feature matching, in order to ensure the panoramic stitching effect, the stitched panoramic image can be detected. Specifically, the Canny edge detection algorithm can be used to identify an obvious line in the panoramic image, and then detect whether the line is interrupted or unnaturally bent at the junction of the sub-images. If the line is interrupted or unnaturally bent at the junction, it means that the panoramic stitching effect is not good and needs to be adjusted; if the line is not interrupted or bent at the junction, it means that the panoramic stitching effect is good and no adjustment is needed. During specific detection, a horizontal line or a vertical line with a length greater than a preset length can be selected, such as a horizontal contour line or a vertical contour line of an object such as a building, a tree, or a roadside. Then, the stitching points at the junction of the sub-images are detected along the contour line to see if they are aligned. If the distance between the stitching points at any junction on the contour line is greater than the preset distance, it means that the contour line is interrupted or unnaturally bent at the junction of the sub-images, which means that there is a problem with the position of the stitching points at the junction and they are not aligned. At this time, the position of the stitching points needs to be adjusted to ensure the panoramic stitching effect. On the contrary, if the distance between the stitching points at each junction on the contour line is less than or equal to the preset distance, it means that the contour line is not interrupted or unnaturally bent at the junction of the sub-images, which means that there is no problem with the position of the stitching points at each junction and the panoramic stitching effect is good.

[0125] A panoramic stitching method provided by an embodiment of the present invention can obtain multiple adjacent image pairs to be matched by querying a preset image adjacency table to determine the adjacent images corresponding to each sub-image, and then realize panoramic stitching of multiple sub-images by performing feature matching on the multiple adjacent image pairs respectively. Since the embodiment of the present invention only performs feature matching on sub-image pairs with an adjacency relationship, instead of performing feature matching on all image pairs like the prior art, it is possible to optimize the image matching process of drone panoramic stitching and simplify the complexity of feature matching, thereby reducing the amount of calculation, improving the efficiency of drone panoramic stitching, and thus meeting application requirements. In addition, the embodiment of the present invention can also perform anomaly detection, so that the algorithm can not only run effectively on drones, but also improve the robustness of the algorithm.

[0126] Further, as Figure 1 and Figure 4 The specific implementation of the method shown in the embodiment provides an image registration device, such as Figure 6 As shown, the device includes: an acquisition unit 41, a determination unit 42, a matching unit 43 and a registration unit 44.

[0127] The acquisition unit 41 can be used to acquire each sub-image taken by the drone, and each sub-image comes from a different shooting direction.

[0128] The determining unit 42 may be used to query a preset image adjacency relationship table to determine adjacent images corresponding to each sub-image, wherein the preset image adjacency relationship table records the adjacency relationship between each sub-image.

[0129] The determining unit 42 may also be configured to determine a plurality of adjacent image pairs to be matched according to the adjacent images corresponding to the respective sub-images.

[0130] The matching unit 43 may be used to perform feature matching on the plurality of adjacent image pairs respectively to obtain feature matching results of the plurality of adjacent image pairs;

[0131] The registration unit 44 may be configured to perform image registration on the plurality of sub-images according to the plurality of feature matching results.

[0132] In some embodiments, Figure 7 As shown, the device further includes: a construction unit 45.

[0133] The construction unit 45 may be configured to determine the adjacency relationship between the sub-images according to the image shooting sequence of the drone; and construct the preset image adjacency relationship table based on the adjacency relationship between the sub-images.

[0134] In some embodiments, the device further includes: a termination unit 46 .

[0135] The determining unit 42 may also be configured to determine the input order of the sub-images according to the identification information corresponding to the sub-images.

[0136] The determining unit 42 may also be configured to query the preset image adjacency table to determine adjacent images corresponding to each sub-image if the input sequence is consistent with the image shooting sequence of the drone.

[0137] The termination unit 46 may be configured to terminate the current image registration if the input sequence is inconsistent with the image shooting sequence of the drone.

[0138] In some embodiments, the matching unit 43 can be specifically used to extract the feature points of the two sub-images of each adjacent image pair in the plurality of adjacent image pairs, and calculate the feature descriptors corresponding to the feature points; based on the feature descriptors, match the feature points in the two sub-images to obtain matching feature point pairs in each adjacent image pair; and determine the feature matching results of each adjacent image pair based on the matching feature point pairs in each adjacent image pair.

[0139] In some embodiments, the registration unit 44 includes: a calculation module 441 and a registration module 442 .

[0140] The calculation module 441 may be configured to calculate homography matrices corresponding to the plurality of adjacent image pairs according to the plurality of feature matching results.

[0141] The registration module 442 can be used to estimate the position and posture of the drone camera device according to the homography matrix to complete the image registration of the multiple adjacent image pairs.

[0142] In some embodiments, the registration unit 44 further includes: a determination module 443 , a judgment module 444 , and a prompt module 445 .

[0143] The determination module 443 may be configured to determine valid adjacent image pairs based on the plurality of feature matching results.

[0144] The determination module 444 may be used to determine whether the number of the valid adjacent image pairs is greater than a preset number of image pairs.

[0145] The prompt module 445 may be configured to prompt that the registration has failed if the number of the valid adjacent image pairs is less than or equal to the preset number of image pairs.

[0146] The registration module 442 can also be used to estimate the posture of the drone camera device according to the homography matrix if the number of the valid adjacent image pairs is greater than the preset number of image pairs, so as to complete the image registration of multiple adjacent image pairs.

[0147] In some embodiments, the determination module 443 may also be used to determine the upward-facing sub-image among the sub-images.

[0148] The determination module 443 may also be configured to determine the number of upward-facing image pairs according to the number of the upward-facing sub-images.

[0149] The calculation module 441 may also be configured to subtract the number of the adjacent image pairs from the number of the upward-facing image pairs to obtain the preset number of image pairs.

[0150] In some embodiments, the determination module 443 can be specifically used to determine the number of matching feature point pairs in the two sub-images of any one of the multiple adjacent image pairs based on the feature matching results of the any one of the multiple adjacent image pairs; determine the number of inliers of the homography matrix corresponding to the any one of the adjacent image pairs; determine whether the number of matching feature point pairs in the any one of the adjacent image pairs is greater than or equal to a first preset number threshold, and whether the number of inliers is greater than or equal to a second preset number threshold; if the number of matching feature point pairs in the any one of the adjacent image pairs is greater than or equal to the first preset number threshold, and the number of inliers is greater than or equal to the second preset number threshold, determine that the any one of the adjacent image pairs is valid; if the number of matching feature point pairs in the any one of the adjacent image pairs is less than the first preset number threshold, or the number of inliers is less than the second preset number threshold, determine that the any one of the adjacent image pairs is invalid.

[0151] It should be noted that for other corresponding descriptions of the functional units involved in the image registration device provided in this embodiment, reference can be made to Figure 1 and Figure 4 The corresponding description in will not be repeated here.

[0152] Based on the above Figure 1 and Figure 4 The method shown in the embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned Figure 1 and Figure 4 The image registration method shown.

[0153] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0154] Based on the above Figure 1 and Figure 4 The method shown, and Figure 7 and Figure 8 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, which can be a personal computer, a tablet computer, a server, or other network equipment, etc. The device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 and Figure 4The image registration method shown.

[0155] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0156] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0157] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0158] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware.

[0159] The embodiment of the present invention can obtain multiple adjacent image pairs to be matched by querying a preset image adjacency table to determine the adjacent images corresponding to each sub-image, and then perform feature matching on the multiple adjacent image pairs respectively to achieve image registration of the multiple sub-images. Since the embodiment of the present invention only performs feature matching on sub-image pairs with an adjacency relationship, instead of performing feature matching on all image pairs like the prior art, it can optimize the feature matching process of image registration and simplify the complexity of feature matching, thereby reducing the amount of calculation, improving the efficiency of image registration, and thus meeting application requirements.

[0160] Further, as Figure 5 The specific implementation of the method shown in the embodiment provides a panoramic stitching device, such as Figure 8 As shown, the device includes: an acquisition unit 51, a determination unit 52 and a fusion unit 53.

[0161] The acquisition unit 51 may be used to acquire a plurality of registered sub-images, wherein the plurality of registered sub-images are Figure 1 and Figure 4 The registration method shown is determined.

[0162] The determining unit 52 may be configured to determine seams of the registered multiple sub-images.

[0163] The fusion unit 53 may be configured to fuse the registered multiple sub-images based on the calculated seams to obtain a panoramic image.

[0164] In some embodiments, Fig. 9 As shown, the device further includes: a compensation unit 54.

[0165] The compensation unit 54 may be used to perform gain compensation on the registered multiple sub-images to obtain compensated multiple sub-images.

[0166] In some embodiments, the fusion unit 53 includes a fusion module 531 and a trimming module 532 .

[0167] The fusion module 531 may be configured to perform image fusion on the compensated multiple sub-images based on the calculated seams to obtain a fused image.

[0168] The cropping module 532 may be used to crop the fused image to obtain the panoramic image.

[0169] It should be noted that for other corresponding descriptions of the functional units involved in the panoramic stitching device provided in this embodiment, please refer to Figure 5 The corresponding description in will not be repeated here.

[0170] Based on the above Figure 5 The method shown in the embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned Figure 5 The panorama stitching method shown.

[0171] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.

[0172] Based on the above Figure 5 The method shown, and Figure 8 and Fig. 9In order to achieve the above-mentioned purpose, the embodiment of the present application also provides an electronic device, which can be a personal computer, a tablet computer, a server, or other network equipment, etc. The device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 5 The panorama stitching method shown.

[0173] Optionally, the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0174] Those skilled in the art will appreciate that the above-mentioned physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different arrangements of components.

[0175] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and / or programs. The network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware.

[0177] The embodiment of the present invention can obtain multiple adjacent image pairs to be matched by querying a preset image adjacency table to determine the adjacent images corresponding to each sub-image, and then can achieve panoramic stitching of multiple sub-images by performing feature matching on the multiple adjacent image pairs respectively. Since the embodiment of the present invention only performs feature matching on sub-image pairs with an adjacency relationship, instead of performing feature matching on all image pairs like the prior art, it can optimize the image matching process of drone panoramic stitching, simplify the complexity of feature matching, thereby reducing the amount of calculation, improving the efficiency of drone panoramic stitching, and thus meeting application requirements.

[0178] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0179] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. An image registration method, characterized in that: include: Acquire each sub-image taken by the drone, wherein each sub-image comes from a different shooting position; Querying a preset image adjacency relationship table to determine adjacent images corresponding to each of the sub-images, wherein the preset image adjacency relationship table records the adjacency relationship between the sub-images; Determining a plurality of adjacent image pairs to be matched according to the adjacent images corresponding to the respective sub-images; Performing feature matching on the plurality of adjacent image pairs respectively to obtain feature matching results of the plurality of adjacent image pairs; Image registration is performed on the multiple sub-images according to the multiple feature matching results.

2. The method according to claim 1, characterized in that Before querying the preset image adjacency table to determine the adjacent images corresponding to each sub-image, the method further includes: Determining the adjacency relationship between the sub-images according to the image shooting order of the drone; Based on the adjacency relationship between the sub-images, the preset image adjacency relationship table is constructed.

3. The method according to claim 1, characterized in that After acquiring each sub-image photographed by the drone, the method further includes: Determining an input order of the sub-images according to identification information corresponding to the sub-images; If the input sequence is consistent with the image shooting sequence of the drone, query the preset image adjacency table to determine the adjacent images corresponding to each sub-image; If the input sequence is inconsistent with the image shooting sequence of the drone, the image registration is terminated.

4. A panoramic stitching method, characterized in that: include: Acquire a plurality of registered sub-images, wherein the plurality of registered sub-images are determined by the registration method according to any one of claims 1 to 3; Determining seams of the registered plurality of sub-images; Based on the calculated seams, the registered multiple sub-images are fused to obtain a panoramic image.

5. An image registration device, characterized in that: include: An acquisition unit, used to acquire each sub-image taken by the drone, wherein each sub-image comes from a different shooting direction; a determining unit, configured to query a preset image adjacency relationship table to determine adjacent images corresponding to each of the sub-images, wherein the preset image adjacency relationship table records the adjacency relationship between the sub-images; The determining unit is further configured to determine a plurality of adjacent image pairs to be matched according to the adjacent images corresponding to the respective sub-images; A matching unit, used to perform feature matching on the plurality of adjacent image pairs respectively to obtain feature matching results of the plurality of adjacent image pairs; A registration unit is used to perform image registration on the multiple sub-images according to the multiple feature matching results.

6. A panoramic stitching device, characterized in that: include: An acquisition unit, used for acquiring a plurality of registered sub-images, wherein the plurality of registered sub-images are determined by the registration method according to any one of claims 1 to 3; A determination unit, configured to determine seams of the registered plurality of sub-images; A fusion unit is used to fuse the registered multiple sub-images based on the calculated seams to obtain a panoramic image.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

8. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to claim 4 is implemented.

10. An electronic device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to claim 4 is implemented.

Images