Image stitching method, image stitching device and computer readable storage medium

By matching and local weight calculation of the stitched image, the rotation matrix is ​​optimized, and the problem of misalignment of multi-camera monitoring screen stitching is solved, the accuracy and efficiency of image stitching is improved, and it is suitable for multi-view monitoring scenarios.

CN114463171BActive Publication Date: 2024-11-29ZHEJIANG DAHUA TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202111636842.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-29
Publication Date
2024-11-29
Estimated Expiration
2041-12-29

AI Technical Summary

Technical Problem

In the field of security, styling images are prone to misalignment when styling the surveillance images collected by multiple cameras, which is difficult to meet users' needs for monitoring vision.

Method used

By matching the stitching image, matching feature point groups are obtained, and local weights are calculated based on the central positions of these feature points and the grid area, reprojection errors are obtained, and the area rotation matrix is ​​optimized to realize image stitching.

Benefits of technology

It improves the accuracy and efficiency of image stitching, reduces the phenomenon of stitching misalignment, adapts to various application scenarios, and has strong versatility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114463171B_ABST
    Figure CN114463171B_ABST
Patent Text Reader

Abstract

The present application discloses an image stitching method, an image stitching device and a computer-readable storage medium, the method comprising: matching a first image to be stitched with a second image to be stitched to obtain a plurality of matching feature point groups, the matching feature point groups comprising a first matching feature point in the first image to be stitched and a second matching feature point in the second image to be stitched, the first image to be stitched comprising a plurality of grid areas; based on the first matching feature point and the center position of the grid area, a local weight is calculated; based on the local weight, each matching feature point group is processed to obtain a first reprojection error; based on the first reprojection error, a rotation matrix corresponding to the grid area is optimized to obtain a regional rotation matrix; based on all regional rotation matrices, the first image to be stitched with the second image to be stitched is stitched to obtain a stitched image. In the above manner, the present application can improve the accuracy of image stitching and avoid the problem of misalignment of stitched images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to an image stitching method, an image stitching device and a computer-readable storage medium. Background Art

[0002] In the security field, since the field of view of a single camera is limited and cannot meet the user's monitoring needs, it is necessary to set up multiple cameras to capture images from multiple perspectives, and then stitch the monitoring images of multiple cameras together to form a picture with a wider field of view. However, in related technologies, when splicing multiple monitoring images, the problem of misalignment of the spliced ​​images is prone to occur. Summary of the invention

[0003] The present application provides an image stitching method, an image stitching device and a computer-readable storage medium, which can improve the accuracy of image stitching and avoid the problem of stitching image misalignment.

[0004] In order to solve the above technical problems, the technical solution adopted in the present application is: to provide an image stitching method, which includes: matching a first image to be stitched with a second image to be stitched to obtain multiple matching feature point groups, the matching feature point groups include first matching feature points in the first image to be stitched and second matching feature points in the second image to be stitched, and the first image to be stitched includes multiple grid areas; based on the first matching feature points and the center positions of the grid areas, local weights are calculated; based on the local weights, each matching feature point group is processed to obtain a first reprojection error; based on the first reprojection error, a rotation matrix corresponding to the grid area is optimized to obtain a regional rotation matrix; based on all regional rotation matrices, the first image to be stitched with the second image to be stitched to obtain a stitched image.

[0005] To solve the above technical problems, another technical solution adopted in the present application is: to provide an image stitching device, the device includes a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the image stitching method in the above technical solution.

[0006] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, it is used to implement the image stitching method in the above technical solution.

[0007] Through the above scheme, the beneficial effects of the present application are: first, the image to be stitched is divided into multiple grid areas; then, the regional rotation matrix corresponding to each grid area is used to describe the positional relationship between the corresponding grid areas in the image to be stitched that needs to be stitched; then, the image is stitched through the regional rotation matrix; at the same time, the first matching feature point and the center position of the grid area are used as local weights to process each matching feature point group to obtain a first reprojection error, so as to facilitate the use of the first reprojection error to automatically optimize the regional rotation matrix, and can refer to the matching feature points near the grid area. The parameter influence of the regional rotation matrix of the grid area can make the regional rotation matrix accurately describe the positional relationship between the grid area and the image to be stitched, thereby improving the accuracy of the regional rotation matrix, and then improving the accuracy of image stitching; moreover, the image stitching scheme provided by the present application does not require manual adjustment of parameters, can improve the efficiency of image stitching, and is suitable for various application scenarios, and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0009] Figure 1 It is a flowchart of an embodiment of an image stitching method provided by the present application;

[0010] Figure 2(a) to Figure 2(d) It is a schematic diagram of four images to be stitched;

[0011] Figure 3 is a schematic diagram of stitching images;

[0012] Figure 4 is a schematic diagram of the grid area of ​​the image to be stitched in FIG2( a );

[0013] Figure 5 is a flowchart of another embodiment of the image stitching method provided by the present application;

[0014] Figure 6 is a flowchart of step 52 provided by this application;

[0015] Figure 7 It is a structural schematic diagram of an embodiment of an image stitching device provided by the present application;

[0016] Figure 8 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0017] The present application is further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present application, but are not intended to limit the scope of the present application. Similarly, the following examples are only some embodiments of the present application rather than all embodiments, and all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0018] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0019] It should be noted that the terms "first", "second" and "third" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second" and "third" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0020] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of an image stitching method provided by the present application, the method comprising:

[0021] Step 11: performing matching processing on the first image to be stitched and the second image to be stitched to obtain a plurality of matching feature point groups.

[0022] The images to be stitched (including the first image to be stitched and the second image to be stitched) can be acquired by a multi-camera set in the application scene. Multiple cameras can be set to acquire images of different viewing angles in the application scene. Generally, the viewing angle range can be in the range of 0° to 180°. 2 to 8 cameras can be set within the viewing angle range, so that 2 to 8 surveillance images with different viewing angles can be acquired. For example, in an airport, four cameras can be set to acquire images such as Figure 2(a) to Figure 2(d)The four surveillance images shown are then stitched together using the image stitching method provided in this embodiment.

[0023] Specifically, the first image to be stitched and the second image to be stitched may be two adjacent images; when there are three or more images to be stitched, that is, when multiple cameras capture multiple surveillance images, before stitching the multiple images to be stitched, the stitching order of the multiple images to be stitched may be arranged first, and then two adjacent first images to be stitched and second images to be stitched are selected in sequence according to the stitching order for stitching, and then the next group of adjacent first images to be stitched and second images to be stitched are selected for stitching, and so on, until all the images to be stitched are stitched.

[0024] by Figure 2(a) to Figure 2(d) Take four images to be stitched as an example. Figure 2(a) to Figure 2(d) By sorting the stitching order, Figure 2(a) and Figure 2(b) can be used as the first image to be stitched and the second image to be stitched, and Figure 2(a) and Figure 2(b) can be stitched; then Figure 2(b) and Figure 2(c) can be used as the first image to be stitched and the second image to be stitched, and Figure 2(b) and Figure 2(c) can be stitched; then Figure 2(c) and Figure 2(d) can be used as the first image to be stitched and the second image to be stitched, and Figure 2(c) and Figure 2(d) can be stitched, thereby completing the stitching of the four images to be stitched, and the following is obtained: Figure 3 It is understandable that the order of the images can be set according to the camera's viewing angle. For example, the above order is performed in the order of viewing angle from left to right. In other embodiments, the order can also be performed in the order of viewing angle from right to left, that is, the order of the stitching is Figure 2(d) to Figure 2(a) .

[0025] Further, the matching feature point group includes a first matching feature point in the first image to be stitched and a second matching feature point in the second image to be stitched; when the image to be stitched is obtained, the image to be stitched may be firstly subjected to a dedistortion process to correct the image to be stitched; then, feature points are extracted from the dedistorted image to be stitched to obtain a plurality of feature points on the image to be stitched; then, feature points of the adjacent first image to be stitched and the second image to be stitched are matched to obtain a first matching feature point in the first image to be stitched and a second matching feature point in the second image to be stitched; it can be understood that there may be a plurality of first matching feature points and a second matching feature point, and the first matching feature point in the first image to be stitched corresponds one to one to the second matching feature point in the second image to be stitched.

[0026] In a specific implementation, the first image to be stitched may include multiple grid areas, such as Figure 4As shown, a plurality of grid areas can be divided in the first image to be stitched, and then each grid area in the first image to be stitched is processed to obtain a regional rotation matrix corresponding to each grid area; and then the regional rotation matrix corresponding to each grid area is used to stitch the image to improve the accuracy of image stitching; it can be understood that the division method and number of grid areas can be set according to actual conditions, and the more grid areas there are, the higher the accuracy of image stitching.

[0027] Step 12: Calculate the local weight based on the first matching feature point and the center position of the grid area.

[0028] The local weight can be calculated based on the position coordinates of the first matching feature point and the center position coordinates of the grid area, so that each matching feature point group can be processed by the local weight. By referring to the positional relationship between the first matching feature point and the center position of the grid area, the accuracy of the regional rotation matrix corresponding to the calculated grid area is improved. The specific steps for calculating the regional rotation matrix are shown in the following steps 13 to 14.

[0029] Step 13: Process each matching feature point group based on the local weight to obtain the first reprojection error.

[0030] The reprojection error may be calculated for each matching feature point group, and then the reprojection error corresponding to each matching feature point group may be weighted using a local weight, thereby obtaining a first reprojection error corresponding to the grid area.

[0031] Step 14: Optimize the rotation matrix corresponding to the grid area based on the first reprojection error to obtain the regional rotation matrix.

[0032] The camera position relationship between the adjacent first image to be stitched and the second image to be stitched is a rotation relationship. The rotation position relationship between the two adjacent images to be stitched or the two adjacent cameras can be described by a rotation matrix. The rotation position relationship can be expressed by the following formula (1):

[0033] I2=R i *I1 formula (1)

[0034] Where I1 and I2 in the above formula (1) are the pixel coordinates in adjacent images, R i is the rotation matrix between adjacent images.

[0035] In this embodiment, the image to be stitched can be divided into multiple grid areas, each grid area corresponds to a regional rotation matrix, and the matching feature points can be projected through the regional rotation matrix corresponding to each grid area, so as to achieve image stitching and improve the accuracy of image stitching; specifically, the rotation matrix corresponding to the grid area can be a matrix of three rows and three columns, and the elements contained in its initial state can be customized according to actual conditions, for example: unit matrix, and then the rotation matrix is ​​optimized according to the calculated first projection error.

[0036] Furthermore, the reprojection error refers to the difference between the projection of the matching feature point on the image plane and the reprojection of the matching feature point using the rotation matrix. For example, the position coordinates of the first matching feature point of the first image to be stitched are (x, y), and the position coordinates of the second matching feature point of the second image to be stitched are (u, v). The reprojection error can be expressed by the following formula (2):

[0037]

[0038] The error in the above formula (2) is the reprojection error.

[0039] It can be understood that the smaller the value of the reprojection error (i.e., the closer it is to 0), the more accurate the rotation matrix is. When the rotation matrix is ​​inaccurate, the reprojection error can be used to optimize the rotation matrix until the optimized first reprojection error is close to 0, thereby obtaining the final regional rotation matrix.

[0040] Step 15: stitching the first image to be stitched and the second image to be stitched based on all regional rotation matrices to obtain a stitched image.

[0041] After the regional rotation matrices corresponding to all grid regions are obtained through the above steps, the first image to be stitched and the second image to be stitched can be stitched according to all regional rotation matrices, and the first image to be stitched and the second image to be stitched can be projected into the same spherical coordinate system to obtain a stitched image. Figure 2(a) to Figure 2(d) Take the four images to be stitched as an example, and stitch them one by one to get the following Figure 3 Stitched image shown.

[0042] This embodiment uses regional rotation matrices corresponding to multiple grid areas to perform image stitching, which can reduce errors during image stitching and improve the accuracy of image stitching, thereby reducing the phenomenon of stitching misalignment; at the same time, the first matching feature point and the center position of the grid area are used as local weights to process each matching feature point group to obtain a first reprojection error, and then the first reprojection error is used to automatically optimize the regional rotation matrix, which can refer to the influence of the matching feature points near the grid area on the parameters of the regional rotation matrix of the grid area, so that the regional rotation matrix can accurately describe the positional relationship between the grid area and the image to be stitched, thereby improving the accuracy of the regional rotation matrix, and then improving the accuracy of image stitching; moreover, the image stitching method in this embodiment does not require manual adjustment of parameters, can improve the efficiency of image stitching, and is suitable for various application scenarios, and has strong versatility.

[0043] See also Figure 5 , Figure 5 : is a flow chart of another embodiment of the image stitching method provided by the present application, the method comprising:

[0044] Step 51: performing matching processing on the first image to be stitched and the second image to be stitched to obtain a plurality of matching feature point groups.

[0045] This step 51 is the same as step 11 in the above embodiment and will not be described again.

[0046] Step 52: Calculate the local weight based on the first matching feature point and the center position of the grid area.

[0047] like Figure 6 As shown, according to the first matching feature point and the center position of the grid area, the method for calculating the local weight is as follows:

[0048] Step 61: Calculate the distance between the first matching feature point and the center position of the grid area.

[0049] The distance between the first matching feature point and the center position of the grid area can be calculated according to the position coordinates of the first matching feature point and the center position coordinates of the grid area. Specifically, the center position coordinates of the grid area are (m, n), and the position coordinates of the first matching feature point are (x i ,y i ) as an example, the distance can be calculated using the following formula (3):

[0050]

[0051] Wherein, in the above formula (3), d is the distance.

[0052] Step 62: Perform an exponential power operation with the natural constant as the base and the opposite number of the distance as the exponent to obtain a fifth value.

[0053] After calculating the distance between the first matching feature point and the center position of the grid area, an exponential power operation is performed with a natural constant as the base and the opposite number of the distance as the exponent to obtain the fifth value, that is, the fifth value is calculated using the following formula (4):

[0054] z=e (-d) Formula (4)

[0055] In the above formula (4), e is a natural constant, and z is the fifth value.

[0056] Step 63: Determine whether the fifth value is greater than the first preset value.

[0057] Step 64: If the fifth value is greater than the first preset value, determine the local weight to be the fifth value.

[0058] Step 65: If the fifth value is less than or equal to the first preset value, determine the local weight to be the first preset value.

[0059] The local weight can be determined according to the size of the fifth value. The fifth value is compared with the first preset value. When the fifth value is greater than the first preset value, the fifth value is used as the local weight. When the fifth value is less than or equal to the first preset value, the first preset value can be used as the local weight. Specifically, the first preset value can be 0.1, that is, the local weight can be determined using the following formula (5):

[0060] w=max(e -d ,0.1) Formula (5)

[0061] In the above formula (5), w is the local weight, and max() is the function for finding the maximum value.

[0062] It can be understood that for a grid area, each matching feature point group corresponds to a local weight. The local weight corresponding to each matching feature point group can be calculated by the above method, and then the second reprojection error corresponding to each matching feature point group is obtained. The second reprojection error is weighted based on the corresponding local weight to obtain the first reprojection error, as shown in the following steps 53 to 55.

[0063] Step 53: Based on the rotation matrix and the position coordinates of the first matching feature point, the mapping position coordinates corresponding to the second matching feature point are calculated.

[0064] The position coordinates of the first matching feature point include a first horizontal coordinate value and a first vertical coordinate value, and the mapped position coordinates include a second horizontal coordinate value and a second vertical coordinate value; the first horizontal coordinate value, the first vertical coordinate value and the preset value can be combined into a first vector, and the first row elements are extracted from the rotation matrix to form a second vector; then the second row elements are extracted from the rotation matrix to form a third vector; then the dot product of the first vector and the second vector is calculated to obtain the second horizontal coordinate value; the dot product of the first vector and the third vector is calculated to obtain the second vertical coordinate value.

[0065] Specifically, the rotation matrix can be a three-row and three-column matrix customized according to actual conditions, and the preset value can be 1, that is, the mapping position coordinates corresponding to the second matching feature point are calculated using the following formula (6) and formula (7):

[0066] u rotate =R(0,0)*x i +R(0,1)*y i +R(0,2) Formula (6)

[0067] v rotate =R(1,0)*x i +R(1,1)*y i +R(1,2) Formula (7)

[0068] In the above formula (6) and formula (7), u rorate is the second horizontal coordinate value, v rorate is the second ordinate value, x i is the first horizontal coordinate value, y i is the first ordinate value, R(0,0), R(0,1) and R(0,2) are the first row elements of the rotation matrix, R(0,0) represents the element corresponding to the first row and first column of the rotation matrix, which constitutes the second vector, R(0,1) represents the element corresponding to the first row and second column of the rotation matrix, R(0,2) represents the element corresponding to the first row and third column of the rotation matrix, R(1,0), R(1,1) and R(1,2) are the second row elements of the rotation matrix, which constitute the third vector, R(1,0) represents the element corresponding to the second row and first column of the rotation matrix, R(1,1) represents the element corresponding to the second row and second column of the rotation matrix, and R(1,2) represents the element corresponding to the second row and third column of the rotation matrix.

[0069] Step 54: Calculate a second reprojection error based on the position coordinates of the second matching feature point and the mapped position coordinates.

[0070] The position coordinates of the second matching feature point include a third horizontal coordinate value and a third vertical coordinate value, and the second reprojection error includes a first horizontal coordinate error and a first vertical coordinate error; the difference between the second horizontal coordinate value and the third horizontal coordinate value is calculated to obtain the first horizontal coordinate error; the difference between the second vertical coordinate value and the third vertical coordinate value is calculated to obtain the first vertical coordinate error.

[0071] Step 55: Perform weighted processing on the second reprojection error based on the local weight to obtain the first reprojection error.

[0072] Each matching feature point group corresponds to a local weight, and each matching feature point group corresponds to a second reprojection error. The local weights corresponding to all matching feature point groups and the second reprojection error can be weighted and summed to obtain the first reprojection error corresponding to the grid area. Specifically, the first reprojection error may include a second abscissa error and a second ordinate error. All first abscissa errors and local weights can be weighted and summed to obtain the second abscissa error. All first ordinate errors and local weights can be weighted and summed to obtain the second ordinate error, which is equivalent to calculating the first reprojection error using the following formula (8) and formula (9):

[0073]

[0074]

[0075] In the above formula (8) and formula (9), error0 is the second horizontal coordinate error, error1 is the second vertical coordinate error, m is the total number of matching feature point groups, and w is i is the local weight corresponding to the i-th matching feature point group, (u i , v i ) is the position coordinate of the second matching feature point in the i-th matching feature point group.

[0076] Step 56: Optimize the rotation matrix corresponding to the grid area based on the first reprojection error to obtain the regional rotation matrix.

[0077] Add the square of the second horizontal coordinate error to the square of the second vertical coordinate error to get the first reprojection error, that is, error = error0 2 +error1 2; Specifically, the smaller the value of the first reprojection error is, the more accurate the rotation matrix is. The size of the first reprojection error can be used to determine whether the rotation matrix corresponding to the current grid area needs to be optimized, and whether the first reprojection error is less than the second preset value. If the first reprojection error is less than the second preset value, the rotation matrix is ​​determined to be a regional rotation matrix; if the first reprojection error is greater than or equal to the second preset value, the element array in the rotation matrix is ​​adjusted, and the step of calculating the mapping position coordinates corresponding to the second matching feature point based on the rotation matrix and the position coordinates of the first matching feature point is returned until the first reprojection error is less than the second preset value.

[0078] Furthermore, the second preset value can be set according to actual conditions, and its value range can generally be set within the range of 0.05 to 0.1, that is, to determine whether the first reprojection error is close to 0. If the first reprojection error is close to 0, it means that the rotation matrix at this time does not need to be optimized, and it can be directly determined as the regional rotation matrix; if the first reprojection error is not close to 0, it means that the rotation matrix at this time needs to be optimized, and the first reprojection error can be used to adjust the element array in the rotation matrix to obtain the optimized rotation matrix; then calculate the first reprojection error corresponding to the optimized rotation matrix until the first reprojection error is less than the second preset value, and then determine the optimized rotation matrix as the regional rotation matrix.

[0079] In a specific implementation, a numerical optimization method can be used to optimize the rotation matrix corresponding to the grid area according to the first reprojection error to obtain an accurate regional rotation matrix, wherein the numerical optimization method includes but is not limited to the Gauss-Newton method, and the first reprojection error can be first differentiated, and then the Gauss-Newton method can be used to optimize the rotation matrix.

[0080] Step 57: stitching the first image to be stitched and the second image to be stitched based on all regional rotation matrices to obtain a stitched image.

[0081] This step 57 is the same as the above step 15 and will not be described again.

[0082] This embodiment uses the first matching feature point and the center position of the grid area to perform weighted processing on the second reprojection error to obtain the first reprojection error; then the rotation matrix corresponding to the grid area is optimized based on the first reprojection error to obtain the regional rotation matrix, so as to use the regional rotation matrices corresponding to all grid areas in the image for image stitching to improve the accuracy of image stitching and reduce the phenomenon of stitching misalignment.

[0083] See also Figure 7 , Figure 7It is a structural schematic diagram of an embodiment of an image stitching device provided in the present application. The image stitching device 70 includes a memory 71 and a processor 72 connected to each other. The memory 71 is used to store a computer program. When the computer program is executed by the processor 72, it is used to implement the image stitching method in the above embodiment.

[0084] See also Figure 8 , Figure 8 It is a structural diagram of an embodiment of a computer-readable storage medium provided in the present application. The computer-readable storage medium 80 is used to store a computer program 81. When the computer program 81 is executed by a processor, it is used to implement the image stitching method in the above embodiment.

[0085] The computer-readable storage medium 80 may be a server, a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other media that can store program codes.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0087] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0088] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0089] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image stitching method, characterized in that: include: Performing matching processing on the first image to be stitched and the second image to be stitched to obtain a plurality of matching feature point groups, wherein the matching feature point groups include first matching feature points in the first image to be stitched and second matching feature points in the second image to be stitched, and the first image to be stitched includes a plurality of grid areas; Calculating a local weight based on the first matching feature point and the center position of the grid area; Processing each matching feature point group based on the local weight to obtain a first reprojection error; Optimizing the rotation matrix corresponding to the grid area based on the first reprojection error to obtain a regional rotation matrix; splicing the first image to be spliced ​​and the second image to be spliced ​​based on all the regional rotation matrices to obtain a spliced ​​image; The position coordinates of the first matching feature point include a first abscissa value and a first ordinate value, and the first reprojection error includes a second abscissa error and a second ordinate error; and the step of processing each matching feature point group based on the local weight to obtain the first reprojection error includes: Combining the first horizontal coordinate value, the first vertical coordinate value and a preset value into a first vector; Extracting the first row of elements from the rotation matrix to form a second vector; Extracting the second row of elements from the rotation matrix to form a third vector; Calculate the dot product of the first vector and the second vector to obtain the second horizontal coordinate value of the mapping position coordinate; Calculate the dot product of the first vector and the third vector to obtain the second ordinate value of the mapping position coordinate; Based on the position coordinates of the second matching feature point and the mapping position coordinates, a second reprojection error is calculated; the second reprojection error includes a first horizontal coordinate error and a first vertical coordinate error; Performing a weighted summation on all the first horizontal coordinate errors and the local weights to obtain the second horizontal coordinate error; A weighted sum is performed on all the first ordinate errors and the local weight to obtain the second ordinate error.

2. The image stitching method according to claim 1, characterized in that: The position coordinates of the second matching feature point include a third horizontal coordinate value and a third vertical coordinate value, and the step of calculating the second reprojection error based on the position coordinates of the second matching feature point and the mapping position coordinates includes: Calculate the difference between the second horizontal coordinate value and the third horizontal coordinate value to obtain the first horizontal coordinate error; The difference between the second ordinate value and the third ordinate value is calculated to obtain the first ordinate error.

3. The image stitching method according to claim 1, characterized in that: The step of calculating the local weight based on the first matching feature point and the center position of the grid area includes: Calculating the distance between the first matching feature point and the center position of the grid area; Performing an exponential power operation with a natural constant as a base and a negative number of the distance as an exponent to obtain a fifth value; Determining whether the fifth value is greater than a first preset value; If yes, determining the local weight to be the fifth value; If not, the local weight is determined to be the first preset value.

4. The image stitching method according to claim 1, characterized in that: The step of optimizing the rotation matrix corresponding to the grid area based on the first reprojection error to obtain the regional rotation matrix includes: Adding the square of the second horizontal coordinate error and the square of the second vertical coordinate error to obtain the first reprojection error; Determining whether the first reprojection error is less than a second preset value; If so, determining that the rotation matrix is ​​the regional rotation matrix; If not, adjust the element matrix in the rotation matrix, and return to the step of calculating the mapping position coordinates corresponding to the second matching feature point based on the rotation matrix and the position coordinates of the first matching feature point, until the first reprojection error is less than the second preset value.

5. An image stitching device, characterized in that: The invention comprises a memory and a processor connected to each other, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, it is used to implement the image stitching method according to any one of claims 1 to 4.

6. A computer-readable storage medium for storing a computer program, characterized in that: When the computer program is executed by a processor, it is used to implement the image stitching method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • A method and apparatus for splicing panoramic images

    CN109389555A