Image stitching method, electronic device and readable storage medium
By determining the best suture and deformation parameters in image stitching, the image is deformed, which solves the problem of discontinuity in image stitching and improves the stitching effect.
Patent Information
- Application Number
- CN202110136004.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-01
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-02-01
AI Technical Summary
Existing image stitching methods have discontinuities in transition areas or edge areas, resulting in poor stitching effects.
By obtaining the feature matching point set between two adjacent frames of images in the same shooting scene, the best suture is determined, and deformation parameters are determined based on the suture and feature matching point set, the first image is deformed, and the third image is obtained. , and then stitch with the second image to reduce artifacts and ghosting.
Improve the effect of image stitching, reduce artifacts and ghosting, and improve the stitching effect.
Smart Images

Figure CN112950468B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of image processing, and in particular relates to an image stitching method, an electronic device and a readable storage medium. Background Art
[0002] The purpose of panoramic imaging technology is to match and stitch images taken at different positions and angles in the same shooting scene to form an image with a wider field of view. At present, panoramic images are usually obtained by algorithm matching and stitching. However, due to the differences in shooting angles of different images, the stitched image often appears discontinuous in the transition area or edge area, resulting in poor image stitching effect. Summary of the invention
[0003] The purpose of the embodiments of the present invention is to provide an image stitching method, an electronic device and a readable storage medium to solve the problem of poor stitching effect of the existing image stitching method.
[0004] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:
[0005] In a first aspect, an embodiment of the present invention provides an image stitching method, the method comprising:
[0006] Acquire a feature matching point set between a first image and a second image; wherein the first image and the second image are two adjacent frames of images in the same shooting scene;
[0007] determining an optimal stitching line between the first image and the second image in an overlapping area between the first image and the second image;
[0008] Determine a first deformation parameter according to the optimal stitching line and the feature matching point set, and deform the first image using the first deformation parameter to obtain a third image; wherein the first deformation parameter is a deformation parameter of the first image when aligning the spatial positions of the first image and the second image;
[0009] The second image and the third image are stitched together.
[0010] Optionally, determining the best stitching line between the first image and the second image includes:
[0011] Constructing a difference image according to a first difference and a second difference; wherein the first difference is a difference between the first image and the second image in a color space, and the second difference is a gradient difference between the first image and the second image in a geometric space;
[0012] A path with the smallest difference is selected from the difference image as the optimal stitching line.
[0013] Optionally, the expression of the difference image is: E(x, y) = E c (x, y) 2 +E g (x, y);
[0014] Among them, E c (x, y) = I1(x, y) - I2(x, y), indicating the first difference; E g (x, y) = S x (I1(x, y)-I2(x, y)) 2 +S y (I1(x, y)-I2(x, y)) 2 , represents the second difference; S x Indicates the horizontal gradient value of the overlapping area, S y Indicates the gradient value in the vertical direction of the overlapping area;
[0015] Among them, I1(x, y) represents the first image, and I2(x, y) represents the second image.
[0016] Optionally, determining a first deformation parameter according to the optimal stitching line and the feature matching point set includes:
[0017] Dividing the first image into grids to obtain a plurality of first grids;
[0018] According to the optimal stitching line and the feature matching point set, a first energy equation is constructed through a feature point coordinate constraint term, a local rigid constraint term and an optimal stitching line constraint term;
[0019] For each of the first grids, determining, according to the first energy equation, a position of a first target grid corresponding to the first grid when aligning the spatial positions of the first image and the second image; wherein the position of the first target grid minimizes the value of the first energy equation;
[0020] The first deformation parameter is determined according to the position of the first mesh and the position of the first target mesh.
[0021] Optionally, the expression of the first energy equation is: O1(E)=αE d +βE s +γE l ;
[0022] Among them, E d =∑ i ‖w i V i -P i ‖ 2, represents the feature point coordinate constraint; w i V i represents the weighted average of the coordinates of the four vertices of the first grid where the feature point i is located, P i represents the coordinates of the first target grid corresponding to the first grid where the feature point i is located; α is E d The weight of
[0023] E s (V1) = ‖V1-(V2+u(V3-V2)+v(R 90 (V3-V2)))‖ 2 , represents the local rigid constraint term; V1, V2, V3 respectively represent the coordinates of the three vertices of the triangle enclosing the feature point i in the first grid; u and v represent coefficients; β is E s The weight of
[0024] represents the optimal suture line constraint; n represents the number of equally divided points of the optimal suture line, w j V j represents the weighted average of the coordinates of the four vertices of the first grid where the equally divided point j is located, P j represents the coordinates of the equally divided point j when the spatial positions of the first image and the second image are aligned; γ is E l The weight of .
[0025] Optionally, the stitching the second image and the third image includes:
[0026] According to the position of the optimal stitching line, the second image and the third image are subjected to weighted average fusion processing using the following formula:
[0027] I=f*I2(x,y)+(1-f)*I3(x,y)
[0028] Among them, the value range of f is [0, 1]; f is determined based on the distance between the pixel point in the second image and the optimal stitching line, and the smaller the distance, the larger f; I2(x, y) represents the second image, and I3(x, y) represents the third image.
[0029] Optionally, after stitching the second image and the third image, the method further includes:
[0030] Performing content completion on the fourth image to obtain a target image; wherein the fourth image is an image obtained by splicing the second image and the third image;
[0031] Determining a second deformation parameter according to the optimal stitching line and the feature matching point set; wherein the second deformation parameter is a deformation parameter of the target image when aligning the spatial position of the target image with the fourth image;
[0032] The target image is deformed using the second deformation parameter to obtain a composite image.
[0033] Optionally, determining a second deformation parameter according to the optimal stitching line and the feature matching point set includes:
[0034] Performing grid division on the target image to obtain a plurality of second grids;
[0035] According to the optimal stitching line and the feature matching point set, a second energy equation is constructed through feature point coordinate constraint items and optimal stitching line constraint items;
[0036] For each of the second grids, determining, according to the second energy equation, a position of a second target grid corresponding to the second grid when aligning the spatial positions of the target image and the fourth image; wherein the position of the second target grid minimizes the value of the second energy equation;
[0037] The second deformation parameter is determined according to the position of the second mesh and the position of the second target mesh.
[0038] Optionally, the expression of the second energy equation is: O2(E)=α0E d ′+γ0E l ′;
[0039] Among them, E d ′=∑ i ‖w i V i -P i ′‖ 2 , represents the feature point coordinate constraint; w i V i represents the weighted average sum of the coordinates of the four vertices of the second grid where the feature point i is located, P i ′ represents the coordinates of the second target grid corresponding to the second grid where the feature point i is located; α0 is E d ′ weight;
[0040] represents the optimal suture line constraint; n represents the number of equally divided points of the optimal suture line, w j V j represents the weighted average sum of the coordinates of the four vertices of the second grid where the equally divided point j is located, P j′ represents the coordinates of the equally divided point j when aligning the spatial positions of the target image and the fourth image; γ0 is E l ′’s weight.
[0041] In a second aspect, an embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction can implement the steps of the above-mentioned image stitching method when executed by the processor.
[0042] In a third aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the above-mentioned image stitching method can be implemented.
[0043] In an embodiment of the present invention, after acquiring the first image and the second image of adjacent frames, the deformation parameters of the first image when aligning the spatial positions of the first image and the second image can be determined based on the feature matching point set and the best stitching line between the first image and the second image, and the first image is deformed according to the deformation parameters to obtain a third image, and the second image and the third image are spliced. Therefore, since the third image is obtained by deforming the first image, and the corresponding deformation parameters are determined when aligning the spatial positions of the first image and the second image, the third image is essentially in the space of the second image, and the local corresponding areas of the second image and the third image are more consistent, so the second image and the third image are spliced to achieve a splicing effect of reducing artifacts and ghosting, and improving the splicing effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 is a flow chart of an image stitching method provided by an embodiment of the present invention;
[0045] Figure 2 is a flow chart of a deformation parameter determination process in an embodiment of the present invention;
[0046] Figure 3 is a flow chart of a post-processing process of the spliced images in an embodiment of the present invention;
[0047] Figure 4a is a schematic diagram of the effect of the original panoramic image obtained by stitching in an embodiment of the present invention;
[0048] Figure 4b Yes Figure 4a Schematic diagram of the effect after the image in the figure is supplemented with content;
[0049] Figure 4c Yes Figure 4b Schematic diagram of the effect after the image in is deformed;
[0050] Figure 5 is a structural schematic diagram of an image stitching device provided by an embodiment of the present invention;
[0051] Figure 6 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] The terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0054] The image stitching method provided by the embodiment of the present invention is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.
[0055] See also Figure 1 , Figure 1 is a flow chart of an image stitching method provided by an embodiment of the present invention, the method is applied to electronic devices such as Figure 1 As shown, the method comprises the following steps:
[0056] Step 11: Obtain a feature matching point set between the first image and the second image.
[0057] In this embodiment, the first image and the second image are two adjacent frame images in the same shooting scene. Further, the first image is the current frame image, the second image is the reference frame image, and the first image and the second image are spliced with the second image as a reference.
[0058] It should be noted that the applicable scenarios of the embodiments of the present invention include but are not limited to panoramic imaging scenarios, augmented reality (AR) scenarios, virtual reality (VR) scenarios, etc. For example, in a panoramic imaging scenario, the first image and the second image may be two adjacent frames of images in a plurality of frames captured at different positions and angles in the same shooting scene; or, the second image is a frame image spliced in the process of splicing a plurality of frames, the first image is a frame image adjacent to the spliced frame image, and the image splicing process is continued with the spliced frame image as a reference.
[0059] Optionally, the process of obtaining the feature matching point set in this step 11 may include: first, searching for feature points of the first image and the second image by using the scale-invariant feature transform (SIFT) image feature detection algorithm, and establishing a feature descriptor for each feature point; wherein the searched feature points may be concentrated in places where the grayscale changes are relatively large, such as isolated points, edges, and contours, and the corresponding features include but are not limited to color features, grayscale features, texture features, etc.; then, using the Euclidean distance as the similarity measure of the feature descriptor, matching the feature points of the first image and the second image, and removing abnormal matching points by using the Random Sample Consensus (RANSAC) method; finally, obtaining the feature matching point set between the first image and the second image. Among them, the SIFT image feature detection algorithm mainly has four steps: the first step is to extract extreme points in the Gaussian difference scale space, the second step is to locate the feature points, the third step is to calculate the main direction of the feature points, and the fourth step is to construct the descriptor of the feature points.
[0060] Step 12: In the overlapping area between the first image and the second image, determine the best stitching line between the first image and the second image.
[0061] In this embodiment, in order to minimize the adverse effects of color difference and stitching ghosting between frame images on the stitching result, the overlapping area of the frame images can be determined by first calculating the global homography matrix, and then the optimal stitching line can be calculated in the overlapping area to perform image stitching according to the optimal stitching line, so that the stitching effect of the frame images has a natural transition.
[0062] Optionally, the process of determining the overlapping area between the first image and the second image may be: first, based on a set of feature matching points between the first image and the second image, calculate the relative position of the first image and the second image, and then, based on the relative position, determine the overlapping area between the first image and the second image.
[0063] Step 13: Determine a first deformation parameter according to the optimal stitching line and the feature matching point set, and use the first deformation parameter to deform the first image to obtain a third image.
[0064] In this embodiment, the first deformation parameter is a deformation parameter of the first image when the spatial positions of the first image and the second image are aligned. The above-mentioned spatial alignment of the first image and the second image can be understood as transforming the first image to the space corresponding to the second image, and at this time, the transformed first image is deformed relative to the original first image.
[0065] Step 14: stitch the second image and the third image.
[0066] Optionally, the present embodiment may use a variety of methods to stitch the second image and the third image, including but not limited to weighted average fusion processing according to the position of the best stitching line.
[0067] The image stitching method of the embodiment of the present invention can determine the deformation parameters of the first image when aligning the spatial positions of the first image and the second image according to the feature matching point set and the best stitching line between the first image and the second image after acquiring the first image and the second image of adjacent frames, and deform the first image according to the deformation parameters to obtain the third image, and stitch the second image and the third image. Therefore, since the third image is obtained by deforming the first image, and the corresponding deformation parameters are determined when aligning the spatial positions of the first image and the second image, the third image is essentially in the space of the second image, and the local corresponding areas of the second image and the third image are more consistent, so by stitching the second image and the third image, it is possible to achieve a stitching effect of reducing artifacts and ghosting, and improving the stitching effect.
[0068] In the embodiment of the present invention, the idea of the optimal stitching line is to find a stitching line in the overlapping area of the inter-frame images so that the color difference and the structural difference between the images on both sides of the stitching line are minimized at the same time, so that when the inter-frame images are stitched, the corresponding overlapping area is perfectly transitioned and there is no obvious stitching gap. Therefore, in the overlapping area of the inter-frame images, the optimal stitching line of the inter-frame images can be determined according to the difference between the inter-frame images in the color space and the geometric space.
[0069] Optionally, the process of determining the best stitching line of the first image and the second image may include: first, constructing a difference image according to the first difference and the second difference; wherein the first difference is the difference between the first image and the second image in the color space, and the second difference is the gradient difference between the first image and the second image in the geometric space; and then, selecting the path with the smallest difference from the difference image as the best stitching line. In this way, the best stitching line can be accurately selected with the help of the difference image that integrates color and structural differences.
[0070] Furthermore, the expression of the difference image can be: E(x, y) = E c (x, y)2 +E g (x, y).
[0071] Among them, E c (x, y) = I1(x, y) - I2(x, y), which represents the color difference between the first image I1(x, y) and the second image I2(x, y), that is, the first difference mentioned above. g (x, y) = S x (I1(x, y)-I2(x, y)) 2 +S y (I1(x, y)-I2(x, y)) 2 , represents the spatial gradient difference between the first image I1(x, y) and the second image I2(x, y), that is, the second difference mentioned above. S x Indicates the horizontal gradient value of the overlapping area, S y It represents the vertical gradient value of the overlapping area, which can be realized by the Sobel operator.
[0072] In one implementation, when selecting the best stitching line from the difference image, n pixel points with the smallest values can be selected from the first row in the difference image as starting points, where n is, for example, 10, 15, etc., and the points propagate downward from these starting points until they reach the last row, each of which travels a path from top to bottom. According to this rule, a path can be generated from each starting point, and a path with the smallest difference among the generated paths is selected as the best stitching line.
[0073] In the embodiment of the present invention, the first deformation parameter can be obtained by using a plurality of deformation algorithms such as as rigid as possible algorithm, moving least square method, etc. In addition, the first deformation parameter can also be obtained by meshing the first image and using mesh deformation.
[0074] Optionally, when determining the first deformation parameter based on mesh deformation, an energy equation may be constructed based on energy constraints for protecting mesh deformation, such as feature point coordinate constraints, local rigidity constraints, and / or optimal seam constraints, and the deformation parameter may be determined based on the energy equation. Figure 2 As shown, the process of determining the first deformation parameter according to the best stitching line and the feature matching point set may include:
[0075] Step 21: Divide the first image into grids to obtain a plurality of first grids.
[0076] For example, this step may randomly perform grid division, and may use 8*8, 16*16, or 32*32, etc. Preferably, at least one feature point is contained in one grid.
[0077] Step 22: According to the best seam line and the feature matching point set, a first energy equation is constructed through feature point coordinate constraints, local rigidity constraints and best seam line constraints.
[0078] It should be noted that the first energy equation here is constructed based on the feature matching point set between the first image and the second image, that is, only the deformation of the grid of important feature points is guaranteed to reduce the amount of calculation. However, in addition to this, the first energy equation can also be constructed based on the feature point set of the first image to ensure that the grid of all feature points of the first image is considered, thereby improving the deformation accuracy of the first image.
[0079] Step 23: For each first grid, determine the position of the first target grid corresponding to the first grid when aligning the spatial positions of the first image and the second image according to the first energy equation. The position of the first target grid minimizes the value of the first energy equation. That is, the goal of the first energy equation is to find a set of grid coordinate points that minimizes its value.
[0080] Step 24: Determine the first deformation parameter according to the position of the first grid and the position of the first target grid corresponding to the first grid, that is, determine the first deformation parameter of the first image according to the position coordinates of the first grid before and after the deformation of each group.
[0081] Furthermore, the expression of the first energy equation is: O1(E) = αE d +βE s +γE l .
[0082] Among them, E d =∑ i ‖w i V i -P i ‖ 2 , represents the feature point coordinate constraint item. i V i P represents the weighted average of the coordinates of the four vertices of the first grid where the feature point i is located, that is, the coordinate position of the feature point i. i Represents the coordinates of the first target grid corresponding to the first grid where the feature point i is located. α is E d The weight of .
[0083] E s (V1) = ‖V1-(V2+u(V3-V2)+v(R 90 (V3-V2)))‖ 2, represents the local rigid constraint term. V1, V2, and V3 represent the coordinates of the three vertices of the triangle that wraps the feature point i in the first mesh. Any vertex V1 can be linearly represented by the coordinates of the other two vertices. u and v represent coefficients. β is E s The weight of .
[0084] represents the optimal suture constraint. n represents the number of equally divided points of the optimal suture, and n is an integer greater than 0. j V j P represents the weighted average of the coordinates of the four vertices of the first grid where the equally divided point j is located, that is, the original coordinate position of the equally divided point j. j represents the coordinate position of the equally divided point j when aligning the spatial positions of the first image and the second image. l The weight of .
[0085] It is understandable that the above weights α, β and γ can be adjusted based on actual needs, and this embodiment does not limit this.
[0086] In the embodiment of the present invention, in order to ensure the image stitching effect, when stitching the second image and the third image, a weighted average fusion process may be performed according to the position of the best stitching line. The stitching process in the above step 14 may include:
[0087] According to the position of the optimal stitching line, the second image and the third image are fused by weighted average using the following formula:
[0088] I=f*I2(x,y)+(1-f)*I3(x,y)
[0089] Wherein, the value range of f is [0, 1]. Furthermore, the value range of f is [0.1, 0.9]. f is determined based on the distance between the pixel point in the second image I2 (x, y) and the optimal stitching line. The smaller the distance, the larger f. f is, for example, a Gaussian function. I3 (x, y) represents the third image.
[0090] That is to say, in order to make the transition of the overlapping area of the second image and the third image smooth, this embodiment adopts a weighted average fusion processing method of gradual in and out, that is, in the overlapping area, the superposition weight of the second image and the third image is strongly related to the distance from the best stitching line, that is, the smaller the distance, the greater the corresponding weight. In this way, the local corresponding areas of the second image and the third image can be more consistent, achieving a stitching effect with reduced artifacts and ghosting.
[0091] In the embodiment of the present invention, in order to make the stitched image present a better visual effect while minimizing the loss of the original information of the image, the stitched image can be content-completed. Since the content completion may cause deformation of part of the image, the final stitched image can be obtained by deforming the image after content completion.
[0092] Optionally, after the second image and the third image are spliced, as shown in FIG. Figure 3 The image synthesis method in this embodiment may further include:
[0093] Step 31: Complete the content of the fourth image to obtain a target image, where the fourth image is an image obtained by splicing the second image and the third image.
[0094] In this step, for spliced images with irregular shapes, the image scaling seam carving algorithm can be used to complete the content to form a rectangular image. The seam carving algorithm is a scaling method based on image content, which may cause deformation of the image visual subject when scaling the image.
[0095] Step 32: Determine a second deformation parameter according to the optimal stitching line and the feature matching point set; wherein the second deformation parameter is a deformation parameter of the target image when aligning the spatial position of the target image with that of the fourth image.
[0096] Step 33: deform the target image using the second deformation parameter to obtain a composite image.
[0097] It should be noted that the best stitching line and feature matching point set here are the best stitching line and feature matching point set involved in the above embodiment. However, in addition to this, the feature matching point set can also be replaced by all feature point sets of the target image to improve the accuracy of the determined deformation parameters. The second deformation parameter can be obtained by meshing the target image and using mesh deformation.
[0098] Optionally, the second deformation parameter is determined in a manner similar to the first deformation parameter, and mainly includes the following process:
[0099] Divide the target image into grids to obtain a plurality of second grids; for example, the grid division may be performed randomly, and the grid division may be performed in a manner of 8*8, 16*16, or 32*32;
[0100] According to the best stitching line and the feature matching point set, a second energy equation is constructed through the feature point coordinate constraint term and the best stitching line constraint term;
[0101] For each second grid, determining, according to the second energy equation, a position of a second target grid corresponding to the second grid when the spatial positions of the target image and the fourth image are aligned; wherein the position of the second target grid minimizes the value of the second energy equation;
[0102] The second deformation parameter is determined according to the position of the second grid and the position of the second target grid corresponding to the second grid, that is, the second deformation parameter of the target image is determined according to the position coordinates of the second grid before and after the deformation of each group.
[0103] Furthermore, the expression of the second energy equation is: O2(E) = α0E d ′+γ0E l ′.
[0104] Among them, E d ′=∑ i ‖w i V i -P i ′‖ 2 , represents the feature point coordinate constraint item. i V i P represents the weighted average of the coordinates of the four vertices of the second grid where the feature point i is located, that is, the coordinate position of the feature point i. i ′ represents the coordinates of the second target grid corresponding to the second grid where the feature point i is located. α0 is E d ′’s weight.
[0105] represents the optimal suture constraint. n represents the number of equally divided points of the optimal suture, and n is an integer greater than 0. j V j P represents the weighted average of the coordinates of the four vertices of the second grid where the equally divided point j is located, that is, the original coordinate position of the equally divided point j. j ′ represents the coordinates of the equally divided point j when aligning the spatial positions of the target image and the fourth image. γ0 is E l ′’s weight.
[0106] It can be understood that, taking the application of the image stitching method in the embodiment of the present invention to panoramic imaging with an ultra-wide viewing angle as an example, the process of obtaining a panoramic image may include: S1: for multiple frames of images shot at different positions and angles in the same shooting scene, in the image stitching process, a feature matching point set between adjacent frame images is obtained, and the following process is performed on each inter-frame image (one frame is the current frame image, and one frame is the reference frame image); S2: in the overlapping area of the inter-frame images, the optimal stitching line is determined according to the difference between the inter-frame images in the color space and the geometric space; S3: based on the optimal stitching line and the feature matching point set, an energy equation is constructed through feature point coordinate constraints, local rigid constraints and optimal stitching line constraints, and the current frame image is deformed based on grid deformation to obtain a deformed image; S4: based on the optimal stitching line position, the deformed image and the reference frame image are subjected to weighted average fusion processing to obtain a stitched image. S5: Complete the content of the stitched image to obtain the completed image; S6: For the image before completion (reference frame image) and the image after completion (current frame image), according to the best stitching line and feature matching point set, an energy equation is constructed through the feature point coordinate constraint item and the best stitching line constraint item, and the completed image is deformed based on the grid deformation to obtain the final panoramic image.
[0107] It should be noted that in the process of obtaining the panoramic image, the content completion processing can be performed immediately after each inter-frame image is stitched as described above, or the original panoramic image can be obtained after all inter-frame images are stitched, such as Figure 4a As shown, the original panoramic image is then processed for content completion to obtain a completed panoramic image, such as Figure 4b As shown, the completed panoramic image is deformed to obtain the final panoramic image, as shown in Figure 4c shown.
[0108] See also Figure 5 , Figure 5 is a schematic diagram of the structure of an image splicing device provided by an embodiment of the present invention, and the device is applied to electronic devices such as Figure 5 As shown, the image stitching device 50 includes:
[0109] An acquisition module 51 is used to acquire a feature matching point set between a first image and a second image; wherein the first image and the second image are two adjacent frames of images in the same shooting scene;
[0110] A determination module 52, configured to determine an optimal stitching line between the first image and the second image in an overlapping area between the first image and the second image;
[0111] A first deformation module 53 is used to determine a first deformation parameter according to the optimal stitching line and the feature matching point set, and deform the first image using the first deformation parameter to obtain a third image; wherein the first deformation parameter is a deformation parameter of the first image when aligning the spatial positions of the first image and the second image;
[0112] The stitching module 54 is configured to stitch the second image and the third image.
[0113] Optionally, the first deformation module 53 includes:
[0114] A first construction unit is configured to construct a difference image according to a first difference and a second difference; wherein the first difference is a difference between the first image and the second image in a color space, and the second difference is a gradient difference between the first image and the second image in a geometric space;
[0115] A selection unit is used to select a path with the smallest difference from the difference image as the optimal stitching line.
[0116] Optionally, the expression of the difference image is: E(x, y) = E c (x, y) 2 +E g (x, y);
[0117] Among them, E c (x, y) = I1(x, y) - I2(x, y), indicating the first difference; E g (x, y) = S x (I1(x, y)-I2(x, y)) 2 +S y (I1(x, y)-I2(x, y)) 2 , represents the second difference; S x Indicates the horizontal gradient value of the overlapping area, S y Indicates the gradient value in the vertical direction of the overlapping area;
[0118] Among them, I1(x, y) represents the first image, and I2(x, y) represents the second image.
[0119] Optionally, the first deformation module 53 includes:
[0120] A first division unit, configured to divide the first image into grids to obtain a plurality of first grids;
[0121] A second construction unit is used to construct a first energy equation according to the optimal stitching line and the feature matching point set through a feature point coordinate constraint term, a local rigidity constraint term and an optimal stitching line constraint term;
[0122] A first determining unit is configured to determine, for each of the first grids, a position of a first target grid corresponding to the first grid when aligning the spatial positions of the first image and the second image according to the first energy equation; wherein the position of the first target grid minimizes the value of the first energy equation;
[0123] The second determining unit is used to determine the first deformation parameter according to the position of the first grid and the position of the first target grid.
[0124] Optionally, the expression of the first energy equation is: O1(E)=αE d +βE s +γE l ;
[0125] Among them, E d =∑ i ‖w i V i -P i ‖ 2 , represents the feature point coordinate constraint; w i V i represents the weighted average of the coordinates of the four vertices of the first grid where the feature point i is located, P i represents the coordinates of the first target grid corresponding to the first grid where the feature point i is located; α is E d The weight of
[0126] E s (V1) = ‖V1-(V2+u(V3-V2)+v(R 90 (V3-V2)))‖ 2 , represents the local rigid constraint term; V1, V2, V3 respectively represent the coordinates of the three vertices of the triangle enclosing the feature point i in the first grid; u and v represent coefficients; β is E s The weight of
[0127] represents the optimal suture line constraint; n represents the number of equally divided points of the optimal suture line, w j V j represents the weighted average of the coordinates of the four vertices of the first grid where the equally divided point j is located, P j represents the coordinates of the equally divided point j when the spatial positions of the first image and the second image are aligned; γ is E l The weight of .
[0128] Optionally, the splicing module 54 is specifically used for:
[0129] According to the position of the optimal stitching line, the second image and the third image are subjected to weighted average fusion processing using the following formula:
[0130] I=f*I2(x,y)+(1-f)*I3(x,y)
[0131] Among them, the value range of f is [0, 1]; f is determined based on the distance between the pixel point in the second image and the optimal stitching line, and the smaller the distance, the larger f; I2(x, y) represents the second image, and I3(x, y) represents the third image.
[0132] Optionally, the image stitching device 50 further includes:
[0133] A completion module, used to complete the content of the fourth image to obtain a target image; wherein the fourth image is an image obtained by splicing the second image and the third image;
[0134] A second deformation module is used to determine a second deformation parameter according to the optimal stitching line and the feature matching point set, and use the second deformation parameter to deform the target image to obtain a composite image; wherein the second deformation parameter is a deformation parameter of the target image when aligning the spatial position of the target image with the fourth image.
[0135] Optionally, the second deformation module includes:
[0136] A second division unit, used for performing grid division on the target image to obtain a plurality of second grids;
[0137] A third construction unit is used to construct a second energy equation according to the optimal stitching line and the feature matching point set through a feature point coordinate constraint item and an optimal stitching line constraint item;
[0138] a third determining unit, configured to determine, for each of the second grids, a position of a second target grid corresponding to the second grid when aligning the spatial positions of the target image and the fourth image according to the second energy equation; wherein the position of the second target grid minimizes the value of the second energy equation;
[0139] The fourth determining unit is used to determine the second deformation parameter according to the position of the second grid and the position of the second target grid.
[0140] Optionally, the expression of the second energy equation is: O2(E)=α0E d ′+γ0E l ′;
[0141] Among them, E d ′=∑i ‖w i V i -P i ′‖ 2 , represents the feature point coordinate constraint; w i V i represents the weighted average sum of the coordinates of the four vertices of the second grid where the feature point i is located, P i ′ represents the coordinates of the second target grid corresponding to the second grid where the feature point i is located; α0 is E d ′ weight;
[0142] represents the optimal suture line constraint; n represents the number of equally divided points of the optimal suture line, w j V j represents the weighted average sum of the coordinates of the four vertices of the second grid where the equally divided point j is located, P j ′ represents the coordinates of the equally divided point j when aligning the spatial positions of the target image and the fourth image; γ0 is E l ′’s weight.
[0143] It is understandable that the image stitching device 50 of the embodiment of the present invention can achieve the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.
[0144] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.
[0146] In addition, an embodiment of the present invention further provides an electronic device 60, including a processor 61, a memory 62, and a program or instruction stored in the memory 62 and executable on the processor 61. When the program or instruction is executed by the processor 61, each process of the above-mentioned image stitching method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0147] The embodiment of the present invention further provides a readable storage medium on which a program or instruction is stored, and when the program or instruction is executed by a processor, the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and to avoid repetition, they will not be described again here.
[0148] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0149] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0150] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a service classification device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0152] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. An image stitching method, It is characterized in that include: Acquire a feature matching point set between a first image and a second image; wherein the first image and the second image are two adjacent frames of images in the same shooting scene; determining an optimal stitching line between the first image and the second image in an overlapping area between the first image and the second image; Determine a first deformation parameter according to the optimal stitching line and the feature matching point set, and deform the first image using the first deformation parameter to obtain a third image; wherein the first deformation parameter is a deformation parameter of the first image when aligning the spatial positions of the first image and the second image; splicing the second image and the third image; Wherein, determining the first deformation parameter according to the optimal stitching line and the feature matching point set includes: Dividing the first image into grids to obtain a plurality of first grids; According to the optimal stitching line and the feature matching point set, a first energy equation is constructed through a feature point coordinate constraint term, a local rigid constraint term and an optimal stitching line constraint term; For each of the first grids, determining, according to the first energy equation, a position of a first target grid corresponding to the first grid when aligning the spatial positions of the first image and the second image; wherein the position of the first target grid minimizes the value of the first energy equation; The first deformation parameter is determined according to the position of the first mesh and the position of the first target mesh.
2. The method according to claim 1, It is characterized in that The determining of the best stitching line between the first image and the second image comprises: Constructing a difference image according to a first difference and a second difference; wherein the first difference is a difference between the first image and the second image in a color space, and the second difference is a gradient difference between the first image and the second image in a geometric space; A path with the smallest difference is selected from the difference image as the optimal stitching line.
3. The method according to claim 2, It is characterized in that The expression of the difference image is: E(x, y) = E c (x,y) 2 +E g (x,y); Among them, E c (x,y)=I 1 (x,y)-I 2 (x, y), represents the first difference; E g (x,y)=S x (I 1 (x,y)-I 2 (x,y) 2 +S y (I 1 (x,y)-I 2 (x,y) 2 , represents the second difference; S x Indicates the horizontal gradient value of the overlapping area, S y Indicates the gradient value in the vertical direction of the overlapping area; Among them, I 1 (x, y) represents the first image, I 2 (x, y) represents the second image.
4. The method according to claim 1, It is characterized in that The expression of the first energy equation is: 1 (E) = αE d +βE s +γE l ; Among them, E d =∑ i ‖w i V o -P o ‖ 2 , represents the feature point coordinate constraint; w o V i represents the weighted average of the coordinates of the four vertices of the first grid where the feature point i is located, P i represents the coordinates of the first target grid corresponding to the first grid where the feature point i is located; α is E d The weight of E s =||V 1 -(V 2 +u(V 3 -V 2 )+v(R 90 (V 3 -V 2 )))|| 2 , represents the local rigid constraint term; V 1 、V 2 、V 3 They represent the coordinates of the three vertices of the triangle enclosing the feature point i in the first grid; u and v represent coefficients, R 90 represents the known constraint; β is E s The weight of represents the optimal suture line constraint; n represents the number of equally divided points of the optimal suture line, w j V j represents the weighted average of the coordinates of the four vertices of the first grid where the equally divided point j is located, P j represents the coordinates of the equally divided point j when the spatial positions of the first image and the second image are aligned; γ is E l The weight of .
5. The method according to claim 1, It is characterized in that The step of stitching the second image and the third image comprises: According to the position of the optimal stitching line, the second image and the third image are subjected to weighted average fusion processing using the following formula: I=f*I 2 (x,y)+(1-f)*I 3 (x,y) Wherein, the value range of f is [0,1]; f is determined based on the distance between the pixel point in the second image and the optimal stitching line, and the smaller the distance, the larger f; 2 (x, y) represents the second image, I 3 (x, y) represents the third image, and I represents the image obtained by weighted average fusion processing of the second image and the third image.
6. The method according to claim 1, It is characterized in that After stitching the second image and the third image, the method further includes: Performing content completion on the fourth image to obtain a target image; wherein the fourth image is an image obtained by splicing the second image and the third image; Determining a second deformation parameter according to the optimal stitching line and the feature matching point set; wherein the second deformation parameter is a deformation parameter of the target image when aligning the spatial position of the target image with the fourth image; The target image is deformed using the second deformation parameter to obtain a composite image.
7. The method according to claim 6, It is characterized in that The determining of a second deformation parameter according to the optimal stitching line and the feature matching point set comprises: Performing grid division on the target image to obtain a plurality of second grids; According to the optimal stitching line and the feature matching point set, a second energy equation is constructed through feature point coordinate constraint items and optimal stitching line constraint items; For each of the second grids, determining, according to the second energy equation, a position of a second target grid corresponding to the second grid when aligning the spatial positions of the target image and the fourth image; wherein the position of the second target grid minimizes the value of the second energy equation; The second deformation parameter is determined according to the position of the second mesh and the position of the second target mesh.
8. The method according to claim 7, It is characterized in that The expression of the second energy equation is: 2 (E) = α 0 E d ′ +γ 0 E l ′ ; Among them, E d ′ =∑ i ‖w i V i -P i ′ ‖ 2 , represents the feature point coordinate constraint; w i V i represents the weighted average sum of the coordinates of the four vertices of the second grid where the feature point i is located, P i ′ represents the coordinates of the second target grid corresponding to the second grid where the feature point i is located; α 0 For E d ′ The weight of represents the optimal suture line constraint; n represents the number of equally divided points of the optimal suture line, w j V j represents the weighted average sum of the coordinates of the four vertices of the second grid where the equally divided point j is located, P j ′ represents the coordinates of the equally divided point j when aligning the spatial positions of the target image and the fourth image; γ 0 For E l ′ The weight of .
9. An electronic device, It is characterized in that The method comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image stitching method according to any one of claims 1 to 8.
10. A readable storage medium, It is characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image stitching method according to any one of claims 1 to 8 are implemented.
