Image stitching method, electronic device, and storage medium

Abstract

The application discloses an image splicing method, an electronic device and a storage medium, wherein the image splicing method comprises the following steps: determining an overlapping area and an intersection between a first to-be-spliced image and a second to-be-spliced image; dividing the overlapping area into a plurality of preset areas, determining a starting area and an ending area based on a preset area where the intersection is located; respectively calculating path energy values between each two adjacent preset areas, finding a plurality of target preset areas corresponding to a path with the minimum path energy value from the starting area to the ending area; determining a suture line based on the intersection and the center points of the target preset areas, and splicing the first to-be-spliced image and the second to-be-spliced image by using the suture line to obtain a spliced image. In the foregoing manner, the application can improve the naturalness of splicing and improve the image splicing accuracy.

Description

Technical Field

[0001] This invention relates to the field of image stitching, and in particular to image stitching methods, electronic devices, and storage media. Background Technology

[0002] Image stitching is a technique that combines a series of partially overlapping images depicting the same scene into a single wide-view image. It resolves the conflict between field of view and resolution, allowing for the creation of wide-view, high-resolution images. Image stitching technology has a wide range of applications.

[0003] Due to the complex and varied shooting environment and the strict requirements for shooting equipment, the existing image registration and mapping will likely have problems such as misalignment of overlapping areas and artifacts in the fused result after shooting from different angles.

[0004] Therefore, there is an urgent need to provide a stitching solution that can improve the accuracy of image stitching. Summary of the Invention

[0005] This invention provides an image stitching method, an electronic device, and a storage medium to solve the problems of misalignment of overlapping areas and artifacts in the fused result.

[0006] To address the aforementioned technical problems, this invention provides an image stitching method, comprising: acquiring a first image to be stitched and a second image to be stitched; determining the overlapping region and intersection point between the first image to be stitched and the second image to be stitched; dividing the overlapping region into multiple preset regions, and determining a starting region and an ending region based on the preset region where the intersection point is located; calculating the path energy value between each pair of adjacent preset regions, and finding multiple target preset regions corresponding to the path with the smallest total path energy value from the starting region to the ending region based on the path energy value; determining a stitching line between the first image to be stitched and the second image to be stitched based on the intersection point and the center point of each target preset region, and stitching the first image to be stitched and the second image to be stitched together using the stitching line to obtain a stitched image.

[0007] The steps of dividing the overlapping region into multiple preset regions and determining the starting and ending regions based on the preset regions where the intersection points are located include: dividing the overlapping region into multiple preset regions, and using graph theory to determine each preset region as a point, and determining the line connecting each two adjacent points as an arc; and determining the preset regions where the two intersection points are located as the starting point and the ending point, respectively; calculating the path energy value between each pair of adjacent preset regions, and finding multiple target preset regions corresponding to the path with the smallest total path energy value from the starting region to the ending region based on the path energy value, including: calculating the path energy value of each arc, and finding multiple target arcs corresponding to the path with the smallest total path energy value from the starting point to the ending point based on the path energy value; determining the corresponding target points based on the multiple target arcs, and determining the preset regions corresponding to each target point as target preset regions.

[0008] The process includes: calculating the path energy value between each pair of adjacent preset regions; finding multiple target preset regions corresponding to the path with the smallest total path energy value from the starting region to the ending region based on the path energy value; calculating the color difference and gradient difference between each preset region on the first stitched image and the corresponding preset region on the second stitched image; determining the region energy value of each preset region based on the color difference and gradient difference; determining the path energy value between each pair of adjacent preset regions using the region energy value of each preset region; and finding multiple target preset regions corresponding to the path with the smallest total path energy value from the starting region to the ending region based on the path energy value of each pair of adjacent preset regions.

[0009] The calculation of color differences and gradient differences between preset regions on the first stitched image and corresponding preset regions on the second stitched image includes: obtaining the size of the preset regions and the red, green, and blue channel values ​​of the pixels in each preset region on the first and second stitched images respectively; calculating the color differences of each preset region using the differences between the red, green, and blue channel values ​​of the pixels in each preset region on the first and second stitched images respectively, and the size of the preset regions; obtaining the gradient values ​​of the pixels in each preset region on the first and second stitched images respectively; and calculating the gradient differences of the corresponding preset regions using the differences between the gradient values ​​of the pixels in each preset region on the first and second stitched images respectively, and the size of the preset regions.

[0010] The determination of the regional energy value of each preset region based on the color difference and gradient difference of each preset region includes: determining the regional energy value of each preset region based on the sum of the squared value of the color difference and the gradient difference of each preset region.

[0011] Specifically, the path energy value between two adjacent preset regions is determined by using the norm of the difference between the regional energy value of each preset region and the regional energy value of the adjacent preset regions.

[0012] The process of acquiring a first image to be stitched and a second image to be stitched, and determining the overlapping area and intersection points between the first image to be stitched and the second image to be stitched, includes: acquiring the first image to be stitched and the second image to be stitched, and extracting features from the first image to be stitched and the second image to be stitched respectively to obtain multiple first feature points of the first image to be stitched and multiple second feature points of the second image to be stitched; mapping the first image to be stitched using the multiple first feature points and the multiple second feature points to obtain a mapped image with the same angle as the second image to be stitched; and determining the overlapping area and two intersection points between the mapped image and the second image to be stitched.

[0013] The process involves mapping a first image to be stitched together using multiple first feature points and multiple second feature points to obtain a mapped image with the same angle as a second image to be stitched together. This includes: performing feature matching on multiple first feature points and multiple second feature points, and calculating a homography matrix between the first and second images to be stitched together based on the successfully matched first and second feature points; and using an interpolation algorithm to perform inverse mapping on the first image to be stitched together using the homography matrix to obtain a mapped image with the same angle as the second image to be stitched together.

[0014] The process of determining the overlapping area and two intersection points between the mapped image and the second image to be stitched includes: obtaining the pixel values ​​of each pixel point on the second image to be stitched and the mapped image; determining the overlapping area using the pixels with the same pixel values ​​on the second image to be stitched and the mapped image; and determining the intersection point as the point where the edge of the second image to be stitched and the edge of the mapped image intersect.

[0015] Specifically, a stitching line is determined between the first image to be stitched and the second image to be stitched based on two intersection points and the center points of each target preset region. The stitching line is then used to stitch the first image to be stitched and the second image to be stitched to obtain a stitched image. This includes: sequentially connecting the center points of the next adjacent target preset regions from one intersection point until the line reaches another intersection point to obtain a stitching line; and stitching and fusing the mapped image on one side of the stitching line and the second image to be stitched on the other side of the stitching line to obtain a stitched image.

[0016] To address the aforementioned technical problems, the present invention also provides an electronic device comprising: a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the image stitching method described above.

[0017] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing program data that can be executed to implement the image stitching method described in any of the above embodiments.

[0018] The beneficial effects of the present invention are as follows: Unlike the prior art, the present invention determines the overlapping area and intersection point between the first image to be stitched and the second image to be stitched, divides the overlapping area into multiple preset areas, and determines the starting area and the ending area based on the preset area where the two intersection points are located. Thus, the preset area is used as a unit to determine the stitch line, which can reduce the number of units to determine the stitch line, improve the determination speed of the stitch line, reduce resource consumption, and improve the efficiency of image stitching. Furthermore, the path energy value between each pair of adjacent preset regions is calculated. Based on the path energy value, multiple target preset regions corresponding to the path with the smallest total path energy value are found from the starting region to the ending region. Finally, the stitching line between the first image to be stitched and the second image to be stitched is determined based on the intersection point and the center point of each target preset region. The stitching line is then used to stitch the first image to be stitched and the second image to be stitched to obtain the stitched image. The smallest total path energy value indicates that the similarity of the overlapping areas of the two images on that path is the highest. Determining the stitching line based on the path with the smallest total energy value makes the images on both sides of the stitching line most similar, thereby effectively reducing the artifact phenomenon generated during the image stitching process, improving the stitching naturalness, improving the image stitching accuracy, and making the stitched panoramic image have a good visual effect. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating an embodiment of the image stitching method provided by the present invention;

[0020] Figure 2 This is a flowchart illustrating another embodiment of the image stitching method provided by the present invention;

[0021] Figure 3 yes Figure 2 A schematic diagram of an embodiment of the overlapping area between the mapped image and the second image to be stitched;

[0022] Figure 4 yes Figure 2 A schematic diagram of an implementation method showing that the overlapping area of ​​the embodiment is divided into multiple preset areas;

[0023] Figure 5 yes Figure 2 The embodiments have rights without direction Figure 1 A schematic diagram of the implementation method;

[0024] Figure 6 yes Figure 2 A schematic diagram of one embodiment of the suture thread;

[0025] Figure 7 yes Figure 2 A schematic diagram of an embodiment of the stitched images;

[0026] Figure 8 This is a schematic diagram of the frame of an embodiment of the image stitching device of the present invention;

[0027] Figure 9 This is a schematic diagram of the structure of an embodiment of the electronic device provided by the present invention;

[0028] Figure 10 This is a schematic diagram of an embodiment of the computer-readable storage medium provided by the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0030] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the image stitching method provided by the present invention.

[0031] Step S11: Obtain the first image to be stitched and the second image to be stitched, and determine the overlapping area and intersection point between the first image to be stitched and the second image to be stitched.

[0032] The first image to be stitched and the second image to be stitched are obtained. The first image to be stitched and the second image to be stitched are images that include at least the same object, such as photos of the same object taken from different angles, or different images of the same object. The object can include objects, backgrounds, scenes, etc., and the specifics are not limited here.

[0033] Determine the overlapping area and intersection points between the first image to be stitched and the second image to be stitched. The intersection points can refer to the vertices of the overlapping area or the overlapping points of the overlapping area.

[0034] In one specific application scenario, features can be extracted from the first and second images to be stitched separately, and then the extracted features of the two images can be compared to identify the overlapping regions where the same features are found. In another specific application scenario, the first and second images to be stitched can also be overlapped to obtain the overlapping region. The method for determining the overlapping region is not limited here.

[0035] Step S12: Divide the overlapping area into multiple preset areas, and determine the starting area and ending area based on the preset area where the intersection point is located.

[0036] The overlapping area is divided into multiple preset areas. In this embodiment, the preset areas refer to areas of preset size and preset shape. That is, the overlapping area is divided into multiple areas of the same size and shape. The specific size of the preset area can be set based on actual needs and is not limited here. The preset shape includes regular shapes such as triangles, rectangles, circles, and trapezoids. Preferably, when the preset shape is a rectangle, it can reduce the amount of subsequent calculations and improve image stitching efficiency.

[0037] In a specific application scenario, when the intersection point is the vertex of the overlapping region, there are two intersection points. The preset regions where the two intersection points are located can be determined as the starting region and the ending region, respectively. This allows the starting region and the ending region to include as many preset regions as possible, improving the comprehensiveness of the subsequent suture line.

[0038] In another specific application scenario, when the intersection point is the overlapping point of the overlapping area, the two overlapping points with the greatest distance can be selected, and the preset areas where the two overlapping points are located can be determined as the starting area and the ending area, respectively. This can make the starting area and the ending area include as many preset areas as possible, thereby improving the comprehensiveness of the subsequent suture.

[0039] In another specific application scenario, when the intersection point is the overlapping point of the overlapping area, multiple overlapping points can be selected. The line connecting the multiple overlapping points can encompass most of the overlapping area. The route area is determined based on the preset area where the multiple overlapping points are located, and the starting area and the ending area are determined from the route area.

[0040] Step S13: Calculate the path energy value between each pair of adjacent preset regions, and find multiple target preset regions corresponding to the path with the smallest total path energy value from the starting region to the ending region based on the path energy value.

[0041] The path energy value between each pair of adjacent preset regions is calculated separately; that is, the path energy value from a given preset region to its adjacent preset regions. In one specific application scenario, the path energy value between each pair of adjacent preset regions can be determined by constructing an image energy function. In another specific application scenario, the path energy value between each pair of adjacent preset regions can also be detected using a pre-trained image detection model. This is not a limitation.

[0042] After determining the path energy value between each pair of adjacent preset regions, multiple target preset regions are found based on the path energy value from the starting region to the ending region, corresponding to the path with the smallest total path energy value.

[0043] There may be multiple paths between the starting region and the ending region. The preset region on the path with the smallest total energy value is determined as the target preset region.

[0044] The minimum total path energy value indicates the highest similarity between the overlapping regions of the two images along that path. Determining the target preset region based on this path can effectively reduce artifacts during image stitching and improve the naturalness of the stitching.

[0045] Step S14: Determine the stitching line between the first image to be stitched and the second image to be stitched based on the intersection point and the center point of each target preset area, and use the stitching line to stitch the first image to be stitched and the second image to be stitched to obtain the stitched image.

[0046] After determining the target region, a seam line is determined between the first image to be stitched and the second image to be stitched, based on the intersection point and the center point of each preset target region. Determining the seam line by preset region reduces the number of seam lines, increases the speed of seam line determination, and improves the efficiency of image stitching.

[0047] In a specific application scenario, when the intersection point is the vertex of the overlapping region, there are two intersection points. The line connecting the two intersection points and the center point of each target preset region is determined as the stitching line between the first image to be stitched and the second image to be stitched.

[0048] In another specific application scenario, when the intersection point is the overlap point of the overlapping area, the stitching line between the first image to be stitched and the second image to be stitched can be determined based on the two overlapping points with the greatest distance and the center point of each target preset area.

[0049] In another specific application scenario, when the intersection point is the overlap point of an overlapping region, the stitching line between the first image to be stitched and the second image to be stitched can also be determined based on the lines connecting multiple selected overlap points to the center points of each target preset region. No further limitations are imposed here.

[0050] Finally, the first and second images to be stitched together are joined using a suture line to obtain a stitched image. Specifically, the stitching line can be used as a boundary, with the first stitched image taken from one side and the second stitched image taken from the other side. The images are then fused and stitched together based on the positional boundary of the stitching line to obtain the stitched image.

[0051] Through the above steps, the image stitching method of this embodiment determines the overlapping area and intersection point between the first image to be stitched and the second image to be stitched, divides the overlapping area into multiple preset areas, determines the starting area and the ending area based on the preset area where the intersection point is located, and uses the preset area as a unit to determine the stitching line. This reduces the number of units to determine the stitching line, increases the speed of determining the stitching line, reduces resource consumption, and improves the efficiency of image stitching. Furthermore, the path energy value between each pair of adjacent preset regions is calculated. Based on the path energy value, multiple target preset regions corresponding to the path with the smallest total path energy value are found from the starting region to the ending region. Finally, the stitching line between the first image to be stitched and the second image to be stitched is determined based on the intersection point and the center point of each target preset region. The stitching line is then used to stitch the first image to be stitched and the second image to be stitched to obtain the stitched image. The smallest total path energy value indicates that the similarity of the overlapping areas of the two images on that path is the highest. Determining the stitching line based on the path with the smallest total energy value makes the images on both sides of the stitching line most similar, thereby effectively reducing the artifact phenomenon generated during the image stitching process, improving the stitching naturalness, enhancing the image stitching quality, and giving the stitched panoramic image a good visual effect.

[0052] Please see Figure 2 , Figure 2 This is a flowchart illustrating another embodiment of the image stitching method provided by the present invention.

[0053] Step S21: Obtain the first image to be stitched and the second image to be stitched, and extract features from the first image to be stitched and the second image to be stitched respectively to obtain multiple first feature points of the first image to be stitched and multiple second feature points of the second image to be stitched; use the multiple first feature points and multiple second feature points to map the first image to be stitched to obtain a mapped image with the same angle as the second image to be stitched; determine the overlapping area and two intersection points between the mapped image and the second image to be stitched.

[0054] The first image to be stitched and the second image to be stitched are obtained. The first image to be stitched and the second image to be stitched may contain at least 30% of the same objects. Specifically, they may include 30%, 40%, 60%, 80%, 90%, etc., which are not limited here.

[0055] Feature extraction is performed on both the first and second images to be stitched, resulting in multiple first feature points in the first image and multiple second feature points in the second image. In one specific application scenario, the SIFT feature detection algorithm can be used to determine multiple first SIFT feature points in the first image and multiple second SIFT feature points in the second image. In another specific application scenario, the FAST (features from accelerated segment test, corner detection algorithm) algorithm can be used to detect ORB feature points, resulting in multiple first ORB feature points in the first image and multiple second ORB feature points in the second image. The type and acquisition method of the feature points are not limited here.

[0056] The first image to be stitched is mapped using multiple first feature points and multiple second feature points to obtain a mapped image with the same angle as the second image to be stitched. Specifically, feature matching can be performed on the multiple first feature points and multiple second feature points, and the homography matrix between the first image to be stitched and the second image to be stitched can be calculated based on the successfully matched first feature points and second feature points. Finally, the first image to be stitched is reverse-mapped using the homography matrix through an interpolation algorithm to map the first image to be stitched onto the plane of the second image to be stitched, obtaining a mapped image with the same angle as the second image to be stitched, so as to facilitate subsequent stitching.

[0057] In a specific application scenario, the RANSAC (Random Sample Consensus) algorithm can be used to perform feature matching on multiple first feature points and multiple second feature points, filtering out mismatched first and second feature points. Then, based on the successfully matched first and second feature points, the homography matrix between the first and second images to be stitched can be calculated. Finally, a bilinear interpolation algorithm with inverse mapping is used to map the first image to be stitched, resulting in a mapped image with the same angle as the second image to be stitched. That is, multiplying the coordinates of each pixel in the mapped image by the inverse of the homography matrix yields the coordinates of each pixel in the first image to be stitched.

[0058] During the mapping process, the coordinates of each pixel in the first image to be stitched are calculated and processed with the homography matrix to obtain the coordinates of the processed pixels, forming a mapped image. Then, the empty points on the mapped image are filled in using a bilinear interpolation algorithm.

[0059] By first mapping the first image to be stitched to obtain a mapped image with the same angle as the second image to be stitched, and then determining the overlapping area and two intersection points between the mapped image and the second image to be stitched, the angular distortion of the stitched image can be reduced, and the mapped image and the second image to be stitched can be located on the same plane, which facilitates the determination of the subsequent stitching line and improves the naturalness of the stitched image.

[0060] After obtaining the mapped image of the first image to be stitched, the overlapping area and two intersection points between the mapped image and the second image to be stitched are determined.

[0061] In a specific application scenario, the pixel values ​​of each pixel on the second image to be stitched and the mapped image can be obtained. Pixels with the same pixel values ​​on both images are used to determine the overlapping region. The points where the edges of the second image to be stitched and the edges of the mapped image intersect are defined as the intersection points, i.e., the vertices of the overlapping region. In other application scenarios, other methods can also be used to determine the overlapping region.

[0062] In this step, the overlapping area is roughly determined based on pixel points. If stitching is performed directly based on this overlapping area, there is a high probability of problems such as misalignment of the overlapping areas and artifacts in the fused result. Therefore, this embodiment determines the subsequent stitching line based on the approximate overlapping area to improve the accuracy and effect of stitching.

[0063] Please see Figure 3 , Figure 3 yes Figure 2 A schematic diagram of an embodiment of the overlapping area between the mapped image and the second image to be stitched.

[0064] In this embodiment, the overlapping area 30 contains an artifact area 31, which requires subsequent processing to remove the artifacts in the artifact area 31 and improve the naturalness of the stitching.

[0065] Step S22: Divide the overlapping area into multiple preset areas, and determine the preset areas where the two intersection points are located as the starting area and the ending area, respectively.

[0066] The overlapping region is divided into multiple predefined regions. In a specific application scenario, the overlapping region can be divided into n rectangular regions with width and height (w, h), and the set of multiple predefined regions is denoted as S = {s1, s2, ... sn}. n The specific values ​​of n, w, and h can be set based on actual needs and are not limited here.

[0067] Please see Figure 4 , Figure 4 yes Figure 2 The overlapping area of ​​the embodiment is divided into multiple preset areas - a schematic diagram of an implementation method.

[0068] In this embodiment, the overlapping region 30 is evenly divided into a rectangular preset region 32 with a width of w and a height of h. If the overlapping region 30 is an irregular shape and the preset region 32 extends beyond the boundary of the overlapping region 30 during division, the pixel value of the portion extending beyond the boundary is set to 0, i.e., black.

[0069] In a specific application scenario, after dividing the overlapping region into n preset regions, graph theory is used to define each preset region as a point, and the line connecting any two adjacent points is defined as an arc, denoted as (si, sj), where i and j belong to 1-n. The preset regions containing the two intersection points are then defined as the starting and ending points, respectively. In other words, each preset region is represented in the form of a weighted undirected graph.

[0070] Please see Figure 5 , Figure 5 yes Figure 2 The embodiments have rights without direction Figure 1 A schematic diagram of the implementation method.

[0071] This implementation method is illustrated using a preset area, i.e., 11 points, as an example.

[0072] Point 1 is the starting point, and point 11 is the ending point. The lines connecting any two adjacent points are arcs. For example, the arc between point 1 and point 2 is denoted as (s1, s2). The arc between point 2 and point 5 is denoted as (s2, s5).

[0073] By constructing each preset region as a weighted undirected graph, it becomes easier to solve for the optimal path from the starting point to the ending point.

[0074] Step S23: Calculate the color difference and gradient difference between each preset region on the first stitched image and the corresponding preset region on the second stitched image; determine the region energy value of each preset region based on the color difference and gradient difference of each preset region; determine the path energy value between each two adjacent preset regions using the region energy value of each preset region; find multiple target preset regions corresponding to the path with the smallest total path energy value from the starting region to the ending region based on the path energy value of each two adjacent preset regions.

[0075] In a specific application scenario, the path energy value of each arc is calculated separately. Based on the path energy value, multiple target arcs corresponding to the path with the smallest total path energy value are found from the starting point to the ending point. Based on the multiple target arcs, the corresponding target points are determined, and the preset areas corresponding to each target point are determined as the target preset areas.

[0076] Specifically, the color difference and gradient difference between each preset region on the stitching area corresponding to the first stitched image and the corresponding preset region on the second stitched image are calculated.

[0077] In a specific application scenario, the dimensions of a preset region, namely w and h, are obtained, along with the red, green, and blue channel values ​​of the pixels in each preset region on both the first and second images to be stitched. Then, using the differences between the red, green, and blue channel values ​​of the pixels in each preset region on both images, and the dimensions of the preset region, the color difference between each preset region is calculated. The preset region s... i The formula for calculating color difference is as follows:

[0078]

[0079] Among them, s i ∈S,i=(1,2...,n),C(s) i ) represents the preset region s i Color differences, P is the preset area s i The pixel coordinates are given, and RGB represents the three color channels of the image. r0(p) represents the red channel value of the mapped image at point p, and r1(p) represents the red channel value of the second stitched image at point p. Similarly, g0(p) represents the green channel value of the mapped image at point p, g1(p) represents the green channel value of the second stitched image at point p, b0(p) represents the blue channel value of the mapped image at point p, and b1(p) represents the blue channel value of the second stitched image at point p. w represents the preset region s. i Width, h preset area s i of high.

[0080] The above formula is used to calculate each preset region s. i The average color difference between each pixel in the preset region s is obtained. i The color differences.

[0081] In a specific application scenario, the gradient values ​​of pixels in each preset region on the first and second images to be stitched are obtained. The gradient difference of the corresponding preset region is calculated using the difference between the gradient values ​​of pixels in each preset region on the first and second images to be stitched, and the size of the preset region. The gradient values ​​can be calculated using the Sobel operator, the general formula for image gradients, or the median difference formula, etc., and are not limited here.

[0082] Among them, the preset area s i The formula for calculating the gradient difference is as follows:

[0083]

[0084] Among them, s i∈S,i=(1,2...,n),G(s i ) represents the preset region s i The gradient difference is given by t0(p), which represents the gradient value of the mapped image at point p, and t1(p), which represents the gradient value of the second image to be stitched at point p.

[0085] The above formula is used to calculate each preset region s. i The average gradient difference between each pixel in the preset region s is used to obtain the results. i The gradient difference.

[0086] The region energy value of each preset region is determined based on the color and gradient differences between them. Specifically, the region energy value of each preset region is determined by the sum of the squared value of the color difference and the gradient difference, i.e., the region energy value of each point. Preset region s i The formula for calculating the regional energy value is as follows:

[0087] E(s i )=C(s i ) 2 +G(s i )

[0088] Among them, E(s) i ) represents the preset region s i The regional energy value.

[0089] The path energy value between any two adjacent preset regions is determined using the region energy value of each preset region. Specifically, the path energy value between any two adjacent preset regions, i.e., the path energy value of each arc, is determined using the norm of the difference between the region energy value of each preset region and the region energy values ​​of adjacent preset regions. The formula for calculating the path energy value between two adjacent preset regions is as follows:

[0090] E(s i ,s j )=||E(s i )-E(s j )||

[0091] Where i≠j,s i ,s j ∈S,E(s i ,s j ) represents the preset region s i With preset area s j The path energy value between them.

[0092] Based on the path energy values ​​of each pair of adjacent preset regions, multiple target preset regions corresponding to the path with the minimum total path energy value between the starting region and the ending region are found. That is, the target points corresponding to multiple target arcs corresponding to the path with the minimum total path energy value between the starting point and the ending point are found. The objective function can be expressed as the path energy value from the starting point to the ending point being minimized.

[0093] min E(s 起 ,s 终 )=∑E(s i ,s j )

[0094] Wherein, min E(s) 起 ,s 终 The path with the minimum energy value is the path between the preset area where the starting point is located and the preset area where the ending point is located.

[0095] In a specific application scenario, in order to find a path with minimum energy, the Floyd-Warshall algorithm or other computational methods can be used to find the path with minimum energy from the starting point to the ending point in a weighted undirected graph.

[0096] Based on the above path, the target points are determined, and the preset areas corresponding to each target point are determined as the target preset areas.

[0097] Step S24: Connect the center points of the next adjacent target preset areas sequentially from one intersection point until the line reaches another intersection point to obtain a suture line; stitch the mapped image on one side of the suture line and the second image to be stitched on the other side of the suture line to obtain a stitched image.

[0098] Since the shortest path obtained in step S23 is a strip-shaped path with rectangular areas as units, this step needs to refine it into a line. Specifically, connect the center points of the next adjacent target preset areas from the starting point to the end point to obtain the stitching line.

[0099] Please see Figure 6 , Figure 6 yes Figure 2 A schematic diagram of one embodiment of the suture.

[0100] In this embodiment, the left side is the stitched image 61, the right side is the second image to be stitched 62, and the stitching is performed in the middle based on the position of the suture line 63, that is, different images to be stitched are taken on both sides of the suture line 63.

[0101] The mapped image on one side of the suture line and the second image to be stitched on the other side of the suture line are stitched together to obtain the stitched image.

[0102] In a specific application scenario, the Laplacian pyramid fusion method can be chosen to merge overlapping areas, eliminating unnatural transitions between two images to be stitched due to exposure differences, ultimately resulting in a stitched image. Other fusion methods can also be used in other application scenarios, and no specific method is specified here.

[0103] Please see Figure 7 , Figure 7 yes Figure 2 A stitched image of an embodiment—a schematic diagram of an implementation method.

[0104] The stitched image obtained after merging has no artifacts, a natural transition, and a good stitching effect.

[0105] In a specific application scenario, when the image stitching method is applied to stitching multiple images, the image stitching method of this embodiment can be used to stitch the images one by one until the multiple images are stitched into a single stitched image, thereby reducing artifacts on the stitched image and improving the stitching effect.

[0106] Through the above steps, the image stitching method of this embodiment first maps the first image to be stitched to obtain a mapped image with the same angle as the second image to be stitched. Then, it determines the overlapping area and two intersection points between the mapped image and the second image to be stitched. This reduces angular distortion of the stitched image, homogenizes the characteristics of the mapped image and the second image to be stitched, facilitates the determination of subsequent stitching lines, and improves the naturalness of the stitched image. Furthermore, the overlapping area is divided into multiple preset areas, allowing the determination of subsequent stitching lines to be performed on a unit-by-unit basis, thereby improving the efficiency of stitching line determination. Moreover, by constructing each preset area as a weighted undirected graph, it is easier to solve for the optimal path from the starting point to the ending point. This embodiment also calculates the color difference and gradient difference of each preset region to determine the region energy value of each preset region, and then uses the region energy value of each preset region to determine the path energy value between each two adjacent preset regions; finally, it finds multiple target preset regions corresponding to the path with the smallest total path energy value, and can use the multiple target preset regions corresponding to the path with the smallest total path energy value to determine the stitching line, which can effectively reduce artifacts, alignment errors, visible stitching areas and other problems in the stitched image, improve the image stitching quality, and make the stitched panoramic image have a good visual effect.

[0107] Please see Figure 8 , Figure 8This is a schematic diagram of the framework of an embodiment of the image stitching device of the present invention. The image stitching device 80 includes an acquisition module 81, a division module 82, a calculation module 83, and a stitching module 84. The acquisition module 81 is used to acquire a first image to be stitched and a second image to be stitched, and to determine the overlapping area and intersection point between the first image to be stitched and the second image to be stitched; the division module 82 is used to divide the overlapping area into multiple preset areas, and to determine the starting area and the ending area based on the preset area where the intersection point is located; the calculation module 83 is used to calculate the path energy value between each pair of adjacent preset areas, and to find multiple target preset areas corresponding to the path with the smallest total path energy value from the starting area to the ending area based on the path energy value; the stitching module 84 is used to determine the stitching line between the first image to be stitched and the second image to be stitched based on the intersection point and the center point of each target preset area, and to stitch the first image to be stitched and the second image to be stitched using the stitching line to obtain a stitched image.

[0108] The partitioning module 82 is also used to divide the overlapping area into multiple preset areas, and use graph theory to determine each preset area as a point, and the line connecting each two adjacent points as an arc; and to determine the preset areas where the two intersection points are located as the starting point and the ending point, respectively; the calculation module 83 is also used to calculate the path energy value of each arc, and based on the path energy value, find the multiple target arcs corresponding to the path with the smallest total path energy value from the starting point to the ending point; determine the corresponding target points based on the multiple target arcs, and determine the preset areas corresponding to each target point as target preset areas.

[0109] The calculation module 83 is also used to calculate the color difference and gradient difference between each preset region on the first stitched image and the corresponding preset region on the second stitched image; determine the region energy value of each preset region based on the color difference and gradient difference of each preset region; determine the path energy value between each two adjacent preset regions using the region energy value of each preset region; and find multiple target preset regions corresponding to the path with the smallest total path energy value from the starting region to the ending region based on the path energy value of each two adjacent preset regions.

[0110] The calculation module 83 is further configured to obtain the size of the preset region and the red channel value, green channel value, and blue channel value of the pixels in each preset region on the first image to be stitched and the second image to be stitched, respectively; calculate the color difference of each preset region by using the corresponding differences between the red channel value, green channel value, and blue channel value of the pixels in each preset region on the first image to be stitched and the second image to be stitched, respectively, and the size of the preset region; and obtain the gradient value of the pixels in each preset region on the first image to be stitched and the second image to be stitched, respectively; calculate the gradient difference of the corresponding preset region by using the differences between the gradient values ​​of the pixels in each preset region on the first image to be stitched and the second image to be stitched, respectively, and the size of the preset region.

[0111] The calculation module 83 is also used to determine the regional energy value of each preset region based on the sum of the squared value of the color difference of each preset region and the gradient difference.

[0112] The calculation module 83 is also used to determine the path energy value between two adjacent preset regions by using the norm of the difference between the regional energy value of each preset region and the regional energy value of the adjacent preset regions.

[0113] The acquisition module 81 is further configured to acquire a first image to be stitched and a second image to be stitched, and to extract features from the first image to be stitched and the second image to be stitched respectively, to obtain multiple first feature points of the first image to be stitched and multiple second feature points of the second image to be stitched; to map the first image to be stitched using the multiple first feature points and the multiple second feature points, to obtain a mapped image with the same angle as the second image to be stitched; and to determine the overlapping area and two intersection points between the mapped image and the second image to be stitched.

[0114] The acquisition module 81 is also used to perform feature matching on multiple first feature points and multiple second feature points, and calculate the homography matrix between the first image to be stitched and the second image to be stitched based on the successfully matched first feature points and second feature points; and use the homography matrix to perform inverse mapping on the first image to be stitched using an interpolation algorithm to obtain a mapped image with the same angle as the second image to be stitched.

[0115] The acquisition module 81 is also used to acquire the pixel values ​​of each pixel point on the second image to be stitched and the mapped image, and to determine the overlapping area using the pixels with the same pixel values ​​on the second image to be stitched and the mapped image; and to determine the intersection point between the edge of the second image to be stitched and the edge of the mapped image as the intersection point.

[0116] The stitching module 84 is also used to sequentially connect the center points of the next adjacent target preset areas from one intersection point until the line reaches another intersection point to obtain a stitch line; and to stitch and fuse the mapped image on one side of the stitch line and the second image to be stitched on the other side of the stitch line to obtain a stitched image.

[0117] The above solution can improve the naturalness of the stitching and enhance the accuracy of image stitching.

[0118] Based on the same inventive concept, the present invention also proposes an electronic device capable of executing the image stitching method of any of the above embodiments. Please refer to [link to relevant documentation]. Figure 9 , Figure 9 This is a schematic diagram of an embodiment of the electronic device provided by the present invention. The electronic device includes a processor 91 and a memory 92.

[0119] The processor 91 executes the program instructions stored in the memory 92 to implement the steps of any of the above-described image stitching methods. In a specific implementation scenario, the electronic device may include, but is not limited to, a microcomputer or a server. In addition, the electronic device may also include mobile devices such as laptops and tablets, which are not limited here.

[0120] Specifically, processor 91 controls itself and memory 92 to implement the steps of any of the above embodiments. Processor 91 may also be referred to as a CPU (Central Processing Unit). Processor 91 may be an integrated circuit chip with signal processing capabilities. Processor 91 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 91 may be implemented using integrated circuit chips.

[0121] The above solution can improve the naturalness of the stitching and enhance the accuracy of image stitching.

[0122] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium, please refer to [link to relevant documentation]. Figure 10 , Figure 10This is a schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. The computer-readable storage medium 100 stores at least one program data 101, which is used to implement any of the methods described above. In one embodiment, the computer-readable storage medium 100 includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0123] In the several embodiments provided by this invention, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0124] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0125] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0126] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium.

[0127] The above are merely embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

[0128] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. An image stitching method, characterized in that, include: Obtain a first image to be stitched and a second image to be stitched, and determine the overlapping area and intersection point between the first image to be stitched and the second image to be stitched; The overlapping region is divided into multiple preset regions, and the starting region and the ending region are determined based on the preset region where the intersection point is located; wherein, graph theory is used to determine each preset region as a point, and the line connecting each two adjacent points is determined as an arc; and the preset regions where the two intersection points are located are respectively determined as the starting point and the ending point; The path energy value between each pair of adjacent preset regions is calculated. Based on the path energy value, multiple target preset regions corresponding to the path with the smallest total path energy value are found from the starting region to the ending region. Specifically, the color difference and gradient difference between each preset region on the first image to be stitched and the corresponding preset region on the second image to be stitched are calculated. The region energy value of each preset region is determined based on the color difference and gradient difference. The path energy value between each pair of adjacent preset regions is determined using the region energy value of each preset region. Based on the intersection point and the center point of each of the target preset regions, a suture line is determined between the first image to be stitched and the second image to be stitched, and the suture line is used to stitch the first image to be stitched and the second image to be stitched to obtain a stitched image.

2. The image stitching method according to claim 1, characterized in that, The step of calculating the path energy value between each pair of adjacent preset regions, and finding multiple target preset regions corresponding to the path with the smallest total path energy value from the starting region to the ending region based on the path energy value, includes: Calculate the path energy value of each arc, and based on the path energy value, find the multiple target arcs corresponding to the path with the smallest total path energy value from the starting point to the ending point; Based on the multiple target arcs, corresponding target points are determined, and the preset areas corresponding to each target point are determined as the target preset areas.

3. The image stitching method according to claim 1, characterized in that, The calculation of color differences and gradient differences between each preset region on the first image to be stitched and the corresponding preset region on the second image to be stitched includes: The size of the preset region and the red channel value, green channel value and blue channel value of the pixels in each preset region on the first image to be stitched and the second image to be stitched are obtained respectively; By using the differences between the red, green, and blue channel values ​​of the pixels in each of the preset regions in the first and second images to be stitched, respectively, and the size of the preset regions, the color difference of each preset region is calculated; and The gradient values ​​of the pixels in each of the preset regions are obtained in the first image to be stitched and the second image to be stitched. The gradient difference of the preset region is calculated by using the difference between the gradient values ​​of the pixels in each preset region on the first image to be stitched and the second image to be stitched, and the size of the preset region.

4. The image stitching method according to claim 1, characterized in that, The determination of the regional energy value of each preset region based on the color difference and gradient difference of each preset region includes: The regional energy value of each preset region is determined based on the sum of the squared value of the color difference and the gradient difference of each preset region.

5. The image stitching method according to claim 1, characterized in that, The step of determining the path energy value between two adjacent preset regions using the regional energy values ​​of each preset region includes: The path energy value between two adjacent preset regions is determined by using the norm of the difference between the regional energy value of each preset region and the regional energy value of an adjacent preset region.

6. The image stitching method according to claim 1, characterized in that, The process of acquiring the first image to be stitched and the second image to be stitched, and determining the overlapping area and intersection points between the first image to be stitched and the second image to be stitched, includes: A first image to be stitched and a second image to be stitched are obtained, and feature extraction is performed on the first image to be stitched and the second image to be stitched respectively to obtain multiple first feature points of the first image to be stitched and multiple second feature points of the second image to be stitched. The first image to be stitched is mapped using the plurality of first feature points and the plurality of second feature points to obtain a mapped image with the same angle as the second image to be stitched. Determine the overlapping area between the mapped image and the second image to be stitched together, as well as the two intersection points.

7. The image stitching method according to claim 6, characterized in that, The step of mapping the first image to be stitched together using the plurality of first feature points and the plurality of second feature points to obtain a mapped image with the same angle as the second image to be stitched together includes: The plurality of first feature points and the plurality of second feature points are matched, and the homography matrix between the first image to be stitched and the second image to be stitched is calculated based on the successfully matched first feature points and second feature points. The first image to be stitched is reverse-mapped using the homography matrix through an interpolation algorithm to obtain a mapped image with the same angle as the second image to be stitched.

8. The image stitching method according to claim 6, characterized in that, Determining the overlapping region between the mapped image and the second image to be stitched, as well as the two intersection points, includes: The pixel values ​​of each pixel on the second image to be stitched and the mapped image are obtained, and the overlapping area is determined by using the pixels with the same pixel values ​​on the second image to be stitched and the mapped image. The intersection point is determined as the point where the edge of the second image to be stitched intersects with the edge of the mapped image.

9. The image stitching method according to claim 6, characterized in that, The step of determining the suture line between the first image to be stitched and the second image to be stitched based on the intersection point and the center point of each of the target preset regions, and stitching the first image to be stitched and the second image to be stitched together using the suture line to obtain a stitched image, includes: The suture line is obtained by sequentially connecting the center points of the next adjacent target preset areas from one intersection point until another intersection point is reached. The mapped image on one side of the suture and the second image to be stitched on the other side of the suture are stitched together to obtain the stitched image.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the image stitching method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program data that can be executed to implement the image stitching method as described in any one of claims 1 to 9.

Images