Image splicing method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an image splicing method and device, equipment and a storage medium. The image splicing method can comprise the steps that the displacement distance between a first picture and a second picture and the confidence coefficient corresponding to the displacement distance are determined; and according to the displacement distance and the confidence coefficient, splicing the first picture and the second picture to obtain a target picture. By implementing the method, errors during image splicing can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the technical field of electronic equipment, and in particular to an image stitching method, device, equipment and storage medium. Background Art

[0002] With the development of smart terminals, the application scenarios of image stitching technology are increasing. Therefore, how to accurately perform image stitching is particularly important. Summary of the invention

[0003] The embodiments of the present application provide an image stitching method, apparatus, device and storage medium, which can effectively reduce the error during image stitching.

[0004] A first aspect of an embodiment of the present application provides an image stitching method, comprising:

[0005] Determine a displacement distance between the first image and the second image, and a confidence level corresponding to the displacement distance;

[0006] The first image and the second image are spliced ​​according to the displacement distance and the confidence level to obtain a target image.

[0007] A second aspect of an embodiment of the present application provides an image stitching device, comprising:

[0008] A displacement determination unit, used to determine a displacement distance between the first image and the second image, and a confidence level corresponding to the displacement distance;

[0009] An image stitching unit is used to stitch the first image and the second image together according to the displacement distance and the confidence level to obtain a target image.

[0010] A third aspect of the embodiments of the present application provides an electronic device, including:

[0011] A memory storing executable program code;

[0012] and a processor coupled to the memory;

[0013] The processor calls the executable program code stored in the memory, and when the executable program code is executed by the processor, the processor implements the method described in the first aspect of the embodiment of the present application.

[0014] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having executable program code stored thereon. When the executable program code is executed by a processor, the method described in the first aspect of the embodiment of the present application is implemented.

[0015] A fifth aspect of an embodiment of the present application discloses a computer program product. When the computer program product runs on a computer, the computer executes the method described in the first aspect of the embodiment of the present application.

[0016] A sixth aspect of an embodiment of the present application discloses an application publishing platform, which is used to publish a computer program product. When the computer program product runs on a computer, the computer executes the method described in the first aspect of the embodiment of the present application.

[0017] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0018] In an embodiment of the present application, a displacement distance between a first image and a second image and a confidence level corresponding to the displacement distance are determined; and the first image and the second image are spliced ​​according to the displacement distance and the confidence level to obtain a target image.

[0019] By implementing this method, the displacement distance between the first image and the second image and the confidence level corresponding to the displacement distance are first determined, and then the first image and the second image are spliced ​​based on the displacement distance and the corresponding confidence level. This can effectively reduce the error during image splicing, thereby improving the accuracy of image splicing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments and the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained based on these drawings.

[0021] Figure 1 It is a scene diagram of the image stitching method disclosed in the embodiment of the present application;

[0022] Figure 2 It is a schematic diagram of a long screenshot scenario disclosed in an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a panoramic image scene disclosed in an embodiment of the present application;

[0024] Figure 4 It is a flowchart of the image stitching method disclosed in the embodiment of the present application;

[0025] Figure 5 is another flowchart of the image stitching method disclosed in the embodiment of the present application;

[0026] Figure 6 It is another flowchart of the image stitching method disclosed in the embodiment of the present application;

[0027] Figure 7 is a structural diagram of the image stitching device disclosed in the embodiment of the present application;

[0028] Figure 8 It is a structural diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The embodiments of the present application provide an image stitching method, apparatus, device and storage medium, which can effectively reduce the error during image stitching.

[0030] In order to make the technical personnel in the technical field better understand the scheme of the present application, the technical scheme in the embodiment of the present application will be described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, they should all fall within the scope of protection of the present application.

[0031] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0032] "At least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c, where a, b, c can be single or plural.

[0033] See also Figure 1 , Figure 1 is a scene diagram of the image stitching method disclosed in the embodiment of the present application. Figure 1 The scene diagram shown includes an electronic device 10 , on which the image stitching method can be applied.

[0034] Image stitching refers to stitching multiple images with overlapping areas into a larger image. The output of image stitching is the union of multiple input images. With the development of smart terminals, the application scenarios of image stitching technology are increasing. Therefore, how to stitch images accurately is particularly important.

[0035] Exemplary usage scenarios of image stitching technology are as follows:

[0036] Usage scenario 1: long screenshot scenario, that is, the user wants to capture a complete image of the current application interface from beginning to end, but this complete image cannot be displayed on the screen of the electronic device 10 at one time, and needs to be displayed by sliding based on the user's sliding operation. In order to capture the complete image, the electronic device 10 can periodically perform the screenshot operation during the sliding display of the current application interface, and then stitch all the captured images through the image stitching technology, so as to achieve the purpose of capturing a complete image of the current application interface from beginning to end.

[0037] For example, see Figure 2 , Figure 2 Schematic diagram of a long screenshot scenario disclosed in the embodiment of the present application. Figure 2 The user updates the display content on the display screen of the electronic device 10 by sliding up. The electronic device 10 can periodically perform a screenshot operation when the display screen updates the display content, and splice the obtained screenshots using a splicing technology.

[0038] Use scenario 2, the panoramic image scenario, that is, through the image stitching technology, a group of small-viewing angle images with overlapping areas are stitched into a new image with a wide field of view.

[0039] For example, see Figure 3 , Figure 3 It is a schematic diagram of a panoramic image scene disclosed in an embodiment of the present application. Figure 3 The electronic device 10 may use image stitching technology to stitch pictures 1, 2, 3 and 4 into a new picture with a wide viewing angle.

[0040] Among them, the electronic device 10 may include general handheld screen electronic devices, such as mobile phones, smart phones, portable terminals, terminals, personal digital assistants (PDA), portable multimedia players (Personal Media Player, PMP) devices, laptop computers, notebooks (Note Pad), wireless broadband (Wireless Broadband, Wibro) terminals, tablet computers (Personal Computer, PC), smart PCs, sales terminals (Point of Sales, POS) and car computers, etc.

[0041] The electronic device 10 may also include a wearable device. A wearable device is a portable electronic device that can be worn directly on the user or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but can also realize powerful intelligent functions through software support, data interaction, and cloud server interaction, such as: computing function, positioning function, and alarm function. At the same time, they can also connect to mobile phones and various terminals. Wearable devices may include but are not limited to watches supported by wrists (such as watches, wrists, etc.), shoes supported by feet (such as shoes, socks or other products worn on the legs), glasses supported by the head (such as glasses, helmets, headbands, etc.), and various non-mainstream product forms such as smart clothing, school bags, crutches, accessories, etc.

[0042] In the technical solution of the present application, the electronic device 10 first determines the displacement distance between the first image and the second image, and the confidence level corresponding to the displacement distance, and then splices the first image and the second image based on the displacement distance and the corresponding confidence level, which can effectively reduce the error during image stitching and thereby improve the accuracy of image stitching.

[0043] It should be noted that the confidence level can be used to measure the degree of deviation of the displacement distance. The larger the absolute value of the confidence level, the smaller the degree of deviation of the displacement distance.

[0044] See also Figure 4 , Figure 4 FIG. 1 is a flowchart of the image stitching method disclosed in the embodiment of the present application. Figure 4 The image stitching method shown may include the following steps:

[0045] 401. The electronic device determines a displacement distance between a first image and a second image, and a confidence level corresponding to the displacement distance.

[0046] Among them, the first picture and the second picture can be obtained by taking pictures of the electronic device, or by performing a screenshot operation on the electronic device, which is not limited in the embodiments of the present application.

[0047] It is understandable that in the long screenshot scenario, the first picture and the second picture are from the screenshot screen obtained by the electronic device performing the screenshot operation. In the full scene image scenario, the first picture and the second picture are from pictures taken by the electronic device.

[0048] The displacement distance may refer to the displacement distance of the first picture relative to the second picture, or may refer to the displacement distance of the second picture relative to the first picture. The displacement distance may include a horizontal displacement distance and / or a vertical displacement distance.

[0049] In some embodiments, the electronic device determines the displacement distance between the first image and the second image, and the confidence level corresponding to the displacement distance, which may include:

[0050] The electronic device uses a preset feature extraction algorithm to extract feature points of the first image and the second image respectively;

[0051] According to the feature vectors of the feature points of the first image and the second image, the feature points of the first image and the second image are matched to obtain a plurality of feature point matching pairs; wherein the feature vector errors of the feature points in each feature point matching pair are less than a vector error threshold;

[0052] Determine the displacement distance between the first image and the second image according to the displacement distance of each feature point matching pair;

[0053] According to the feature vector error of each feature point matching pair, the confidence level corresponding to the displacement distance is determined.

[0054] Among them, feature points can also be called key points, which can identify objects in the picture.

[0055] In some embodiments, the preset feature extraction algorithm may include but is not limited to at least one of the following: SURF algorithm, SIFT feature extraction algorithm, Harris corner feature extraction algorithm, ORB feature extraction algorithm, FAST algorithm, etc.

[0056] In some embodiments, determining the displacement distance between the first image and the second image based on the displacement distance of each feature point matching pair may include: averaging the displacement distance of each feature point matching pair to obtain the displacement distance between the first image and the second image.

[0057] It should be noted that, when the displacement distance between the first image and the second image includes a lateral displacement distance and a longitudinal displacement distance, correspondingly, the displacement distance of each feature point matching pair may include the lateral displacement distance of each feature point matching pair, and the longitudinal displacement distance of each feature point matching pair, and averaging the displacement distance of each feature point matching pair may refer to averaging the lateral displacement distance of each feature point matching pair and averaging the longitudinal displacement distance of each feature point matching pair.

[0058] In some embodiments, the confidence level corresponding to the displacement distance is determined based on the feature vector error of each feature point matching pair, which may include but is not limited to the following methods:

[0059] Method 1: Determine the proportion of feature point matching pairs whose feature vector errors are less than the vector error threshold in all feature point matching pairs, and determine the confidence corresponding to the displacement distance based on the proportion; the larger the proportion, the larger the absolute value of the confidence.

[0060] Method 2: average the feature vector errors of each feature point matching pair to obtain an average vector error; and, based on the average vector error, determine the confidence corresponding to the displacement distance; wherein, the smaller the average vector error, the greater the absolute value of the confidence.

[0061] It should be noted that determining the displacement distance between the first image and the second image based on the feature points of the images and the corresponding confidence often requires a lot of calculations, which affects the efficiency of image stitching. In order to reduce the complexity of the calculation, the displacement distance and the corresponding confidence can be obtained based on the phase correlation method, that is, the electronic device first determines the maximum coordinates of the overlapping area of ​​the first image and the second image, and then obtains the displacement distance between the first image and the second image and the confidence corresponding to the displacement distance based on the maximum coordinates.

[0062] It should be noted that the specific process of the phase correlation method will be introduced in the following embodiments and will not be described in detail here.

[0063] 402. The electronic device splices the first image and the second image according to the displacement distance and the corresponding confidence level to obtain a target image.

[0064] In some embodiments, the electronic device splices the first image and the second image according to the displacement distance and the corresponding confidence level to obtain the target image, which may include:

[0065] When the absolute value of the corresponding confidence is greater than the confidence threshold, the electronic device splices the first image and the second image according to the displacement distance to obtain the target image.

[0066] The confidence threshold may be obtained through a large number of experiments.

[0067] In some embodiments, when the absolute value of the corresponding confidence is less than or equal to the confidence threshold, the electronic device can also perform IOU matching on the first image and the second image to obtain a matching result, and splice the first image and the second image based on the matching result to obtain the target image.

[0068] In some embodiments, the electronic device performs IOU matching on the first image and the second image to obtain a matching result, which may include:

[0069] The electronic device performs threshold segmentation on the first image and the second image respectively to obtain a binary image corresponding to the first image and a binary image corresponding to the second image;

[0070] The electronic device performs dynamic IOU matching on the binary image corresponding to the first image and the binary image corresponding to the second image according to the specified number of cycles, the specified step length and the initial distance value to obtain a matching result.

[0071] Among them, the specified number of cycles, the specified step size and the initial distance value can be obtained through a large number of experiments, and the embodiments of the present application are not limited thereto.

[0072] In the long screenshot scenario, the specified number of cycles, the specified step length, and the initial distance value may also be related to the screenshot cycle of the electronic device and / or the sliding distance of the current application interface. The screenshot cycle is proportional to the specified number of cycles, the specified step length, and the initial distance value; the sliding distance is also proportional to the specified number of cycles, the specified step length, and the initial distance value.

[0073] Exemplarily, the specified number of loops may be 10 times, 15 times, 20 times, etc., and the specified step size may be 5 pixels, 6 pixels, 7 pixels, etc.

[0074] In the embodiment of the present application, the initial distance value may include an initial height value and / or an initial width value. Correspondingly, the specified step length may include a height step length and / or a width step length.

[0075] For example, in a long screenshot scenario, the initial distance value may include an initial height value or an initial width value. In a panoramic image scenario, the initial distance value may include an initial height value and an initial width value.

[0076] In some embodiments, the long screenshot scene may include any of the following: a long screenshot scene of a downward sliding display mode, a long screenshot scene of an upward sliding display mode, a long screenshot scene of a left sliding display mode, and a long screenshot scene of a right sliding display mode.

[0077] It can be understood that in the long screenshot scenario of the downward / upward sliding display mode, the initial distance value may include an initial height value; in the long screenshot scenario of the left / right sliding display mode, the initial distance value may include an initial width value.

[0078] The downward sliding display mode means that the content on the display screen is updated and displayed from top to bottom, and the upward sliding display mode means that the content on the display screen is updated and displayed from bottom to top. The left sliding display mode means that the content on the display screen is updated and displayed from right to left, and the right sliding display mode means that the content on the display screen is updated and displayed from left to right.

[0079] In some embodiments, the electronic device performs dynamic IOU matching on the binary image corresponding to the first image and the binary image corresponding to the second image according to the specified number of cycles, the specified step length, and the initial distance value, and the matching result may include:

[0080] According to the initial distance value, respectively extracting a first region and a second region from the binary image of the first picture and the binary image of the second picture;

[0081] Perform IOU calculation on the first area and the second area to obtain an IOU value;

[0082] If the IOU value is greater than the IOU threshold, the initial distance value is used as the matching result;

[0083] If the IOU value is less than or equal to the IOU threshold, the sum of the initial distance value and the specified step size is used as the updated initial distance value, and the above operation of respectively cutting out the first area and the second area from the binary image of the first picture and the binary image of the second picture according to the initial distance value is continued until the IOU value is greater than the IOU threshold.

[0084] The performing IOU calculation on the first region and the second region to obtain the IOU value may include: performing a dot product operation on the first region and the second region to obtain the IOU value.

[0085] In the long screenshot scenario, if the screenshot time corresponding to the first picture is earlier than the screenshot time corresponding to the second picture, the first area and the second area are described as follows:

[0086] In the long screenshot scenario of the sliding display mode, the first area is the area corresponding to the initial height value from the bottom of the binary image of the first picture, and the second area is the area corresponding to the initial height value from the top of the binary image of the second picture.

[0087] In the long screenshot scenario of the upward sliding display mode, the first area is the area corresponding to the initial height value from the top of the binary image of the first picture, and the second area is the area corresponding to the initial height value from the bottom of the binary image of the second picture.

[0088] In the long screenshot scenario of the left-slide display mode, the first area is the area corresponding to the initial width value of the right part of the binary image of the first picture, and the second area is the area corresponding to the initial width value of the left part of the binary image of the second picture.

[0089] In the long screenshot scenario of the right-slide display mode, the first area is the area corresponding to the initial width value from the left to the right of the binary image of the first picture, and the second area is the area corresponding to the initial width value from the right to the left of the binary image of the second picture.

[0090] In the long screenshot scenario of the sliding display mode, when the initial height value when the IOU value is greater than the IOU threshold is obtained, splicing the first picture and the second picture according to the matching result to obtain the target picture may include:

[0091] The area corresponding to the initial height value below the top of the second image is cropped and removed, and the remaining image is spliced ​​to the bottom of the first image to obtain the target image.

[0092] or,

[0093] The area corresponding to the initial height value above the bottom of the first image is cropped and removed, and the second image is spliced ​​to the bottom of the processed first image to obtain the target image.

[0094] In some embodiments, if the above-mentioned displacement distance and corresponding confidence are obtained based on the phase correlation method, then when the absolute value of the corresponding confidence is less than or equal to the confidence threshold, the electronic device can also re-determine the new displacement distance based on the feature points of the first picture and the feature points of the second picture, and then splice the first picture and the second picture based on the new displacement distance.

[0095] It should be noted that, regarding the method of re-determining a new displacement distance based on the feature points of the first image and the feature points of the second image, reference can be made to the above content and will not be repeated here.

[0096] By implementing the above method, the displacement distance between the first image and the second image and the confidence level corresponding to the displacement distance are first determined, and then the first image and the second image are spliced ​​based on the displacement distance and the corresponding confidence level, which can effectively reduce the error during image splicing and achieve a better splicing effect.

[0097] Please refer to Figure 5 , Figure 5 FIG. 1 is another flowchart of the image stitching method disclosed in the embodiment of the present application. Figure 5 The image stitching method shown uses the phase correlation method, such as Figure 5 The image stitching method shown may include the following steps:

[0098] 501. The electronic device obtains a cross-power spectrum of a first picture and a second picture.

[0099] In the long screenshot scenario, the first picture and the second picture may come from the screenshots obtained from the first two screenshot operations; or, the first picture may come from one or more spliced ​​pictures, and the second picture may come from the current screenshot.

[0100] In the case where the first picture and the second picture are from the screenshots obtained by the previous two screenshot operations, before step 501, the electronic device may further perform the following operations:

[0101] Get the first screenshot and the second screenshot obtained by performing the screenshot operations twice before;

[0102] Convert the first screenshot image and the second screenshot image into grayscale images respectively to obtain a first grayscale image and a second grayscale image;

[0103] The resolutions of the first grayscale image and the second grayscale image are respectively reduced to obtain a first picture and a second picture.

[0104] In some embodiments, the electronic device can use a resize function to reduce the resolution of the first grayscale image and the second grayscale image respectively to obtain the first picture and the second picture. The resize function is used to change the size of the picture.

[0105] It should be noted that when the first picture comes from one or more spliced ​​pictures and the second picture comes from the current screenshot, the process of obtaining the second picture by processing the current screenshot can refer to the above-mentioned processing process of the second screenshot, which will not be repeated here.

[0106] By implementing this method, the resolution of the screenshot can be reduced, the amount of calculation during image stitching can be reduced, and the efficiency of image stitching can be improved.

[0107] In the embodiment of the present application, the electronic device obtains the cross power spectrum of the first picture and the second picture, which may include:

[0108] The electronic device converts the first image and the second image into frequency domains respectively to obtain a first frequency spectrum corresponding to the first image and a second frequency spectrum corresponding to the second image;

[0109] The electronic device obtains a cross-power spectrum of the first picture and the second picture according to the first frequency spectrum and the second frequency spectrum.

[0110] 502. The electronic device performs inverse Fourier transform on the cross power spectrum to obtain a transformation matrix.

[0111] Exemplarily, the first spectrum can be expressed as: Ga=DFT[scr1]; the second spectrum can be expressed as: Gb=DFT[scr2].

[0112] The cross power spectrum can be expressed as: in, is the conjugate of the first spectrum Ga, is the conjugate of the second spectrum Gb.

[0113] The transformation matrix can be expressed as: r=DFT-1[R].

[0114] 503. The electronic device uses the coordinates of the maximum value in the transformation matrix as the maximum coordinates.

[0115] 504. The electronic device determines the target window from the transformation matrix with the maximum coordinate as the center.

[0116] In the embodiment of the present application, the size of the target window can be obtained through a large number of experiments. For example, the size of the target window can be 4*4, 5*5 or 6*6.

[0117] 505. The electronic device calculates the displacement distance between the first image and the second image and the confidence level corresponding to the displacement distance according to the specified function and the target window.

[0118] The specified function may include a longitudinal displacement function and a lateral displacement function. The longitudinal displacement function is used to calculate the longitudinal displacement of the first image and the second image according to the target window; and the lateral displacement function is used to calculate the lateral displacement of the first image and the second image according to the target window.

[0119] In the embodiment of the present application, the electronic device calculates the displacement distance between the first image and the second image and the confidence level corresponding to the displacement distance according to the specified function and the target window, which may include:

[0120] The electronic device obtains a first score according to the lateral displacement function and the target window;

[0121] The electronic device obtains a second score according to the longitudinal displacement function and the target window;

[0122] Using the fractional value of the first fraction as the horizontal displacement amount between the first picture and the second picture, and using the fractional value of the second fraction as the vertical displacement amount between the first picture and the second picture;

[0123] The denominators of the first fraction and the second fraction are normalized to obtain the confidence level corresponding to the displacement distance.

[0124] For example, when the target window size is 5*5:

[0125] The lateral displacement function can be expressed as:

[0126] The longitudinal displacement function can be expressed as: Among them, f(i,j) is the coordinates of each point in the target window.

[0127] 506. The electronic device splices the first image and the second image according to the displacement distance and the corresponding confidence level to obtain a target image.

[0128] In the long screenshot scenario of the sliding display mode, the electronic device splices the first picture and the second picture according to the displacement distance and the corresponding confidence level to obtain the target picture, which may include:

[0129] If the absolute value of the confidence is greater than the confidence threshold and the displacement distance is a positive number, the area corresponding to the displacement distance is removed from the top of the second image downwards, and the remaining area of ​​the second image is spliced ​​to the bottom of the first image to obtain the target image;

[0130] If the absolute value of the confidence is greater than the confidence threshold and the displacement distance is a negative number, the target area corresponding to the displacement distance is cut from the bottom of the second image upwards, and the target area is spliced ​​to the bottom of the first image to obtain the target image.

[0131] It should be noted that when the absolute value of the confidence is greater than the confidence threshold, the method of splicing the first image and the second image according to the displacement distance in other long screenshot scenarios can be analogous to the method in the long screenshot scenario of the above-mentioned sliding display mode, and will not be repeated here.

[0132] For the case where the absolute value of the confidence is less than the confidence threshold, see Figure 4 The following description of step 402 is omitted here.

[0133] In the long screenshot scenario, after the electronic device subsequently performs the screenshot operation to obtain the screenshot, it is also necessary to splice the screenshot onto the target image. After step 506, the electronic device may further perform the following steps:

[0134] When the electronic device performs a third screenshot operation to obtain a third screenshot image, converting the third screenshot image into a grayscale image to obtain a third grayscale image, and reducing a resolution of the third grayscale image to obtain a third picture;

[0135] According to the height of the third image, the target image is cropped to obtain a fourth image;

[0136] The fourth picture is used as the first picture, and the third picture is used as the second picture, and a displacement distance between the first picture and the second picture and a confidence level corresponding to the displacement distance are calculated based on a mutual power spectrum of the first picture and the second picture.

[0137] Combined with the above description, in the long screenshot scenario, the process of the image stitching method can be seen in Figure 6 , Figure 6FIG. 1 is another flowchart of the image stitching method disclosed in the embodiment of the present application. Figure 6 The image stitching method shown may include the following steps:

[0138] 611. Obtain a first screenshot image and a second screenshot image obtained by executing the screenshot operations twice before.

[0139] 612. Convert the first screenshot and the second screenshot into grayscale images respectively to obtain a first grayscale image and a second grayscale image.

[0140] 613. Reduce the resolutions of the first grayscale image and the second grayscale image respectively to obtain a first picture and a second picture.

[0141] 614. Convert the first picture and the second picture into the frequency domain to obtain a first spectrum corresponding to the first picture and a second spectrum corresponding to the second picture, and obtain a cross-power spectrum of the first picture and the second picture according to the first spectrum and the second spectrum.

[0142] 615. Perform inverse Fourier transform on the mutual power spectrum to obtain a transformation matrix, and use the coordinates of the maximum value in the transformation matrix as the maximum coordinates, and determine the target window from the transformation matrix with the maximum coordinates as the center, and calculate the displacement distance between the first image and the second image and the confidence corresponding to the displacement distance according to the specified function and the target window.

[0143] 616. Determine whether the corresponding confidence is greater than the confidence threshold. If so, execute step 617; if less than or equal to, execute step 618.

[0144] 617. According to the displacement distance, the first image and the second image are spliced ​​to obtain a target image.

[0145] 618. Perform threshold segmentation on the first image and the second image respectively to obtain a binary image corresponding to the first image and a binary image corresponding to the second image.

[0146] 619. According to the initial distance value, cut out a first region and a second region from the binary image of the first picture and the binary image of the second picture respectively.

[0147] 620. Perform a dot product operation on the first region and the second region to obtain an IOU value.

[0148] 621. Determine whether the IOU value is greater than the IOU threshold. If so, execute step 622; if not, execute step 623.

[0149] 622. Splice the first image and the second image according to the initial distance value to obtain a target image.

[0150] 623. The sum of the initial distance value and the specified step length is used as the updated initial distance value, and step 619 is continued.

[0151] By implementing the above method, the electronic device first determines the displacement distance between the first image and the second image, and the confidence level corresponding to the displacement distance, and then splices the first image and the second image based on the displacement distance and the corresponding confidence level, which can effectively reduce the error during image splicing, thereby achieving a better splicing effect. In addition, by calculating the displacement distance and the corresponding confidence level based on the cross-power spectrum, the amount of calculation can be effectively reduced, thereby facilitating an increase in the rate of image splicing.

[0152] See also Figure 7 , Figure 7 is a structural diagram of the image stitching device disclosed in the embodiment of the present application. Figure 7 The image stitching device shown may include a displacement determination unit 701 and an image stitching unit 702; wherein:

[0153] A displacement determination unit 701, configured to determine a displacement distance between a first image and a second image, and a confidence level corresponding to the displacement distance;

[0154] The image stitching unit 702 is used to stitch the first image and the second image according to the displacement distance and the confidence level to obtain a target image.

[0155] In some embodiments, the displacement determination unit 701 may be used to determine the displacement distance between the first image and the second image, and the confidence level corresponding to the displacement distance, which may specifically include:

[0156] The displacement determination unit 701 is used to determine the maximum coordinates of the overlapping area of ​​the first picture and the second picture; and, based on the maximum coordinates, determine the displacement distance between the first picture and the second picture, and the confidence level corresponding to the displacement distance.

[0157] In some embodiments, the method in which the displacement determining unit 701 determines the maximum coordinates of the overlapping area of ​​the first picture and the second picture may specifically include:

[0158] The displacement determination unit 701 is used to obtain the mutual power spectrum of the first picture and the second picture; and perform inverse Fourier transformation on the mutual power spectrum to obtain a transformation matrix; and use the coordinates of the maximum value in the transformation matrix as the maximum coordinates.

[0159] In some embodiments, the displacement determining unit 701 is used to determine the displacement distance between the first image and the second image and the confidence level corresponding to the displacement distance according to the maximum coordinate, which may specifically include:

[0160] The displacement determination unit 701 is used to determine the target window from the transformation matrix with the maximum coordinate as the center; and, according to the specified function and the target window, calculate the displacement distance between the first image and the second image, and the confidence corresponding to the displacement distance.

[0161] In some embodiments, the image stitching unit 702 is configured to stitch the first image and the second image according to the displacement distance and the confidence level to obtain the target image in a manner that specifically includes:

[0162] The image stitching unit 702 is used to stitch the first image and the second image according to the displacement distance to obtain a target image if the absolute value of the confidence is greater than the confidence threshold; and, if the absolute value of the confidence is less than or equal to the confidence threshold, perform IOU matching on the first image and the second image to obtain a matching result, and stitch the first image and the second image according to the matching result to obtain the target image.

[0163] In some embodiments, the image stitching unit 702 is used to perform IOU matching on the first image and the second image, and the way to obtain the matching result may specifically include:

[0164] The image stitching unit 702 is used to perform threshold segmentation on the first image and the second image respectively to obtain a binary image corresponding to the first image and a binary image corresponding to the second image; and, according to a specified number of loops, a specified step size and an initial distance value, perform dynamic IOU matching on the binary image corresponding to the first image and the binary image corresponding to the second image to obtain a matching result.

[0165] In some embodiments, Figure 7 The image stitching device shown may also include a picture processing unit ( Figure 7 (not shown in the figure), an image processing unit is used to obtain, in a long screenshot scenario, the first screenshot image and the second screenshot image obtained by the first two screenshot operations before the displacement determination unit 701 determines the displacement distance between the first image and the second image, and the confidence level corresponding to the displacement distance; and, convert the first screenshot image and the second screenshot image into grayscale images, respectively, to obtain the first grayscale image and the second grayscale image; and, respectively reduce the resolutions of the first grayscale image and the second grayscale image, to obtain the first image and the second image.

[0166] In some embodiments, the long screenshot scene includes a long screenshot scene in a sliding display mode, and the image stitching unit 702 is used to stitch the first picture and the second picture according to the displacement distance and the confidence level to obtain the target picture. Specifically, the method may include:

[0167] The image stitching unit 702 is used to, if the absolute value of the confidence is greater than the confidence threshold and the displacement distance is a positive number, remove the area corresponding to the displacement distance from the top of the second image downward, and stitch the remaining area of ​​the second image to the bottom of the first image to obtain the target image; and, if the absolute value of the confidence is greater than the confidence threshold and the displacement distance is a negative number, intercept the target area corresponding to the displacement distance from the bottom of the second image upward, and stitch the target area to the bottom of the first image to obtain the target image.

[0168] In some embodiments, the image processing unit is further configured to cause the image stitching unit 702 to stitch the first image and the second image according to the displacement distance and the confidence level to obtain a target image, and when the electronic device performs a third screenshot operation to obtain a third screenshot, convert the third screenshot into a grayscale image to obtain a third grayscale image, and reduce the resolution of the third grayscale image to obtain a third image; and, according to the height of the third image, crop the target image to obtain a fourth image;

[0169] Furthermore, the displacement determination unit 701 is also used to take the fourth picture as the first picture and the third picture as the second picture, and to calculate the displacement distance between the first picture and the second picture, and the confidence corresponding to the displacement distance based on the mutual power spectrum of the first picture and the second picture.

[0170] See also Figure 8 , Figure 8 FIG. 1 is a structural diagram of an electronic device disclosed in an embodiment of the present application. Figure 8 The electronic device shown includes: a processor 810, a memory 820, a display unit 830, an input unit 840, a sensor 850, an audio circuit 860 and other components.

[0171] The processor 810 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 820, and calling data stored in the memory 820, so as to monitor the electronic device as a whole. Optionally, the processor 810 may include one or more processing units; optionally, the processor 810 may integrate an application processor, which mainly processes operating devices, user interfaces, and application programs, etc. Of course, other processors may also be included, which are not listed here one by one.

[0172] The memory 820 can be used to store software programs and modules. The processor 810 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 820. The memory 820 can mainly include a program storage area and a data storage area, wherein the program storage area can store operating devices, at least one application required for a function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory 820 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0173] The display unit 830 can be used to display information input by the user or information provided to the user and various menus of the electronic device. The display unit 830 may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel may cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 810 to determine the type of touch event. Then, the processor 810 provides a corresponding visual output on the display panel according to the type of touch event. Among them, the touch panel and the display panel are not in contact with each other. Figure 8 The touch panel and the display panel can be used as two independent components to realize the input and output functions of the electronic device, or the touch panel and the display panel can be integrated to realize the input and output functions of the electronic device.

[0174] The input unit 840 can be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the electronic device. Specifically, the input unit 840 may include a touch panel and other input devices. A touch panel, also known as a touch screen, can collect user touch operations on or near it (such as operations performed by a user using any suitable object or accessory such as a finger, stylus, etc. on or near the touch panel), and drive the corresponding connection device according to a pre-set program. In addition, a variety of types such as resistive, capacitive, infrared, and surface acoustic waves can be used to implement the touch panel. In addition to the touch panel, the input unit 840 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of function keys (such as volume control keys, switch keys, etc.), trackballs, joysticks, etc.

[0175] The electronic device may also include at least one sensor 850, such as a magnetometer, a gyroscope sensor, a motion sensor, and other sensors. Specifically, the magnetometer is used to determine the orientation of the electronic device, and the gyroscope sensor can be used to determine the motion posture of the electronic device. It can be used for anti-shake shooting, and can also be used for navigation and somatosensory game scenes. As a type of motion sensor, the acceleration sensor can detect the magnitude of acceleration in all directions, and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the electronic device, such as horizontal and vertical screen switching, related games, magnetometer posture calibration, etc.; as for other sensors that the electronic device can also be configured with, such as pressure gauges, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0176] The audio circuit 860 may include a speaker and a microphone, and may provide an audio interface between a user and an electronic device. The audio circuit 860 may transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 860 and converted into audio data, and then the audio data is output to the processor 810 for processing, and then sent to another device through the video circuit, or the audio data is output to the memory 820 for further processing.

[0177] Although not shown, the electronic device may further include a power supply and a camera. Optionally, the camera may be located at the front or rear of the electronic device, which is not limited in the present embodiment.

[0178] It is to be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0179] In the embodiment of the present application, the processor 810 also has the following functions:

[0180] Determine a displacement distance between the first image and the second image, and a confidence level corresponding to the displacement distance;

[0181] The first image and the second image are spliced ​​according to the displacement distance and the confidence level to obtain a target image.

[0182] In the embodiment of the present application, the processor 810 also has the following functions:

[0183] Determine the maximum coordinates of the overlapping area of ​​the first image and the second image;

[0184] According to the maximum coordinate, a displacement distance between the first image and the second image and a confidence level corresponding to the displacement distance are determined.

[0185] In the embodiment of the present application, the processor 810 also has the following functions:

[0186] Obtain a cross power spectrum of the first picture and the second picture;

[0187] Performing inverse Fourier transform on the cross power spectrum to obtain a transformation matrix;

[0188] The coordinates of the maximum value in the transformation matrix are taken as the maximum coordinates.

[0189] In the embodiment of the present application, the processor 810 also has the following functions:

[0190] Taking the maximum coordinate as the center, determining the target window from the transformation matrix;

[0191] According to the specified function and the target window, the displacement distance between the first image and the second image and the confidence level corresponding to the displacement distance are calculated.

[0192] In the embodiment of the present application, the processor 810 also has the following functions:

[0193] If the absolute value of the confidence is greater than a confidence threshold, splicing the first picture and the second picture according to the displacement distance to obtain a target picture;

[0194] If the absolute value of the confidence is less than or equal to the confidence threshold, IOU matching is performed on the first image and the second image to obtain a matching result, and the first image and the second image are spliced ​​according to the matching result to obtain a target image.

[0195] In the embodiment of the present application, the processor 810 also has the following functions:

[0196] Performing threshold segmentation on the first picture and the second picture respectively to obtain a binary image corresponding to the first picture and a binary image corresponding to the second picture;

[0197] According to the specified number of loops, the specified step size, and the initial distance value, dynamic IOU matching is performed on the binary image corresponding to the first image and the binary image corresponding to the second image to obtain a matching result.

[0198] In the embodiment of the present application, the processor 810 also has the following functions:

[0199] In a long screenshot scenario, obtain the first screenshot and the second screenshot obtained by the first two screenshot operations;

[0200] Converting the first screenshot and the second screenshot into grayscale images respectively to obtain a first grayscale image and a second grayscale image;

[0201] The resolutions of the first grayscale image and the second grayscale image are respectively reduced to obtain a first picture and a second picture.

[0202] In the embodiment of the present application, the processor 810 also has the following functions:

[0203] If the absolute value of the confidence is greater than the confidence threshold and the displacement distance is a positive number, remove the area corresponding to the displacement distance from the top of the second image downward, and splice the remaining area of ​​the second image to the bottom of the first image to obtain the target image;

[0204] If the absolute value of the confidence is greater than the confidence threshold and the displacement distance is a negative number, a target area corresponding to the displacement distance is cut upward from the bottom of the second image, and the target area is spliced ​​to the bottom of the first image to obtain a target image.

[0205] In the embodiment of the present application, the processor 810 also has the following functions:

[0206] When the electronic device performs a third screenshot operation to obtain a third screenshot image, converting the third screenshot image into a grayscale image to obtain a third grayscale image, and reducing a resolution of the third grayscale image to obtain a third picture;

[0207] Cropping the target image according to the height of the third image to obtain a fourth image;

[0208] The fourth picture is used as the first picture, and the third picture is used as the second picture, and the displacement distance between the first picture and the second picture and the confidence corresponding to the displacement distance are calculated according to the mutual power spectrum of the first picture and the second picture.

[0209] The embodiment of the present application discloses a computer-readable storage medium on which executable program code is stored. When the executable program code is executed by a processor, the method described in the electronic device in the embodiment of the present application is implemented.

[0210] The embodiment of the present application discloses a computer program product. When the computer program product is executed on a computer, the computer implements the method described in the electronic device in the embodiment of the present application.

[0211] The embodiment of the present application discloses an application publishing platform, which is used to publish a computer program product. When the computer program product runs on a computer, the computer implements the method described in the electronic device in the embodiment of the present application.

[0212] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0213] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0214] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0215] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0216] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0217] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0218] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0219] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0220] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0221] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0222] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0223] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0224] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image stitching method, characterized in that: include: Determine a displacement distance between the first image and the second image, and a confidence level corresponding to the displacement distance; The first image and the second image are spliced ​​according to the displacement distance and the confidence level to obtain a target image.

2. The method according to claim 1, characterized in that The determining of the displacement distance between the first image and the second image, and the confidence level corresponding to the displacement distance, includes: Determine the maximum coordinates of the overlapping area of ​​the first image and the second image; According to the maximum coordinate, a displacement distance between the first image and the second image and a confidence level corresponding to the displacement distance are determined.

3. The method according to claim 2, characterized in that The determining the maximum coordinates of the overlapping area of ​​the first picture and the second picture includes: Obtain a cross power spectrum of the first picture and the second picture; Performing inverse Fourier transform on the cross power spectrum to obtain a transformation matrix; The coordinates of the maximum value in the transformation matrix are taken as the maximum coordinates.

4. The method according to claim 3, characterized in that The determining, according to the maximum coordinate, a displacement distance between the first picture and the second picture, and a confidence level corresponding to the displacement distance includes: Taking the maximum coordinate as the center, determining the target window from the transformation matrix; According to the specified function and the target window, the displacement distance between the first image and the second image and the confidence level corresponding to the displacement distance are calculated.

5. The method according to claim 1, characterized in that The step of splicing the first image and the second image according to the displacement distance and the confidence level to obtain a target image includes: If the absolute value of the confidence is greater than a confidence threshold, splicing the first picture and the second picture according to the displacement distance to obtain a target picture; If the absolute value of the confidence is less than or equal to the confidence threshold, IOU matching is performed on the first image and the second image to obtain a matching result, and the first image and the second image are spliced ​​according to the matching result to obtain a target image.

6. The method according to claim 5, characterized in that The performing IOU matching on the first image and the second image to obtain a matching result includes: Performing threshold segmentation on the first picture and the second picture respectively to obtain a binary image corresponding to the first picture and a binary image corresponding to the second picture; According to the specified number of loops, the specified step size, and the initial distance value, dynamic IOU matching is performed on the binary image corresponding to the first image and the binary image corresponding to the second image to obtain a matching result.

7. The method according to any one of claims 1 to 6, characterized in that Before determining the displacement distance between the first image and the second image, and the confidence level corresponding to the displacement distance, the method further includes: In a long screenshot scenario, obtain the first screenshot and the second screenshot obtained by the first two screenshot operations; Converting the first screenshot and the second screenshot into grayscale images respectively to obtain a first grayscale image and a second grayscale image; The resolutions of the first grayscale image and the second grayscale image are respectively reduced to obtain a first picture and a second picture.

8. The method according to claim 7, characterized in that The long screenshot scene includes a long screenshot scene in a sliding display mode, and the splicing of the first picture and the second picture according to the displacement distance and the confidence level to obtain a target picture includes: If the absolute value of the confidence is greater than the confidence threshold and the displacement distance is a positive number, remove the area corresponding to the displacement distance from the top of the second image downward, and splice the remaining area of ​​the second image to the bottom of the first image to obtain the target image; If the absolute value of the confidence is greater than the confidence threshold and the displacement distance is a negative number, a target area corresponding to the displacement distance is cut upward from the bottom of the second image, and the target area is spliced ​​to the bottom of the first image to obtain a target image.

9. The method according to claim 7, characterized in that: After the first image and the second image are spliced ​​according to the displacement distance and the confidence level to obtain a target image, the method further includes: When the electronic device performs a third screenshot operation to obtain a third screenshot image, converting the third screenshot image into a grayscale image to obtain a third grayscale image, and reducing a resolution of the third grayscale image to obtain a third picture; Cropping the target image according to the height of the third image to obtain a fourth image; The fourth picture is used as the first picture, and the third picture is used as the second picture, and the displacement distance between the first picture and the second picture and the confidence corresponding to the displacement distance are calculated according to the mutual power spectrum of the first picture and the second picture.

10. An image stitching device, characterized in that: include: A displacement determination unit, used to determine a displacement distance between the first image and the second image, and a confidence level corresponding to the displacement distance; An image stitching unit is used to stitch the first image and the second image together according to the displacement distance and the confidence level to obtain a target image.

11. An electronic device, characterized in that: include: A memory storing executable program code; and a processor coupled to the memory; The processor calls the executable program code stored in the memory, and when the executable program code is executed by the processor, the processor implements the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having executable program code stored thereon, characterized in that: When the executable program code is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images