An image stitching method, device, equipment and storage medium

Abstract

The application provides an image splicing method, device and equipment and a storage medium. The method comprises the following steps: obtaining a perspective feature map of a first image frame, performing perspective correction on the first image frame according to the perspective feature map of the first image frame to obtain a first perspective correction image frame, and obtaining a second perspective correction image frame of a second image frame. The first perspective correction image frame and the second perspective correction image frame are spliced to obtain a spliced image. According to the technical scheme, the perspective effect of the image frame is weakened by performing perspective correction on the image frame, the matching accuracy between two image frames is improved when the image frames are spliced, and therefore the splicing effect of the scanning image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image splicing method and device, equipment and a storage medium. BACKGROUND

[0002] With the continuous upgrading and development of information technology in recent years, the recognition of characters in images has been realized, so that the information of books, files and other materials can be recognized by scanning images and then recognizing the characters in the images. For example, a scanning device such as a scanning pen continuously acquires images and splices all the images to recognize the spliced images. The better the spliced image, the more conducive to improving the recognition accuracy, so how to improve the splicing effect of the scanned image is a problem that needs to be solved by those skilled in the art. SUMMARY

[0003] Based on the defects and deficiencies of the prior art, the present application provides an image splicing method, device, equipment and storage medium, which can improve the splicing effect of the scanned image.

[0004] The first aspect of the present application provides an image splicing method, comprising:

[0005] obtaining a perspective feature map of a first image frame; wherein the perspective feature map is used to represent the perspective effect feature of the image content in the image frame;

[0006] performing perspective correction on the first image frame according to the perspective feature map of the first image frame to obtain a first perspective corrected image frame, and obtaining a second perspective corrected image frame of a second image frame; the perspective correction aims to eliminate the perspective effect of the image content in the image frame;

[0007] splicing the first perspective corrected image frame and the second perspective corrected image frame to obtain a spliced image.

[0008] Optionally, the second perspective corrected image frame is the last frame of the pre-stored spliced image, and the second perspective corrected image frame is obtained by performing perspective correction on the second image frame according to the perspective feature map of the second image frame.

[0009] Optionally, the obtaining of the perspective feature map of the first image frame comprises:

[0010] extracting gradient information of each pixel point of the first image frame;

[0011] constructing a gradient superposition map of the first image frame as the perspective feature map of the first image frame according to the gradient information of each pixel point of the first image frame.

[0012] Optionally, the first image frame is perspective corrected according to the perspective feature map of the first image frame to obtain a first perspective corrected image frame, comprising:

[0013] From the perspective feature map of the first image frame, a to-be-corrected region needing perspective correction is extracted;

[0014] Based on the to-be-corrected region, a deformation field corresponding to the first image frame is calculated;

[0015] According to the deformation field, the first image frame is perspective corrected to obtain a first perspective corrected image frame corresponding to the first image frame.

[0016] Optionally, the calculation of the deformation field corresponding to the first image frame based on the to-be-corrected region comprises:

[0017] Based on the vertices of the to-be-corrected region and the upper and lower boundaries of the to-be-corrected region, a straight line fitting is performed to obtain a fitting straight line;

[0018] The fitting straight line is adjusted to a horizontal state to obtain a horizontal fitting straight line, and an intersection point of the horizontal fitting straight line and the boundary of the first image frame is determined as a target reference point;

[0019] According to the fitting straight line and the target reference point, a transformation matrix is calculated, and the transformation matrix is taken as the deformation field of the first image frame.

[0020] Optionally, before the first image frame is perspective corrected according to the perspective feature map of the first image frame to obtain a first perspective corrected image frame, it further comprises:

[0021] According to the perspective feature map of the first image frame and a pre-set correction condition region, it is determined whether the first image frame has perspective correction requirement; the correction condition region includes a set region on the leftmost side of the image frame;

[0022] If the first image frame has perspective correction requirement, the first image frame is perspective corrected according to the perspective feature map of the first image frame to obtain a first perspective corrected image frame.

[0023] Optionally, the perspective feature map of the first image frame is a gradient superposition map of the first image frame;

[0024] According to the perspective feature map of the first image frame and a pre-set correction condition region, it is determined whether the first image frame has perspective correction requirement, comprising:

[0025] If there is a to-be-recognized character in the pre-set correction condition region in the first image frame, and the connected domain area of the gradient superposition map reaches a preset threshold, it is determined that the first image frame has perspective correction requirement.

[0026] If there is no character to be recognized in the preset correction condition region in the first image frame, or the area of the connected domain of the gradient superposition graph does not reach the preset threshold, it is determined that the first image frame does not have a perspective correction requirement.

[0027] Optionally, the first perspective correction image frame and the second perspective correction image frame are spliced to obtain a spliced image, including:

[0028] The first perspective correction image frame and the second perspective correction image frame are compared to determine an overlapping region of the two image frames.

[0029] Based on the overlapping region, the first perspective correction image frame and the second perspective correction image frame are spliced to obtain a spliced image.

[0030] Optionally, the first perspective correction image frame and the second perspective correction image frame are compared to determine an overlapping region of the two image frames, including:

[0031] A preset image region is extracted from the first perspective correction image frame as a matching region;

[0032] A feature vector of the matching region is determined, and a feature vector of each sub-region of the second perspective correction image frame is determined.

[0033] According to the feature vector of the matching region and the feature vector of each sub-region, a sub-region with the highest similarity to the matching region is searched from the each sub-region as an overlapping region.

[0034] Optionally, based on the overlapping region, the first perspective correction image frame and the second perspective correction image frame are spliced to obtain a spliced image, including:

[0035] According to the overlapping region and the matching region, an offset between the first perspective correction image frame and the second perspective correction image frame is determined.

[0036] Based on the offset, the first perspective correction image frame and the second perspective correction image frame are spliced to obtain a spliced image.

[0037] Optionally, the feature vector of the matching region is determined, including:

[0038] The matching region is divided into at least one image block, and a weight of each image block is determined.

[0039] A direction gradient histogram feature of each image block is calculated.

[0040] According to the weight of each image block, the directional gradient histogram features of all the image blocks are weighted and normalized to obtain a feature vector of the matching region.

[0041] Optionally, before the perspective feature map of the first image frame is acquired, the method further comprises:

[0042] analyzing a target similarity between the acquired current scanning image and the second image frame;

[0043] determining a similarity relationship between the current scanning image and the second image frame according to the target similarity and a preset similarity threshold value;

[0044] if the similarity relationship indicates dissimilarity, taking the current scanning image as the first image frame;

[0045] if the similarity relationship indicates similarity, taking a next frame scanning image as the current scanning image.

[0046] Optionally, the analyzing a target similarity between the acquired current scanning image and the second image frame comprises:

[0047] determining image features of the acquired current scanning image and image features of the second image frame;

[0048] calculating a target similarity between the image features of the current scanning image and the image features of the second image frame.

[0049] Optionally, if the image features comprise global features and local features corresponding to a preset local region;

[0050] the similarity threshold value comprises a first similarity threshold value and a second similarity threshold value;

[0051] the target similarity comprises a first target similarity between global features of the current scanning image and global features of the second image frame, and a second target similarity between local features of the current scanning image and local features of the second image frame;

[0052] determining a similarity relationship between the current scanning image and the second image frame according to the target similarity and a preset similarity threshold value comprises:

[0053] if the first target similarity is less than the first similarity threshold value and the second target similarity is less than the second similarity threshold value, determining that the similarity relationship between the current scanning image and the previous image frame indicates dissimilarity;

[0054] If the first target similarity is not less than the first similarity threshold, and / or the second target similarity is not less than the second similarity threshold, it is determined that the similarity relationship between the current scan image and the second image frame represents similarity.

[0055] Optionally, the image stitching method further comprises:

[0056] determining a boundary contour of image content in the stitched image;

[0057] performing boundary alignment correction in the vertical direction on the stitched image by using the point coordinates on the boundary contour, to obtain a corrected stitched image.

[0058] The second aspect of the present application provides an image stitching device, comprising:

[0059] an acquisition module configured to acquire a perspective feature map of a first image frame; wherein the perspective feature map is used to represent a perspective effect feature of image content in the image frame;

[0060] a perspective correction module configured to perform perspective correction on the first image frame according to the perspective feature map of the first image frame, to obtain a first perspective corrected image frame, and to acquire a second perspective corrected image frame of a second image frame; the perspective correction aims to eliminate the perspective effect of the image content in the image frame;

[0061] an image stitching module configured to stitch the first perspective corrected image frame and the second perspective corrected image frame, to obtain a stitched image.

[0062] The third aspect of the present application provides an image stitching device, comprising:

[0063] a memory and a processor;

[0064] The memory is connected with the processor, and is configured to store a program.

[0065] The processor is configured to realize the above-mentioned image stitching method by running the program in the memory.

[0066] The fourth aspect of the present application provides a storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to realize the above-mentioned image stitching method.

[0067] The image splicing method provided in the application obtains a perspective feature map of a first image frame, performs perspective correction on the first image frame according to the perspective feature map of the first image frame to obtain a first perspective correction image frame, and obtains a second perspective correction image frame of a second image frame. The first perspective correction image frame and the second perspective correction image frame are spliced to obtain a spliced image. By performing perspective correction on the image frame, the perspective effect of the image frame is weakened, the matching accuracy between two image frames is improved when the image frames are spliced, and thus the splicing effect of the scanning image is improved. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present application, and other drawings can be obtained by the drawings provided by the person skilled in the art without creative labor.

[0069] Figure 1 is a flowchart of an image splicing method provided by an embodiment of the present application;

[0070] Figure 2 is a processing flowchart of obtaining a perspective feature map of a first image frame provided by an embodiment of the present application;

[0071] Figure 3 is an acquisition effect diagram of a perspective feature map provided by an embodiment of the present application;

[0072] Figure 4 is a processing flowchart of performing perspective correction on a first image frame provided by an embodiment of the present application;

[0073] Figure 5 is a target reference point selection diagram of perspective correction provided by an embodiment of the present application;

[0074] Figure 6 is a comparison diagram before and after perspective correction provided by an embodiment of the present application;

[0075] Figure 7 is a flowchart of another image splicing method provided by an embodiment of the present application;

[0076] Figure 8 is a processing flowchart of splicing a first perspective correction image frame and a second perspective correction image frame provided by an embodiment of the present application;

[0077] Figure 9 is a processing flowchart of determining an overlapping area of a first perspective correction image frame and a second perspective correction image frame provided by an embodiment of the present application;

[0078] Figure 10 This is a schematic diagram of the matching region segmentation process provided in the embodiments of this application;

[0079] Figure 11 This is a flowchart illustrating another image stitching method provided in an embodiment of this application;

[0080] Figure 12 This is a flowchart illustrating another image stitching method provided in an embodiment of this application;

[0081] Figure 13 These are comparison images of the stitched image boundary alignment correction before and after provided in the embodiments of this application;

[0082] Figure 14 This is a schematic diagram of the structure of an image stitching device provided in an embodiment of this application;

[0083] Figure 15 This is a schematic diagram of the structure of an image stitching device provided in an embodiment of this application. Detailed Implementation

[0084] The technical solution of this application is applicable to image stitching scenarios. By performing perspective correction on image frames, the perspective effect of image frames is reduced, which can improve the matching accuracy between two image frames during image stitching, thereby improving the stitching effect of scanned images.

[0085] Image recognition technology is an important field of artificial intelligence, and its applications are becoming increasingly widespread with its advancements. For example, scanning devices such as pens, mobile phones, and scanners can recognize images after scanning them. When there is a large amount of content to be recognized, the scanning device can perform continuous scanning, and then the scanned images can be stitched together for recognition. The better the stitching effect, the higher the recognition accuracy.

[0086] When a scanning device scans an image, the angle between it and the scanned plane causes varying degrees of perspective in the scanned image. For example, when the scanning device is at a 90-degree angle to the scanned plane, there is no perspective; however, when the angle is less than 60 degrees, a significant perspective effect occurs. When stitching two images together, similar areas are matched to determine if the same content has been scanned, thus preventing duplicate content in the stitched image. However, if perspective exists in the images, even if the two images scanned the same content, the perspective will cause an unsatisfactory matching effect, increasing the probability of duplicate content in the stitched image and affecting its overall quality.

[0087] In view of the deficiencies of the prior art and the actual existence of the scanning image perspective, the splicing effect of the scanning image is not high, the present application inventors have proposed an image splicing method which can weaken the perspective effect of the image frame, improve the matching accuracy between two image frames when splicing the image frame, and thus improve the splicing effect of the scanning image.

[0088] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0089] The present application embodiment proposes an image splicing method, as shown in Figure 1 The method comprises the following steps:

[0090] S101, obtaining the perspective feature map of the first image frame.

[0091] Specifically, the first image frame is the image frame scanned by the scanning device. After obtaining the first image frame, the perspective of the first image frame needs to be judged, that is, the perspective feature map of the first image frame is obtained. The perspective feature map is used to represent the perspective effect feature of the image content in the image frame. The foreground connected domain in the gradient binary cumulative superimposed graph (i.e. gradient superimposed graph) of the image frame can clearly reflect the perspective effect of the image content in the image frame. Therefore, the present embodiment can construct the gradient superimposed graph of the first image frame as the perspective feature map of the first image frame.

[0092] S102, according to the perspective feature map of the first image frame, performing perspective correction on the first image frame to obtain the first perspective corrected image frame, and obtaining the second perspective corrected image frame of the second image frame.

[0093] Specifically, the perspective feature map of the first image frame can reflect the perspective effect of the image content in the first image frame. Since the perspective feature map adopts the gradient superposition map, and the gradient superposition map is preferably a map obtained by superimposing the binary map of the first image frame after binarization and the binary map of the image frame before the first image frame according to a preset number. This embodiment needs to determine the correction mode of the region where the image content is located in the perspective feature map first, and then correct the original image (i.e., the first image frame) according to the correction mode corresponding to the perspective feature map. That is, according to the perspective situation of the image content shown in the perspective feature map, the correction mode that can eliminate the perspective effect of the image content in the perspective feature map is determined, and then the region where the image content in the first image frame is located is corrected according to the correction mode, so as to eliminate the perspective effect of the image content in the first image frame, and the image frame after the perspective correction is taken as the first perspective correction image frame.

[0094] The first image frame scanned by the scanning device can be a grayscale image or a color image. When the first image frame is a grayscale image, since the grayscale image is a single-channel picture, it only needs to be corrected according to the correction mode corresponding to the pre-determined perspective feature map. When the first image frame is a color image, since the color image is a three-channel picture (i.e., R, G, and B correspond to three channels), it needs to be corrected according to the correction mode corresponding to the pre-determined perspective feature map, that is, the correction value corresponding to each channel of the color image is determined according to the correction mode, and the three channels are corrected according to the correction values of the three channels. Figure 3

[0095] If the first image frame and the second image frame are to be spliced, the second perspective correction image frame of the second image frame also needs to be obtained. If the second perspective correction image frame has been obtained by completing the perspective correction of the second image frame, the second perspective correction image frame is directly obtained. If the second image frame has not been corrected, the perspective feature map of the second image frame also needs to be obtained. Specifically, the perspective correction of the second image frame can be performed according to the perspective feature map of the second image frame in the manner of the perspective correction of the first image frame, to obtain the second perspective correction image frame.

[0096] S103, splicing the first perspective correction image frame and the second perspective correction image frame to obtain a spliced image.

[0097] ​Specifically, when the image frames are spliced, there is the same content between two adjacent image frames. The slower the scanning speed or the faster the scanning frame rate of the scanning device, the more the same content between two adjacent image frames. Therefore, when the two image frames are spliced, directly splicing the two image frames at the head and tail will cause the same content to appear repeatedly in the spliced image. Therefore, when the image frames are spliced, the area with the same content in the two image frames needs to be found, and then the areas with the same content in the two image frames are spliced together, so that the same content in the two image frames appears only once in the spliced image.

[0098] If the image frames have a large perspective effect, when searching for the area with the same content in the two image frames, the contrast matching between the two image frames may be inaccurate due to the perspective. Even if the contents are the same, they cannot be detected as the same under the perspective effect. Therefore, in order to improve the accuracy of image frame splicing and improve the effect of the spliced image, the embodiment needs to use the first perspective correction image frame obtained by perspective correction of the first image frame and the second perspective correction image frame obtained by perspective correction of the second image frame to realize image splicing, search for the area with the same content in the first perspective correction image frame and the second perspective correction image frame, and then overlap the areas with the same content in the two image frames together, so as to obtain the spliced image.

[0099] As can be seen from the above introduction, the image splicing method proposed in the embodiment of the present application obtains the perspective feature map of the first image frame, performs perspective correction on the first image frame according to the perspective feature map of the first image frame to obtain the first perspective correction image frame, and obtains the second perspective correction image frame of the second image frame. The first perspective correction image frame and the second perspective correction image frame are spliced to obtain a spliced image. By using the technical solution of the embodiment, the perspective effect of the image frame is weakened by performing perspective correction on the image frame, which can improve the correctness of matching between the two image frames when the image frames are spliced, thereby improving the splicing effect of the scanned image.

[0100] As an optional implementation, another embodiment of the present application discloses that the second perspective correction image frame obtained in the step S102 is the last frame of the pre-stored spliced image. For example, in the case of real-time scanning and real-time splicing of the scanning image, the first image frame is the currently obtained image frame, and the second image frame is the image frame obtained last time before the first image frame. When the second image frame is obtained last time, the perspective correction is performed on the second image frame according to the perspective feature map of the second image frame, and the second perspective correction image frame is obtained. The spliced image stored is spliced by the image frames obtained before the perspective correction, and the last frame of the currently stored spliced image is the second perspective correction image frame. At this time, the first perspective correction image frame and the second perspective correction image frame are spliced, that is, the first perspective correction image frame and the stored spliced image are spliced, and the spliced image obtained after the splicing replaces the previously stored spliced image.

[0101] As an optional implementation, referring to FIG. 1, Figure 2 The step S101 of obtaining the perspective feature map of the first image frame includes:

[0102] S201, gradient information of each pixel point of the first image frame is extracted.

[0103] Specifically, in order to obtain the perspective feature map of the first image frame, gradient extraction needs to be performed on the first image frame, that is, the gradient information of each pixel point in the first image frame is extracted. The gradient refers to the difference between the intensities of two adjacent positions, and in the image frame, the gradient refers to the pixel difference between two adjacent pixel points.

[0104] The gradient information of the pixel point includes the gradient angle and the gradient amplitude of the pixel point. The gradient angle of the pixel point is determined by the gradient in the x direction and the gradient in the y direction. For example, for a pixel point with coordinates (x, y), the gradient g x x in the x direction is f(x+1, y)-f(x, y), the gradient g y y in the y direction is f(x, y+1)-f(x, y), and the gradient angle a(x, y) of the pixel point with coordinates (x, y) is arctan[g y / g x ], and the gradient amplitude Wherein, f(x+1, y) represents the pixel value of the pixel point with coordinate (x+1, y), f(x, y) represents the pixel value of the pixel point with coordinate (x, y), and f(x, y+1) represents the pixel value of the pixel point with coordinate (x, y+1). Generally, when extracting the gradient of each direction of the pixel point in the image, the gradient operator can be used for extraction, wherein the gradient operator includes Roberts operator, Sobel operator, Prewitt operator, Laplace operator, etc., and any one of them can be selected. In this embodiment, the Sobel operator is preferably used.

[0105] S202, constructing the gradient superposition graph of the first image frame as the perspective feature map of the first image frame according to the gradient information of each pixel point of the first image frame.

[0106] Specifically, after calculating the gradient information of each pixel point in the first image frame, all the gradient information is used to construct the gradient binary cumulative superposition graph of the first image frame, i.e. the gradient superposition graph, and the gradient superposition graph is taken as the perspective feature map of the first image frame. To construct the gradient binary cumulative superposition graph of the first image frame using all the gradient information, first, the gradient binary graph corresponding to the first image frame is constructed using the gradient information of all the pixel points in the first image frame, and then the gradient binary graph of the first image frame is superimposed with the gradient binary graphs of the image frames collected before the first image frame according to a preset number, so as to obtain the gradient binary cumulative superposition graph of the first image frame. For example, if the preset number is 50, the gradient binary graph of the first image frame is superimposed with the gradient binary graphs of 49 image frames collected before the first image frame, and the 50 gradient binary graphs are superimposed together to obtain the gradient binary cumulative superposition graph of the first image frame, i.e. the gradient superposition graph.

[0107] Referring to Figure 3 As shown in the figure, the first row of images are original image frames, the original image frames in the first column are image frames acquired at the first frame moment, the original image frames in the second column are image frames acquired at the n-th frame moment, and the original image frames in the third column are image frames acquired at the m-th frame moment. The second row of images are the gradient binary graphs corresponding to the original image frames in the first row, and the third row of images are the gradient binary cumulative superposition graphs corresponding to the original image frames in the first row. Figure 4 In the gradient binary cumulative superposition graph at the m-th frame moment, there is a relatively obvious difference between the width of the left two vertices of the connected domain (i.e. the white area in the figure) and the width of the right two vertices of the connected domain, so as to reflect that the original image frame corresponding to the gradient binary cumulative superposition graph has a relatively obvious perspective effect. Therefore, in this embodiment, the gradient superposition graph is taken as the perspective feature map to represent the perspective effect feature of the image content in the image frame.

[0108] As an optional implementation, referring to Figure 3As shown, another embodiment of this application discloses that in step S102 above, perspective correction is performed on the first image frame based on the perspective feature map of the first image frame to obtain a first perspective-corrected image frame, including:

[0109] S401. Extract the region to be corrected from the perspective feature map of the first image frame.

[0110] Specifically, in order to determine the perspective correction method for the image content in the first image frame, this embodiment needs to determine, based on the perspective of the image content shown in the perspective feature map, the correction method for the region where the image content in the perspective feature map is located, which can eliminate the perspective effect of the image content in the perspective feature map. Therefore, it is first necessary to extract the foreground connected components (such as...) from the perspective feature map of the first image frame. Figure 5 (The white area in the gradient overlay map at time m). If, in addition to the complete character in the middle of the image frame, there is a portion of a character from the previous or next line that was accidentally scanned above or below the image frame, when constructing the gradient overlay map, the middle of the gradient overlay map will have a foreground connected component corresponding to the complete character, and the top or bottom of the gradient overlay map will also form a foreground connected component corresponding to a portion of the character from the previous or next line. Since the foreground connected component corresponding to the complete character will definitely be larger than the foreground connected component corresponding to a portion of the character, in order to perform perspective correction on the complete character in the middle of the first image frame, it is necessary to extract the largest foreground connected component from all foreground connected components in the perspective feature map (i.e., the gradient overlay map) of the first image frame as the region to be corrected for perspective correction.

[0111] S402. Based on the region to be corrected, calculate the deformation field corresponding to the first image frame.

[0112] Specifically, the region to be corrected extracted from the perspective feature map of the first image frame is a region with perspective effect. The region to be corrected needs to be corrected with the goal of eliminating perspective effect. The transformation matrix corresponding to correcting the region to a region without perspective effect is determined, and the transformation matrix is ​​used as the deformation field corresponding to the first image frame.

[0113] Furthermore, the specific steps are as follows:

[0114] First, based on the vertices and upper and lower boundaries of the region to be corrected, a straight line is fitted to obtain the fitted straight line.

[0115] The embodiment needs to obtain a plurality of vertices from the boundary of the to-be-corrected region in the perspective feature map, and then perform straight line fitting on the upper and lower boundaries of the to-be-corrected region through the vertices of the to-be-corrected region and the upper and lower boundaries of the to-be-corrected region, so as to obtain a fitting straight line corresponding to the upper boundary of the to-be-corrected region and a fitting straight line corresponding to the lower boundary of the to-be-corrected region. In the embodiment, four vertices of the to-be-corrected region are preferably obtained, two vertices are obtained for the upper boundary, and two vertices are obtained for the lower boundary. The fitting straight line corresponding to the upper boundary is fitted according to the upper boundary vertices, and the fitting straight line corresponding to the lower boundary is fitted according to the lower boundary vertices. As shown in Figure 5 Figure 5 In the left figure, the straight line corresponding to the two points a1 and b1 is the fitting straight line corresponding to the upper boundary of the to-be-corrected region, and the straight line corresponding to the two points c1 and d1 is the fitting straight line corresponding to the lower boundary of the to-be-corrected region. Among them, a1, b1, c1 and d1 are the intersection points between the two fitting straight lines and the boundary of the perspective feature map. When the perspective feature map embodies a larger degree of perspective effect of the first image frame, the distance between a1 and c1 is greatly different from the distance between b1 and d1. Therefore, the distance between a1 and c1 and the distance between b1 and d1 can be adjusted to reduce the difference between the distance between a1 and c1 and the distance between b1 and d1, so as to correct the to-be-corrected region and weaken the perspective effect of the perspective feature map.

[0116] Secondly, the fitting straight line is adjusted to a horizontal state to obtain a horizontal fitting straight line, and the intersection point of the horizontal fitting straight line and the boundary of the first image frame is determined as a target reference point.

[0117] The embodiment can adjust the fitting straight line corresponding to the upper boundary of the to-be-corrected region and the fitting straight line corresponding to the lower boundary of the to-be-corrected region of the perspective feature map to a horizontal state. In this way, the two straight lines remain parallel, the distance between a1 and c1 is adjusted to be the same as the distance between b1 and d1, and both of the two fitting straight lines are in a horizontal state. The to-be-corrected region will be in a horizontal state, and the image content in the to-be-corrected region corresponding to the first image frame will also remain in a horizontal state, so as to avoid the image content to be recognized being inclined and improve the accuracy during recognition. After the fitting straight line is adjusted to a horizontal state, a corresponding horizontal fitting straight line can be obtained. The embodiment needs to determine the intersection point of the horizontal fitting straight line and the boundary of the perspective feature map, which is also the intersection point of the horizontal fitting straight line and the boundary of the first image frame. All the intersection points are determined as target reference points.

[0118] In the embodiment, it is assumed that the width of the first image frame is w, and the width of the perspective feature map corresponding to the first image frame is also w. At this time, the coordinates of a1 can be determined as (0, y1), the coordinates of b1 can be determined as (w-1, y2), the coordinates of c1 can be determined as (0, y3), and the coordinates of d1 can be determined as (w-1, y4). As shown in Figure 5 ​As shown in the right side of the figure, four points a2, b2, c2 and d2 are target reference points, and a1 and b1 are adjusted according to the height of the highest point in a1 and b1 to obtain a2 and b2, Figure 5 In the case where the highest point in a1 and b1 is a1, a1 remains in the original position, and b1 is adjusted according to the height of a1, that is, the coordinates of a2 are (0, y1), and the coordinates of b2 are (w-1, y1). c1 and d1 are adjusted according to the height of the lowest point in c1 and d1 to obtain c2 and d2, Figure 6 In the case where the lowest point in c1 and d1 is d1, d1 remains in the original position, and c1 is adjusted according to the height of d1, that is, the coordinates of c2 are (0, y4), and the coordinates of d2 are (w-1, y4).

[0119] Thirdly, a transformation matrix is calculated according to the fitting straight line and the target reference points, and the transformation matrix is used as the deformation field of the first image frame.

[0120] In this embodiment, the transformation matrix can be calculated by using the DLT (direct linear transformation) method according to the intersection of the fitting straight line and the boundary of the perspective feature map and the target reference points, that is, the four points a1-a2, b1-b2, c1-c2 and d1-d2. The perspective feature map of the first image frame can be corrected to a perspective feature map without perspective effect according to the transformation matrix, and the first image frame can also be corrected to an image frame without perspective effect according to the transformation matrix. Therefore, the transformation matrix can be used as the deformation field of the first image frame.

[0121] S403, according to the deformation field, the first image frame is perspective corrected to obtain a first perspective corrected image frame corresponding to the first image frame.

[0122] In this embodiment, the first image frame can be perspective corrected according to the deformation field determined in the above steps, that is, the image content corresponding to the to-be-corrected region in the first image frame is corrected according to the above transformation matrix, so as to obtain the first perspective corrected image frame after perspective correction of the first image frame. As shown in the figure, Figure 7 As shown in the figure, the left side of the figure is the original image frame, and the right side is the perspective corrected image frame after perspective correction of the original image frame. The perspective corrected image frame eliminates the perspective effect in the original image frame.

[0123] As an optional implementation, referring to Figure 3 As shown in the figure, another embodiment of the present application discloses that the following steps need to be performed before the above step S102:

[0124] S702, according to the perspective feature map of the first image frame and the pre-set correction condition region, it is determined whether the first image frame has perspective correction requirement.

[0125] Specifically, the first image frame needs to be perspective corrected only when the perspective effect of the first image frame reaches a situation requiring perspective correction, and therefore, it is further needed to determine whether the first image frame has a perspective correction requirement before the perspective correction is performed on the first image frame. In the embodiment, a correction condition region is pre-set, and whether the first image frame has a perspective correction requirement needs to be determined according to the image content in the correction condition region and the to-be-corrected region in the perspective feature map of the first image frame. When the image is scanned, the characters appear in the right, middle and left of the image in turn during the movement of the scanning device, and when the scanning angle of the scanning device causes the scanned image frame to have a perspective effect, the image content (characters) in the image frame is closer to the left side, and the perspective effect of the image frame is more obvious, and therefore, the pre-set correction condition region is preferably a set region at the leftmost side of the image frame. As shown in FIG. 1, the box region at the left side of each image in the figure is the pre-set correction condition region. Figure 3

[0126] Further, the specific steps are as follows:

[0127] First, if the to-be-recognized characters exist in the pre-set correction condition region in the first image frame, and the connected domain area of the gradient superposition map reaches a preset threshold, it is determined that the first image frame has a perspective correction requirement.

[0128] In the embodiment, the set region at the leftmost side of the image frame is taken as the region to which the image content needs to reach when the perspective correction is needed, that is, when the to-be-recognized characters in the image content reach the set region at the leftmost side (the pre-set correction condition region), the perspective correction of the image frame is necessary. As shown in FIG. 1, the to-be-recognized characters do not exist in the correction condition region in the image frame at the first frame moment and the nth frame moment, and therefore, the image frame at the first frame moment and the nth frame moment does not need to be perspective corrected. Figure 7

[0129] In addition, when the image content in the image frame does not have a perspective effect, the connected domain of the image content in the gradient superposition map corresponding to the image frame is in a horizontal state, and is more similar to a rectangle, and if the image content in the image frame has a perspective effect, the connected domain of the image content in the gradient superposition map corresponding to the image frame is similar to a trapezoid, and at this time, the area of the connected domain is greater than that when the perspective effect does not exist, and therefore, the preset threshold is set according to the connected domain area of the image content in the gradient superposition map corresponding to the image frame when the perspective effect does not exist. Only when the connected domain area of the gradient superposition map corresponding to the first image frame reaches the preset threshold, it is indicated that the perspective correction of the first image frame is necessary.

[0130] ​​This embodiment needs to determine the perspective correction requirement of the first image frame based on two aspects: the correction condition region and the area of ​​the connected components of the gradient overlay map. When there is a character to be recognized in the correction condition region of the first image frame and the area of ​​the connected components of the gradient overlay map corresponding to the first image frame reaches a preset threshold, it is determined that the first image frame has a perspective correction requirement.

[0131] Second, if there is no character to be recognized within the pre-set correction condition area in the first image frame, or if the area of ​​the connected components of the gradient overlay map does not reach the preset threshold, then it is determined that the first image frame does not have a perspective correction requirement.

[0132] If there is no character to be recognized within the correction condition region of the first image frame, it means that there is no need to perform perspective correction on the first image frame. If the area of ​​the connected component of the gradient overlay graph corresponding to the first image frame does not reach the preset threshold, it also means that there is no need to perform perspective correction on the first image frame. Therefore, when there is no character to be recognized within the preset correction condition region of the first image frame, or when the area of ​​the connected component of the gradient overlay graph does not reach the preset threshold, it is determined that the first image frame does not have a perspective correction requirement.

[0133] S703. If the first image frame has a perspective correction requirement, then perform perspective correction on the first image frame according to the perspective feature map of the first image frame to obtain the first perspective-corrected image frame, and obtain the second perspective-corrected image frame of the second image frame.

[0134] If it is determined through the above steps that the first image frame does not require perspective correction, then perspective correction is not necessary for the first image frame. In this case, the first image frame is directly used as the first perspective-corrected image frame, and the second perspective-corrected image frame of the second image frame is obtained for subsequent image stitching steps. If it is determined that the first image frame requires perspective correction, then perspective correction is performed on the first image frame according to its perspective feature map to obtain the first perspective-corrected image frame, and the second perspective-corrected image frame of the second image frame is obtained. The specific steps for performing perspective correction on the first image frame and obtaining the second perspective-corrected image frame of the second image frame have been described in step S102 of the above embodiment and will not be elaborated here.

[0135] Figure 1 Step S701 and Figure 7 The steps in step S101 are the same. Figure 1 Step S704 and Figure 8 The steps S103 are the same as those in the previous embodiment, and steps S701 and S704 will not be described in detail in this embodiment.

[0136] As an optional implementation, see [link to implementation details]. Figure 9As shown, another embodiment of the present application discloses that the step S103 of splicing the first perspective correction image frame and the second perspective correction image frame to obtain a spliced image comprises:

[0137] S801, comparing the first perspective correction image frame and the second perspective correction image frame to determine the overlapping area of the two image frames.

[0138] Specifically, during the scanning process, the scanning frame rate is usually greater than the moving speed of the scanning device to avoid image missing, and when the scanning frame rate is greater than the moving speed of the scanning device, the same content in the adjacent two or more frames of images scanned will appear. Therefore, in order to avoid the repetition of the image content after the splicing of the two image frames, the same image content in the adjacent frames can be overlapped and spliced, so that only one copy of the repeated image content is retained. Therefore, the embodiment needs to determine the overlapping area between the two image frames to be spliced, and since the same image content is located in different positions in the two image frames, the image features corresponding to the same image content will also be different under the influence of the perspective effect. If the two image frames are image frames with obvious perspective effect, the matching accuracy will be low when the two image frames are compared and matched, that is, the accuracy of the overlapping area of the two image frames obtained by comparison will be low, thereby causing poor splicing effect between the two image frames and affecting image recognition. Therefore, when splicing the image, the first perspective correction image frame and the second perspective correction image frame after perspective correction are directly compared to determine the overlapping area of the two image frames, and the accuracy of the overlapping area is higher.

[0139] Due to the scanning characteristics of the scanning device, if the two adjacent image frames have the same image content, the image content of the left part of the current image frame is the same as the image content of a certain region of the previous image frame. Therefore, when searching for the overlapping area of the two image frames, the image region on the left side of the image frame can be set as the matching region for image matching and comparison, and the region with the highest image similarity to the matching region of the first perspective correction image frame in the second perspective correction image frame is searched as the overlapping area.

[0140] S802, based on the overlapping area, splicing the first perspective correction image frame and the second perspective correction image frame to obtain a spliced image.

[0141] Specifically, after the overlapping area between the first perspective corrected image frame and the second perspective corrected image is determined, the matching area in the first perspective corrected image frame and the overlapping area in the second perspective corrected image frame need to be overlapped and spliced, that is, the first perspective corrected image frame is directly spliced to the second perspective corrected image frame based on the overlapping of the matching area in the first perspective corrected image frame and the overlapping area in the second perspective corrected image frame.

[0142] Further, the specific steps are as follows:

[0143] First, according to the overlapping area and the matching area, the offset between the first perspective corrected image frame and the second perspective corrected image frame is determined.

[0144] By comparing the above steps, it can be concluded that the image similarity between the overlapping area in the second perspective corrected image and the matching area in the first perspective corrected image frame is the highest, which means that the image in the overlapping area of the second perspective corrected image and the image in the matching area of the first perspective corrected image frame are the same image content. In order to realize the overlapping splicing, the offset between the two positions needs to be determined according to the position of the overlapping area in the second perspective corrected image frame and the position of the matching area in the first perspective corrected image frame, wherein the offset includes: the horizontal deviation value between the horizontal position of the overlapping area in the second perspective corrected image frame and the horizontal position of the matching area in the first perspective corrected image frame, and the vertical deviation value between the vertical position of the overlapping area in the second perspective corrected image frame and the vertical position of the matching area in the first perspective corrected image frame.

[0145] Second, based on the offset, the first perspective corrected image frame and the second perspective corrected image frame are spliced to obtain a spliced image.

[0146] When splicing the first perspective corrected image frame and the second perspective corrected image frame, the first perspective corrected image frame needs to be moved by a distance corresponding to the horizontal deviation value in the horizontal direction, and then be moved by a distance corresponding to the vertical deviation value in the vertical direction, based on the overlapping and alignment of the first perspective corrected image frame and the second perspective corrected image frame, so as to realize the overlapping between the matching area in the first perspective corrected image frame and the overlapping area in the second perspective corrected image frame. Then, based on the current position, the first perspective corrected image frame and the second perspective corrected image frame are spliced together to obtain a spliced image.

[0147] As an optional implementation, referring to Figure 10 As shown in FIG. 8, another embodiment of the present application discloses that the step S801 of comparing the first perspective corrected image frame with the second perspective corrected image frame to determine the overlapping area of the two image frames includes:

[0148] S901, extract a preset image region from the first perspective correction image frame as a matching region.

[0149] Specifically, due to the scanning characteristics of the scanning device, if the two adjacent image frames have the same image content, that is, the image content of the left part of the current image frame is the same as the image content of a certain region of the previous image frame, therefore, when searching for the overlapping region of the two image frames, the image region on the left side of the image frame can be preset, and the preset image region is extracted as the matching region for image matching comparison.

[0150] S902, determine the feature vector of the matching region, and determine the feature vector of each sub-region of the second perspective correction image frame.

[0151] Specifically, in order to match the region with the highest image similarity with the image in the matching region from the second perspective correction image frame, the embodiment needs to divide the region of the second perspective correction image frame, so as to obtain a plurality of sub-regions, and then calculate the image similarity between the matching region and each sub-region according to the image features in the matching region of the first perspective correction image frame and the image features in each sub-region of the second perspective correction image frame. Wherein, the feature vector of the image in the region can be calculated, and the feature vector is taken as the image feature of the region for similarity analysis. Therefore, the embodiment needs to determine the feature vector of the matching region of the first perspective correction image frame and the feature vector of each sub-region of the second perspective correction image frame.

[0152] Further, the feature vector of the matching region is determined, which specifically includes the following steps:

[0153] First, divide the matching region into at least one image block, and determine the weight of each image block.

[0154] The embodiment can divide the image of the matching region of the first perspective correction image frame into at least one image block, and then set the corresponding weight for each image block according to the position of each image block, wherein the weight of the image block is determined according to the importance of the position of the image block in image recognition. For example, in the image frame, the middle part is usually the image content that needs to be recognized, and the upper boundary and the lower boundary of the image frame may have the character edge of the previous row or the next row scanned by accident, so the weight of the middle region of the image frame can be assigned to be larger, and the weight of the region near the upper boundary and the lower boundary can be assigned to be smaller. In this way, when the number of strokes of the character to be recognized is small and the contrast is difficult, the accuracy of image contrast can be improved. The embodiment can adopt irregular size division method when dividing the matching region into image blocks, such as Figure 10As shown, if the left image represents the matching region, the middle image represents the nine image blocks resulting from the matching process, which are divided into irregularly sized blocks of varying sizes. The right image represents the weight values ​​assigned to each image block. The central region of the image frame is the primary area to be compared, thus receiving a larger weight value. Conversely, the upper and lower boundaries are less important in image comparison and therefore receive smaller weight values.

[0155] Second, calculate the gradient orientation histogram features for each image patch.

[0156] In this embodiment, after dividing the image of the matching region into several image blocks, it is necessary to calculate the oriented gradient histogram (HOG) feature of each image block. Since the 8-directional gradient histogram feature can effectively distinguish different characters, and in object detection tasks, the appearance and shape of local objects can be well described by the directional density distribution of gradients or edges, this embodiment preferably calculates the 8-directional gradient histogram feature of each image block.

[0157] Third, based on the weight of each image patch, the directional gradient histogram features of all image patches are weighted and normalized to obtain the feature vector of the matching region.

[0158] Since each image patch has a corresponding weight, this embodiment requires weighted normalization of the histogram features of all image patches, concatenating all histogram features into a feature vector, i.e., the feature vector of the matching region. Specifically, the histogram features of image patches with a weight of 0 are discarded; only the histogram features of image patches with non-zero weights are weighted and normalized directly. Figure 11 Of the nine image blocks in the matching region shown, four have a weight of 0. Therefore, only the directional feature histograms of the remaining five image blocks are weighted and normalized. Since each image block is calculated as an 8-directional gradient histogram feature, the feature vector obtained after weighting and normalizing the five image blocks is a 40-dimensional feature vector.

[0159] In addition, in this embodiment, the calculation method of the feature vector of each sub-region of the second perspective correction image frame is the same as the calculation method of the feature vector of the matching region, and will not be described in detail in this embodiment.

[0160] S903. Based on the feature vector of the matching region and the feature vector of each sub-region, search for the sub-region with the highest similarity to the matching region from each sub-region as the overlapping region.

[0161] Specifically, after calculating the feature vector of the matching region of the first perspective-corrected image frame and the feature vectors of each sub-region of the second perspective-corrected image frame through the above steps, it is necessary to calculate the similarity between the matching region and each sub-region. In this embodiment, the cosine similarity calculation method is used to calculate the cosine similarity between the feature vector of the matching region and the feature vectors of each sub-region, thereby determining the similarity between the matching region and each sub-region. Based on all similarities, the sub-region with the highest similarity is extracted and regarded as the overlapping region.

[0162] In this embodiment, when dividing the second perspective correction image frame into sub-regions, the second perspective correction image frame can be completely divided directly, and then a traversal search method can be used to traverse each sub-region and select the sub-region with the highest similarity. Alternatively, the second perspective correction image frame can be divided vertically first, and the vertical region with the highest similarity can be selected from each vertically divided region. Then, the second perspective correction image frame can be divided horizontally, and the horizontal region with the highest similarity can be selected from each horizontally divided region. The region where the vertical region with the highest similarity and the horizontal region with the highest similarity overlap are then taken as the sub-region with the highest similarity, i.e., the overlapping region.

[0163] As an optional implementation, see [link to implementation details]. Figure 11 As shown, another embodiment of this application discloses that, in the case of real-time scanning and real-time stitching of scanned images, the second perspective-corrected image frame is the last frame of the pre-stored stitched image. Since scanning devices typically set a high scanning frame rate (e.g., 120 frames / second) to avoid missing images during scanning, the similarity between adjacent scanned image frames is high. If each image frame is matched and stitched, it would put a heavy burden on real-time stitching, affecting stitching efficiency. Therefore, when the similarity between the current scanned image and the previous stitched image (i.e., the last frame of the pre-stored stitched image) is high before perspective correction, frame skipping can be used to abandon the stitching of the current scanned image, reducing the burden on real-time stitching. Since the image features change after perspective correction, and the current scanned image is an uncorrected image, when judging the similarity between the current scanned image and its previous frame, it is also necessary to compare it with the previous frame before perspective correction. This avoids inaccurate judgments caused by comparing the similarity between perspective-corrected images.

[0164] That is, before obtaining the perspective feature map of the first image frame in step S101 above, the method further includes:

[0165] S1101. Analyze the target similarity between the acquired current scanned image and the second image frame.

[0166] Specifically, in this embodiment, the last frame of the pre-stored spliced image is a second perspective correction image frame, and the second image frame is an original image frame before perspective correction of the second perspective correction image frame. When the scanning device performs real-time scanning, the current scanning image obtained by the scanning device needs to be compared with the original image frame (i.e., the second image frame) before perspective correction corresponding to the previous frame of the spliced image to obtain the target similarity between the current scanning image and the second image frame.

[0167] Further, the specific steps are as follows:

[0168] First, the image features of the obtained current scanning image and the image features of the second image frame are determined.

[0169] In this embodiment, the image features of the current scanning image and the second image frame need to be calculated. In this embodiment, the perceptual hash feature (phash feature) can be calculated as the image feature, or the histogram of oriented gradients feature (HOG feature) can be calculated as the image feature, but the image features of the current scanning image and the second image frame need to be of the same type. The calculation of the perceptual hash feature first needs to divide the image into image blocks, then perform DCT (discrete cosine transform) on the image blocks, extract the low-frequency features of each image block, and obtain the phash feature value by comparing the low-frequency features of the image blocks with the average low-frequency features of all the image blocks. In this embodiment, the image features include global features and / or local features, wherein the local region corresponding to the local features is a pre-set region, and the local region is preferably a region with a width of 1 / 3 of the left side of the image.

[0170] Second, the target similarity between the image features of the current scanning image and the image features of the second image frame is calculated.

[0171] In this embodiment, the target similarity between the image features of the current scanning image and the image features of the second image frame can be determined by calculating the Bhattacharyya distance, or the target similarity between the image features of the current scanning image and the image features of the second image frame can be determined by using the cosine similarity, and the embodiment does not make specific limitations. If the image features only include global features, the target similarity is the similarity between the global features of the current scanning image and the global features of the second image frame. If the image features only include local features, the target similarity is the similarity between the local features of the current scanning image and the local features of the second image frame. If the image features include global features and local features, the target similarity includes the similarity between the global features of the current scanning image and the global features of the second image frame, and the similarity between the local features of the current scanning image and the local features of the second image frame.

[0172] S1102, determine the similarity relationship between the current scanning image and the second image frame according to the target similarity and the pre-set similarity threshold.

[0173] The embodiment needs to compare the target similarity between the image features of the current scanning image and the image features of the second image frame with the pre-set similarity threshold, so as to determine the similarity relationship between the current scanning image and the second image frame. If the target similarity is less than the similarity threshold, it indicates that the similarity relationship between the current scanning image and the second image frame represents dissimilarity; if the target similarity is not less than the similarity threshold, it indicates that the similarity relationship between the current scanning image and the second image frame represents similarity.

[0174] Specifically, the embodiment can take the similarity between the whole image features of the current scanning image and the whole image features of the second image frame as the first target similarity, and the first similarity threshold corresponding to the first target similarity is set as the first similarity threshold. The similarity between the local features of the current scanning image and the local features of the second image frame is taken as the second target similarity, and the second similarity threshold corresponding to the second target similarity is set as the second similarity threshold.

[0175] Then, when the image features only include the whole image features, if the first target similarity is less than the first similarity threshold, it indicates that the similarity relationship between the current scanning image and the second image frame represents dissimilarity; if the first target similarity is not less than the first similarity threshold, it indicates that the similarity relationship between the current scanning image and the second image frame represents similarity.

[0176] When the image features only include the local features, if the second target similarity is less than the second similarity threshold, it indicates that the similarity relationship between the current scanning image and the second image frame represents dissimilarity; if the second target similarity is not less than the second similarity threshold, it indicates that the similarity relationship between the current scanning image and the second image frame represents similarity.

[0177] When the image features include the whole image features and the local features, if the first target similarity is less than the first similarity threshold, and the second target similarity is less than the second similarity threshold, it is determined that the similarity relationship between the current scanning image and the previous image frame represents dissimilarity; if the first target similarity is not less than the first similarity threshold, and or, the second target similarity is not less than the second similarity threshold, it is determined that the similarity relationship between the current scanning image and the second image frame represents similarity. The first similarity threshold is preferably set to 0.95, and the second similarity threshold is preferably set to 0.90.

[0178] S1103, if the similarity relationship represents similarity, the next frame scanning image is taken as the current scanning image.

[0179] If it is judged that the similarity relation between the current scanning image and the second image frame indicates similarity through the above steps, it is indicated that the similarity between the current scanning image and the second image frame is high, and thus the current scanning image can be directly skipped, the next scanning image is directly obtained and taken as the current scanning image, and the target similarity calculation is continued between the current scanning image and the second image frame.

[0180] S1104, if the similarity relation indicates dissimilarity, the current scanning image is taken as the first image frame.

[0181] If it is judged that the similarity relation between the current scanning image and the second image frame indicates dissimilarity through the above steps, it is indicated that the similarity between the current scanning image and the second image frame is not high, and thus the image content can be lost if the current scanning image is discarded, and thus the current scanning image is not skipped, and the current scanning image is taken as the first image frame, and then the subsequent perspective correction and image splicing steps are performed.

[0182] Figure 1 The steps S1105-S1107 in the embodiment are the same as the steps S101-S103 in the embodiment, and thus the steps S1105-S1107 are not described in detail. Figure 12

[0183] As an optional implementation, referring to FIG. 12, another embodiment of the present application discloses that the image splicing method further includes: Figure 12

[0184] S1204, determining the boundary contour of the image content in the spliced image.

[0185] When the first perspective correction image frame and the second perspective correction image frame are spliced, the scanning paths of the two image frames can be inconsistent, and thus the spliced image can be curved. In order to ensure the accuracy of image recognition, the spliced image needs to be corrected. First, the boundary contour of the image content in the spliced image needs to be determined. The boundary contour of the image content can be determined by gradient extraction. As shown in FIG. 12, the upper left image in FIG. 12 is the spliced image before correction, and the contour around the text is the determined boundary contour of the image content. Figure 12 Figure 12

[0186] S1205, performing boundary alignment correction in the vertical direction on the spliced image by using the point coordinates on the boundary contour, to obtain the spliced image after correction.

[0187] ​​​​After the boundary contour of the image content in the spliced image is determined, the difference of each point in the vertical direction can be calculated according to the point coordinates on the boundary contour, the transformation matrix of the image content in the spliced image in the vertical direction is determined according to the difference, and then the spliced image is corrected in the vertical direction according to the transformation matrix to obtain the corrected spliced image. For example Figure 14 The lower right image is the corrected spliced image.

[0188] Corresponding to the image splicing method described above, the embodiment of the application also provides an image splicing device, as shown in Figure 15 The device comprises:

[0189] The acquisition module 100 is configured to acquire a perspective feature map of the first image frame; wherein the perspective feature map is used to represent the perspective effect feature of the image content in the image frame;

[0190] The perspective correction module 110 is configured to perform perspective correction on the first image frame according to the perspective feature map of the first image frame to obtain a first perspective corrected image frame, and obtain a second perspective corrected image frame of the second image frame; the perspective correction aims to eliminate the perspective effect of the image content in the image frame;

[0191] The image splicing module 120 is configured to splice the first perspective corrected image frame and the second perspective corrected image frame to obtain a spliced image.

[0192] The image splicing device provided by the embodiment of the application acquires the perspective feature map of the first image frame by using the acquisition module 100, performs perspective correction on the first image frame according to the perspective feature map of the first image frame by using the perspective correction module 110 to obtain the first perspective corrected image frame, and obtains the second perspective corrected image frame of the second image frame. The first perspective corrected image frame and the second perspective corrected image frame are spliced by using the image splicing module 120 to obtain a spliced image. By using the technical solution of the embodiment, the perspective effect of the image frame is weakened by performing perspective correction on the image frame, which can improve the matching accuracy between two image frames when the image frames are spliced, thereby improving the splicing effect of the scanned image.

[0193] As an optional implementation, another embodiment of the application further discloses that the second perspective corrected image frame is the last frame of the pre-stored spliced image, and the second perspective corrected image frame is obtained by performing perspective correction on the second image frame according to the perspective feature map of the second image frame.

[0194] As an optional implementation, another embodiment of the application further discloses that the acquisition module 100 is specifically configured to:

[0195] extract gradient information of each pixel point of the first image frame;

[0196] According to gradient information of each pixel point of the first image frame, a gradient superposition graph of the first image frame is constructed as a perspective feature graph of the first image frame.

[0197] As an optional implementation, another embodiment of the present application further discloses that the perspective correction module 110 comprises a region extraction unit, a calculation unit and a correction unit.

[0198] The region extraction unit is configured to extract a to-be-corrected region requiring perspective correction from the perspective feature graph of the first image frame.

[0199] The calculation unit is configured to calculate a deformation field corresponding to the first image frame based on the to-be-corrected region.

[0200] The correction unit is configured to perform perspective correction on the first image frame according to the deformation field to obtain a first perspective-corrected image frame corresponding to the first image frame.

[0201] As an optional implementation, another embodiment of the present application further discloses that the calculation unit is specifically configured to:

[0202] Perform straight line fitting based on a vertex of the to-be-corrected region and upper and lower boundaries of the to-be-corrected region to obtain a fitting straight line;

[0203] Adjust the fitting straight line to a horizontal state to obtain a horizontal fitting straight line, and determine an intersection point of the horizontal fitting straight line and a boundary of the first image frame as a target reference point;

[0204] Calculate a transformation matrix according to the fitting straight line and the target reference point, and take the transformation matrix as the deformation field of the first image frame.

[0205] As an optional implementation, another embodiment of the present application further discloses that the image stitching device further comprises a correction requirement determination module configured to determine whether the first image frame has a perspective correction requirement according to the perspective feature graph of the first image frame and a pre-set correction condition region; the correction condition region comprises a set region on the leftmost side of the image frame.

[0206] The perspective correction module 110 is specifically configured to perform perspective correction on the first image frame according to the perspective feature graph of the first image frame to obtain a first perspective-corrected image frame if the first image frame has a perspective correction requirement.

[0207] As an optional implementation, another embodiment of the present application further discloses that the perspective feature graph of the first image frame is a gradient superposition graph of the first image frame.

[0208] The correction requirement determination module is specifically configured to:

[0209] If there is a character to be recognized in the preset correction condition region in the first image frame, and the connected domain area of the gradient superposition graph reaches the preset threshold, it is determined that the first image frame has a perspective correction requirement;

[0210] If there is no character to be recognized in the preset correction condition region in the first image frame, or the connected domain area of the gradient superposition graph does not reach the preset threshold, it is determined that the first image frame does not have a perspective correction requirement.

[0211] As an optional implementation, another embodiment of the present application further discloses that the image stitching module 120 comprises an overlapping region determination unit and a stitching unit.

[0212] The overlapping region determination unit is configured to compare the first perspective correction image frame and the second perspective correction image frame, and determine the overlapping region of the two image frames.

[0213] The stitching unit is configured to stitch the first perspective correction image frame and the second perspective correction image frame based on the overlapping region, to obtain a stitched image.

[0214] As an optional implementation, another embodiment of the present application further discloses that the overlapping region determination unit is specifically configured to:

[0215] extract a preset image region from the first perspective correction image frame as a matching region;

[0216] determine the feature vector of the matching region, and determine the feature vector of each sub-region of the second perspective correction image frame;

[0217] According to the feature vector of the matching region and the feature vector of each sub-region, search for a sub-region with the highest similarity to the matching region from each sub-region as the overlapping region.

[0218] As an optional implementation, another embodiment of the present application further discloses that the stitching unit is specifically configured to:

[0219] determine the offset between the first perspective correction image frame and the second perspective correction image frame according to the overlapping region and the matching region;

[0220] stitch the first perspective correction image frame and the second perspective correction image frame based on the offset, to obtain a stitched image.

[0221] As an optional implementation, another embodiment of the present application further discloses that the overlapping region determination unit determines the feature vector of the matching region, comprising:

[0222] dividing the matching region into at least one image block, and determining the weight of each image block;

[0223] calculating the direction gradient histogram feature of each image block;

[0224] The direction gradient histogram features of all the image blocks are weighted and normalized according to the weight of each image block to obtain a feature vector of the matching region.

[0225] As an optional implementation, the image splicing device further includes an analysis module, a similarity relationship determination module, and an image frame determination module.

[0226] The analysis module is configured to analyze a target similarity between the acquired current scanning image and the second image frame.

[0227] The similarity relationship determination module is configured to determine, according to the target similarity and a pre-set similarity threshold, a similarity relationship between the current scanning image and the second image frame.

[0228] The image frame determination module is configured to, if the similarity relationship indicates dissimilarity, take the current scanning image as the first image frame; and if the similarity relationship indicates similarity, take a next frame scanning image as the current scanning image.

[0229] As an optional implementation, the analysis module is specifically configured to:

[0230] determine image features of the acquired current scanning image and image features of the second image frame.

[0231] calculate a target similarity between the image features of the current scanning image and the image features of the second image frame.

[0232] As an optional implementation, the image features include global image features and pre-set local features corresponding to local regions.

[0233] The similarity threshold includes a first similarity threshold and a second similarity threshold.

[0234] The target similarity includes a first target similarity between the global image features of the current scanning image and the global image features of the second image frame, and a second target similarity between the local features of the current scanning image and the local features of the second image frame.

[0235] The similarity relationship determination module is specifically configured to:

[0236] if the first target similarity is less than the first similarity threshold and the second target similarity is less than the second similarity threshold, determine that the similarity relationship between the current scanning image and the previous image frame indicates dissimilarity.

[0237] If the first target similarity is not less than the first similarity threshold, and / or the second target similarity is not less than the second similarity threshold, it is determined that the similarity relationship between the current scanning image and the second image frame represents similarity.

[0238] As an optional implementation, the image stitching device further includes a boundary determination module and a boundary correction module.

[0239] The boundary determination module is configured to determine a boundary contour of the image content in the stitched image.

[0240] The boundary correction module is configured to perform boundary alignment correction in the vertical direction on the stitched image by using the point coordinates on the boundary contour, to obtain a corrected stitched image.

[0241] The image stitching device provided in the embodiment belongs to the same application concept as the image stitching method provided in the above embodiments of the application, can execute the image stitching method provided in any of the above embodiments of the application, and has the corresponding function modules and beneficial effects of executing the image stitching method. Technical details not described in detail in the embodiment can be referred to the specific processing content of the image stitching method provided in the above embodiments of the application, which will not be described here.

[0242] Another embodiment of the application further discloses an image stitching device, as shown in ​ The device includes:

[0243] a memory 200 and a processor 210.

[0244] The memory 200 is connected with the processor 210, and is configured to store programs.

[0245] The processor 210 is configured to realize the image stitching method disclosed in any of the above embodiments by running the programs stored in the memory 200.

[0246] Specifically, the image stitching device can further include a bus, a communication interface 220, an input device 230 and an output device 240.

[0247] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected with each other through the bus.

[0248] The bus can include a path for transmitting information between various components of the computer system.

[0249] The processor 210 can be a general processor, such as a general central processing unit (CPU), a microprocessor, or the like, or can be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the present application. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-to-use programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0250] The processor 210 can include a main processor and can also include a baseband chip, a modem, and the like.

[0251] The memory 200 stores programs for executing the technical solutions of the present application, and can also store operating systems and other key services. Specifically, the programs can include program codes, and the program codes include computer operation instructions. More specifically, the memory 200 can include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash, and the like.

[0252] The input device 230 can include a device that receives data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor, and the like.

[0253] The output device 240 can include a device that allows information to be output to a user, such as a display screen, a printer, a speaker, and the like.

[0254] The communication interface 220 can include a device using any transceiver, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), and the like, to communicate with other devices or communication networks.

[0255] The processor 2102 executes the programs stored in the memory 200 and calls other devices, which can be used to implement the steps of the image stitching method provided by the embodiments of the present application.

[0256] The image stitching device described above can be a scanning pen, a scanner, or other electronic devices with image scanning functions. By executing the image stitching method described above, the electronic devices can achieve better image stitching effects. Alternatively, the image stitching device described above can also be used in scanning pens, scanners, servers, image processing devices, and other electronic devices. By applying the image stitching device, the other electronic devices can achieve better image stitching effects.

[0257] Another embodiment of the present application further provides a storage medium, which stores a computer program. The computer program is executed by a processor to implement each step of the image stitching method provided by any of the above embodiments.

[0258] For each method embodiment described above, in order to simply describe, each method embodiment is described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of actions described, because according to the present application, some steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0259] It should be noted that each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts between each embodiment can be known by referring to each other. For the device embodiment, since it is basically similar to the method embodiment, it is described relatively simply, and the relevant parts can be known by referring to the part of the method embodiment.

[0260] The steps in the method embodiments of the present application can be adjusted in order, combined and reduced according to actual needs.

[0261] The modules and sub-modules in the device and terminal in each embodiment of the present application can be combined, divided and reduced according to actual needs.

[0262] In several embodiments provided by the present application, it should be understood that the disclosed terminal, device and method can be implemented by other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or sub-modules is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or modules, and can be electrical, mechanical or other forms.

[0263] The modules or sub-modules described as separate components can or can not be physically separate, and the components of the modules or sub-modules can or can not be physical modules or sub-modules, that is, they can be located in one place, or can be distributed on a plurality of network modules or sub-modules. Some or all modules or sub-modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0264] In addition, each functional module or sub-module in each embodiment of the present application can be integrated in one processing module, or each module or sub-module can exist physically alone, or two or more modules or sub-modules can be integrated in one module. The integrated module or sub-module can be realized in the form of hardware or in the form of a software functional module or sub-module.

[0265] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or any combination thereof. To clearly illustrate the interchangeability of hardware and software, various components and steps have been described above generally in terms of their functionality, without limitation. The specific implementation of the described functionality no matter whether it is implemented in hardware or in software depends on the particular application and design constraints imposed on the overall system. Skilled persons can use various methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0266] The steps of the methods or algorithms described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination thereof. A software module can be located in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0267] Finally, it should be noted that, in the present text, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0268] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image stitching method, characterized in that, include: Extract the gradient information of each pixel in the first image frame; Based on the gradient information of each pixel in the first image frame, construct a gradient binary map of the first image frame; The gradient binary map of the first image frame is superimposed with the gradient binary maps of a preset number of image frames acquired before the first image frame to obtain the gradient superimposed map of the first image frame as the perspective feature map of the first image frame; wherein, the perspective feature map is used to characterize the perspective effect features of the image content in the image frame. Based on the perspective feature map of the first image frame, perspective correction is performed on the first image frame to obtain a first perspective-corrected image frame, and a second perspective-corrected image frame of the second image frame is obtained; the perspective correction aims to eliminate the perspective effect of the image content in the image frame. The first perspective-corrected image frame and the second perspective-corrected image frame are stitched together to obtain a stitched image.

2. The method according to claim 1, characterized in that, The second perspective-corrected image frame is the last frame of the pre-stored stitched image, and the second perspective-corrected image frame is obtained by performing perspective correction on the second image frame according to the perspective feature map of the second image frame.

3. The method according to claim 1, characterized in that, Based on the perspective feature map of the first image frame, perspective correction is performed on the first image frame to obtain a first perspective-corrected image frame, including: Extract the region to be corrected that needs perspective correction from the perspective feature map of the first image frame; Based on the region to be corrected, the deformation field corresponding to the first image frame is calculated; Based on the deformation field, perspective correction is performed on the first image frame to obtain the first perspective-corrected image frame corresponding to the first image frame.

4. The method according to claim 3, characterized in that, The step of calculating the deformation field corresponding to the first image frame based on the region to be corrected includes: Based on the vertices and upper and lower boundaries of the region to be corrected, a straight line is fitted to obtain the fitted straight line; The fitted line is adjusted to a horizontal state to obtain a horizontal fitted line, and the intersection point of the horizontal fitted line and the boundary of the first image frame is determined as the target reference point. The transformation matrix is ​​calculated based on the fitted straight line and the target reference point, and the transformation matrix is ​​used as the deformation field of the first image frame.

5. The method according to claim 1, characterized in that, Before performing perspective correction on the first image frame based on the perspective feature map of the first image frame to obtain the first perspective-corrected image frame, the method further includes: Based on the perspective feature map of the first image frame and the preset correction condition area, determine whether the first image frame has a perspective correction requirement; the correction condition area includes the set area on the far left of the image frame. If the first image frame requires perspective correction, then the first image frame is subjected to perspective correction based on the perspective feature map of the first image frame to obtain the first perspective-corrected image frame.

6. The method according to claim 5, characterized in that, The perspective feature map of the first image frame is a gradient overlay map of the first image frame; Based on the perspective feature map of the first image frame and the pre-set correction condition region, determine whether the first image frame has a perspective correction requirement, including: If there is a character to be recognized in the pre-set correction condition area in the first image frame, and the area of ​​the connected components of the gradient overlay map reaches a preset threshold, then it is determined that the first image frame has a perspective correction requirement. If there is no character to be recognized within the pre-set correction condition area in the first image frame, or if the area of ​​the connected components of the gradient overlay map does not reach the preset threshold, then it is determined that the first image frame does not have a perspective correction requirement.

7. The method according to claim 1, characterized in that, The first perspective-corrected image frame and the second perspective-corrected image frame are stitched together to obtain a stitched image, including: The first perspective-corrected image frame and the second perspective-corrected image frame are compared to determine the overlapping area of ​​the two images; Based on the overlapping area, the first perspective correction image frame and the second perspective correction image frame are stitched together to obtain a stitched image.

8. The method according to claim 7, characterized in that, The first perspective-corrected image frame and the second perspective-corrected image frame are compared to determine the overlapping area of ​​the two images, including: Extract a preset image region from the first perspective-corrected image frame as the matching region; Determine the feature vector of the matching region, and determine the feature vector of each sub-region of the second perspective-corrected image frame; Based on the feature vector of the matching region and the feature vector of each sub-region, the sub-region with the highest similarity to the matching region is searched from each sub-region as the overlapping region.

9. The method according to claim 8, characterized in that, Based on the overlapping region, the first perspective-corrected image frame and the second perspective-corrected image frame are stitched together to obtain a stitched image, including: The offset between the first perspective-corrected image frame and the second perspective-corrected image frame is determined based on the overlapping region and the matching region. Based on the offset, the first perspective-corrected image frame and the second perspective-corrected image frame are stitched together to obtain a stitched image.

10. The method according to claim 8, characterized in that, Determining the feature vector of the matching region includes: The matching region is divided into at least one image block, and the weight of each image block is determined; Calculate the directional gradient histogram features for each image patch; Based on the weight of each image block, the directional gradient histogram features of all image blocks are weighted and normalized to obtain the feature vector of the matching region.

11. The method according to claim 2, characterized in that, Before obtaining the perspective feature map of the first image frame, the following steps are also included: Analyze the target similarity between the acquired current scanned image and the second image frame; Based on the target similarity and a preset similarity threshold, the similarity relationship between the current scanned image and the second image frame is determined; If the similarity relationship indicates dissimilarity, then the currently scanned image is taken as the first image frame; If the similarity relationship indicates similarity, then the next frame scanned image will be used as the current scanned image.

12. The method according to claim 11, characterized in that, The target similarity between the currently scanned image and the second image frame obtained by the analysis includes: Determine the image features of the acquired current scanned image and the image features of the second image frame; Calculate the target similarity between the image features of the currently scanned image and the image features of the second image frame.

13. The method according to claim 1, characterized in that, Also includes: Determine the boundary contours of the image content in the stitched image; Using the coordinates of points on the boundary contour, the vertical boundary alignment of the stitched image is corrected to obtain the corrected stitched image.

14. An image stitching device, characterized in that, include: The acquisition module is used to extract the gradient information of each pixel in the first image frame; Based on the gradient information of each pixel in the first image frame, a gradient binary map of the first image frame is constructed; the gradient binary map of the first image frame is superimposed with the gradient binary maps of a preset number of image frames previously acquired, and the resulting gradient superimposed map of the first image frame is used as the perspective feature map of the first image frame; wherein, the perspective feature map is used to characterize the perspective effect features of the image content in the image frame. A perspective correction module is used to perform perspective correction on the first image frame based on the perspective feature map of the first image frame to obtain a first perspective-corrected image frame, and to obtain a second perspective-corrected image frame of the second image frame; the perspective correction aims to eliminate the perspective effect of the image content in the image frame. The image stitching module is used to stitch the first perspective correction image frame and the second perspective correction image frame together to obtain a stitched image.

15. An image stitching device, characterized in that, include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the image stitching method as described in any one of claims 1 to 13 by running a program in the memory.

16. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the image stitching method as described in any one of claims 1 to 13.

Images