Image processing method, medium and electronic device

Abstract

This invention provides an image processing method, a medium, and an electronic device. The image processing method includes: acquiring a first frame image and a second frame image; acquiring the cross-power spectrum of the first frame image and the second frame image; acquiring a corresponding impulse response map based on the cross-power spectrum; acquiring a first offset between the first frame image and the second frame image based on the response values ​​of pixels in the impulse response map; acquiring an actual offset between the first frame image and the second frame image based on the first offset; and stitching the first frame image and the second frame image together based on the actual offset. The image processing method can achieve the stitching of the first frame image and the second frame image.

Description

Technical Field

[0001] This invention relates to an image processing method, and more particularly to an image processing method, a medium, and an electronic device. Background Technology

[0002] With the continuous development of technology, scanning devices are gradually becoming smaller and more intelligent. The working principle of existing scanning devices (such as miniature scanners, scanning pens, and reading pens) is as follows: the scanning device continuously scans the target object while moving, obtaining multiple frames of images. Based on this, an image processor stitches the acquired multiple frames together to obtain a scanned image of the target object. However, in practical applications, the inventors have found that there are often overlapping areas between two or more consecutive frames acquired by the scanning device. These frames cannot be easily stitched together. Therefore, how to provide a technical solution that can stitch frames together has become one of the technical problems that urgently needs to be solved by relevant technicians. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an image processing method, medium, and electronic device to solve the aforementioned problems in the prior art.

[0004] To achieve the above and other related objectives, a first aspect of the present invention provides an image processing method, the image processing method comprising: acquiring a first frame image and a second frame image; acquiring the cross-power spectrum of the first frame image and the second frame image; acquiring a corresponding impulse response map based on the cross-power spectrum; acquiring a first offset between the first frame image and the second frame image based on the response values ​​of pixels in the impulse response map; acquiring an actual offset between the first frame image and the second frame image based on the first offset; and stitching the first frame image and the second frame image together based on the actual offset.

[0005] In one embodiment of the first aspect, the method for obtaining a first offset of the first frame image and the second frame image based on the response values ​​of pixels in the impact response map includes: obtaining a first pixel in the impact response map, wherein the first pixel is the pixel with the largest response value in the impact response map; filtering the impact response map to obtain a filtered map; obtaining a second pixel in the filtered map, wherein the second pixel is the pixel with the largest response value in the filtered map; obtaining a third pixel in the impact response map, wherein the third pixel is the pixel in the impact response map corresponding to the second pixel; obtaining the total response value of all pixels in a first region, wherein the first region includes the first pixel; obtaining the total response value of all pixels in a second region, wherein the second region includes the third pixel, and the shape and size of the second region are the same as those of the first region; selecting a reference region from the first region and the second region based on the total response values ​​of all pixels in the first region and the total response values ​​of all pixels in the second region; and obtaining the first offset based on the centroid coordinates of the reference region and the center coordinates of the impact response map.

[0006] In one embodiment of the first aspect, the method for stitching the first frame image and the second frame image according to the first offset includes: obtaining a matching response value based on the total response value of all pixels in the reference area, the width and height of the impact response map; determining whether the first frame image and the second frame image can be stitched together based on the matching response value and the first offset; if the first frame image and the second frame image can be stitched together, then stitching the first frame image and the second frame image together according to the first offset; if the first frame image and the second frame image cannot be stitched together, obtaining the next frame of the second frame image as a new second frame image, and stitching the first frame image together with the new second frame image.

[0007] In one embodiment of the first aspect, before acquiring the cross-power spectrum of the first frame image and the second frame image, the image processing method further includes: expanding the left side, top side and bottom side of the first frame image and the second frame image; and windowing the expanded first frame image and the second frame image.

[0008] In one embodiment of the first aspect, the method for expanding the left side of the first frame image and the second frame image includes: expanding the left side of the first frame image and the left side of the second frame image based on at least one column of pixels on the left side of the first frame image; and / or the method for expanding the top and bottom of the first frame image and the second frame image includes: expanding the bottom of the first frame image based on multiple rows of pixels in the upper part of the first frame image; expanding the top of the first frame image based on multiple rows of pixels in the lower part of the first frame image; expanding the bottom of the second frame image based on multiple rows of pixels in the upper part of the second frame image; and expanding the top of the second frame image based on multiple rows of pixels in the lower part of the second frame image.

[0009] In one embodiment of the first aspect, after stitching the first frame image and the second frame image to obtain a stitched image, the image processing method further includes: using the second frame image as a new first frame image; obtaining a new second frame image, wherein the new second frame image refers to the next frame image of the new first frame image; obtaining a new first offset; and stitching the new second frame image with the stitched image according to the new first offset to obtain a new stitched image.

[0010] In one embodiment of the first aspect, after stitching the first frame image and the second frame image together, the image processing method further includes: obtaining connected regions in the stitched image, wherein the stitched image is obtained by stitching at least the first frame image and the second frame image together; obtaining the bounding rectangles of each of the connected regions; and segmenting the stitched image according to the bounding rectangles to obtain at least one image block, wherein each image block contains an integer number of the bounding rectangles.

[0011] In one embodiment of the first aspect, the method for obtaining the cross-power spectrum of the first frame image and the second frame image includes: obtaining a second offset; obtaining a sub-image of the first frame image, wherein the sub-image is located on the side of the first frame image away from the offset direction, and the width of the sub-image is determined by the difference between the width of the first frame image and the second offset; and obtaining the cross-power spectrum of the sub-image and the second frame image as the cross-power spectrum of the first frame image and the second frame image.

[0012] A second aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image processing method according to any one of the first aspects of the present invention.

[0013] A third aspect of the present invention provides an electronic device, the electronic device comprising: a memory storing a computer program; and a processor communicatively connected to the memory, which executes the image processing method according to any one of the first aspects of the present invention when the computer program is invoked.

[0014] As described above, the image processing method in one or more embodiments of the present invention has the following beneficial effects:

[0015] The image processing method can obtain the corresponding impulse response map based on the cross-power spectrum of the first frame image and the second frame image, and obtain a first offset based on the response value of the pixel in the impulse response map. Based on the first offset, the image processing method can stitch the first frame image and the second frame image together to obtain a stitched image. Attached Figure Description

[0016] Figure 1A The flowchart shown is a specific embodiment of the image processing method described in this invention.

[0017] Figure 1B The image shown is an example of an impact response map obtained in a specific embodiment of the image processing method described in this invention.

[0018] Figure 1C The image shown is a schematic diagram of a frame image in a specific embodiment of the image processing method described in this invention.

[0019] Figure 1D The image shown is a schematic diagram of a stitched image in a specific embodiment of the image processing method described in this invention.

[0020] Figure 2A The diagram shows a detailed flowchart of step S14 in a specific embodiment of the image processing method described in this invention.

[0021] Figure 2B The image shown is an example of a filtered image in a specific embodiment of the image processing method described in this invention.

[0022] Figure 2C The flowchart shown is a key step of the image processing method described in this invention in a specific embodiment.

[0023] Figure 3A The flowchart shown is a key step of the image processing method described in this invention in a specific embodiment.

[0024] Figures 3B-3G The image shown is a schematic diagram of a frame image in a specific embodiment of the image processing method described in this invention.

[0025] Figure 4 The flowchart shown is a key step of the image processing method described in this invention in a specific embodiment.

[0026] Figure 5A The image shown is an example of a stitched image obtained in a specific embodiment of the image processing method described in this invention.

[0027] Figure 5B The image shown is an example of a segmented image obtained from related technologies.

[0028] Figure 5C The flowchart shown is a process for segmenting a stitched image in a specific embodiment of the image processing method described in this invention.

[0029] Figure 5D The image shown is an example of an outer rectangular frame obtained in a specific embodiment of the image processing method described in this invention.

[0030] Figure 5E The image shown is an example of a segmented image obtained in a specific embodiment of the image processing method described in this invention.

[0031] Figure 5F The diagram shows a detailed flowchart of step S53 in a specific embodiment of the image processing method described in this invention.

[0032] Figure 6A The flowchart shown is a key step of the image processing method described in this invention in a specific embodiment.

[0033] Figure 6B The image shown is a schematic diagram of a frame image obtained by the image processing method described in this invention in a specific embodiment.

[0034] Figure 7A The diagram shown is a detailed flowchart of step S61 in a specific embodiment of the image processing method described in this invention.

[0035] Figure 7B The image shown is an example of a straight line image in a specific embodiment of the image processing method described in this invention.

[0036] Figure 7C The diagram shows a detailed flowchart of step S611 in a specific embodiment of the image processing method described in this invention.

[0037] Figure 7D The image shown is an example of a frame image obtained in a specific embodiment of the image processing method described in this invention.

[0038] Figure 7E The image shown is an example of a binarized gradient image obtained in a specific embodiment of the image processing method described in this invention.

[0039] Figure 7F and Figure 7GThe image shown is an example of a binarized image in a specific embodiment of the image processing method described in this invention.

[0040] Figure 7H The diagram shown is a detailed flowchart of step S612 in a specific embodiment of the image processing method described in this invention.

[0041] Figure 8 The flowchart shown is a specific embodiment of the image processing method described in this invention.

[0042] Figure 9 The diagram shown is a structural schematic of the electronic device described in a specific embodiment of the present invention.

[0043] Component designation explanation

[0044] 11 First frame image

[0045] 111 sub-images

[0046] 12 Second frame image

[0047] 900 electronic devices

[0048] 910 memory

[0049] 920 processor

[0050] 930 monitor

[0051] Steps S11 to S16

[0052] Steps S141~S148

[0053] Steps S21 to S24

[0054] Steps S31 to S32

[0055] Steps S41 to S44

[0056] Steps S51 to S53

[0057] Steps S531~S532

[0058] Steps S61 to S63

[0059] Steps S611~S612

[0060] Steps S6111~S6114

[0061] Steps S6121~S6123

[0062] Steps S801~S813 Detailed Implementation

[0063] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0064] It should be noted that the illustrations provided in the following embodiments are merely schematic representations of the basic concept of the present invention. The illustrations only show components relevant to the present invention and are not drawn according to the actual number, shape, and size of the components in implementation. In actual implementation, the form, quantity, and proportion of each component can be arbitrarily changed, and the component layout may also be more complex. Furthermore, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0065] Scanning devices often acquire overlapping regions between two or more consecutive image frames, which cannot be easily stitched together. To address this issue, an image processing method is provided in one embodiment of the present invention. Please refer to [link to relevant documentation]. Figure 1A The image processing method described in this embodiment includes:

[0066] S11, acquire the first frame image and the second frame image, wherein the first frame image and the second frame image can be two consecutive frames scanned by the scanning device, or two frames with an interval of one or more frames.

[0067] Optionally, after acquiring the first frame image and the second frame image, the image processing method in this embodiment further includes: performing perspective transformation pre-correction on the first frame image and the second frame image to make the text in the first frame image and the second frame image more balanced.

[0068] S12, obtain the cross-power spectrum of the first frame image and the second frame image. Optionally, the cross-power spectrum is a normalized cross-power spectrum, and the method for obtaining the normalized cross-power spectrum is, for example: Wherein, F1(u,v) and F2(u,v) are the Fourier transforms of f1(x,y) and f2(x,y) respectively, f1(x,y) is the function representation of the first frame image, whose parameters are the coordinates of the points in the first frame image and whose function value is the pixel value of the corresponding point, and f2(x,y) is the function representation of the second frame image, whose parameters are the coordinates of the points in the second frame image and whose function value is the pixel value of the corresponding point.

[0069] S13, Obtain the corresponding impulse response diagram based on the cross-power spectrum. Specifically, the impulse response diagram can be obtained by performing an inverse Fourier transform on the cross-power spectrum. For example, please refer to [link to relevant documentation]. Figure 1B The image shown is an example of an impact response diagram obtained in this embodiment.

[0070] S14, obtain a first offset between the first frame image and the second frame image based on the response values ​​of the pixels in the impact response map. For example, the pixel with the largest response value in the impact response map can be selected as the first pixel, and the distance from the first pixel to the center point of the impact response map can be obtained as the first offset.

[0071] S15, obtain the actual offset between the first frame image and the second frame image based on the first offset. Specifically, when the cross-power spectrum is obtained based on the complete first frame image and second frame image, the first offset is the actual offset between the first frame image and the second frame image.

[0072] S16, the first frame image and the second frame image are stitched together according to the actual offset.

[0073] Optionally, step S16 can extract a sub-image with a width equal to the actual offset from one side of the first frame image along the offset direction, and stitch this sub-image to the second frame image along the offset direction to obtain a stitched image of the first and second frames. Alternatively, extract a sub-image with a width equal to the actual offset from one side of the second frame image away from the offset direction, and stitch this sub-image to the first frame image away from the offset direction to obtain a stitched image of the first and second frames. Here, the offset direction is the opposite direction of the scanning device's movement direction, and the offset direction is, for example, from left to right or from right to left. When the offset direction is from left to right, the side along the offset direction is the right side, and the side away from the offset direction is the left side; when the offset direction is from right to left, the side along the offset direction is the left side, and the side away from the offset direction is the right side.

[0074] For example, please see Figure 1C The actual offset between the first frame and the second frame is 2L. Furthermore, Figure 1C The scanning device moves from left to right, and therefore the offset direction is from right to left. Thus, a sub-image with a width of 2L can be extracted from the left side of the first frame image 11 and stitched to the left side of the second frame image 12 to obtain... Figure 1DThe stitched image shown can also be obtained by extracting a sub-image with a width of 2L from the right side of the second frame image 12 and stitching it to the right side of the first frame image 11. Figure 1D The stitched image shown.

[0075] Optionally, step S16 may also obtain the overlapping area between the first frame image and the second frame image based on the actual offset, and stitch the second frame image to the first frame image in a manner that covers the overlapping area.

[0076] As described above, the image processing method in this embodiment can obtain the corresponding impulse response map based on the cross-power spectrum of the first frame image and the second frame image, and obtain a first offset based on the response value of the pixel in the impulse response map. Based on the first offset, the image processing method can stitch the first frame image and the second frame image together to obtain a stitched image.

[0077] In one embodiment of the present invention, considering that directly selecting the pixel with the largest response value as the reference pixel in step S14 may result in errors, please refer to [link to relevant documentation]. Figure 2A In this embodiment, a preferred method for obtaining the first offset based on the response values ​​of pixels in the impact response map is as follows:

[0078] S141, Obtain the first pixel in the impact response map, wherein the first pixel refers to the pixel with the largest response value in the impact response map. In practical applications, the first pixel can be obtained in the impact response map by traversal or other methods.

[0079] It should be noted that selecting the pixel with the largest response value in the impact response diagram as the first pixel is only a preferred implementation method, but the present invention is not limited to this. In practical applications, the pixel with a relatively large response value in the impact response diagram can also be selected as the first pixel, such as the second largest pixel, the third largest pixel, etc.

[0080] S142, the impulse response map is filtered to obtain a filtered map, wherein the filtering is, for example, mean filtering, and the size of the filtered map is preferably the same as that of the impulse response map. (See also...) Figure 2B The image shown is an example of a filtered image obtained in this embodiment.

[0081] S143, Obtain the second pixel in the filtered image, wherein the second pixel is the pixel with the largest response value in the filtered image. In practical applications, the second pixel can be obtained in the filtered image by traversal or other methods.

[0082] It should be noted that selecting the pixel with the largest response value in the filtered image as the second pixel is only a preferred implementation method, but the present invention is not limited to this. In practical applications, the pixel with a relatively large response value in the filtered image can also be selected as the second pixel, such as the second largest pixel, the third largest pixel, etc.

[0083] S144, Obtain the third pixel in the impact response map, wherein the third pixel is the pixel in the impact response map that corresponds to the second pixel. For example, the third pixel can be a pixel in the impact response map that is at the same position as the second pixel.

[0084] S145, obtain the total response value of all pixels in the first region, wherein the first region includes the first pixel. For example, a geometric region around the first pixel, such as a 3×3 rectangular region, can be selected as the first region. Specifically, step S145 can add the response values ​​of all pixels in the first region to obtain the total response value of all pixels in the first region.

[0085] S146, obtain the total response value of all pixels in the second region, wherein the second region includes the third pixel, and the shape, size, and position of the second region are the same as those of the first region. Specifically, step S146 can add the response values ​​of all pixels in the second region to obtain the total response value of all pixels in the second region.

[0086] S147, a reference region is selected from the first region and the second region based on the total response value of all pixels in the first region and the total response value of all pixels in the second region. For example, the region with the largest total response value between the first region and the second region can be selected as the reference region.

[0087] S148, the first offset is obtained based on the centroid coordinates of the reference region and the center coordinates of the impact response map. Specifically, if the reference region is the first region, the distance between the centroid of the first region and the center point of the impact response map is obtained as the first offset; if the reference region is the second region, the distance between the centroid of the second region and the center point of the filter map is obtained as the first offset.

[0088] As can be seen from the above description, steps S141 to S148 provide a method for obtaining the first offset, and the first offset obtained in this way has high accuracy.

[0089] Optionally, the actual image may develop moiré patterns after scaling, which could affect the accuracy of the first offset. For details on this issue, please refer to [link to relevant documentation]. Figure 2C The image processing method described in this embodiment further includes:

[0090] S21, a matching response value is obtained based on the total response value of all pixels within the reference area, and the width and height of the impact response map. The matching response value represents the degree of matching between the first frame image and the second frame image; a larger value indicates a higher degree of matching. In this embodiment, the matching response value is preferably... Wherein, sum_max is the total response value of all pixels in the reference area, and imgR_w and imgR_h are the width and height of the impact response map, respectively.

[0091] S22 determines whether the first frame image and the second frame image can be stitched together based on the matching response value and the first offset. Specifically, if the matching response value is greater than a response threshold and the first offset is greater than an offset threshold, then the first frame image and the second frame image can be stitched together; otherwise, the matching result error between the first frame image and the second frame image is too large, and they cannot be stitched together. The response threshold and the offset threshold can be set according to actual needs or experience.

[0092] S23, if the first frame image and the second frame image can be stitched together, then the first frame image and the second frame image are stitched together according to the second offset and the first offset.

[0093] S24, if the first frame image and the second frame image cannot be stitched together, obtain the next frame of the second frame image as a new second frame image, and stitch the first frame image with the new second frame image. At this time, it is necessary to... Figure 1A The steps S12 to S14 shown are used to obtain a new cross-power spectrum, a new impulse response map, and a new first offset, and the first frame image and the new second frame image are stitched together based on the new first offset.

[0094] As described above, this embodiment can determine whether to reselect a new second frame image based on the matching response value. This method can avoid stitching together mismatched frame images, thereby eliminating the interference of error factors such as moiré patterns, which is beneficial for obtaining a more accurate first offset.

[0095] Please see Figure 3A In one embodiment of the present invention, the offset direction is from right to left, and before acquiring the cross-power spectrum of the first frame image and the second frame image, the image processing method further includes:

[0096] S31, the left, top, and bottom sides of the first frame image and the second frame image are expanded.

[0097] S32, window the expanded first frame image and the second frame image, for example, add a Hamming window to the first frame image and the second frame image.

[0098] In this embodiment, step S31 expands the first frame image and the second frame image so that the windowing operation in step S32 does not affect the substantive content of the first frame image and the second frame image. Furthermore, step S32, by windowing the first frame image and the second frame image, can prevent spectral leakage, thus enabling accurate acquisition of the cross-power spectrum of the first frame image and the second frame image.

[0099] It should be noted that when the offset direction is from left to right, step S31 can be modified to expand the right side, top and bottom of the first frame image and the second frame image.

[0100] Optionally, the method for expanding the left side of the first frame image in this embodiment includes: expanding the left side of the first frame image based on at least one column of pixels on the left side of the first frame image.

[0101] Specifically, the method for expanding the left side of the first frame image based on at least one column of pixels from the left side of the first frame image includes: copying at least one column of pixels from the left side of the first frame image at least once and then adding it to the left side of the first frame image. For example, please refer to... Figure 3B and Figure 3C ,in, Figure 3B An example image shown is of the first frame, containing pixels 11 to 89. Figure 3C This is an example image obtained by expanding the left side of the first frame image. Specifically, it shows... Figure 3B The first column of pixels is copied twice and added to the left side of the first frame image to complete the expansion of the left side of the first frame image. It should be noted that the above is only one example, but the invention is not limited thereto. For example, two or more columns of pixels from the left side of the first frame image can be copied and added to the left side of the first frame image. Furthermore, the number of times at least one column of pixels on the left side of the first frame image is copied can be set according to actual needs. Preferably, the width of the expanded area on the left side of the first frame image is 1 / 4 of the width of the first frame image.

[0102] Furthermore, the second frame image can also be implemented using a similar method as described above. Preferably, in order to improve the regional consistency and matching degree between the second frame image and the first frame image, in this embodiment, the left side of the second frame image can be expanded based on at least one column of pixels on the left side of the first frame image.

[0103] Specifically, one method for extending the left side of the second frame image based on at least one column of pixels from the left side of the first frame image includes: copying at least one column of pixels from the left side of the first frame image at least once and then adding it to the left side of the second frame image. For example, please refer to... Figure 3D and Figure 3E ,in, Figure 3D An example image shown is of the second frame, containing pixels 14 to 8C. Figure 3E The image shown is an example of an image obtained by expanding the left side of the second frame image. Specifically, the first column of pixels in the first frame image is copied twice and then added to the left side of the second frame image. It should be noted that this is only one example, but the invention is not limited thereto. For example, two or more columns of pixels from the left side of the first frame image can be copied and added to the left side of the second frame image. Furthermore, the number of times at least one column of pixels from the left side of the first frame image is copied can be set according to actual needs. Preferably, the width of the expanded area on the left side of the second frame image is 1 / 4 of the width of the second frame image.

[0104] In addition, another implementation method for extending the left side of the second frame image based on at least one column of pixels on the left side of the first frame image includes copying the extended left area of ​​the first frame image to the left side of the second frame image.

[0105] Optionally, in this embodiment, the lower part of the first frame image is expanded based on the upper multi-row pixels in the first frame image, and the upper part of the first frame image is expanded based on the lower multi-row pixels in the first frame image. For example, please refer to... Figure 3F The bottom two rows of pixels in the first frame image can be copied to the top of the first frame image to expand the upper part of the first frame image, and the top two rows of pixels in the first frame image can be copied to the bottom of the first frame image to expand the lower part of the first frame image. It should be noted that the above is only one example, but the invention is not limited thereto. In practical applications, the top two or more rows of pixels in the first frame image can be copied to the bottom of the first frame image, or the bottom two or more rows of pixels in the first frame image can be copied to the top of the first frame image, depending on actual needs. Preferably, the height of the upper expansion area and / or the height of the lower expansion area of ​​the first frame image is 1 / 4 of the height of the first frame image.

[0106] Furthermore, in this embodiment, the lower part of the second frame image is expanded based on the upper multi-row pixels in the second frame image, and the upper part of the second frame image is expanded based on the lower multi-row pixels in the second frame image. The specific method is similar to that of the first frame image, and will not be described in detail here.

[0107] An example image obtained by expanding the left, top, and bottom of the first frame image in this embodiment is shown below. Figure 3G As shown, windowing after expansion does not result in the loss of edge region information. Furthermore, this embodiment can expand the left side of the second frame image based on at least one column of pixels from the left side of the first frame image. This method can improve the matching degree between the expanded first and second frame images, thereby increasing the matching speed and stitching speed. Moreover, this method can also improve the matching score and stitching accuracy.

[0108] Please see Figure 4 In one embodiment of the present invention, after stitching the first frame image and the second frame image to obtain the stitched image, the image processing method further includes:

[0109] S41, the second frame image is used as the new first frame image.

[0110] S42, acquire a new second frame image, wherein the new second frame image refers to the next frame image after the new first frame image.

[0111] S43, obtain the new first offset, specifically, through... Figure 1A The new first offset is obtained using a method similar to steps S12 to S14 shown.

[0112] S45, the new second frame image is stitched together with the stitched image according to the new first offset to obtain a new stitched image. Specifically, a sub-image can be extracted from the side of the new second frame image away from the offset direction and stitched onto the side of the current stitched image away from the offset direction to obtain a new stitched image, wherein the width of the sub-image is the actual offset between the new first frame image and the new second frame image.

[0113] It is understandable that after step S45, the above steps S41 to S45 can be executed repeatedly. In each iteration, the second frame image obtained in step S42 of the previous iteration is used as the new first frame image in step S41 of the current iteration. In this way, the frame images obtained by the scanning device are stitched together one by one to the stitched image, thereby obtaining the final stitched image.

[0114] In practical applications, after stitching frame images, a long stitched image will be obtained. For example, Figure 5A as shown, it is then necessary to segment the stitched image to obtain multiple image blocks and send them to other modules, such as an optical character recognition module, etc. In related technologies, when segmenting the stitched image, the situation often occurs that a single character is segmented into two parts. For example, Figure 5B the character "筑" in Figure 5C . To address this problem, refer to

[0115] S51. Obtain the connected regions in the stitched image. Here, the stitched image can be obtained by stitching the first frame image and the second frame image, or can be obtained by stitching the first frame image, the second frame image, and one or more subsequent frame images.

[0116] Optionally, the implementation method of obtaining the connected regions in the stitched image may include: obtaining the gradient map of the stitched image and performing binary processing on the gradient map. Then, using the relevant algorithm for connected regions to extract the contours of each independent block in the white region of the gradient map can obtain the connected regions in the stitched image. Preferably, after extracting the white contours, independent blocks with smaller contours can also be deleted to reduce the influence of noise.

[0117] S52. Obtain the bounding rectangles of each of the connected regions. Specifically, the relevant algorithm for bounding rectangles can be used to obtain the bounding rectangles of each of the connected regions. For example, refer to Figure 5D which shows an example diagram of the bounding rectangles obtained in this embodiment. Preferably, the row where the bounding rectangle is located can be determined according to the vertical position of the bounding rectangle or the degree of overlap of adjacent rectangles in the vertical direction. In addition, different colored bounding rectangles can be used to represent different rows. Ideally, each of the bounding rectangles contains a complete character or punctuation mark.

[0118] S53. Segment the stitched image according to the bounding rectangles to obtain at least one image block, where each image block contains an integer number of the bounding rectangles. For example, refer to Figure 5E which shows an example diagram of an image block obtained in this embodiment.

[0119] According to the above description, it can be known that this embodiment can ensure that the image blocks obtained after segmenting the stitched image contain an integer number of bounding rectangles, thus being able to ensure as much as possible that characters are not segmented into two parts in the horizontal direction, which is beneficial to the processing of the image blocks by other modules.

[0120] Optionally, refer to Figure 5F The method for segmenting the stitched image based on the circumscribed rectangle includes:

[0121] S531, obtain the target rectangle based on the vertical position of the circumscribed rectangle, wherein the target rectangle refers to the circumscribed rectangle corresponding to the target row. Specifically, the target row can be specified by the user, or by default, it can be the row containing complete text in the spliced ​​image, for example... Figure 5B The first row is shown. The target rectangle can be obtained based on the relationship between the vertical position of the circumscribed rectangle and the vertical position of the target row.

[0122] S532, select an integer number of target rectangles on one side of the stitched image according to a preset width, and segment the stitched image according to the selection result to obtain one image block and a new stitched image. For example, it can be done in the opposite direction of the offset direction (e.g. Figure 5D From left to right, the distance between each target rectangle and the leftmost end of the stitched image is obtained sequentially. If the distance between a target rectangle and the leftmost end of the stitched image is greater than the preset width, then any vertical line between the right side of the previous target rectangle and the left side of the target rectangle is selected as the dividing line to divide the stitched image to obtain one image block and a new stitched image.

[0123] It is understandable that the above step S532 can be executed cyclically. Specifically, in each cycle, the new stitched image obtained in the previous cycle is used as the stitched image for the current cycle. An integer number of the target rectangles are selected on one side of the stitched image for the current cycle according to the preset width. The stitched image for the current cycle is then divided according to the selection result to obtain an image block and a new stitched image.

[0124] Please see Figure 6A In one embodiment of the present invention, the method for obtaining the cross-power spectrum of the first frame image and the second frame image includes:

[0125] S61, obtain a second offset, wherein the second offset is an estimated value used to roughly describe the actual offset between the first frame image and the second frame image. At this time, the first offset is used to correct the second offset to obtain the total offset between the first frame image and the second frame image, which is the actual offset between the first frame image and the second frame image. In this embodiment, the first offset can be positive or negative: when the first offset is positive, it indicates that the second offset is less than the actual offset, and the second offset needs to be corrected in the positive direction; when the first offset is negative, it indicates that the second offset is greater than the actual offset, and the first offset needs to be corrected in the negative direction.

[0126] S62, acquire a sub-image of the first frame image, wherein the sub-image is located on the side of the first frame image away from the offset direction, and the width of the sub-image is determined by the difference between the width of the first frame image and the second offset. Preferably, the width of the sub-image is the width of the first frame image minus the second offset. For example, please refer to... Figure 6B If the second offset is 1.5L, then a sub-image 111 with a width of 3.5L is selected on the right side of the first frame image 11.

[0127] S63, obtain the cross-power spectrum of the sub-image and the second frame image as the cross-power spectrum of the first frame image and the second frame image.

[0128] Based on the cross-power spectrum obtained in step S63 above, the image processing method can obtain a first offset. At this time, the sum of the first offset and the second offset is the actual offset between the first frame image and the second frame image.

[0129] As described above, the image processing method in this embodiment only needs to match the second frame image of the sub-image, without needing to perform full image matching of the first frame image and the second frame image. Compared with related technologies, the image processing method in this embodiment can reduce the amount of computation and improve the image stitching speed. Furthermore, the image processing method in this embodiment can make the matching in image stitching more accurate, thereby obtaining a more accurate stitched image.

[0130] In particular, when the scanning device moves at a high speed, the non-overlapping area between the first and second frame images is large, which can lead to unstable stitching. The image processing method described in this embodiment has a high stitching speed, so the image processor can stitch each frame image in a timely manner, which helps to improve the accuracy of stitching and achieve stable stitching.

[0131] In one embodiment of the present invention, the second offset is obtained based on the first frame image and its previous frame image. The specific implementation method includes: if the first frame image does not have a previous frame image, then the second offset is 0; if the first frame image has a previous frame image, then the second offset is the offset between the first frame image and its previous frame image.

[0132] Please see Figure 7A In one embodiment of the present invention, the method for obtaining the second offset includes:

[0133] S611, determine whether the first frame image and the second frame image are straight line images. For example, when scanning paper documents using a scanning device, it often happens that one or more frames contain only horizontal or near-horizontal straight lines; such frame images are straight line images. Figure 7B As shown. It should be noted that in practical applications, the obtained straight line may be a horizontal straight line, an inclined straight line, or a vertical straight line, and the straight line may have inclination, horizontal displacement, or vertical displacement. Since the image processing method mainly focuses on the horizontal direction when obtaining the offset, this embodiment needs to determine whether the frame image contains only horizontal or approximately horizontal long straight lines through step S611, that is, to determine whether the frame image is a straight line image.

[0134] S612, obtain the second offset based on the judgment result. Specifically, when the first frame image and / or the second frame image are straight line images, the straight line portions in the first frame image and the second frame image can be considered to overlap. Therefore, the offset between the two can be obtained as the second offset based on the distribution of the straight line regions in the first frame image and the second frame image. When neither the first frame image nor the second frame image are straight line images, the second offset can be obtained based on the first frame image and its previous frame image, or a preset value can be selected as the second offset.

[0135] Optionally, please refer to Figure 7C In this embodiment, the method for determining whether a frame image (the first frame image or the second frame image) is a straight line image includes:

[0136] S6111, Obtain the gradient value of each pixel in the frame image. The gradient value of a pixel is used to identify the difference in pixel value between the pixel and its surrounding or adjacent pixels. Preferably, step S6111 can obtain the gradient value of each pixel in the frame image.

[0137] Optionally, for a pixel at coordinates (x,y) in a frame image, its gradient value is G(x,y)=|dx(x,y)|+|dy(x,y)|, where dx(x,y)=I(x+1,y)-I(x,y) represents the gradient of the pixel at coordinates (x,y) in the horizontal direction, dy(x,y)=I(x,y+1)-I(x,y) represents the gradient of the pixel at coordinates (x,y) in the vertical direction, and I(x,y) is the pixel value of the pixel at coordinates (x,y).

[0138] S6112, obtain a binarized gradient image based on the gradient values ​​of pixels in the frame image. Specifically, based on the relationship between the gradient value of each pixel and a gradient threshold, set the pixel value of each pixel to black (0) or white (255), wherein the gradient threshold can be set according to actual needs or experience. For example, please refer to... Figure 7D and Figure 7E ,in, Figure 7D The image shown is an example of the gradient image obtained in this embodiment. Figure 7E The image shown is an example of the corresponding binarized gradient image.

[0139] S6113, obtain the covariance matrix based on the white pixels in the binarized gradient image. Specifically, obtain white regions based on the white pixels in the binarized gradient image, and calculate the covariance matrix based on the coordinate set of points within the white regions, wherein the covariance matrix is ​​used to reflect the relationship between different white pixels.

[0140] Optionally, the method for obtaining the covariance matrix C based on the white pixels in the binarized gradient image is C = [V0, V1, ..., V...]. N ]×[V0,V i ,...,V N ] T ,in, (X i ,Y i Let be the coordinates of the i-th white pixel. Let N be the average coordinates of each white pixel, and N be the number of white pixels in the binarized gradient image.

[0141] S6114, determine whether the frame image is a straight line image based on the eigenvalues ​​of the covariance matrix. Specifically, select a minimum eigenvalue t_min from the eigenvalues ​​of the covariance matrix, and determine whether the frame image is a straight line image based on the relationship between the minimum eigenvalue t_min and an eigenvalue threshold T_cov: if the minimum eigenvalue t_min is less than the eigenvalue threshold T_cov, then the frame image is a straight line image; otherwise, the frame image is not a straight line image. The eigenvalue threshold T_cov can be set according to actual needs or experience.

[0142] Optionally, in this embodiment, there are four combinations of whether the first frame image and the second frame image are straight line images. The second offset corresponding to each of these four combinations will be described in detail below.

[0143] If the first frame image is not a straight line image and the second frame image is a straight line image, then the starting point column of the overlapping area is obtained according to the vertical span of the white pixels in the first image, and the second offset is obtained according to the horizontal coordinate of the starting point column, wherein the first image is a binarized image of the first frame image.

[0144] Specifically, the binarized image of the first frame image is obtained as the first image, for example... Figure 7F As shown, along the offset direction (e.g.) Figure 7F Starting from the first column of the first image (from right to left), the vertical span of each white pixel in each column is obtained sequentially. When the vertical span of a white pixel in a column is greater than a span threshold, that column is the starting column of the overlapping region. The horizontal coordinate xi_bias of the starting column is obtained as the second offset. The span threshold can be set according to actual needs or experience.

[0145] Preferably, the span threshold is: Where max_span_y is the maximum span of white pixels in the vertical direction in the binarized gradient image corresponding to the first frame image. For example, the span of each column of white pixels in the vertical direction in the binarized gradient image corresponding to the first frame image can be obtained and a maximum value can be selected as max_span_y. T_cov is the feature value threshold and t_min is the minimum feature value.

[0146] If the first frame image is a straight line image and the second frame image is not a straight line image, then the endpoint of the overlapping region is obtained according to the vertical span of the white pixels in the second image, and the second offset is obtained according to the horizontal coordinate of the endpoint and the width of the second frame image, wherein the second image is a binarized image of the second frame image.

[0147] Specifically, the binarized image of the second frame image is obtained as the second image, for example... Figure 7G As shown, in the opposite direction of the offset direction (e.g.) Figure 7G Starting from the first column of the second image (from left to right), the vertical span of each white pixel in each column is obtained. When the vertical span of white pixels in a column is greater than the span threshold corresponding to the first frame image, that column is the endpoint column of the overlapping region. The horizontal coordinate xi_bias of the endpoint column is obtained. The second offset is obtained based on the width img_w and xi_bias of the second frame image. For example, the second offset can be img_w minus xi_bias. The span threshold corresponding to the first frame image can be calculated using the formula for T_line mentioned above, or it can be an empirical value or a preset value.

[0148] If both the first frame image and the second frame image are straight line images, then the second offset is obtained based on the white region contours in the first image and the white region contours in the second image. For details, please refer to... Figure 7H One method for obtaining the second offset at this point includes:

[0149] S6121, obtain the outline of the white region in the first image, and obtain the maximum value max_x' and minimum value min_x' of the coordinates of the first white region outline in the horizontal direction along the opposite direction of the offset direction. For example, when the offset direction is from right to left, step S6121 obtains the maximum value and minimum value of the coordinates of the first white region outline on the left side of the first image in the horizontal direction as max_x' and min_x', respectively.

[0150] S6122, obtain the outline of the white region in the second image, and obtain the maximum value max_x and minimum value min_x of the coordinates of the first white region outline in the horizontal direction along the opposite direction of the offset direction. For example, when the offset direction is from right to left, step S6122 obtains the maximum value and minimum value of the coordinates of the first white region outline on the left side of the second image in the horizontal direction as max_x and min_x, respectively.

[0151] S6123, obtain the second offset based on the width img_w of the second frame image and the two maximum and two minimum values ​​mentioned above. Specifically, determine whether the first image and the second image satisfy the following conditions: If the conditions are met, the second offset is obtained based on max_x and max_x'. For example, the second offset can be max_x' - max_x. If the conditions are not met, the second offset is obtained in another way, such as obtaining the second offset based on the first frame image and its previous frame image, or selecting a preset value as the second offset. Here, α and β are two weight values, and their values ​​can be set according to actual needs. For example, α can be 0.75 and β can be 0.5.

[0152] If neither the first frame image nor the second frame image is a straight line image, the second offset can be obtained based on the first frame image and its previous frame image, or a preset value can be selected as the second offset.

[0153] As can be seen from the above description, this embodiment provides a method for obtaining the second offset. When the first frame image and / or the second frame image are straight line images, the second offset can be obtained based on the distribution of the straight line (or line segment) region in the first frame image. The second offset obtained in this way is closer to the true offset and therefore has higher accuracy.

[0154] Please see Figure 8 In one embodiment of the present invention, the scanning device moves from left to right, and the image processing method includes:

[0155] S801, acquire a first frame image and a second frame image, wherein the first frame image and the second frame image are, for example, images of two adjacent frames. Optionally, after acquiring the first frame image and the second frame image, the image processing method further includes: performing perspective transformation pre-correction on the first frame image and the second frame image to make the text size in the first frame image and the second frame image more balanced.

[0156] S802, determine whether the first frame image and the second frame image are straight line images, and obtain the second offset based on the determination result. Specifically, this can be achieved by... Figure 7C The steps S6111 to S6114 shown are used to determine whether the first frame image and the second frame image are straight line images. When the first frame image and / or the second frame image are straight line images, the second offset is obtained according to the distribution of the straight line area. Otherwise, the offset between the first frame image and its previous frame image is used as the second offset. When the first frame image does not have a previous frame image, the second offset is 0.

[0157] S803, a sub-image is obtained on the right side of the first frame image, the width of the sub-image being the width of the first frame image minus the second offset.

[0158] S804, the left, top, and bottom edges of the sub-image and the second frame image are expanded, and the expanded sub-image and the second frame image are windowed. Step S804 can be performed using... Figure 3A The extension is implemented in the manner shown in step S31.

[0159] S805, obtain the cross-power spectrum of the sub-image and the second frame image as the cross-power spectrum of the first frame image and the second frame image.

[0160] S806, obtain the corresponding impact response diagram based on the cross power spectrum, and then obtain the second offset.

[0161] S807, obtain the actual offset between the first frame image and the second frame image according to the first offset and the second offset, and stitch the first frame image and the second frame image according to the actual offset to obtain a stitched image.

[0162] S808, the second frame image is used as the new first frame image, and the next frame of the new first frame image is obtained as the new second frame image.

[0163] S809, obtain a new second offset and a new first offset, wherein the method for obtaining the new second offset is the same as in step S802, and the method for obtaining the new first offset is the same as in steps S803 to S706.

[0164] S810, the stitched image and the new second frame image are stitched together according to the new second offset and the new first offset to obtain a new stitched image.

[0165] S811, repeat steps S807 to S810. In each iteration, step S807 obtains the new second frame image from the previous iteration as the new first frame image for the current iteration.

[0166] S812, obtain the connected regions in the stitched image, and obtain the bounding rectangle of each connected region.

[0167] S813, the stitched image is segmented according to the circumscribed rectangle to obtain one or more image blocks.

[0168] It should be noted that steps S812 to S813 can be performed after the complete stitched image is acquired, or they can be performed simultaneously with steps S807 to S811. When performed simultaneously, steps S807 to S811 continuously stitch new image frames into the stitched image, while steps S812 to S813 continuously divide the stitched image into one or more image blocks and send them to other modules, thereby achieving synchronization of scanning and processing.

[0169] Based on the above description of the image processing method, the present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement... Figure 1A or Figure 8 The image processing method shown.

[0170] Based on the above description of the image processing method, the present invention also provides an electronic device. Specifically, please refer to... Figure 9 The electronic device 900 includes a memory 910 and a processor 920. The memory 910 stores a computer program, and the processor 920 is communicatively connected to the memory 910 for executing the computer program. Figure 1A or Figure 8 The image processing method shown.

[0171] Optionally, the electronic device 900 further includes a display 930, which is communicatively connected to the memory 910 and the processor 920, and is used to display the relevant GUI interactive interface of the image processing method.

[0172] Optionally, the electronic device is a scanning pen, which includes an imaging sensor and an image processor, wherein the imaging sensor is used to acquire frame images, and the image processor is used to employ... Figure 1A or Figure 8 The image processing method shown stitches together frame images.

[0173] Optionally, the electronic device is a scanning pen, which includes a light source emitter, an optical imaging sensor, and an image processor. The light source emitter and the optical imaging sensor are used to acquire frame images optically, and the image processor is used to... Figure 1A or Figure 8 The image processing method shown stitches together frame images.

[0174] Optionally, the scanning pen also includes a transparent dust cover.

[0175] The scope of protection of the image processing method described in this invention is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this invention is included within the scope of protection of this invention.

[0176] The image processing method provided in one or more embodiments of the present invention can obtain a corresponding impulse response map based on the cross-power spectrum of a first frame image and a second frame image, and obtain a first offset based on the response values ​​of pixels in the impulse response map. Based on the first offset, the image processing method can stitch the first frame image and the second frame image together to obtain a stitched image.

[0177] Furthermore, the image processing method can achieve product-level stitching results by estimating inter-frame offsets, expanding frame images, improving the calculation method for offsets between two frames, and filtering inter-frame offset results. Under normal usage conditions, it is virtually error-free. Simultaneously, the image processing method significantly reduces stitching time, reducing it to a few milliseconds on a single Cortex-A7 processor, ensuring real-time stitching. Moreover, the image processing method can significantly improve the matching accuracy of the image stitching process while reducing the amount of matching computation.

[0178] In summary, this invention effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0179] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An image processing method, characterized in that, The image processing method includes: Acquire the first frame image and the second frame image; Obtain the cross-power spectrum of the first frame image and the second frame image; The corresponding impact response diagram is obtained based on the cross power spectrum. The first offset between the first frame image and the second frame image is obtained based on the response values ​​of the pixels in the impact response image; The actual offset between the first frame image and the second frame image is obtained based on the first offset. The first frame image and the second frame image are stitched together according to the actual offset. The first offset between the first frame image and the second frame image is obtained based on the response values ​​of pixels in the impact response map, including: The first pixel is obtained from the impact response map, where the first pixel is the pixel with the largest response value in the impact response map. The impact response diagram is filtered to obtain a filtered diagram; The second pixel is obtained from the filtered image, where the second pixel refers to the pixel with the largest response value in the filtered image. A third pixel is obtained from the impact response map, where the third pixel refers to the pixel in the impact response map that corresponds to the second pixel. Obtain the total response value of all pixels within a first region, where the first region includes the first pixel; Obtain the total response value of all pixels in the second region, which includes the third pixel, and the shape and size of the second region are the same as those of the first region; A reference region is selected from the first region and the second region based on the total response value of all pixels in the first region and the total response value of all pixels in the second region; The first offset is obtained based on the centroid coordinates of the reference area and the center coordinates of the impact response diagram.

2. The image processing method according to claim 1, characterized in that, The method for stitching the first frame image and the second frame image according to the first offset includes: The matching response value is obtained based on the total response value of all pixels in the reference area, and the width and height of the impact response map; Based on the matching response value and the first offset, it is determined whether the first frame image and the second frame image can be stitched together; If the first frame image and the second frame image can be stitched together, then the first frame image and the second frame image are stitched together according to the first offset. If the first frame image and the second frame image cannot be stitched together, the next frame of the second frame image is obtained as a new second frame image, and the first frame image is stitched together with the new second frame image.

3. The image processing method according to claim 1, characterized in that, Before acquiring the cross-power spectrum of the first frame image and the second frame image, the image processing method further includes: Expand the left, top, and bottom of the first frame image and the second frame image; Windowing is applied to the expanded first and second frame images.

4. The image processing method according to claim 3, characterized in that: The method for expanding the left side of the first frame image and the second frame image includes: expanding the left side of the first frame image and the left side of the second frame image based on at least one column of pixels on the left side of the first frame image; and / or The methods for expanding the top and bottom of the first frame image and the second frame image include: The lower part of the first frame image is expanded based on the upper multi-row pixels in the first frame image; The upper part of the first frame image is expanded based on the lower multi-row pixels in the first frame image; The lower part of the second frame image is expanded based on the upper multi-row pixels in the second frame image; The upper part of the second frame image is expanded based on the lower multi-row pixels in the second frame image.

5. The image processing method according to claim 1, characterized in that, After stitching the first frame image and the second frame image to obtain the stitched image, the image processing method further includes: Use the second frame image as the new first frame image; Acquire a new second frame image, wherein the new second frame image refers to the frame image following the new first frame image; Get the new first offset; The new second frame image is stitched together with the stitched image according to the new first offset to obtain a new stitched image.

6. The image processing method according to claim 1, characterized in that, After stitching the first frame image and the second frame image together, the image processing method further includes: Obtain connected regions in the stitched image, wherein the stitched image is obtained by stitching together at least the first frame image and the second frame image; Obtain the bounding rectangle of each of the connected regions; The stitched image is segmented according to the circumscribed rectangle to obtain at least one image block, wherein each image block contains an integer number of the circumscribed rectangles.

7. The image processing method according to claim 1, characterized in that, The method for obtaining the cross-power spectrum of the first frame image and the second frame image includes: Get the second offset; Obtain a sub-image of the first frame image, wherein the sub-image is located on the side of the first frame image away from the offset direction, and the width of the sub-image is determined by the difference between the width of the first frame image and the second offset. The cross-power spectrum of the sub-image and the second frame image is obtained as the cross-power spectrum of the first frame image and the second frame image.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the image processing method according to any one of claims 1 to 7.

9. An electronic device, characterized in that, The electronic device includes: A memory that stores a computer program; The processor, which is communicatively connected to the memory, executes the image processing method according to any one of claims 1 to 7 when the computer program is invoked.

Images