Image splicing method and electronic equipment

Abstract

One or more embodiments of the invention provide an image splicing method and an electronic device. The image splicing method based on the scanning pen comprises the following steps: acquiring a first scanning image and a second scanning image in sequence scanning images acquired by the scanning pen; if it is detected that the scanning pen deviates from the preset scanning direction in the process of collecting the sequence scanning images, registration processing in the first direction and the second direction is carried out on the first scanning image and the second scanning image, and a registration result is obtained; if it is detected that the scanning pen collects the sequence scanning images according to the scanning direction, registration processing in the first direction is carried out on the first scanning image and the second scanning image, and a registration result is obtained; and splicing the first scanning image and the second scanning image according to the registration result to obtain a spliced image. And character dislocation in the spliced image is avoided or reduced.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of image processing technology, and in particular, to an image stitching method and electronic device. Background Art

[0002] In the related art, image stitching technology is a technology that stitches together several images with overlapping parts (which may be obtained at different times, different perspectives or different sensors) into a seamless stitched image. Image stitching technology is applied to a variety of image processing scenarios. For example, in the scanning pen scenario, some scanning pens have a scanning and word recognition function. When the pen tip is pressed, the camera starts taking pictures and sampling, and the shooting ends after the pen is lifted. In this process, the multiple photos collected are stitched together, and finally the text recognition of long sentences is performed. However, the image stitching solution in the related art has a misalignment problem when performing image stitching; for example, in the scanning pen scenario, the misalignment of characters in the stitched image greatly affects the text recognition rate. Summary of the invention

[0003] In view of this, one or more embodiments of the present specification provide an image stitching method, an electronic device, and a computer-readable storage medium.

[0004] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, a scanning pen-based image stitching method is proposed, comprising:

[0006] Acquire a first scan image and a second scan image in a sequence of scan images captured by a scanning pen; wherein the first scan image and the second scan image have overlapping image contents;

[0007] If it is detected that the scanning pen deviates from the preset scanning direction during the process of acquiring the sequence scanned images, the first scanned image and the second scanned image are registered in the first direction and the second direction to obtain a registration result;

[0008] If it is detected that the scanning pen collects a sequence of scanned images in the scanning direction, a registration process is performed on the first scanned image and the second scanned image in the first direction to obtain a registration result;

[0009] The first scanned image and the second scanned image are spliced ​​according to the registration result to obtain a spliced ​​image.

[0010] Beneficial effect: This embodiment implements registration processing in two directions when the scanning pen shakes or tilts. For example, the first direction is a direction parallel to the scanning direction, and the second direction is a direction perpendicular to the scanning direction. The image alignment deviation can be corrected so that the two scanned images are aligned in two directions, avoiding or reducing the problem of character misalignment in the spliced ​​image; and when the scanning pen does not shake or tilt, only registration processing in one direction is required, which is beneficial to reduce the amount of registration calculation and improve the registration efficiency.

[0011] According to a second aspect of one or more embodiments of this specification, a scanning pen-based image stitching method is proposed, comprising:

[0012] Acquire a first scan image and a second scan image in a sequence of scan images captured by a scanning pen; wherein the first scan image and the second scan image have overlapping image contents;

[0013] Performing registration processing on the first scanned image and the second scanned image in a first direction and a second direction to obtain a registration result;

[0014] The first scanned image and the second scanned image are spliced ​​according to the registration result to obtain a spliced ​​image.

[0015] Beneficial effect: This embodiment does not need to detect whether the scanning pen is shaking or tilted, and handles it according to the situation where the scanning pen is shaking or tilted. The image alignment deviation can be corrected so that the two scanned images are aligned in two directions, avoiding or reducing the problem of character misalignment in the spliced ​​image.

[0016] According to a third aspect of one or more embodiments of this specification, an image stitching method is proposed, including:

[0017] Acquire a first image and a second image in a sequence of images captured by a photographing device; wherein the first image and the second image have overlapping image content;

[0018] If it is detected that the shooting device deviates from the preset shooting direction during the process of capturing the sequence of images, the first image and the second image are registered in the first direction and the second direction to obtain a registration result;

[0019] If it is detected that the shooting device acquires a sequence of images according to the shooting direction, performing a registration process in a first direction on the first image and the second image to obtain a registration result;

[0020] The first image and the second image are spliced ​​according to the registration result to obtain a spliced ​​image.

[0021] Beneficial effect: This embodiment realizes registration processing in two directions when the shooting device is shaken or tilted. For example, the first direction is a direction parallel to the shooting direction, and the second direction is a direction perpendicular to the shooting direction. The image alignment deviation can be corrected so that the two images are aligned in two directions, avoiding or reducing the misalignment problem in the stitched image; and when the shooting device is not shaken or tilted, only registration processing in one direction needs to be performed, which is beneficial to reducing the amount of registration calculation and improving the registration efficiency.

[0022] According to a fourth aspect of one or more embodiments of this specification, an image stitching method is proposed, including:

[0023] Acquire a first image and a second image in the acquired sequence of images; wherein the first image and the second image have overlapping image content;

[0024] Performing registration processing on the first image and the second image in a first direction and a second direction to obtain a registration result; wherein;

[0025] The first image and the second image are spliced ​​according to the registration result to obtain a spliced ​​image.

[0026] Beneficial effect: This embodiment does not need to detect whether the shooting device is shaking or tilted, and processes it according to the situation where the shooting device is shaking or tilted. The image alignment deviation can be corrected so that the two images are aligned in two directions, avoiding or reducing the problem of character misalignment in the stitched image.

[0027] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0028] processor;

[0029] a memory for storing processor-executable instructions;

[0030] When the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0031] According to a sixth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above methods are implemented.

[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of character misalignment in a spliced ​​image in the related art.

[0034] Figure 2 It is a flowchart of an image stitching method based on a scanning pen provided by an exemplary embodiment.

[0035] Figure 3 is a schematic diagram of a first scan image and a second scan image provided by an exemplary embodiment.

[0036] Figure 4 is another schematic diagram of a first scan image and a second scan image provided by an exemplary embodiment.

[0037] Figure 5 It is a schematic diagram of a stitched image provided by an exemplary embodiment.

[0038] Figure 6 It is a flowchart of another image stitching method based on a scanning pen provided by an exemplary embodiment.

[0039] Figure 7 It is a flowchart of an image stitching method provided by an exemplary embodiment.

[0040] Figure 8 It is a flowchart of another image stitching method provided by an exemplary embodiment.

[0041] Fig. 9 It is a structural schematic diagram of a device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0042] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0043] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0044] Image stitching technology is applied to a variety of image processing scenarios. For example, in the scanning pen scenario, some scanning pens have the function of scanning and recognizing words. When the pen tip is pressed, the camera starts taking pictures and sampling, and the picture taking ends after the pen is lifted. In this process, the multiple photos collected are stitched together, and finally the text recognition of long sentences is performed. For another example, in the panoramic photography scenario, when there is no wide-angle camera to capture panoramic images, a single camera can be used to rotate and shoot scenes in different directions, and the panoramic picture that can be seen at the current position can be obtained by image stitching.

[0045] However, the image stitching solutions in the related art do not take into account the situation that the user shakes or tilts when shooting, which leads to the problem of misalignment of the stitched images. For example, Figure 1 The figure shows the character misalignment in the stitched image obtained in the scanning pen scene. The character misalignment in the stitched image greatly affects the text recognition rate. For example, the misalignment phenomenon in the stitching position in the panoramic image affects the visual effect.

[0046] In response to the problems in the related art, an embodiment of the present specification provides an image stitching method based on a scanning pen. During the image stitching process, if it is detected that the scanning pen deviates from a preset scanning direction during the process of acquiring sequential scanning images, indicating that the scanning pen is shaking or tilted, then the first scanning image and the second scanning image to be stitched are aligned in the first direction and the second direction to obtain an alignment result; if it is detected that the scanning pen acquires sequential scanning images according to the scanning direction, indicating that the scanning pen is not shaking or tilted, then the first scanning image and the second scanning image to be stitched are aligned in the first direction to obtain an alignment result; finally, the first scanning image and the second scanning image are stitched according to the alignment result to obtain a stitched image. This embodiment implements registration processing in two directions when the scanning pen shakes or tilts. For example, the first direction is a direction parallel to the scanning direction, and the second direction is a direction perpendicular to the scanning direction. The image alignment deviation can be corrected so that the two scanned images are aligned in two directions (such as horizontal and vertical directions), avoiding or reducing the problem of character misalignment in the spliced ​​image; and when the scanning pen does not shake or tilt, only registration processing in one direction is required, which is beneficial to reduce the amount of registration calculation and improve the registration efficiency.

[0047] In response to the problems in the related art, the embodiments of this specification provide another image stitching method based on a scanning pen. During the image stitching process, the first scanned image and the second scanned image to be stitched are directly registered in the first direction and the second direction to obtain a registration result; for example, the first direction is a direction parallel to the preset scanning direction of the scanning pen, and the second direction is a direction perpendicular to the scanning direction; the first scanned image and the second scanned image are stitched according to the registration result to obtain a stitched image. This embodiment does not need to detect whether the scanning pen is shaking or tilted, and processes according to the situation where the scanning pen is shaking or tilted. The image alignment deviation can be corrected so that the two scanned images are aligned in two directions (such as horizontal and vertical directions), avoiding or reducing the problem of character misalignment in the stitched image.

[0048] Exemplarily, the scanning pen includes a camera, a processor and a memory, the memory stores executable instructions that can be run on the processor, the camera collects sequential scan images based on a sliding scanning operation, and the processor implements the image stitching method provided in the embodiment of this specification when executing the executable instructions, thereby providing a guarantee for subsequent text recognition.

[0049] like Figure 2 As stated, Figure 2 : is a flowchart of an image stitching method based on a scanning pen provided in an embodiment of this specification, the method comprising:

[0050] In S101, a first scan image and a second scan image in a sequence of scan images captured by a scanning pen are acquired; wherein the first scan image and the second scan image have overlapping image contents.

[0051] The scanning pen can collect sequential scan images based on the user's sliding scanning operation, and then the sequential scan images need to be spliced ​​in real time to improve the accuracy and efficiency of splicing, so as to provide guarantee for subsequent text recognition. The sequential scan images include at least two scan images with overlapping image content. Exemplarily, the first scan image and the second scan image to be spliced ​​are images with overlapping image content collected by the scanning pen at adjacent times.

[0052] It can be understood that this embodiment does not impose any restrictions on the capture time of the first scan image and the second scan image. For example, the scanning pen may capture the first scan image at time T and then capture the second scan image at time T+1; or the scanning pen may capture the second scan image at time T and then capture the first scan image at time T+1.

[0053] In S102, if it is detected that the scanning pen deviates from a preset scanning direction during the process of acquiring the sequence of scanned images, the first scanned image and the second scanned image are registered in the first direction and the second direction to obtain a registration result.

[0054] In S103, if it is detected that the scanning pen collects a sequence of scanned images according to the scanning direction, the first scanned image and the second scanned image are registered in the first direction to obtain a registration result.

[0055] Wherein, the first direction is a direction parallel to the scanning direction, and the second direction is a direction perpendicular to the scanning direction. Exemplarily, if the reading direction of the text is horizontal, the user can slide the scanning pen in the horizontal direction, and the preset scanning direction includes the horizontal direction, then the first direction is the horizontal direction, and the second direction is the vertical direction. Alternatively, if the reading direction of the text is vertical, the user can slide the scanning pen in the vertical direction, and the preset scanning direction may include the vertical direction, then the first direction is the vertical direction, and the second direction is the horizontal direction. Alternatively, if the reading direction of the text is an inclined direction that is neither horizontal nor vertical, and the user slides the scanning pen in the inclined direction, and the preset scanning direction may include the inclined direction, then the first direction is the inclined direction, and the second direction is a direction perpendicular to the inclined direction.

[0056] In S104, the first scanned image and the second scanned image are spliced ​​according to the registration result to obtain a spliced ​​image.

[0057] This embodiment implements registration processing in two directions when the scanning pen shakes or tilts. The first direction is a direction parallel to the scanning direction, and the second direction is a direction perpendicular to the scanning direction. The alignment deviation of the image can be corrected so that the two scanned images are aligned in the horizontal and vertical directions, thereby avoiding or reducing the problem of character misalignment in the spliced ​​image; and when the scanning pen does not shake or tilt, only registration processing in one direction is required, which is beneficial to reducing the registration workload and improving the registration efficiency.

[0058] In some embodiments, the scanning pen is equipped with an acceleration sensor. When the scanning pen scans the sequence scanning image, the acceleration data collected by the acceleration sensor can be used to determine the scanning trajectory of the scanning pen; then, according to the scanning trajectory of the scanning pen, it is determined whether the scanning pen deviates from the preset scanning direction. For example, the extension direction of the scanning trajectory can be compared with the preset scanning direction to determine whether the scanning pen deviates from the preset scanning direction, thereby realizing accurate detection of the scanning direction of the scanning pen.

[0059] For example, if the extension direction of the scanning track is exactly the same as the preset scanning direction or the angle difference between the two is within the allowable error range, it is determined that the scanning pen has not deviated from the preset scanning direction; otherwise, it is determined that the scanning pen has deviated from the preset scanning direction. The allowable error range can be specifically set according to the actual application scenario.

[0060] It is understandable that the preset scanning direction can be set before image stitching, for example, it can be set before leaving the factory, or it can be pre-input by the user based on actual scanning requirements. The user input can be the user manually inputting the scanning direction information or the scanning direction can be set with the help of an acceleration sensor. Here is an example of setting the scanning direction with the help of an acceleration sensor: the scanning pen can preset a scanning direction setting mode, and after entering the scanning direction setting mode, the acceleration data collected by the acceleration sensor can be used to determine the scanning trajectory of the scanning pen, and then the extension direction of the scanning trajectory is set as the scanning direction in the subsequent image stitching process.

[0061] In some embodiments, considering that the scanning pen will bring a large amount of registration calculation to the subsequent image registration process when the deviation angle is large, and it is not conducive to the subsequent text recognition process, therefore, if it is detected that the scanning pen deviates from the preset scanning direction during the process of acquiring the sequence scanned image, and the angle difference between the deviated direction and the scanning direction is greater than the preset angle, a reminder message can be output to remind the user to correct the direction, such as reminding the user to scan in the preset scanning direction. The scanning pen can be equipped with a speaker and / or a display screen, and the reminder message can be auditory information or visual information, and this embodiment does not impose any restrictions on this.

[0062] In some embodiments, in order to improve the efficiency of image registration, the first scanned image and the second scanned image may be preprocessed before the registration process. Image preprocessing is the preparatory work before image registration, which is used to eliminate irrelevant information that affects the accuracy of image registration, improve the efficiency of image registration, and reduce the difficulty of registration, including image denoising, geometric correction, image enhancement, image cropping, etc.

[0063] Next, the registration process is described as follows:

[0064] In some embodiments, when it is detected that the scanning pen deviates from a preset scanning direction during the process of acquiring a sequence of scanned images, the scanning pen can determine a template image from the first scanned image; then, the scanning pen can move the window in the second scanned image along a first direction and a second direction respectively to select multiple candidate images, where the first direction is a direction parallel to the scanning direction, and the second direction is a direction perpendicular to the scanning direction.

[0065] Exemplarily, assuming that the scanning direction is a horizontal direction, the first direction may be a horizontal direction, and the second direction may be a vertical direction. Exemplarily, assuming that the scanning direction is a vertical direction, the first direction may be a vertical direction, and the second direction may be a horizontal direction.

[0066] The size of the template image is smaller than or equal to the first scanned image, and the template image contains image content that overlaps with certain areas in the second scanned image. The size of the candidate image is smaller than or equal to the second scanned image. The size of the template image is equal to the size of the candidate image. Exemplarily, a window can be moved along a first direction and a second direction in a selected area in the second scanned image to select multiple candidate images. The selected area in the second scanned image is an area that overlaps with certain areas in the first scanned image.

[0067] The template image in the first scanned image and the selected area in the second scanned image can be determined based on the scanning direction. For example, see Figure 3 , assuming that the user operates the scanning pen to scan horizontally from left to right, first collects the first scan image, and then collects the second scan image, then the right area in the first scan image ( Figure 3 The gray part in the image) has an overlapped part with the second scan area. The left area of ​​the second scan image ( Figure 3 The gray part in the image) has a part that overlaps with the first scanned image. For another example, see Figure 4 , assuming that the user operates the scanning pen to scan from top to bottom in the vertical direction, first collects the second scan image, and then collects the first scan image, then the upper area ( Figure 4 The gray part in the image) has a part overlapping with the second scan area, and the lower area of ​​the second scan image ( Figure 4 The gray part in FIG. 11A and FIG. 12B ) has a portion overlapping with the first scanned image.

[0068] After determining the template image and the candidate image, the scanning pen can calculate the similarity between the template image and each of the candidate images; finally, the registration result is obtained according to the template image and the candidate image whose similarity is higher than the preset threshold. This embodiment implements registration processing in two directions when the scanning pen is shaken or tilted. The first direction is a direction parallel to the scanning direction, and the second direction is a direction perpendicular to the scanning direction. The alignment deviation of the image can be corrected so that the two scanned images are aligned in the horizontal direction and the vertical direction, thereby avoiding or reducing the problem of character misalignment in the spliced ​​image.

[0069] For example, see Figure 1 and Figure 5 , Figure 5The stitched image shown is obtained by stitching using the image stitching method of the embodiment of this specification. Figure 5 and Figure 1 By comparison, it can be seen that there is no misalignment of characters in the spliced ​​image obtained by the image splicing method according to the embodiment of this specification.

[0070] In other embodiments, when it is detected that the scanning pen captures a sequence of scanned images in a predicted scanning direction, the scanning pen can determine a template image from the first scanned image; then, the scanning pen can move the window in the second scanned image along a first direction to select multiple candidate images, where the first direction is a direction parallel to the scanning direction. After determining the template image and the candidate images, the scanning pen can calculate the similarity between the template image and each of the candidate images; finally, a registration result is obtained based on the template image and the candidate image whose similarity is higher than a preset threshold. In this embodiment, when the scanning pen does not shake or tilt, only one direction of registration processing needs to be performed, which helps to reduce the registration workload and improve the registration efficiency.

[0071] Here, the process of calculating the similarity between the template image and each of the candidate images is exemplarily described: in order to avoid inaccurate registration results caused by the influence of light source, the NCC (normalized cross correlation) matching algorithm is used to calculate the similarity.

[0072] For any candidate image, the scanning pen first calculates the mean of the pixel values ​​at corresponding positions in the template image and the candidate image; then, calculates the standard deviation of the pixel values ​​at corresponding positions in the template image and the candidate image.

[0073] Then, a decentralization and normalization process is performed: the scanning pen subtracts the mean of the corresponding position from each pixel value in the template image and then divides the result by the standard deviation of the corresponding position to obtain a normalized template image; and subtracts the mean of the corresponding position from each pixel value in the candidate image and then divides the result by the standard deviation of the corresponding position to obtain a normalized candidate image;

[0074] Finally, the cross-correlation normalized cross-correlation is calculated: the scanner performs cross-correlation operation on the normalized template image and the normalized candidate image, and normalizes the result of the cross-correlation operation; the cross-correlation operation can be regarded as pixel-by-pixel multiplication and summation of the template image and the candidate image; the result of the cross-correlation operation is divided by the number of pixels of the template image and the candidate image (or the normalization factor) so that the range of the normalized cross-correlation result is between [-1,1].

[0075] The result of normalized cross-correlation indicates the similarity between the template image and the corresponding position in the candidate image. A value of 1 indicates a perfect match, a value of -1 indicates a complete opposite, and a value of 0 indicates no correlation.

[0076] Using normalized cross-correlation as a metric, it has a certain degree of robustness to brightness changes and linear illumination changes, and can find the most similar area between the template image and the candidate image, thereby achieving the purpose of image registration.

[0077] In some embodiments, after determining the similarities between the template image and each of the candidate images, a registration result may be obtained based on the template image and the candidate image whose similarities are higher than a preset threshold.

[0078] In a possible implementation, in order to further improve the registration accuracy, after determining the similarity between the template image and each of the candidate images, further optimization search can be performed on the template image and the candidate image whose similarity is higher than a preset threshold. A particle swarm search algorithm can be used for search optimization to achieve a jump search, thereby reducing the amount of registration calculation and improving the registration efficiency.

[0079] The scanning pen can initialize a group of particles, each particle includes a position and a speed, the position represents the position information of the candidate image whose similarity is higher than a preset threshold, and the speed indicates the moving step of the particle; the fitness value of each particle is the similarity between the candidate image and the template image; and the individual optimal position and the group optimal position are determined according to the similarity of the particles.

[0080] When the conditions for ending the iteration are not met, the scanning pen can update the position and speed of the particle according to the individual optimal position, the group optimal position, and the current position and speed of the particle; the update of the speed and position can be achieved using the formula in the particle swarm algorithm. Next, the similarity between the candidate image indicated by the updated position of the particle and the template image is calculated; the individual optimal position and the group optimal position are determined according to the updated similarity of the particle. When the conditions for ending the iteration are met, the registration result is obtained according to the candidate image indicated by the group optimal position determined in the last round of the cycle and the template image.

[0081] The iteration termination condition may be that a preset number of iterations is reached; or that a difference between an individual optimal position and a group optimal position is smaller than a preset difference.

[0082] By using the particle swarm search algorithm for optimization, the best combination can be searched to maximize the matching score (similarity) between the candidate image and the template image. Finally, the optimal transformation parameters are obtained based on the candidate image and template image with the highest similarity. The optimal transformation parameters indicate how to transform the candidate image (such as translation, rotation, scaling, etc.) to match the template image to obtain the best registration result and achieve geometric alignment between the candidate image and the template image.

[0083] In some embodiments, after obtaining the registration result, the scanning pen can stitch the first scanned image and the second scanned image according to the registration result to obtain a stitched image. During the stitching process, in order to improve the visual effect and quality of the stitched image, the stitched image can be fused and corrected for differences, and useful information in the registered images can be integrated into one image and the display details can be enhanced, so that the stitched image is more natural and smooth.

[0084] See also Figure 6 The embodiment of this specification also provides a flowchart of another image stitching method based on a scanning pen, the method comprising:

[0085] In S201, a first scan image and a second scan image in a sequence of scan images captured by a scanning pen are acquired; wherein the first scan image and the second scan image have overlapping image contents.

[0086] The scanning pen can collect sequential scan images based on the user's sliding scanning operation, and then the sequential scan images need to be spliced ​​in real time to improve the accuracy and efficiency of splicing, so as to provide guarantee for subsequent text recognition. The sequential scan images include at least two scan images with overlapping image content. Exemplarily, the first scan image and the second scan image to be spliced ​​are images with overlapping image content collected by the scanning pen at adjacent times.

[0087] It can be understood that this embodiment does not impose any restrictions on the capture time of the first scan image and the second scan image. For example, the scanning pen may capture the first scan image at time T and then capture the second scan image at time T+1; or the scanning pen may capture the second scan image at time T and then capture the first scan image at time T+1.

[0088] In S202, the first scanned image and the second scanned image are registered in a first direction and a second direction to obtain a registration result.

[0089] For example, the first direction is a direction parallel to the preset scanning direction of the scanning pen, and the second direction is a direction perpendicular to the scanning direction. Exemplarily, if the reading direction of the text is horizontal, the user can slide the scanning pen in the horizontal direction, and the preset scanning direction includes the horizontal direction, then the first direction is the horizontal direction, and the second direction is the vertical direction. Alternatively, if the reading direction of the text is vertical, the user can slide the scanning pen in the vertical direction, and the preset scanning direction may include the vertical direction, then the first direction is the vertical direction, and the second direction is the horizontal direction. Alternatively, if the reading direction of the text is an inclined direction that is neither horizontal nor vertical, and the user slides the scanning pen in the inclined direction, and the preset scanning direction may include the inclined direction, then the first direction is the inclined direction, and the second direction is a direction perpendicular to the inclined direction.

[0090] In S203, the first scanned image and the second scanned image are spliced ​​according to the registration result to obtain a spliced ​​image.

[0091] This embodiment does not need to detect whether the scanning pen is shaking or tilted. It handles it according to the situation where the scanning pen is shaking or tilted. The first scan image and the second scan image are aligned in the first direction and the second direction. The image alignment deviation can be corrected so that the two scan images are aligned in the horizontal direction and the vertical direction, avoiding or reducing the problem of character misalignment in the spliced ​​image.

[0092] In some embodiments, in order to improve the efficiency of image registration, the first scanned image and the second scanned image may be preprocessed before the registration process. Image preprocessing is the preparatory work before image registration, which is used to eliminate irrelevant information that affects the accuracy of image registration, improve the efficiency of image registration, and reduce the difficulty of registration, including image denoising, geometric correction, image enhancement, image cropping, etc.

[0093] Next, the registration process is described as follows:

[0094] In some embodiments, the scanning pen can determine a template image from the first scanned image; then, the scanning pen can move a window in the second scanned image along a first direction and a second direction respectively to select multiple candidate images, wherein the first direction is a direction parallel to the scanning direction, and the second direction is a direction perpendicular to the scanning direction.

[0095] The size of the template image is smaller than or equal to the first scanned image, and the template image contains image content that overlaps with certain areas in the second scanned image. The size of the candidate image is smaller than or equal to the second scanned image. The size of the template image is equal to the size of the candidate image. Exemplarily, a window can be moved along a first direction and a second direction in a selected area in the second scanned image to select multiple candidate images. The selected area in the second scanned image is an area that overlaps with certain areas in the first scanned image.

[0096] After determining the template image and the candidate image, the scanning pen can calculate the similarity between the template image and each of the candidate images; finally, the registration result is obtained according to the template image and the candidate image whose similarity is higher than the preset threshold. This embodiment implements registration processing in two directions when the scanning pen is shaken or tilted. The first direction is a direction parallel to the scanning direction, and the second direction is a direction perpendicular to the scanning direction. The alignment deviation of the image can be corrected so that the two scanned images are aligned in the horizontal direction and the vertical direction, thereby avoiding or reducing the problem of character misalignment in the spliced ​​image.

[0097] For example, see Figure 1 and Figure 5 , Figure 5 The stitched image shown is obtained by stitching using the image stitching method of the embodiment of this specification. Figure 5 and Figure 1 By comparison, it can be seen that there is no misalignment of characters in the spliced ​​image obtained by the image splicing method according to the embodiment of this specification.

[0098] Here, the process of calculating the similarity between the template image and each of the candidate images is exemplarily described: in order to avoid inaccurate registration results caused by the influence of light source, the NCC (normalized cross correlation) matching algorithm is used to calculate the similarity.

[0099] For any candidate image, the scanning pen first calculates the mean of the pixel values ​​at corresponding positions in the template image and the candidate image; then, calculates the standard deviation of the pixel values ​​at corresponding positions in the template image and the candidate image.

[0100] Then, a decentralization and normalization process is performed: the scanning pen subtracts the mean of the corresponding position from each pixel value in the template image and then divides the result by the standard deviation of the corresponding position to obtain a normalized template image; and subtracts the mean of the corresponding position from each pixel value in the candidate image and then divides the result by the standard deviation of the corresponding position to obtain a normalized candidate image;

[0101] Finally, the cross-correlation normalized cross-correlation is calculated: the scanner performs cross-correlation operation on the normalized template image and the normalized candidate image, and normalizes the result of the cross-correlation operation; the cross-correlation operation can be regarded as pixel-by-pixel multiplication and summation of the template image and the candidate image; the result of the cross-correlation operation is divided by the number of pixels of the template image and the candidate image (or the normalization factor) so that the range of the normalized cross-correlation result is between [-1,1].

[0102] The result of normalized cross-correlation indicates the similarity between the template image and the corresponding position in the candidate image. A value of 1 indicates a perfect match, a value of -1 indicates a complete opposite, and a value of 0 indicates no correlation.

[0103] Using normalized cross-correlation as a metric, it has a certain degree of robustness to brightness changes and linear illumination changes, and can find the most similar area between the template image and the candidate image, thereby achieving the purpose of image registration.

[0104] In some embodiments, after determining the similarities between the template image and each of the candidate images, a registration result may be obtained based on the template image and the candidate image whose similarities are higher than a preset threshold.

[0105] In a possible implementation, in order to further improve the registration accuracy, after determining the similarity between the template image and each of the candidate images, further optimization search can be performed on the template image and the candidate image whose similarity is higher than a preset threshold. A particle swarm search algorithm can be used for search optimization to achieve a jump search, thereby reducing the amount of registration calculation and improving the registration efficiency.

[0106] The scanning pen can initialize a group of particles, each particle includes a position and a speed, the position represents the position information of the candidate image whose similarity is higher than a preset threshold, and the speed indicates the moving step of the particle; the fitness value of each particle is the similarity between the candidate image and the template image; and the individual optimal position and the group optimal position are determined according to the similarity of the particles.

[0107] When the conditions for ending the iteration are not met, the scanning pen can update the position and speed of the particle according to the individual optimal position, the group optimal position, and the current position and speed of the particle; the update of the speed and position can be achieved using the formula in the particle swarm algorithm. Next, the similarity between the candidate image indicated by the updated position of the particle and the template image is calculated; the individual optimal position and the group optimal position are determined according to the updated similarity of the particle. When the conditions for ending the iteration are met, the registration result is obtained according to the candidate image indicated by the group optimal position determined in the last round of the cycle and the template image.

[0108] The iteration termination condition may be that a preset number of iterations is reached; or that a difference between an individual optimal position and a group optimal position is smaller than a preset difference.

[0109] By using the particle swarm search algorithm for optimization, the best combination can be searched to maximize the matching score (similarity) between the candidate image and the template image. Finally, the optimal transformation parameters are obtained based on the candidate image and template image with the highest similarity. The optimal transformation parameters indicate how to transform the candidate image (such as translation, rotation, scaling, etc.) to match the template image to obtain the best registration result and achieve geometric alignment between the candidate image and the template image.

[0110] In some embodiments, after obtaining the registration result, the scanning pen can stitch the first scanned image and the second scanned image according to the registration result to obtain a stitched image. During the stitching process, in order to improve the visual effect and quality of the stitched image, the stitched image can be fused and corrected for differences, and useful information in the registered images can be integrated into one image and the display details can be enhanced, so that the stitched image is more natural and smooth.

[0111] Considering that the image stitching method is not only applicable to the scanning pen scenario, but also to other image stitching scenarios. Figure 7 , the embodiment of this specification provides an image stitching method, including:

[0112] In S301, a first image and a second image in a sequence of images captured by a photographing device are acquired; wherein the first image and the second image have overlapping image contents.

[0113] In S302, if it is detected that the shooting device deviates from the preset shooting direction during the process of capturing the sequence of images, the first image and the second image are registered in the first direction and the second direction to obtain a registration result.

[0114] In S303, if it is detected that the shooting device captures a sequence of images according to the shooting direction, the first image and the second image are registered in a first direction to obtain a registration result.

[0115] The first direction is a direction parallel to the shooting direction, and the second direction is a direction perpendicular to the shooting direction.

[0116] In S304, the first image and the second image are spliced ​​according to the registration result to obtain a spliced ​​image.

[0117] In some embodiments, it also includes:

[0118] If it is detected that the shooting device deviates from the preset shooting direction during the process of capturing the sequence of images, and the angle difference between the deviated direction and the shooting direction is greater than the preset angle, a reminder message is output to remind the user to correct the direction.

[0119] In some embodiments, the camera device is equipped with an acceleration sensor;

[0120] The method further includes: in the process of the shooting device capturing sequence images, using acceleration data captured by the acceleration sensor to determine the shooting trajectory of the shooting device; and determining whether the shooting device deviates from a preset shooting direction according to the shooting trajectory of the shooting device.

[0121] In some embodiments, the registration processing of the first image and the second image in the first direction and the second direction to obtain the registration result includes: determining a template image from the first image; moving a window in the second image along the first direction and the second direction respectively to select multiple candidate images; calculating the similarity between the template image and each of the candidate images; and obtaining the registration result based on the template image and the candidate image whose similarity is higher than a preset threshold.

[0122] In some embodiments, the first image and the second image are registered in a first direction to obtain a registration result, including: determining a template image from the first image; moving windows in the second image along the first direction to select multiple candidate images; calculating the similarity between the template image and each of the candidate images; and obtaining a registration result based on the template image and the candidate image whose similarity is higher than a preset threshold.

[0123] In some embodiments, the calculating the similarity between the template image and each of the candidate images includes: for any candidate image, calculating the mean and standard deviation of the pixel values ​​at corresponding positions between the template image and the candidate image; subtracting the mean of the corresponding position from each pixel value in the template image and dividing the result by the standard deviation of the corresponding position to obtain a normalized template image; and subtracting the mean of the corresponding position from each pixel value in the candidate image and dividing the result by the standard deviation of the corresponding position to obtain a normalized candidate image; performing a cross-correlation operation on the normalized template image and the normalized candidate image, and normalizing the result of the cross-correlation operation to obtain the similarity between the template image and the candidate image.

[0124] In some embodiments, obtaining the registration result based on the template image and the candidate image whose similarity is higher than a preset threshold comprises: initializing a group of particles, each particle comprising a position and a speed, the position representing the position information of the candidate image whose similarity is higher than the preset threshold, and the speed indicating the moving step of the particle; the fitness value of each particle is the similarity between the candidate image and the template image; and determining the individual optimal position and the group optimal position according to the similarity of the particle; when the iteration end condition is not met, updating the position and speed of the particle according to the individual optimal position, the group optimal position, and the current position and speed of the particle; calculating the similarity between the candidate image indicated by the updated position of the particle and the template image; determining the individual optimal position and the group optimal position according to the updated similarity of the particle; when the iteration end condition is met, obtaining the registration result according to the candidate image and the template image indicated by the group optimal position determined in the last round of the cycle.

[0125] In some embodiments, the shooting direction includes a horizontal direction, then the first direction is a horizontal direction, and the second direction is a vertical direction; or the shooting direction includes a vertical direction, then the first direction is a vertical direction, and the second direction is a horizontal direction.

[0126] See also Figure 8 , the embodiment of this specification provides an image stitching method, including:

[0127] In S401, a first image and a second image in a captured sequence of images are acquired; wherein the first image and the second image have overlapping image content.

[0128] In S402, the first image and the second image are registered in a first direction and a second direction to obtain a registration result.

[0129] The first direction is a direction parallel to a preset shooting direction, and the second direction is a direction perpendicular to the shooting direction.

[0130] In S403, the first image and the second image are spliced ​​according to the registration result to obtain a spliced ​​image.

[0131] The various technical features in the above embodiments can be arbitrarily combined as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, any combination of the various technical features in the above embodiments also falls within the scope of this specification.

[0132] Fig. 9 is a schematic structural diagram of a device provided by an exemplary embodiment. Fig. 9At the hardware level, the device includes a processor 902, an internal bus 904, a network interface 906, a memory 908, and a non-volatile memory 910, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 902 reading the corresponding computer program from the non-volatile memory 910 into the memory 908 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0133] In some embodiments, the embodiments of this specification also provide an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements any of the above methods by running the executable instructions.

[0134] Exemplarily, the electronic device includes a scanning pen, a photographing device, or a device communicatively connected to the photographing device.

[0135] In some embodiments, the embodiments of this specification further provide a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the steps of any of the methods described above are implemented.

[0136] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0137] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0138] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0139] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0140] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0141] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0142] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0143] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0144] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0145] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A scanning pen-based image stitching method, characterized in that: include: Acquire a first scan image and a second scan image in a sequence of scan images captured by a scanning pen; wherein the first scan image and the second scan image have overlapping image contents; If it is detected that the scanning pen deviates from the preset scanning direction during the process of acquiring the sequence scanned images, the first scanned image and the second scanned image are registered in the first direction and the second direction to obtain a registration result; If it is detected that the scanning pen collects a sequence of scanned images in the scanning direction, a registration process is performed on the first scanned image and the second scanned image in the first direction to obtain a registration result; The first scanned image and the second scanned image are spliced ​​according to the registration result to obtain a spliced ​​image.

2. The method according to claim 1, characterized in that Also includes: If it is detected that the scanning pen deviates from the preset scanning direction during the process of acquiring the sequential scanning images, and the angle difference between the deviated direction and the scanning direction is greater than the preset angle, a reminder message is output to remind the user to correct the direction.

3. The method according to claim 1, characterized in that The scanning pen is equipped with an acceleration sensor; The method further comprises: In the process of the scanning pen collecting the sequential scanning images, the scanning trajectory of the scanning pen is determined by using the acceleration data collected by the acceleration sensor; Determine whether the scanning pen deviates from a preset scanning direction according to the scanning track of the scanning pen.

4. The method according to claim 1, characterized in that The performing registration processing on the first scanned image and the second scanned image in the first direction and the second direction to obtain a registration result includes: determining a template image from the first scanned image; Moving a window in the second scanned image along a first direction and a second direction respectively to select a plurality of candidate images; Calculating the similarity between the template image and each of the candidate images; The registration result is obtained based on the template image and the candidate image whose similarity is higher than a preset threshold.

5. The method according to claim 1, characterized in that The performing registration processing on the first scanned image and the second scanned image in a first direction to obtain a registration result includes: determining a template image from the first scanned image; Moving windows respectively along a first direction in the second scanned image to select a plurality of candidate images; Calculating the similarity between the template image and each of the candidate images; The registration result is obtained based on the template image and the candidate image whose similarity is higher than a preset threshold.

6. The method according to claim 4 or 5, characterized in that: The calculating the similarity between the template image and each of the candidate images includes: For each candidate image, calculating the mean and standard deviation of pixel values ​​at corresponding positions in the template image and the candidate image; Subtracting the mean of the corresponding position from each pixel value in the template image and dividing the result by the standard deviation of the corresponding position to obtain a normalized template image; and subtracting the mean of the corresponding position from each pixel value in the candidate image and dividing the result by the standard deviation of the corresponding position to obtain a normalized candidate image; A cross-correlation operation is performed on the normalized template image and the normalized candidate image, and the result of the cross-correlation operation is normalized to obtain the similarity between the template image and the candidate image.

7. The method according to claim 4 or 5, characterized in that: The step of obtaining a registration result based on the template image and the candidate image whose similarity is higher than a preset threshold comprises: Initialize a group of particles, each particle includes a position and a speed, the position represents the position information of the candidate image whose similarity is higher than a preset threshold, and the speed indicates the moving step of the particle; the fitness value of each particle is the similarity between the candidate image and the template image; and determine the individual optimal position and the group optimal position according to the similarity of the particles; When the iteration termination condition is not met, the position and speed of the particle are updated according to the individual optimal position, the group optimal position, and the current position and speed of the particle; the similarity between the candidate image indicated by the updated position of the particle and the template image is calculated; and the individual optimal position and the group optimal position are determined according to the updated similarity of the particle; When the iteration termination condition is met, the registration result is obtained according to the candidate image and the template image indicated by the optimal position of the group determined in the last round of the cycle.

8. The method according to claim 1, characterized in that The scanning direction includes a horizontal direction, then the first direction is a horizontal direction, and the second direction is a vertical direction; or The scanning direction includes a vertical direction, then the first direction is the vertical direction, and the second direction is the horizontal direction.

9. An image stitching method based on a scanning pen, characterized in that: include: Acquire a first scan image and a second scan image in a sequence of scan images captured by a scanning pen; wherein the first scan image and the second scan image have overlapping image contents; Performing registration processing on the first scanned image and the second scanned image in a first direction and a second direction to obtain a registration result; The first scanned image and the second scanned image are spliced ​​according to the registration result to obtain a spliced ​​image.

10. An image stitching method, characterized in that: include: Acquire a first image and a second image in a sequence of images captured by a photographing device; wherein the first image and the second image have overlapping image content; If it is detected that the shooting device deviates from the preset shooting direction during the process of capturing the sequence of images, the first image and the second image are registered in the first direction and the second direction to obtain a registration result; If it is detected that the shooting device acquires a sequence of images according to the shooting direction, performing a registration process in a first direction on the first image and the second image to obtain a registration result; The first image and the second image are spliced ​​according to the registration result to obtain a spliced ​​image.

11. An image stitching method, characterized in that: include: Acquire a first image and a second image in the acquired sequence of images; wherein the first image and the second image have overlapping image content; Performing registration processing on the first image and the second image in a first direction and a second direction to obtain a registration result; The first image and the second image are spliced ​​according to the registration result to obtain a spliced ​​image.

12. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 11 by running the executable instructions.

Images