Image splicing method and device and storage medium

Abstract

The invention provides an image stitching method, an image stitching device and a medium. The method comprises the steps that a plurality of images arranged in an array are obtained, a plurality of image columns are included in the first direction, at least one image row is included in the second direction different from the first direction, and the adjacent images are overlapped within at least a preset range; the method comprises the following steps: in the process of splicing a plurality of images one by one along a first direction by rows, acquiring a first splicing homography matrix corresponding to each image pair, evaluating the splicing results of the image pairs according to the first splicing homography matrix corresponding to each image pair, and when the splicing results do not meet a preset condition, judging whether the splicing results meet the preset condition or not. Splicing the image pairs by using a first splicing homography matrix corresponding to the image pairs in the same column in the previously spliced row, or obtaining a first initial homography matrix corresponding to the image column where the image pairs are located, and splicing the image pairs; and when the plurality of images comprise a plurality of image rows, further splicing according to columns to obtain a spliced image.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image stitching method, device and computer-readable storage medium. Background Art

[0002] In the process of stitching images through deep neural networks, it is usually necessary to first extract the features of the images to be stitched, match the features of all two adjacent images and calculate the homography matrix that deforms the images, then deform the images according to the homography matrix and fuse the images to finally obtain the stitched image.

[0003] However, current image stitching methods are prone to false matching of feature points due to various factors, such as camera angle, the ratio of near-field to far-field images, or high image similarity. This can lead to image deformation and fusion results that do not meet expectations, resulting in stitching failure. Furthermore, during the stitching process, there is no way to predict and correct the image stitching results, making it impossible to correct or adjust stitching errors.

[0004] Therefore, a more robust image stitching method is needed, which can timely correct the errors generated in the image stitching process and adjust the homography matrix, thereby reducing the stitching error and improving the image stitching quality. Summary of the Invention

[0005] To solve the above technical problems, according to one aspect of the present invention, there is provided an image stitching method, comprising: acquiring a plurality of images arranged in an array, wherein the array of the plurality of images includes a plurality of image columns along a first direction and at least one image row along a second direction different from the first direction, and adjacent images arranged along the first direction and the second direction overlap by at least a preset range; in a process of stitching the plurality of images row by row along the first direction, acquiring a first stitching homography matrix corresponding to each image pair, wherein stitching results of the image pairs are evaluated based on the first stitching homography matrix corresponding to each image pair; when the stitching results do not meet a preset condition, stitching the image pairs using the first stitching homography matrix corresponding to the image pair in the same column in a previously stitched row, or acquiring a first initial homography matrix corresponding to the image column where the image pair is located, and stitching the image pairs using the first initial homography matrix; and when the plurality of images include a plurality of image rows, further stitching the plurality of images column by column along the second direction, and acquiring a stitched image.

[0006] According to another aspect of the present invention, an image stitching device is provided, comprising: an acquisition unit configured to acquire a plurality of images arranged in an array, wherein the array of the plurality of images includes a plurality of image columns along a first direction and at least one image row along a second direction different from the first direction, and adjacent images arranged along the first direction and the second direction overlap by at least a preset range; a stitching unit configured to acquire a first stitching homography matrix corresponding to each image pair during a process of stitching the plurality of images row by row along the first direction, wherein stitching results of the image pairs are evaluated based on the first stitching homography matrix corresponding to each image pair, and when the stitching results do not meet a preset condition, the image pairs are stitched using a first stitching homography matrix corresponding to an image pair in the same column in a previously stitched row, or a first initial homography matrix corresponding to the image column where the image pair is located is acquired and the image pairs are stitched using the first initial homography matrix; and when the plurality of images include a plurality of image rows, the plurality of images are further stitched column by column along the second direction to acquire a stitched image.

[0007] According to another aspect of the present invention, an image stitching device is provided, comprising: a processor; and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: acquiring a plurality of images arranged in an array, wherein the array of the plurality of images includes a plurality of image columns along a first direction, includes at least one image row along a second direction different from the first direction, and adjacent images arranged along the first direction and the second direction respectively have an overlap of at least a preset range; in a process of stitching the plurality of images row by row along the first direction, obtaining Obtain a first stitching homography matrix corresponding to each image pair, wherein the stitching results of the image pairs are evaluated respectively according to the first stitching homography matrix corresponding to each image pair; when the stitching result does not meet a preset condition, the image pairs are stitched using the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched row, or a first initial homography matrix corresponding to the image column where the image pair is located is obtained, and the image pairs are stitched using the first initial homography matrix; and when the multiple images include multiple image rows, the multiple images are further stitched by column along a second direction, and a stitched image is obtained.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: acquiring a plurality of images arranged in an array, wherein the array of the plurality of images includes a plurality of image columns along a first direction, includes at least one image row along a second direction different from the first direction, and has at least a preset range of overlap between adjacent images arranged along the first direction and the second direction; in a process of stitching the plurality of images row by row along the first direction, acquiring the image pairs corresponding to each image pair A first stitching homography matrix, wherein the stitching results of the image pairs are evaluated respectively according to the first stitching homography matrix corresponding to each image pair. When the stitching result does not meet the preset conditions, the image pairs are stitched using the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched row, or a first initial homography matrix corresponding to the image column where the image pair is located is obtained, and the image pairs are stitched using the first initial homography matrix; and when the multiple images include multiple image rows, the multiple images are further stitched by column along a second direction to obtain a stitched image.

[0009] According to the above-mentioned image stitching method, device and computer-readable storage medium of the present invention, it is possible to select an appropriate homography matrix for stitching to stitch images by timely evaluating the stitching results and adaptively adjusting the homography matrix during the image stitching process, thereby reducing stitching errors and improving image stitching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail the embodiments of the present invention with reference to the accompanying drawings.

[0011] Figure 1 (a) and (b) in the figure respectively show two images of a shelf with a certain range of overlapping areas acquired by a camera;

[0012] Figure 2 Show the Figure 1 After the images in (a) and (b) are stitched together, the incorrect stitched image is caused by the false matching of feature points;

[0013] Figure 3 Show the Figure 1 The correct stitched image is obtained by stitching the images in (a) and (b);

[0014] Figure 4 A flowchart of an image stitching method according to an embodiment of the present invention is shown;

[0015] Figure 5An example of specific steps of performing row-by-row image pair stitching along a first direction according to an embodiment of the present invention is shown;

[0016] Figure 6 A schematic diagram showing an example camera array according to an embodiment of the present invention;

[0017] Figure 7 A schematic diagram illustrating a plurality of images arranged in an M×N array acquired according to an example of an embodiment of the present invention;

[0018] Figure 8 A schematic diagram showing a row-stitched image after row-wise stitching according to an example of an embodiment of the present invention;

[0019] Figure 9 A schematic diagram showing column-based splicing according to an example of an embodiment of the present invention is shown;

[0020] Figure 10 A schematic diagram showing a stitched image according to an example of an embodiment of the present invention;

[0021] Figure 11 A block diagram of an image stitching device according to an embodiment of the present invention is shown;

[0022] Figure 12 A block diagram of an image stitching apparatus according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0023] The following describes an image stitching method, apparatus, and computer-readable storage medium according to embodiments of the present invention with reference to the accompanying drawings. Like reference numerals denote like elements throughout the accompanying drawings. It should be understood that the embodiments described herein are merely illustrative and should not be construed as limiting the scope of the present invention.

[0024] In various application scenarios such as warehouses, stores, and shelf management, it is often necessary to stitch together multiple captured images to obtain a stitched image. However, due to various factors such as the camera shooting angle, the ratio of near-field to far-field images, the high degree of image similarity, and the lack of significant features, false matching of feature points may occur, resulting in image distortion, unsatisfactory fusion results, and stitching failure. Figure 1 (a) and (b) in the figure respectively show two images of a shelf with a certain range of overlapping areas acquired by a camera. Figure 2 The traditional image stitching method is shown. Figure 1 After the images in (a) and (b) are stitched together, the incorrect stitched image is caused by the false matching of feature points. Figure 2 It can be seen that due to Figure 1The two images in (a) and (b) are very similar, which will affect the image stitching results, resulting in the inability to correctly match the feature points and the inability to obtain the correct shelf image.

[0025] Therefore, there is a need for an image stitching method, device and computer readable storage medium that can promptly correct errors generated during the image stitching process and obtain a correct stitched image. According to the image stitching method in the embodiment of the present invention, the following can be obtained: Figure 3 Shown for Figure 1 The correct stitched image is obtained by stitching the images in (a) and (b).

[0026] Figure 4 4. A flow chart of an image stitching method 400 according to an embodiment of the present invention is shown. Figure 4 An image stitching method according to an embodiment of the present invention is described.

[0027] In step S401, a plurality of images arranged in an array are acquired, wherein the array of the plurality of images includes a plurality of image columns along a first direction, and includes at least one image row along a second direction different from the first direction, and adjacent images arranged along the first direction and the second direction respectively have an overlap of at least a preset range.

[0028] According to an embodiment of the present invention, the plurality of images arranged in an array may have a plurality of image columns (e.g., M image columns) and may have one or more image rows (e.g., N image rows). The image columns of the plurality of images may be arranged along a first direction, and the image rows of the plurality of images may be arranged along a second direction, the first direction being different from the second direction. Optionally, the first direction may be perpendicular to the second direction. Furthermore, the first direction may optionally have a specific angle with the second direction, such as 30 degrees, 45 degrees, 60 degrees, 80 degrees, or other angles, which are not limited herein.

[0029] According to embodiments of the present invention, the multiple images arranged in an M×N array can be acquired using one or more cameras. In one example, a single camera can be used to acquire M images in each row and repeat this process for N rows of images to acquire the multiple images arranged in an M×N array. In another example, N cameras can be used to acquire N images in M ​​columns at a time to acquire the multiple images arranged in an M×N array. Of course, in yet another example, the multiple images arranged in an M×N array can also be acquired using various methods, such as a single camera acquiring images through a zigzag motion or a zigzag motion. Alternatively, in yet another example, the multiple images arranged in an M×N array can be acquired using N / a cameras to repeatedly acquire N / a images in M ​​columns a times, where a can be, for example, 2 or 3. The above-mentioned methods for acquiring multiple images arranged in an array are merely examples. In actual applications, different image acquisition methods can be adopted based on specific requirements, such as different application scenarios, camera parameter settings, and image size, to meet the needs of subsequent image stitching, and these are not limitations herein.

[0030] In an embodiment of the present invention, among the multiple images acquired and arranged in an M×N array, there may be at least a preset range of overlap between adjacent images in the first direction and the second direction. Optionally, the overlap of the preset range may be 40%. In addition, optionally, the overlap of the preset range may also be different values ​​such as 45%, 50%, etc. Further, the overlap of the preset range between adjacent images in the first direction and the second direction may be the same or different. For example, in the first direction, the overlap of the preset range may be 45%, while in the second direction, the overlap of the preset range may be 55%, etc. The specific limitation on the overlap of the preset range between adjacent images arranged in the first direction and the second direction is only an example. In actual applications, specific settings can be made according to different application scenarios and image acquisition requirements, and are not limited here.

[0031] In step S402, during the process of stitching the multiple images row by row along the first direction, a first stitching homography matrix corresponding to each image pair is obtained, wherein the stitching results of the image pairs are evaluated respectively according to the first stitching homography matrix corresponding to each image pair. When the stitching result does not meet the preset conditions, the image pairs are stitched using the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched row, or a first initial homography matrix corresponding to the image column where the image pair is located is obtained, and the image pairs are stitched using the first initial homography matrix.

[0032] In an embodiment of the present invention, for a plurality of images arranged in an M×N array, image pair-by-image splicing can be performed for each row in the N rows. Specifically, for each row in the plurality of images, the corresponding first stitching homography matrix for stitching can be calculated for each of the (M-1) image pairs formed in pairs, and image pair-by-image splicing can be performed accordingly. That is to say, for each of the N rows in the plurality of images, first, the image pair consisting of the first image and the second image in the row can be stitched, that is, by calculating the first stitching homography matrix of the image pair, the second image is deformed for the fusion of the first image and the second image; then, the image pair consisting of the second image and the third image can be stitched in the same way, and so on, until the calculation of the first stitching homography matrix of all (M-1) image pairs in the row is completed. Optionally, in the process of calculating the first stitching homography matrix, a neural network can be used to calculate the first stitching homography matrix based on a deep learning method.

[0033] In an embodiment of the present invention, during the stitching process of each image pair, the stitching results of the image pairs can be evaluated based on the first stitching homography matrix corresponding to each image pair. When the stitching result of the image pair obtained by the evaluation meets a preset condition, it is determined that the first stitching homography matrix calculated above is used to apply to the image pair; and when the stitching result does not meet the preset condition, the first stitching homography matrix calculated above is not used, and the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched row is selected to stitch the image pair, or a first initial homography matrix corresponding to the image column where the image pair is located is obtained, and the image pair is stitched using the first initial homography matrix.

[0034] In an embodiment of the present invention, after acquiring multiple images arranged in an M×N array, for the above-mentioned row-by-row stitching step, a first initial homography matrix required for stitching corresponding to each image pair in a row may be acquired when necessary.

[0035] Optionally, the first initial homography matrix required for corresponding stitching can be determined for each of the (M-1) image pairs formed in each of the M columns in each of the N rows. That is, a first initial homography matrix can be determined for the image pair consisting of the first and second images in each row, and a first initial homography matrix can also be determined for the image pair consisting of the second and third images, and so on. Therefore, a total of (M-1) first initial homography matrices can be formed for the (M-1) image pairs formed in each of the N rows. Since the relative distance between multiple cameras in the same column in the camera array is fixed, the image pairs corresponding to the same image column can share the same first initial homography matrix. That is, the (M-1) first initial homography matrices can be determined only for the M image pairs in a specific row, while the image pairs in other rows can share the first initial homography matrix corresponding to the image column in which the image pair is located, among the (M-1) first initial homography matrices determined for the image pairs in that specific row.

[0036] When determining these first initial homography matrices, optionally, only the change in displacement between the two images in the image pair can be considered, without considering other relationships between adjacent images. Therefore, in one example, the first initial homography matrix representing the relative displacement can be obtained based on the relative displacement between the images in each image pair in a row. In one example, a neural network can be used to calculate the first initial homography matrix representing the relative displacement. Of course, the above-mentioned method of determining the first initial homography matrix is ​​only an example, and the first initial homography matrix can be determined in different ways according to actual applications, and there is no limitation here. In addition, the time for obtaining the first initial homography matrix can be before the acquisition of the image pairs or evaluation results used for row-by-row image pair stitching, or after the acquisition of the above-mentioned image pairs or evaluation results, and there is no limitation here. Further, if the splicing method of the first splicing homography matrix can be applied to each image pair when stitching by row, it is also possible to directly use each first splicing homography matrix to complete the splicing without obtaining the corresponding first initial homography matrix.

[0037] In the above operation of obtaining a first initial homography matrix representing the relative displacement between images in each image pair in a row, the following various exemplary methods may be used to calculate the relative displacement and / or determine the first initial homography matrix:

[0038] In one example, the relative displacement between images in a row can be determined based on the camera's movement distance when capturing images in each image pair. For example, if it is known that a device for controlling camera movement, equipped with a camera, moved 30 centimeters in a first direction while capturing both images in an image pair, then it can be determined that the camera also moved 30 centimeters in the first direction. Therefore, the relative displacement between the two images in the image pair can be determined to be 30 centimeters. Furthermore, if the relative displacement between images needs to be determined in units such as pixels, a correspondence between the camera's movement distance and the relative displacement between images expressed in units such as pixels can be pre-set to convert the camera's movement distance into a relative displacement in units such as pixels. For example, it can be assumed that for every 10 centimeters of camera movement in the first direction, the captured images will shift by 1000 pixels. Based on this correspondence, if the camera moved 50 centimeters in the first direction while capturing the image pair, the relative displacement between the images in the image pair can be calculated to be 1000 * 50 / 10 = 5000 pixels. Finally, a first initial homography matrix corresponding to the image may be obtained according to the relative displacement between the images in each image pair in a row.

[0039] In another example, as a special case of the above example, if the device for controlling the movement of the camera is equipped with a camera, and the distance of movement is always a fixed value in the process of capturing two images in an image pair in one or more rows, then the first initial homography matrix corresponding to a specific image pair can be obtained first, and applied to all image pairs in the row or rows. That is to say, when stitching images in the first direction, only one first initial homography matrix can be used for multiple image pairs in one or more rows. The specific image pair mentioned above can be the first image pair in a row or the middle image pair. The specific image pair can be selected and set arbitrarily without limitation.

[0040] In another example, the relative displacement can also be determined based on the overlap between pixels of one image in an image pair and pixels of the other image in the adjacent image pair. For example, if an image includes 5000 pixels in a first direction and the pixel overlap between one image and the other image in the image pair is 40%, the relative displacement between the images in the image pair can be calculated as 5000 * (1-40%) = 3000 pixels. Based on this, a first initial homography matrix corresponding to the image pair is obtained.

[0041] In another example, the first initial homography matrix used in stitching a previously stitched image can be applied to the stitching of other images. For example, in the process of stitching shelf images, if the acquisition method for each shelf image is the same, for example, the camera acquires multiple images of each shelf arranged in an M×N array for stitching, and if the stitching result of the image of a particular shelf is satisfactory, the first initial homography matrix used in stitching the image of that shelf can be applied to the stitching process of all subsequent shelf images.

[0042] Of course, in each of the above-mentioned examples, the various parameters used can be pre-set or automatically acquired. For example, the distance that the device used to control camera movement moves when capturing each adjacent image can be manually set in advance or automatically acquired from the device controlling camera movement via a wired or wireless network.

[0043] Optionally, the evaluation of the stitching result of the image pair may include multiple methods. For example, the stitching result may be evaluated for a transformed image or a fused image based on the image pair and the first stitching homography matrix. The transformed image may be obtained by transforming one image in the image pair using the corresponding first stitching homography matrix, and the fused image may be obtained by fusing the two images in the image pair using the corresponding first stitching homography matrix. After obtaining the transformed image or the fused image, the stitching result may be evaluated for the transformed image and / or the fused image using at least one result of target detection, feature extraction, shape detection, structural similarity, and peak signal-to-noise ratio to determine whether the stitching result meets the preset conditions.

[0044] Optionally, when evaluating the transformed image and / or the fused image using target detection or feature extraction, the stitching result can be judged to determine whether it meets the preset conditions based on the specific application scenario of the embodiment of the present invention, for example, by using a pre-trained neural network to detect a specific pattern, target, or feature. When evaluating the transformed image and / or the fused image using shape detection, the shape of the transformed image and / or the fused image can be judged to determine whether the transformed image and / or the fused image meets the expected image shape (such as a rectangle, square, parallelogram, etc.) to determine whether the stitching result meets the preset conditions. When evaluating the transformed image and / or the fused image using structural similarity and / or peak signal-to-noise ratio, the structural similarity and / or peak signal-to-noise ratio parameters of the transformed image and / or the fused image can be judged to determine whether the stitching result meets the preset conditions. In addition, in other embodiments of the present invention, the stitching result can also be judged based on various aspects such as the relative positions of the corner points of the transformed image and / or the fused image, the scanning direction of the camera, and the transformation or fusion quality of the image. The various methods for evaluating the splicing results described above are merely examples. Different methods for evaluating the splicing results may be adopted according to actual application scenarios of the embodiments of the present invention, and are not limited here.

[0045] In an embodiment of the present invention, as described above, when the stitching result does not meet the preset conditions, the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched row or the first initial homography matrix corresponding to the image pair in the same column can be selected for stitching the image pair. Optionally, when there is an image row that has been stitched previously for the currently stitched image row, the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched second row closest to the first row where the image pair is located can be first selected, and the stitching result can be further evaluated in a similar manner as described above, and the image pair can be stitched when the evaluated stitching result meets the preset conditions; at this time, when, after evaluation, the stitching result using the first stitching homography matrix corresponding to the image pair in the same column in the second row still does not meet the preset conditions, the first stitching homography matrix corresponding to the closest previously stitched row can be selected in sequence by analogy, and the image pair can be stitched when the evaluated stitching result meets the preset conditions. When the evaluation results of the first stitching homography matrices corresponding to all previously stitched rows do not meet the preset conditions, the first initial homography matrix corresponding to the image pair in the same column may be selected to stitch the image pair. In other words, when the multiple images include one image row or do not have a previously stitched row, the first initial homography matrix corresponding to the image pair in the same column is selected to stitch the image pair.

[0046] In one example, when stitching the first row of N rows of a plurality of images arranged in an M×N array, for example, for an image pair consisting of the first image and the second image in the row, a first stitching homography matrix for the image pair may be calculated first, and a transformed image and / or fused image obtained using the first stitching homography matrix may be evaluated. If the stitching result does not meet a preset condition, because there is no previously stitched image row, the first initial homography matrix corresponding to the same column of the image pair may be used to stitch the image pair in the row. Subsequently, stitching of all image pairs in the first row may be completed in this manner.

[0047] Afterwards, stitching can be continued for the second row in N rows. When stitching the image pair consisting of the first image and the second image in the second row, it is also necessary to first calculate the corresponding first stitching homography matrix. When the stitching result evaluated in a similar manner as described above does not meet the preset conditions, the first stitching homography matrix corresponding to the same column of the first row that is closest to the second row and has been previously stitched can be first selected and applied, and the evaluation can be continued. If the stitching result still does not meet the preset conditions, if there is no previously stitched row, the first initial homography matrix corresponding to the same column is used for stitching. The stitching of the second row is completed in this way.

[0048] Subsequently, a similar method may be used to continue stitching the image pairs of the remaining rows in the N rows, and a row stitching image corresponding to each row in the N rows may be generated.

[0049] exist Figure 5 In FIG, an example of the specific steps of performing row-by-row image pair stitching along the first direction according to an embodiment of the present invention is shown in detail. Figure 5 As shown, when stitching multiple images arranged in an M×N array, in step S1, a first stitching homography matrix corresponding to a certain image pair in a certain row is first calculated, and stitching results such as transformed images and / or fused images obtained using the first stitching homography matrix are evaluated.

[0050] When the image pair in step S1 is the first image pair, i.e., n = 1, step S2 evaluates whether the stitching result of the nth image pair meets the preset conditions. If the result is yes, i.e., the stitching result meets the preset conditions, then the process proceeds directly to step S7 to determine whether the next image pair exists, i.e., whether the n+1th image pair exists. If the result is yes, n is assigned to n+1, and the evaluation step of step S2 is restarted. If the result is no, the process ends.

[0051] If the stitching result of the nth image pair is evaluated in step S2 to determine whether it meets the preset conditions, that is, the stitching result does not meet the preset conditions, then the process proceeds to step S3 to determine whether there are previously stitched image rows. If the result is no, that is, no previously stitched image rows exist, then the process proceeds to step S6 to select the first initial homography matrix corresponding to the image column in which the image pair is located and stitch the image pair.

[0052] If the judgment result in step S3 is yes, that is, there are image rows that have been previously spliced, then enter step S4, and select the first splicing homography matrix corresponding to the closest previously spliced ​​row in turn (for example, first select the first splicing homography matrix used for the spliced ​​n-1th row, and if the judgment result does not meet the preset conditions, then select the first splicing homography matrix used for the spliced ​​n-2th row, and so on), and splice the image pair when the evaluated splicing result meets the preset conditions.

[0053] After step S4, the process proceeds to step S5 to determine whether the stitching result meets the pre-set conditions. If the result in S5 is yes, the process proceeds directly to step S7 to determine whether the next image pair exists, that is, whether the n+1th image pair exists. If the result is yes, n is assigned to n+1 and the evaluation step in step S2 is restarted. If the result is no, the process ends.

[0054] If the judgment result in step S5 is no, that is, the evaluation results of the first stitching homography matrix corresponding to all previously stitched rows do not meet the preset conditions, then enter step S6, that is, select the first initial homography matrix corresponding to the image column where the image pair is located to stitch the image pair.

[0055] After all the image pairs in one row are stitched together, the image pairs in the remaining N rows can be stitched together in a similar manner, and a row stitching image corresponding to each row in the N rows can be generated.

[0056] In S403 , when the multiple images include multiple image rows, the multiple images are further spliced ​​in columns along the second direction, and a spliced ​​image is acquired.

[0057] In an embodiment of the present invention, after completing row stitching for a plurality of images arranged in an M×N array, column stitching can be further completed for each of the M columns in the plurality of images in a manner similar to the aforementioned row stitching, and ultimately a complete stitched image can be obtained.

[0058] Optionally, during the process of stitching the multiple images column by column along the second direction, a second stitching homography matrix corresponding to each image pair is obtained, wherein the stitching results of the image pairs are evaluated respectively according to the second stitching homography matrix corresponding to each image pair. When the stitching result does not meet the preset conditions, the image pairs are stitched using the second stitching homography matrix corresponding to the image pairs in the same row in the previously stitched column, or a second initial homography matrix corresponding to the image row where the image pair is located is obtained, and the image pairs are stitched using the second initial homography matrix.

[0059] Optionally, when there is a previously stitched image column for the currently stitched image column, the second stitching homography matrix corresponding to the image pair in the same row in the second previously stitched column closest to the first column where the image pair is located can be first selected, and the stitching result can be further evaluated in a similar manner as described above. When the evaluated stitching result meets the preset conditions, the image pair is stitched; at this time, when, after evaluation, the stitching result using the second stitching homography matrix corresponding to the image pair in the same row in the second column still does not meet the preset conditions, the second stitching homography matrix corresponding to the closest previously stitched column can be selected in turn, and when the evaluated stitching result meets the preset conditions, the image pair is stitched. When the evaluation results of the second stitching homography matrices corresponding to all previously stitched columns do not meet the preset conditions, the second initial homography matrix corresponding to the image pair in the same row can be selected to stitch the image pair. That is, when there is no previously stitched column, the second initial homography matrix corresponding to the image pair in the same row is selected to stitch the image pair.

[0060] After the column stitching is completed, a complete M×N array stitching image can be obtained based on all the multiple images arranged in the M×N array, as well as the row stitching images corresponding to each row in the previous N rows and the column stitching images corresponding to each column in the M columns.

[0061] According to the above-mentioned image stitching method of the present invention, it is possible to select an appropriate homography matrix for stitching to stitch images by timely evaluating the stitching results and adaptively adjusting the homography matrix during the image stitching process, thereby reducing stitching errors and improving image stitching quality.

[0062] The following illustrates a specific implementation process of an image stitching method based on deep learning according to an example of an embodiment of the present invention.

[0063] In this example of the embodiment of the present invention, a camera array having N cameras may be firstly used to acquire a plurality of images arranged in an M×N array. Figure 6A schematic diagram showing an example camera array according to an embodiment of the present invention. Figure 7 FIG. 1 is a schematic diagram showing a plurality of images arranged in an M×N array according to an example of an embodiment of the present invention. Figure 6 As shown, in this example, a camera array with 4 cameras arranged in the longitudinal direction (second direction) can be set to capture images multiple times (7 times) in the transverse direction (first direction) to obtain the following images: Figure 7 The plurality of images shown are arranged in a 7×4 array, wherein the first direction and the second direction may be perpendicular to each other, and adjacent images in both the longitudinal and transverse directions may have at least 40% overlap.

[0064] Subsequently, a first initial homography matrix required for stitching corresponding to each image pair in a row during the row-wise stitching process of the multiple images along the first direction is obtained.

[0065] In this example, after acquiring multiple images arranged in a 7×4 array, a neural network can be used to obtain six first initial homography matrices that only represent the relative displacement between the images in each image pair in a row, for each of the six image pairs formed by the seven columns in each of the four rows.

[0066] Afterwards, the multiple images are stitched row by row, image pair by image pair, to obtain the first stitching homography matrix corresponding to each image pair. Among them, for the multiple images arranged in a 7×4 array, the image pair by image pair can be stitched for each of the 4 rows. When stitching the first row of the 4 rows, for example, for the image pair consisting of the first image and the second image in the row, the first stitching homography matrix of the image pair can be first calculated using a neural network, and the transformed image and / or fused image obtained using the first stitching homography matrix can be evaluated. When the stitching result of the evaluation does not meet the preset conditions, because there is no previously stitched image row at this time, the first initial homography matrix corresponding to the same column of the image pair can be used to stitch the image pair in the row. Subsequently, by analogy, the stitching of all 6 image pairs in the first row is completed.

[0067] Afterwards, the stitching can be continued for the second row of the four rows. When stitching the image pair consisting of the first image and the second image in the second row, it is also necessary to first calculate the corresponding first stitching homography matrix. When the stitching result evaluated in a similar manner to the above does not meet the preset conditions, the first stitching homography matrix corresponding to the same column of the first row that is closest to the second row and has been previously stitched can be selected and applied, and the evaluation can be continued. If the stitching result of the evaluation still does not meet the preset conditions, when there is no previously stitched row, the first initial homography matrix corresponding to the same column is used for stitching. And so on to complete the stitching of the six image pairs in the second row. Subsequently, the stitching of the image pairs of the remaining rows in the four rows can be continued in a similar manner, and a row stitching image corresponding to each row in the four rows can be generated.

[0068] In this example, the stitching result can be evaluated for the transformed image or fused image based on the image pair and the first stitching homography matrix. Optionally, in this example, whether the stitching result meets the preset conditions can be determined by respectively judging the shape of the transformed image and / or fused image (e.g., whether it is still rectangular), the peak signal-to-noise ratio (PSNR), and the structural similarity (SSIM) results. Figure 8 FIG. 1 is a schematic diagram showing a row-stitched image after row-stitching according to an example of an embodiment of the present invention. Figure 8 As shown, a row stitching image of each of the four cameras in the example of the embodiment of the present invention can be obtained.

[0069] Finally, the multiple images are further stitched longitudinally along the second direction by columns to obtain a stitched image. After completing the row stitching for the multiple images arranged in a 7×4 array, column stitching can be further performed for each of the seven columns in the multiple images using a method similar to the aforementioned row stitching, ultimately obtaining a complete stitched image.

[0070] Optionally, a neural network may first be used to obtain a second initial homography matrix required for stitching corresponding to each of the three image pairs in a column during the process of stitching the multiple images vertically by column. Subsequently, the multiple images may be similarly stitched image pair by image pair by column, and a second stitching homography matrix corresponding to each image pair may be obtained by using a neural network, wherein the stitching results of the image pairs are evaluated in the aforementioned manner based on the second stitching homography matrix corresponding to each image pair. When the stitching results do not meet the preset conditions, the image pairs are stitched using the second stitching homography matrix corresponding to the image pairs in the same column in the previously stitched column or the second initial homography matrix corresponding to the image pairs in the same column.

[0071] Figure 9A schematic diagram illustrating column-based stitching according to an example embodiment of the present invention is shown. In this example, after acquiring each row stitched image, the multiple images initially acquired in a 7×4 array and the row stitched image can be input separately, and then the column stitching calculation can be performed. Specifically, for each of the four rows in each of the seven columns, three second initial homography matrices representing only the relative displacement between the images in each image pair within a column can be acquired.

[0072] Then, the plurality of images are spliced ​​in pairs by columns to obtain a second splicing homography matrix corresponding to each image pair. For the plurality of images arranged in a 7×4 array, the splicing can be performed on each of the 7 columns. Figure 9 When stitching the first of the seven columns outlined in the middle, for example, for an image pair consisting of the first and second images in the column, a second stitching homography matrix for the image pair can be first calculated, and the transformed image and / or fused image obtained using the second stitching homography matrix can be evaluated. If the stitching result does not meet the preset conditions, because there is no previously stitched image column, the second initial homography matrix corresponding to the same row of the image pair can be used to stitch the image pair in the column. Subsequently, the stitching of all three image pairs in the first column is completed in this way.

[0073] Afterwards, the stitching can be continued for the second column in the 7 columns, for example, to the right of the first column. When stitching the image pair consisting of the first image and the second image in the second column, it is also necessary to first calculate the corresponding second stitching homography matrix. When the stitching result evaluated in a similar manner to the above does not meet the preset conditions, the second stitching homography matrix corresponding to the same row of the first column that is closest to the second column and has been previously stitched can be selected and applied first, and the evaluation can be continued. If the stitching result of the evaluation still does not meet the preset conditions, when there is no previously stitched column, the second initial homography matrix corresponding to the same row is used for stitching. And so on to complete the stitching of the three image pairs in the second column. Subsequently, the stitching of the image pairs of the remaining columns in the 7 columns can be continued in a similar manner, and a column stitching image corresponding to each column in the 7 columns can be generated.

[0074] After the column stitching is completed, a complete 7×4 array stitching image can be obtained based on all the multiple images arranged in a 7×4 array, the row stitching images corresponding to each of the previous 4 rows, and the column stitching images corresponding to each of the 7 columns.

[0075] Figure 10 A schematic diagram showing an example of a stitched image according to an embodiment of the present invention is shown.

[0076] Below, refer to Figure 11 The image stitching device according to an embodiment of the present invention will be described. Figure 11 FIG. 1 shows a block diagram of an image stitching device 1100 according to an embodiment of the present invention. Figure 11 As shown, the image stitching device 1100 includes an acquisition unit 1110 and a stitching unit 1120. In addition to these units, the image stitching device 1100 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the image stitching device 1100 according to the embodiment of the present invention are the same as those described above with reference to FIG. Figure 4 The details described are the same, so repeated description of the same details is omitted here to avoid repetition.

[0077] Figure 11 The acquisition unit 1110 of the image stitching device 1100 acquires a plurality of images arranged in an array, wherein the array of the plurality of images includes a plurality of image columns along a first direction and at least one image row along a second direction different from the first direction, and adjacent images arranged respectively along the first direction and the second direction have an overlap of at least a preset range.

[0078] According to an embodiment of the present invention, the plurality of images arranged in an array may have a plurality of image columns (e.g., M image columns) and may have one or more image rows (e.g., N image rows). The image columns of the plurality of images may be arranged along a first direction, and the image rows of the plurality of images may be arranged along a second direction, the first direction being different from the second direction. Optionally, the first direction may be perpendicular to the second direction. Furthermore, the first direction may optionally have a specific angle with the second direction, such as 30 degrees, 45 degrees, 60 degrees, 80 degrees, or other angles, which are not limited herein.

[0079] According to embodiments of the present invention, the multiple images arranged in an M×N array can be acquired using one or more cameras. In one example, a single camera can be used to acquire M images in each row and repeat this process for N rows of images to acquire the multiple images arranged in an M×N array. In another example, N cameras can be used to acquire N images in M ​​columns at a time to acquire the multiple images arranged in an M×N array. Of course, in yet another example, the multiple images arranged in an M×N array can also be acquired using various methods, such as a single camera acquiring images through a zigzag motion or a zigzag motion. Alternatively, in yet another example, the multiple images arranged in an M×N array can be acquired using N / a cameras to repeatedly acquire N / a images in M ​​columns a times, where a can be, for example, 2 or 3. The above-mentioned methods for acquiring multiple images arranged in an array are merely examples. In actual applications, different image acquisition methods can be adopted based on specific requirements, such as different application scenarios, camera parameter settings, and image size, to meet the needs of subsequent image stitching, and these are not limitations herein.

[0080] In an embodiment of the present invention, among the multiple images acquired and arranged in an M×N array, there may be at least a preset range of overlap between adjacent images in the first direction and the second direction. Optionally, the overlap of the preset range may be 40%. In addition, optionally, the overlap of the preset range may also be different values ​​such as 45%, 50%, etc. Further, the overlap of the preset range between adjacent images in the first direction and the second direction may be the same or different. For example, in the first direction, the overlap of the preset range may be 45%, while in the second direction, the overlap of the preset range may be 55%, etc. The specific limitation on the overlap of the preset range between adjacent images arranged in the first direction and the second direction is only an example. In actual applications, specific settings can be made according to different application scenarios and image acquisition requirements, and are not limited here.

[0081] The stitching unit 1120 obtains a first stitching homography matrix corresponding to each image pair during the process of stitching the multiple images row by row along the first direction, wherein the stitching results of the image pairs are evaluated respectively according to the first stitching homography matrix corresponding to each image pair. When the stitching result does not meet the preset conditions, the image pairs are stitched using the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched row, or the first initial homography matrix corresponding to the image column where the image pair is located is obtained, and the image pairs are stitched using the first initial homography matrix.

[0082] In an embodiment of the present invention, the stitching unit 1120 can stitch image pairs one by one for each row of N rows of a plurality of images arranged in an M×N array. Specifically, for each row of the plurality of images, the corresponding first stitching homography matrix for stitching can be calculated for each of the (M-1) image pairs formed in pairs, and stitching can be performed image pairs accordingly. That is, for each of the N rows of the plurality of images, first, the image pair consisting of the first image and the second image in the row can be stitched, that is, by calculating the first stitching homography matrix of the image pair, the second image is deformed for the fusion of the first image and the second image; then, the image pair consisting of the second image and the third image can be stitched in the same way, and so on, until the calculation of the first stitching homography matrix of all (M-1) image pairs in the row is completed. Optionally, in the process of calculating the first stitching homography matrix, a neural network can be used to calculate the first stitching homography matrix based on a deep learning method.

[0083] In an embodiment of the present invention, during the stitching process of each image pair, the stitching results of the image pairs can be evaluated based on the first stitching homography matrix corresponding to each image pair. When the stitching result of the image pair obtained by the evaluation meets a preset condition, it is determined that the first stitching homography matrix calculated above is used to apply to the image pair; and when the stitching result does not meet the preset condition, the first stitching homography matrix calculated above is not used, and the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched row is selected to stitch the image pair, or a first initial homography matrix corresponding to the image column where the image pair is located is obtained, and the image pair is stitched using the first initial homography matrix.

[0084] In an embodiment of the present invention, after acquiring multiple images arranged in an M×N array, the stitching unit 1120 may, for the above-mentioned row-by-row stitching process, acquire a first initial homography matrix required for stitching corresponding to each image pair in a row when necessary.

[0085] Optionally, the first initial homography matrix required for corresponding stitching can be determined for each of the (M-1) image pairs formed in each of the M columns in each of the N rows. That is, a first initial homography matrix can be determined for the image pair consisting of the first and second images in each row, and a first initial homography matrix can also be determined for the image pair consisting of the second and third images, and so on. Therefore, a total of (M-1) first initial homography matrices can be formed for the (M-1) image pairs formed in each of the N rows. Since the relative distance between multiple cameras in the same column in the camera array is fixed, the image pairs corresponding to the same image column can share the same first initial homography matrix. That is, the (M-1) first initial homography matrices can be determined only for the M image pairs in a specific row, while the image pairs in other rows can share the first initial homography matrix corresponding to the image column in which the image pair is located, among the (M-1) first initial homography matrices determined for the image pairs in that specific row.

[0086] When determining these first initial homography matrices, optionally, only the change in displacement between the two images in the image pair can be considered, without considering other relationships between adjacent images. Therefore, in one example, the first initial homography matrix representing the relative displacement can be obtained based on the relative displacement between the images in each image pair in a row. In one example, a neural network can be used to calculate the first initial homography matrix representing the relative displacement. Of course, the above-mentioned method of determining the first initial homography matrix is ​​only an example, and the first initial homography matrix can be determined in different ways according to actual applications, and there is no limitation here. In addition, the time for obtaining the first initial homography matrix can be before the acquisition of the image pairs or evaluation results used for row-by-row image pair stitching, or after the acquisition of the above-mentioned image pairs or evaluation results, and there is no limitation here. Further, if the splicing method of the first splicing homography matrix can be applied to each image pair when stitching by row, it is also possible to directly use each first splicing homography matrix to complete the splicing without obtaining the corresponding first initial homography matrix.

[0087] In the above operation of obtaining a first initial homography matrix representing the relative displacement between images in each image pair in a row, the following various exemplary methods may be used to calculate the relative displacement and / or determine the first initial homography matrix:

[0088] In one example, the relative displacement between images in a row can be determined based on the camera's movement distance when capturing images in each image pair. For example, if it is known that a device for controlling camera movement, equipped with a camera, moved 30 centimeters in a first direction while capturing both images in an image pair, then it can be determined that the camera also moved 30 centimeters in the first direction. Therefore, the relative displacement between the two images in the image pair can be determined to be 30 centimeters. Furthermore, if the relative displacement between images needs to be determined in units such as pixels, a correspondence between the camera's movement distance and the relative displacement between images expressed in units such as pixels can be pre-set to convert the camera's movement distance into a relative displacement in units such as pixels. For example, it can be assumed that for every 10 centimeters of camera movement in the first direction, the captured images will shift by 1000 pixels. Based on this correspondence, if the camera moved 50 centimeters in the first direction while capturing the image pair, the relative displacement between the images in the image pair can be calculated to be 1000 * 50 / 10 = 5000 pixels. Finally, a first initial homography matrix corresponding to the image may be obtained according to the relative displacement between the images in each image pair in a row.

[0089] In another example, as a special case of the above example, if the device for controlling the movement of the camera is equipped with a camera, and the distance of movement is always a fixed value in the process of capturing two images in an image pair in one or more rows, then the first initial homography matrix corresponding to a specific image pair can be obtained first, and applied to all image pairs in the row or rows. That is to say, when stitching images in the first direction, only one first initial homography matrix can be used for multiple image pairs in one or more rows. The specific image pair mentioned above can be the first image pair in a row or the middle image pair. The specific image pair can be selected and set arbitrarily without limitation.

[0090] In another example, the relative displacement can also be determined based on the overlap between pixels of one image in an image pair and pixels of the other image in the adjacent image pair. For example, if an image includes 5000 pixels in a first direction and the pixel overlap between one image and the other image in the image pair is 40%, the relative displacement between the images in the image pair can be calculated as 5000 * (1-40%) = 3000 pixels. Based on this, a first initial homography matrix corresponding to the image pair is obtained.

[0091] In another example, the first initial homography matrix used in stitching a previously stitched image can be applied to the stitching of other images. For example, in the process of stitching shelf images, if the acquisition method for each shelf image is the same, for example, the camera acquires multiple images of each shelf arranged in an M×N array for stitching, and if the stitching result of the image of a particular shelf is satisfactory, the first initial homography matrix used in stitching the image of that shelf can be applied to the stitching process of all subsequent shelf images.

[0092] Of course, in each of the above-mentioned examples, the various parameters used can be pre-set or automatically acquired. For example, the distance that the device used to control camera movement moves when capturing each adjacent image can be manually set in advance or automatically acquired from the device controlling camera movement via a wired or wireless network.

[0093] Optionally, the stitching unit 1120 may evaluate the stitching result of the image pair in a variety of ways. For example, the stitching result may be evaluated for a transformed image or a fused image based on the image pair and the first stitching homography matrix. The transformed image may be obtained by transforming one image in the image pair using the corresponding first stitching homography matrix, and the fused image may be obtained by fusing the two images in the image pair using the corresponding first stitching homography matrix. After obtaining the transformed image or the fused image, the stitching result may be evaluated for the transformed image and / or the fused image using at least one of the results of target detection, feature extraction, shape detection, structural similarity, and peak signal-to-noise ratio to determine whether the stitching result meets the preset conditions.

[0094] Optionally, when evaluating the transformed image and / or the fused image using target detection or feature extraction, the stitching result can be judged to determine whether it meets the preset conditions based on the specific application scenario of the embodiment of the present invention, for example, by using a pre-trained neural network to detect a specific pattern, target, or feature. When evaluating the transformed image and / or the fused image using shape detection, the shape of the transformed image and / or the fused image can be judged to determine whether the transformed image and / or the fused image meets the expected image shape (such as a rectangle, square, parallelogram, etc.) to determine whether the stitching result meets the preset conditions. When evaluating the transformed image and / or the fused image using structural similarity and / or peak signal-to-noise ratio, the structural similarity and / or peak signal-to-noise ratio parameters of the transformed image and / or the fused image can be judged to determine whether the stitching result meets the preset conditions. In addition, in other embodiments of the present invention, the stitching result can also be judged based on various aspects such as the relative positions of the corner points of the transformed image and / or the fused image, the scanning direction of the camera, and the transformation or fusion quality of the image. The various methods for evaluating the splicing results described above are merely examples. Different methods for evaluating the splicing results may be adopted according to actual application scenarios of the embodiments of the present invention, and are not limited here.

[0095] In an embodiment of the present invention, as described above, when the stitching result does not meet the preset conditions, the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched row or the first initial homography matrix corresponding to the image pair in the same column can be selected for stitching the image pair. Optionally, when there is an image row that has been stitched previously for the currently stitched image row, the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched second row closest to the first row where the image pair is located can be first selected, and the stitching result can be further evaluated in a similar manner as described above, and the image pair can be stitched when the evaluated stitching result meets the preset conditions; at this time, when, after evaluation, the stitching result using the first stitching homography matrix corresponding to the image pair in the same column in the second row still does not meet the preset conditions, the first stitching homography matrix corresponding to the closest previously stitched row can be selected in sequence by analogy, and the image pair can be stitched when the evaluated stitching result meets the preset conditions. When the evaluation results of the first stitching homography matrices corresponding to all previously stitched rows do not meet the preset conditions, the first initial homography matrix corresponding to the image pair in the same column may be selected to stitch the image pair. In other words, when the multiple images include one image row or do not have a previously stitched row, the first initial homography matrix corresponding to the image pair in the same column is selected to stitch the image pair.

[0096] In one example, when stitching the first row of N rows of a plurality of images arranged in an M×N array, for example, for an image pair consisting of the first image and the second image in the row, a first stitching homography matrix for the image pair may be calculated first, and a transformed image and / or fused image obtained using the first stitching homography matrix may be evaluated. If the stitching result does not meet a preset condition, because there is no previously stitched image row, the first initial homography matrix corresponding to the same column of the image pair may be used to stitch the image pair in the row. Subsequently, stitching of all image pairs in the first row may be completed in this manner.

[0097] Afterwards, stitching can be continued for the second row in N rows. When stitching the image pair consisting of the first image and the second image in the second row, it is also necessary to first calculate the corresponding first stitching homography matrix. When the stitching result evaluated in a similar manner as described above does not meet the preset conditions, the first stitching homography matrix corresponding to the same column of the first row that is closest to the second row and has been previously stitched can be first selected and applied, and the evaluation can be continued. If the stitching result still does not meet the preset conditions, if there is no previously stitched row, the first initial homography matrix corresponding to the same column is used for stitching. The stitching of the second row is completed in this way.

[0098] Subsequently, a similar method may be used to continue stitching the image pairs of the remaining rows in the N rows, and a row stitching image corresponding to each row in the N rows may be generated.

[0099] exist Figure 5 , an example of the specific steps of the splicing unit 1120 performing row-by-row image pair splicing along the first direction according to an embodiment of the present invention is shown in detail. Figure 5 As shown, when stitching multiple images arranged in an M×N array, in step S1, the stitching unit 1120 first calculates a first stitching homography matrix corresponding to a certain image pair in a certain row, and evaluates the stitching results such as the transformed image and / or fused image obtained using the first stitching homography matrix.

[0100] When the image pair in step S1 is the first image pair, i.e., n = 1, step S2 evaluates whether the stitching result of the nth image pair meets the preset conditions. If the result is yes, i.e., the stitching result meets the preset conditions, then the process proceeds directly to step S7 to determine whether the next image pair exists, i.e., whether the n+1th image pair exists. If the result is yes, n is assigned to n+1, and the evaluation step of step S2 is restarted. If the result is no, the process ends.

[0101] If the stitching result of the nth image pair is evaluated in step S2 to determine whether it meets the preset conditions, that is, the stitching result does not meet the preset conditions, then the process proceeds to step S3 to determine whether there are previously stitched image rows. If the result is no, that is, no previously stitched image rows exist, then the process proceeds to step S6 to select the first initial homography matrix corresponding to the image column in which the image pair is located and stitch the image pair.

[0102] If the judgment result in step S3 is yes, that is, there are image rows that have been previously spliced, then enter step S4, and select the first splicing homography matrix corresponding to the closest previously spliced ​​row in turn (for example, first select the first splicing homography matrix used for the spliced ​​n-1th row, and if the judgment result does not meet the preset conditions, then select the first splicing homography matrix used for the spliced ​​n-2th row, and so on), and splice the image pair when the evaluated splicing result meets the preset conditions.

[0103] After step S4, the process proceeds to step S5 to determine whether the stitching result meets the pre-set conditions. If the result in S5 is yes, the process proceeds directly to step S7 to determine whether the next image pair exists, that is, whether the n+1th image pair exists. If the result is yes, n is assigned to n+1 and the evaluation step in step S2 is restarted. If the result is no, the process ends.

[0104] If the judgment result in step S5 is no, that is, the evaluation results of the first stitching homography matrix corresponding to all previously stitched rows do not meet the preset conditions, then enter step S6, that is, select the first initial homography matrix corresponding to the image column where the image pair is located to stitch the image pair.

[0105] After all the image pairs in one row are stitched together, the image pairs in the remaining N rows can be stitched together in a similar manner, and a row stitching image corresponding to each row in the N rows can be generated.

[0106] Thereafter, when the plurality of images include a plurality of image rows, the plurality of images are further spliced ​​in columns along the second direction, and a spliced ​​image is acquired.

[0107] In an embodiment of the present invention, after completing row stitching for a plurality of images arranged in an M×N array, column stitching can be further completed for each of the M columns in the plurality of images in a manner similar to the aforementioned row stitching, and ultimately a complete stitched image can be obtained.

[0108] Optionally, during the process of stitching the multiple images column by column along the second direction, a second stitching homography matrix corresponding to each image pair may be obtained. The stitching results of the image pairs are evaluated based on the second stitching homography matrix corresponding to each image pair. When the stitching results do not meet preset conditions, the image pairs are stitched using the second stitching homography matrix corresponding to the image pair in the same row in the previously stitched column, or a second initial homography matrix corresponding to the image row containing the image pair is obtained and stitched using the second initial homography matrix.

[0109] Optionally, when there is a previously stitched image column for the currently stitched image column, the second stitching homography matrix corresponding to the image pair in the same row in the second previously stitched column closest to the first column where the image pair is located can be first selected, and the stitching result can be further evaluated in a similar manner as described above. When the evaluated stitching result meets the preset conditions, the image pair is stitched; at this time, when, after evaluation, the stitching result using the second stitching homography matrix corresponding to the image pair in the same row in the second column still does not meet the preset conditions, the second stitching homography matrix corresponding to the closest previously stitched column can be selected in turn, and when the evaluated stitching result meets the preset conditions, the image pair is stitched. When the evaluation results of the second stitching homography matrices corresponding to all previously stitched columns do not meet the preset conditions, the second initial homography matrix corresponding to the image pair in the same row can be selected to stitch the image pair. That is, when there is no previously stitched column, the second initial homography matrix corresponding to the image pair in the same row is selected to stitch the image pair.

[0110] After the column stitching is completed, a complete M×N array stitching image can be obtained based on all the multiple images arranged in the M×N array, as well as the row stitching images corresponding to each row in the previous N rows and the column stitching images corresponding to each column in the M columns.

[0111] According to the above-mentioned image stitching device of the present invention, it is possible to select an appropriate homography matrix for stitching to stitch images by timely evaluating the stitching results and adaptively adjusting the homography matrix during the image stitching process, thereby reducing stitching errors and improving image stitching quality.

[0112] Below, refer to Figure 12 The image stitching device according to an embodiment of the present invention will be described. Figure 12 FIG. 1 shows a block diagram of an image stitching device 1200 according to an embodiment of the present invention. Figure 12 As shown, the device 1200 may be a computer or a server.

[0113] like Figure 12As shown, the image stitching device 1200 includes one or more processors 1210 and a memory 1220. Of course, in addition to this, the image stitching device 1200 may also include an input device, an output device (not shown), etc. These components can be interconnected through a bus system and / or other forms of connection mechanisms. It should be noted that Figure 12 The components and structure of the image stitching device 1200 shown are merely exemplary and non-limiting. The image stitching device 1200 may also have other components and structures as needed.

[0114] The processor 1210 may be a central processing unit (CPU) or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 1220 to execute desired functions, which may include: acquiring a plurality of images arranged in an array, wherein the array of the plurality of images includes a plurality of image columns along a first direction, includes at least one image row along a second direction different from the first direction, and has at least a preset range of overlap between adjacent images arranged along the first direction and the second direction; in the process of stitching the plurality of images row by row along the first direction, obtaining Obtain a first stitching homography matrix corresponding to each image pair, wherein the stitching results of the image pairs are evaluated respectively according to the first stitching homography matrix corresponding to each image pair; when the stitching result does not meet a preset condition, the image pairs are stitched using the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched row, or a first initial homography matrix corresponding to the image column where the image pair is located is obtained, and the image pairs are stitched using the first initial homography matrix; and when the multiple images include multiple image rows, the multiple images are further stitched by column along a second direction, and a stitched image is obtained.

[0115] The memory 1220 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The computer-readable storage medium may store one or more computer program instructions, and the processor 1210 may execute these program instructions to implement the functions of the image stitching apparatus according to the embodiments of the present invention described above and / or other desired functions, and / or to execute the image stitching method according to the embodiments of the present invention. Various application programs and various data may also be stored in the computer-readable storage medium.

[0116] The following describes a computer-readable storage medium according to an embodiment of the present invention, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: acquiring a plurality of images arranged in an array, wherein the array of the plurality of images includes a plurality of image columns along a first direction, includes at least one image row along a second direction different from the first direction, and has at least a preset range of overlap between adjacent images arranged along the first direction and the second direction; in a process of stitching the plurality of images row by row along the first direction, acquiring a first image corresponding to each image pair. A stitching homography matrix, wherein the stitching results of the image pairs are evaluated respectively according to a first stitching homography matrix corresponding to each image pair; when the stitching result does not meet a preset condition, the image pairs are stitched using a first stitching homography matrix corresponding to an image pair in the same column in a previously stitched row, or a first initial homography matrix corresponding to an image column where the image pair is located is obtained and the image pairs are stitched using the first initial homography matrix; and when the multiple images include multiple image rows, the multiple images are further stitched by column along a second direction to obtain a stitched image.

[0117] Of course, the above-mentioned specific embodiments are merely examples and not limitations, and those skilled in the art can, based on the concept of the present invention, merge and combine some steps and devices from the various embodiments described separately above to achieve the effects of the present invention. Such merged and combined embodiments are also included in the present invention, and such merges and combinations are not described one by one here.

[0118] Note that the advantages, benefits, and effects mentioned in the present invention are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present invention. Furthermore, the specific details of the invention described above are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. The details do not necessarily limit the present invention to being implemented using the specific details.

[0119] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present invention are intended to be illustrative examples only and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems may be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and may be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and may be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and may be used interchangeably therewith.

[0120] The step flow charts and the above method descriptions in the present invention are intended to be illustrative examples only and are not intended to require or imply that the steps of the various embodiments must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter," "then," "next," and the like are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to an element in the singular, such as using the articles "a," "an," or "the," is not to be construed as limiting the element to the singular.

[0121] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, based on the concept of the present invention, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.

[0122] Each operation of the method described above may be performed by any suitable means capable of performing the corresponding functions, which may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs), or processors.

[0123] The various illustrated logic blocks, modules, and circuits may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but as an alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0124] The steps of the method or algorithm described in conjunction with the present invention can be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. The software module can exist in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative embodiment, the storage medium can be integral to the processor. A software module can be a single instruction or many instructions and can be distributed over several different code segments, between different programs, and across multiple storage media.

[0125] The methods herein include one or more actions for implementing the methods described. The methods and / or actions may be interchangeable with one another without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of the specific actions may be modified without departing from the scope of the claims.

[0126] The functions described can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage device or any other tangible medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer. As used herein, disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc.

[0127] Thus, a computer program product can perform the operations presented herein. For example, such a computer program product can be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, the instructions being executable by one or more processors to perform the operations described herein. The computer program product can include packaging materials.

[0128] Software or instructions may also be transmitted via a transmission medium. For example, software may be transmitted from a website, server, or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, or microwave.

[0129] In addition, the modules and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when appropriate. For example, such a device can be coupled to a server to facilitate the transmission of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a CD or floppy disk) so that the user terminal and / or base station can obtain the various methods when being coupled to the device or providing a storage component to the device. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.

[0130] Other examples and implementations are within the scope and spirit of the invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hardwiring, or any combination of these. Features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, "or" used in a list of items that begin with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). In addition, the wording "exemplary" does not mean that the example described is preferred or better than other examples.

[0131] Various changes, substitutions, and modifications may be made to the technology described herein without departing from the teachings as defined by the appended claims. Moreover, the scope of the claims is not limited to the specific aspects of the processes, machines, manufacture, compositions of things, means, methods, and actions described above. Currently existing or later developed processes, machines, manufacture, compositions of things, means, methods, or actions that perform substantially the same function or achieve substantially the same results as the corresponding aspects described herein may be utilized. Accordingly, the appended claims include within their scope such processes, machines, manufacture, compositions of things, means, methods, or actions.

[0132] The above description of the invented aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features of the invention herein.

[0133] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the form of the invention. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations.

Claims

1. An image stitching method, comprising: Acquiring a plurality of images arranged in an array, wherein the array includes a plurality of image columns along a first direction and at least one image row along a second direction different from the first direction, and adjacent images arranged along the first direction and the second direction respectively overlap by at least a predetermined range; In a process of stitching the multiple images row by row along a first direction, obtaining a first stitching homography matrix corresponding to each image pair, wherein stitching results of the image pairs are evaluated respectively based on the first stitching homography matrix corresponding to each image pair; when the stitching result does not meet a preset condition, stitching the image pairs using a first stitching homography matrix corresponding to an image pair in the same column in a previously stitched row, or obtaining a first initial homography matrix corresponding to an image column in which the image pair is located, and stitching the image pairs using the first initial homography matrix; and When the multiple images include multiple image rows, the multiple images are further spliced ​​in columns along the second direction, and a spliced ​​image is acquired.

2. The method according to claim 1, wherein Acquiring multiple images arranged in an array involves: The plurality of images are acquired by moving along the first direction using one camera or using a camera group including a plurality of cameras arranged along the second direction.

3. The method according to claim 1, wherein Acquiring a first initial homography matrix corresponding to the image column where the image pair is located includes: According to the relative displacement between images in each image pair in a row, a first initial homography matrix corresponding to the image column where the image pair is located and representing the relative displacement is obtained.

4. The method according to claim 1, wherein When the stitching result does not meet the preset condition, stitching the image pair using a first stitching homography matrix corresponding to an image pair in the same column in a previously stitched row, or obtaining a first initial homography matrix corresponding to an image column where the image pair is located, and stitching the image pair using the first initial homography matrix includes: When the plurality of images include a plurality of image rows and the rows have been previously spliced, selecting a first splicing homography matrix corresponding to the image pair in the same column in the second previously spliced ​​row closest to the first row where the image pair is located, further evaluating a splicing result, and splicing the image pair when the evaluated splicing result meets a preset condition; When the further evaluated stitching result does not meet the preset condition, sequentially selecting the first stitching homography matrix corresponding to the closest previously stitched row, and stitching the image pair when the evaluated stitching result meets the preset condition; When the evaluation results of the first stitching homography matrices corresponding to all previously stitched rows do not meet the preset conditions, a first initial homography matrix corresponding to the image column where the image pair is located is selected to stitch the image pair.

5. The method according to claim 1, wherein If the stitching result does not meet the preset condition, stitching the image pair using a first stitching homography matrix corresponding to an image pair in the same column in a previously stitched row, or obtaining a first initial homography matrix corresponding to an image column where the image pair is located, and stitching the image pair using the first initial homography matrix includes: When the plurality of images include one image row or do not have a previously spliced ​​row, a first initial homography matrix corresponding to the image column where the image pair is located is selected to splice the image pair.

6. The method of claim 1, wherein: According to the first stitching homography matrix corresponding to each image pair, the stitching results of the image pairs are evaluated separately, including: transforming one image in the image pair according to a first stitching homography corresponding to the image pair to obtain a transformed image, or fusing two images in the image pair to obtain a fused image; The transformed image or the fused image is evaluated for a stitching result using at least one result of target detection, feature extraction, shape detection, structural similarity, and peak signal-to-noise ratio to determine whether the stitching result meets a preset condition.

7. The method of claim 1, wherein: When the multiple images include multiple image rows, further splicing the multiple images by columns along the second direction and acquiring the spliced ​​image includes: In the process of stitching the multiple images image pair by image pair along the second direction by column, a second stitching homography matrix corresponding to each image pair is obtained, wherein the stitching results of the image pairs are evaluated respectively according to the second stitching homography matrix corresponding to each image pair. When the stitching result does not meet the preset conditions, the image pairs are stitched using the second stitching homography matrix corresponding to the image pair in the same row in the previously stitched column, or a second initial homography matrix corresponding to the image row where the image pair is located is obtained, and the image pairs are stitched using the second initial homography matrix.

8. An image stitching device, comprising: an acquisition unit configured to acquire a plurality of images arranged in an array, wherein the array of the plurality of images includes a plurality of image columns along a first direction and at least one image row along a second direction different from the first direction, and adjacent images arranged along the first direction and the second direction respectively overlap by at least a preset range; The stitching unit is configured to obtain a first stitching homography matrix corresponding to each image pair during the process of stitching the multiple images row by row along a first direction, wherein the stitching results of the image pairs are evaluated respectively according to the first stitching homography matrix corresponding to each image pair; when the stitching result does not meet the preset condition, the image pairs are stitched using the first stitching homography matrix corresponding to the image pair in the same column in the previously stitched row, or a first initial homography matrix corresponding to the image column where the image pair is located is obtained, and the image pairs are stitched using the first initial homography matrix; and when the multiple images include multiple image rows, the multiple images are further stitched column by column along a second direction to obtain a stitched image.

9. An image stitching device, comprising: processor; and a memory having computer program instructions stored therein, When the computer program instructions are executed by the processor, the processor is caused to perform the following steps: Acquiring a plurality of images arranged in an array, wherein the array includes a plurality of image columns along a first direction and at least one image row along a second direction different from the first direction, and adjacent images arranged along the first direction and the second direction respectively overlap by at least a predetermined range; In a process of stitching the multiple images row by row along a first direction, obtaining a first stitching homography matrix corresponding to each image pair, wherein stitching results of the image pairs are evaluated respectively based on the first stitching homography matrix corresponding to each image pair; when the stitching result does not meet a preset condition, stitching the image pairs using a first stitching homography matrix corresponding to an image pair in the same column in a previously stitched row, or obtaining a first initial homography matrix corresponding to an image column in which the image pair is located, and stitching the image pairs using the first initial homography matrix; and When the multiple images include multiple image rows, the multiple images are further spliced ​​in columns along the second direction, and a spliced ​​image is acquired.

10. A computer-readable storage medium having computer program instructions stored thereon, wherein: When the computer program instructions are executed by a processor, the following steps are implemented: Acquiring a plurality of images arranged in an array, wherein the array includes a plurality of image columns along a first direction and at least one image row along a second direction different from the first direction, and adjacent images arranged along the first direction and the second direction respectively overlap by at least a predetermined range; In a process of stitching the multiple images row by row along a first direction, obtaining a first stitching homography matrix corresponding to each image pair, wherein stitching results of the image pairs are evaluated respectively based on the first stitching homography matrix corresponding to each image pair; when the stitching result does not meet a preset condition, stitching the image pairs using a first stitching homography matrix corresponding to an image pair in the same column in a previously stitched row, or obtaining a first initial homography matrix corresponding to an image column in which the image pair is located, and stitching the image pairs using the first initial homography matrix; and When the multiple images include multiple image rows, the multiple images are further spliced ​​in columns along the second direction, and a spliced ​​image is acquired.

Images