Image stitching method and apparatus, image processing method, and readable storage medium

Abstract

Disclosed in the present invention are an image stitching method and apparatus, an image processing method, and a readable storage medium. The image stitching method comprises: according to a preset arrangement sequence, performing row arrangement and column arrangement of a plurality of unit images to be stitched; determining one of a row arrangement direction and a column arrangement direction as a first direction, and the other one as a second direction; determining seams between adjacent images among the plurality of unit images in the first direction, and stitching adjacent unit images according to the seams, so as to obtain a plurality of intermediate images; and determining seams between adjacent images among the plurality of intermediate images in the second direction, and stitching adjacent intermediate images according to the seams, so as to obtain a result image. In the present invention, images are stitched in two-dimensional directions by means of generating a seam network, thereby avoiding the problems such as artifacts, misalignments, overlapping and blank spaces that are generated during image stitching.

Description

Image stitching method, image processing method, device and readable storage medium Technical Field

[0001] The present application relates to the field of graphics processing technology, and in particular to an image stitching method, an image processing method, a device, and a readable storage medium. Background Art

[0002] When users observe samples under a high-magnification lens, the field of view is limited, so they can only see part of the sample. Using a camera and an automated mobile platform to scan the sample and using computer imaging technology to stitch multiple high-resolution, small-field images into a large-field image, users can see the entire sample.

[0003] Typical technologies for stitching multiple small-field-of-view images into a large-field-of-view image include Chinese patent application publication number CN110337386A, which discloses a rear-stitching panoramic view technology for rear-view visualization. Specifically, the technology uses multiple on-board cameras to capture images and stitches them together along fixed straight lines to form a panoramic image. However, stitching along straight lines can cause image misalignment because there's no guarantee that image features will be segmented along straight lines, and there's a high probability that image features won't be segmented along straight lines.

[0004] When stitching multiple images, one feasible method is to sequentially find the seams between adjacent images to form a seam network. However, the seam network may be discontinuous at the seam intersection, resulting in unnatural and unreasonable overlaps or holes in the fusion area between the stitched adjacent images, as shown in Figure 3.

[0005] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application, nor does it necessarily provide technical guidance. In the absence of clear evidence that the above content has been disclosed before the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of this application.

[0006] Summary of the Invention

[0007] The purpose of the present invention is to provide an image stitching method, an image processing method, an apparatus and a readable storage medium, which can find a seam network in the overlapping area between adjacent images in rows and columns that minimizes the pixel difference of the stitched image, so that the stitching transition between images is natural.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] An image stitching method comprises the following steps:

[0010] Arranging the multiple unit images to be spliced ​​in rows and columns according to a preset arrangement order;

[0011] determining one of the row arrangement direction and the column arrangement direction as a first direction and the other as a second direction;

[0012] determining seams between adjacent images in a plurality of unit images in a first direction, and splicing the adjacent unit images according to the seams to obtain a plurality of intermediate images;

[0013] Determine seams between adjacent images in the plurality of intermediate images in the second direction, and splice the adjacent intermediate images according to the seams to obtain a result image.

[0014] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, the step of determining the seams between adjacent images includes:

[0015] Determine the overlapping area between two adjacent images;

[0016] A seam consisting of a plurality of pixel units is found in the overlapping area so that the pixel difference between two adjacent images on a path corresponding to the seam is minimized.

[0017] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, the overlapping area is determined by the following method:

[0018] The image positioning algorithm is used to locate the position of each unit image in the global image, and the overlapping area between two adjacent images is determined based on the located position.

[0019] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, the unit images are obtained by scanning with a scanning device, and the overlapping area is determined by the following method:

[0020] According to the preset scanning configuration, the position of each unit image in the global image is obtained, and then the overlapping area between two adjacent images is determined.

[0021] Further, based on any one of the above technical solutions or a combination of multiple technical solutions, the step of finding the seam in the overlapping area includes:

[0022] For each pixel unit in the overlapping area, the difference of the pixel unit is calculated based on the image data of the two adjacent images in the pixel unit, and a difference table corresponding to the overlapping area is obtained;

[0023] Planning a path consisting of adjacent pixel units based on the difference table, and ensuring that the sum of the differences of the pixel units on the path is minimal;

[0024] The planned path is mapped to the overlapping area to obtain the seam.

[0025] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, the difference of a pixel unit is calculated by the following method:

[0026] Acquire the R / G / B value of the channel data of the adjacent first image at the pixel unit and the R / G / B value of the channel data of the adjacent another image at the pixel unit;

[0027] respectively calculating an R value difference, a G value difference, and a B value difference between the first image and another image at the pixel unit;

[0028] The degree of difference of the pixel unit is calculated according to the R value difference, the G value difference and the B value difference.

[0029] Furthermore, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the path is planned in the following manner:

[0030] Starting from the first row or the first column of the difference table, and according to the extension direction of the seam, calculating the difference cumulative value corresponding to each unit in the difference table row by row or column by column;

[0031] Determine the unit with the smallest cumulative difference value in the last row or last column as the starting point for backtracking;

[0032] The path formed by backtracking from the backtracking starting point along the extension direction of the seam in the difference degree table is searched as the planned path.

[0033] Furthermore, based on any one of the above-mentioned technical solutions or a combination of multiple technical solutions, the method of backtracking from the backtracking starting point to search for the path formed in the difference table is:

[0034] The backtracking starting point is the unit with the smallest cumulative difference value in the last row; with the backtracking starting point as the first path unit, the path units of each row are determined row by row from the last row to the first row, wherein the path unit of each row is the unit with the smallest cumulative difference value among two or more units adjacent to the path unit of the previous row;

[0035] Alternatively, the backtracking starting point is the unit with the smallest cumulative difference value in the last column; with the backtracking starting point as the first path unit, the path units of each column are determined column by column from the last column to the first column, wherein the path unit of each column is a unit with the smallest cumulative difference value among two or more units adjacent to the path unit of the previous column.

[0036] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, a graph cut algorithm is used to find the seam in the overlapping area, and the steps include:

[0037] Represent the image of the overlapping area as a graph, where the nodes of the graph represent pixels in the image and the edges of the graph represent the relationship between pixels;

[0038] Assign weights to the nodes and edges of the graph;

[0039] A cut is found on the graph by minimizing a cost function of the cut, and the graph is divided into two parts, the foreground and the background. The cost function is composed of factors including similarity between pixels.

[0040] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, the Voronoi diagram method is used to find the seam in the overlapping area, and the steps include:

[0041] Generate a Voronoi diagram for each unit image to be spliced;

[0042] The seam positions in the overlapping area are determined by stitching the line network according to the Voronoi diagram.

[0043] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, the step of splicing the adjacent first image and the second image according to the seam includes:

[0044] Generate masks for the first image and the second image based on the seam, wherein the mask corresponding to each pixel unit on the side of the seam away from the second image in the first image is 1, and the masks corresponding to the remaining pixel units in the first image are 0; and the mask corresponding to each pixel unit on the side of the seam away from the first image in the second image is 1, and the masks corresponding to the remaining pixel units in the second image are 0;

[0045] A bitwise AND logic operation is performed between the image of the pixel unit and the corresponding mask, and the results of the logic operation are arranged according to the positional relationship of each unit image to obtain a spliced ​​image.

[0046] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, the mask at the joint of the first image and the second image is determined by the following method:

[0047] A pixel unit where the seam is located has a mask of 1 in one of the first image and the second image, and a mask of 0 in the other image;

[0048] Alternatively, the masks corresponding to the pixel units where the seams in the first image and the second image are located are all 1.

[0049] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, the first direction and the second direction are determined by the following method:

[0050] Calculating the number of image columns × the pixel value of a single image column for the unit images to be spliced ​​to obtain a first product, and calculating the number of image rows × the pixel value of a single image row for the unit images to be spliced ​​to obtain a second product;

[0051] If the first product is smaller than the second product, determining the column arrangement direction as the first direction and determining the row arrangement direction as the second direction;

[0052] If the second product is smaller than the first product, the row arrangement direction is determined to be the first direction, and the column arrangement direction is determined to be the second direction.

[0053] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, the plurality of unit images to be spliced ​​are acquired by scanning an object with a scanning device in a preset order;

[0054] The unit images are arranged in rows and columns according to the scanned row and column order, and the multiple unit images to be spliced ​​are arranged in more than two rows and more than two columns.

[0055] According to another aspect of the present invention, there is provided an image processing method, comprising the following steps:

[0056] Scanning and photographing the object to be imaged to obtain multiple sub-images;

[0057] Arranging the multiple sub-images in rows and columns according to a scanning order to obtain a preview image;

[0058] Determining a region of interest based on the preview image, and determining each sub-image corresponding to the region of interest as a unit image to be stitched;

[0059] The unit images to be stitched are stitched using the image stitching method described in any one of the above technical solutions or a combination of multiple technical solutions.

[0060] According to another aspect of the present invention, an image stitching device is provided, comprising the following modules:

[0061] an arrangement module configured to arrange the plurality of unit images to be spliced ​​in rows and columns according to a preset arrangement order;

[0062] a sequence setting module configured to determine one of the row arrangement direction and the column arrangement direction as a first direction and the other as a second direction;

[0063] a first sequential stitching module configured to determine seams between adjacent images in a plurality of unit images in a first direction, and stitch the adjacent unit images according to the seams to obtain a plurality of intermediate images;

[0064] The second sequential stitching module is configured to determine seams between adjacent images in the plurality of intermediate images in the second direction, and stitch the adjacent intermediate images according to the seams to obtain a result image.

[0065] Furthermore, based on any one of the above technical solutions or a combination of multiple technical solutions, the first sequential splicing module and the second sequential splicing module each include the following units:

[0066] an overlapping unit configured to determine an overlapping area between two adjacent images;

[0067] The seam finding unit is configured to find a seam composed of a plurality of pixel units in the overlapping area so that the difference in pixel units between two adjacent images on a path corresponding to the seam is minimized.

[0068] Furthermore, any one of the above-mentioned technical solutions or a combination of multiple technical solutions further includes the following modules:

[0069] a scanning module configured to scan and photograph the object to be imaged to obtain a plurality of sub-images;

[0070] a preview module configured to arrange the plurality of sub-images in rows and columns according to a scanning order to obtain a preview image;

[0071] The region of interest module is configured to determine a region of interest according to the preview image, and determine each sub-image corresponding to the region of interest as a unit image to be spliced.

[0072] According to another aspect of the present invention, a microscopic imaging system is provided, comprising a microscope and an image stitching device as described in any one of the above technical solutions or a combination of multiple technical solutions.

[0073] According to another aspect of the present invention, an electronic device is provided, comprising a processor and a memory, wherein the memory is used to store program instructions, and the processor is configured to run the program instructions, and the program instructions are run to perform the steps of the method described in any one of the above technical solutions or a combination of multiple technical solutions.

[0074] According to another aspect of the present invention, a computer-readable storage medium is provided for storing program instructions, wherein the program instructions are configured to call and execute the steps of the method described in any one of the above technical solutions or a combination of multiple technical solutions.

[0075] According to another aspect of the present invention, a computer program product is provided, comprising a readable and stored computer program, wherein the computer program comprises program instructions. When the program instructions are executed on a computer device, the computer device executes the steps of the method described in any one of the above technical solutions or a combination of multiple technical solutions.

[0076] The beneficial effects brought about by the technical solution provided by the present invention are as follows:

[0077] a. The image stitching method provided by the present invention arranges multiple unit images to be stitched in rows and columns, first stitching multiple images in each row (or column) into an intermediate image, and then stitching the multiple intermediate images in the column direction (or row direction) to obtain a result image. The resulting seam network avoids discontinuities, overlaps, or blanks in the seam intersection area, effectively improving the image stitching quality.

[0078] b. The present invention calculates the number of columns of the unit images to be stitched × the image column pixel value and the number of rows × the image row pixel value, and uses the row / column arrangement direction with the smaller product as the first direction and the other as the second direction. Determining the first and second directions using this method can reduce computational workload and save image stitching time.

[0079] c. This invention proposes a method that determines a dissimilarity table for the overlapping regions and calculates the cumulative dissimilarity value of each cell row or column by row based on the dissimilarity table. Using the cell with the smallest cumulative dissimilarity value in the last row or column as the starting point, a path consisting of adjacent pixel cells is planned, ensuring that the sum of the dissimilarity values ​​of the pixel cells along the path is minimized. This method maximizes the preservation of information within each image and is unaffected by the fusion algorithm.

[0080] d. The image stitching device provided by the present invention can quickly stitch together scanned images to obtain high-quality panoramic images with natural stitching transitions, which is more convenient for users to observe and is particularly suitable for stitching microscopic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technical solutions, the drawings required for use in the description of the embodiments or the conventional technical solutions will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0082] FIG1 is a schematic flow chart of an image stitching method provided by an exemplary embodiment of the present invention;

[0083] FIG2 is a schematic diagram of an abstract process of image stitching provided by an exemplary embodiment of the present invention;

[0084] FIG3 is a diagram showing the result of image stitching by finding the seams of four adjacent images, top, bottom, left, and right, and the corresponding seam network;

[0085] FIG4 is a schematic diagram of a result image and a corresponding seam network obtained by using the image stitching method provided by the present invention;

[0086] FIG5 is a schematic diagram of a preview image obtained by arranging and splicing the scanned images in accordance with the scanning order;

[0087] FIG6 is a result diagram obtained by stitching the multiple images in FIG5 using the image stitching method provided by the present invention;

[0088] FIG7 is a flow chart of an image processing method provided by an exemplary embodiment of the present invention;

[0089] FIG8 is a module diagram of an image stitching device provided by an exemplary embodiment of the present invention;

[0090] FIG9 is a schematic diagram of a system 100 for executing the method described in an embodiment of the present invention.

[0091] The reference numerals include: 100 - system, 110 - microscope, 120 - computer system. DETAILED DESCRIPTION

[0092] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0093] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0094] In one embodiment of the present invention, referring to FIG1 and FIG2 , a method for image stitching is provided, comprising the following steps:

[0095] Arranging the multiple unit images to be spliced ​​in rows and columns according to a preset arrangement order;

[0096] Determine one of the row arrangement direction and the column arrangement direction as the first direction, and the other as the second direction; for example, in FIG2 , the column direction is the first direction, and the row direction is the second direction;

[0097] Determine seams between adjacent images in a plurality of unit images in a first direction, and splice the adjacent unit images according to the seams to obtain a plurality of intermediate images. In the example shown in FIG2 , the number of intermediate images is two.

[0098] The seams between adjacent images in the plurality of intermediate images in the second direction are determined, and the adjacent intermediate images are spliced ​​according to the seams to obtain a result image. The image at the bottom right of FIG2 is the obtained result image.

[0099] In this embodiment, the plurality of unit images to be spliced ​​are arranged in more than two rows and more than two columns. The plurality of unit images to be spliced ​​can be obtained by scanning an object with a scanning device in a preset order, or can be obtained by other means.

[0100] In one embodiment, an image positioning algorithm is used to locate the global position of each unit image. Specifically, this includes, but is not limited to, using an image feature matching algorithm to calculate the relative position between two adjacent images, employing a position fusion strategy to determine the relative position between adjacent images, or determining the global position of a single image based on arrangement priors, position estimation based on relative position information of fused images, and other methods. Based on the global positions of the individual unit images obtained by positioning, the multiple unit images to be spliced ​​are arranged in rows and columns, and the overlapping area between the two adjacent images is determined based on the positioned positions.

[0101] In one embodiment, multiple unit images can be obtained by scanning an object in a preset order through a scanning device, and the global position of each unit image can be obtained according to the scanning configuration. For example, the scanning configuration has a length and width pixel size of 1920×1080, the translation distance of the scanning device in the length direction is set to 1850 pixels, and the mechanical error is -10 pixels (indicating that the actual movement distance will be 10 pixels less than the set target). Assuming that the scanning device does not deviate in the width direction when moving along the length direction, the overlapping area of ​​the two images before and after the translation is a rectangle, and the overlapping area occupies 80 pixels in the length direction and 1080 pixels in the width direction. The unit images are arranged in the length direction according to this overlapping area, and the arrangement in the width direction is the same, which will not be repeated.

[0102] In this embodiment, one of the row arrangement direction and the column arrangement direction is determined as the first direction, and the other as the second direction, in the following manner: for the unit images to be stitched, the number of image columns multiplied by the pixel values ​​of the individual image columns is calculated to obtain a first product, and for the unit images to be stitched, the number of image rows multiplied by the pixel values ​​of the individual image rows is calculated to obtain a second product. If the first product is less than the second product, the column arrangement direction is determined to be the first direction, and the row arrangement direction is determined to be the second direction. If the second product is less than the first product, the row arrangement direction is determined to be the first direction, and the column arrangement direction is determined to be the second direction. Determining the first and second directions using this method can reduce the computational complexity of image stitching and save image stitching time.

[0103] Furthermore, based on the determined overlapping region between two adjacent images, the seams between adjacent images in the plurality of unit images in the first direction are determined. A seam consisting of a plurality of pixel units is found within the overlapping region, such that the pixel difference between the two adjacent images along the path corresponding to the seam is minimized. The adjacent unit images are then spliced ​​along the seams to obtain a plurality of intermediate images. The seams between adjacent intermediate images in the second direction are then found, also following the principle of minimizing the pixel difference between the two adjacent images along the path corresponding to the seam, to ensure the most natural transition in the spliced ​​images.

[0104] The step of finding the seam in the overlapping area includes:

[0105] For each pixel unit in the overlapping area, the pixel difference of the pixel unit is calculated based on the image data of the two adjacent images in the pixel unit. In this embodiment, the adjacent images P1 and P2 have a common pixel unit U in the overlapping area. The channel data of image P1 in pixel unit U is R1, G1, and B1, and the channel data of image P2 in pixel unit U is R2, G2, and B2. The pixel difference of pixel unit U is calculated as: (R1-R2) 2+(G1-G2) 2 +(B1-B2) 2 .

[0106] The above calculation is performed on each pixel unit in the overlapping area one by one to obtain a pixel difference table corresponding to the overlapping area. Taking a rectangular overlapping area A as an example, the pixel difference table corresponding to the overlapping area A is represented as Table 1.

[0107] Table 1 Pixel difference table corresponding to overlapping area A

[0108] It should be noted that the present application is not limited to the above-mentioned method of calculating the difference, and other reasonable methods of calculating the difference also fall within the scope of protection of the present application. For example, the sum of the absolute value of the R value difference, the absolute value of the G value difference, and the absolute value of the B value difference can also be used for calculation. In a specific embodiment, before calculating the difference value, it is not ruled out that the source image is first processed, including but not limited to processing the image using color space conversion, convolution, etc., and then calculating the pixel difference value for the processed image. If the processed image contains only single-channel data, the current single-channel data can be used to calculate the pixel difference value, such as calculating the absolute value of the difference between the channel data of an overlapping pixel unit in the two images as the pixel difference value.

[0109] Furthermore, a path consisting of adjacent pixel units is planned based on the difference table, and the sum of pixel differences of the pixel units on the path is minimized. The planned path is mapped to the overlapping area to obtain the seam.

[0110] In this embodiment, the following path planning method is provided:

[0111] Starting from the first row or the first column of the difference table, and according to the extension direction of the seam, calculating the difference cumulative value corresponding to each unit in the difference table row by row or column by column;

[0112] Determine the unit with the smallest cumulative difference value in the last row or last column as the starting point for backtracking;

[0113] The path formed by backtracking from the backtracking starting point along the extension direction of the seam in the difference degree table is searched as the planned path.

[0114] There are several ways to calculate the cumulative difference value corresponding to each unit:

[0115] The first method is to calculate the cumulative difference value corresponding to each unit from top to bottom row by row using the following formula:

[0116] sum[i,j]=e[i,j]+min(sum[i-1,j-1],sum[i-1,j],sum[i-1,j+1]), where sum[i,j] is the cumulative difference value corresponding to the unit in the i-th row and j-th column, e[i,j] is the difference value of the unit in the i-th row and j-th column, min() is the minimum function, and the cumulative difference value corresponding to each unit in the top row is equal to the difference value of the pixel unit itself; this method is used to calculate the cumulative difference value corresponding to each pixel unit in Table 1 to obtain its corresponding cumulative difference value table as shown in Table 2.

[0117] Table 2 The cumulative difference value corresponding to the overlapping area A

[0118] The second method is to calculate the cumulative difference value corresponding to each unit row by row from bottom to top using the following formula:

[0119] sum[i,j]=e[i,j]+min(sum[i+1,j-1],sum[i+1,j],sum[i+1,j+1]), where sum[i,j] is the cumulative difference value corresponding to the cell in the i-th row and j-th column, e[i,j] is the difference value of the cell in the i-th row and j-th column, and min() is the minimum function. The cumulative difference value corresponding to each pixel cell in the first row at the bottom is equal to the difference value of the pixel cell itself.

[0120] The third method is to calculate the cumulative difference value corresponding to each unit from left to right column by column using the following formula:

[0121] sum[i,j]=e[i,j]+min(sum[i-1,j-1],sum[i,j-1],sum[i+1,j-1]), where sum[i,j] is the cumulative difference value corresponding to the cell in the i-th row and j-th column, e[i,j] is the difference value of the cell in the i-th row and j-th column, and min() is the minimum function. The cumulative difference value corresponding to each pixel cell in the leftmost column is equal to the difference value of the pixel cell itself.

[0122] The fourth method is to calculate the cumulative difference value corresponding to each unit from right to left column by column using the following formula:

[0123] sum[i,j]=e[i,j]+min(sum[i-1,j+1],sum[i,j+1],sum[i+1,j+1]), where sum[i,j] is the cumulative difference value corresponding to the cell in the i-th row and j-th column, e[i,j] is the difference value of the cell in the i-th row and j-th column, min() is the minimum function, and the cumulative difference value corresponding to each pixel cell in the rightmost column is equal to the difference value of the pixel cell itself.

[0124] Taking the difference cumulative value table corresponding to the overlapping area determined by the first method above (i.e., Table 2) as an example, the path planning method is introduced as follows: the unit with the smallest difference cumulative value in its last row is used as the backtracking starting point, and sum[10,2]=495 in the last row of Table 2 is determined as the backtracking starting point.

[0125] After determining the backtracking starting point, backtrack along the extension direction of the seam from the backtracking starting point to find the path formed by the seam in the difference degree table, and use it as the planned path. The method of backtracking from the backtracking starting point to find the path formed by the seam in the difference degree table includes but is not limited to the following methods:

[0126] For example, the backtracking starting point is the unit with the smallest cumulative difference value in the last row; with the backtracking starting point as the first path unit, the path units of each row are determined row by row from the last row to the first row, wherein the path unit of each row is the unit with the smallest cumulative difference value among two or more units adjacent to the path unit of the previous row; according to this method, the planned path of the overlapping area A is determined starting from the backtracking starting point sum[10,2] in Table 2 as shown in the table cells marked with * in Table 3:

[0127] Table 3 Planning paths of overlapping area A

[0128] The other path planning methods from bottom to top, from left to right, and from right to left are similar and will not be described in detail.

[0129] It should be noted that the path planning method provided in the above embodiment, which backtracks from the end point, is only one feasible solution. The above method of path planning can greatly save the time of determining the planned path. However, the present invention is not limited to the above method of determining the path that satisfies the minimum sum of pixel differences of each pixel unit on the path. For example, all paths can also be enumerated, the sum of the differences of each pixel unit on each path is calculated, and the path with the minimum sum of differences can be determined.

[0130] In another embodiment of the present invention, unlike the above embodiment in which the seams are found in the overlapping area by dynamic programming, this embodiment uses a graph cut algorithm to find the seams in the overlapping area, and the steps include:

[0131] Represent the image of the overlapping area as a graph, where the nodes of the graph represent pixels in the image and the edges of the graph represent the relationship between pixels;

[0132] Assign weights to the nodes and edges of the graph;

[0133] A cut (ie, a seam) is found on the image by minimizing a cost function of the cut, and the image is divided into two parts, the foreground and the background. The constituent factors of the cost function include the similarity between pixels.

[0134] In another embodiment of the present invention, the Voronoi diagram method is used to find the seam in the overlapping area, and the steps include:

[0135] Generate a Voronoi diagram for each unit image to be spliced;

[0136] The seam positions in the overlapping area are determined by stitching the line network according to the Voronoi diagram.

[0137] After determining the seam, the adjacent first image and the second image are spliced ​​according to the seam, and the splicing step includes:

[0138] Generate masks for the first image and the second image based on the seam, wherein the mask corresponding to each pixel unit on the side of the seam away from the second image in the first image is 1, and the masks corresponding to the remaining pixel units in the first image are 0; and the mask corresponding to each pixel unit on the side of the seam away from the first image in the second image is 1, and the masks corresponding to the remaining pixel units in the second image are 0;

[0139] The pixels on the seam can come from the left or the right and are randomly set. For example, a pixel unit on the seam has a mask of 1 in the first image and a mask of 0 in the second image. Alternatively, a pixel unit on the seam has a mask of 0 in the first image and a mask of 1 in the second image. Alternatively, the masks corresponding to the pixel units where the seam is located in the first and second images are both 1, and the pixels with both masks set to 1 are averaged, i.e., the average of the corresponding pixels in the first image and the second image is taken.

[0140] In another embodiment of determining the pixel mask on the seam, the center line of the overlapping area between the first image and the second image is determined along the extension direction of the seam; if the pixel unit where the seam is located is closer to the first image than the center line, the corresponding mask of the first image in the pixel unit is 1, and the corresponding mask of the second image in the pixel unit is 0.

[0141] After determining the image mask, a bitwise AND logic operation is performed between the image (pixel information) of the pixel unit and the corresponding mask, and the results of the completed logic operation are arranged according to the positional relationship of each unit image to obtain a spliced ​​image.

[0142] The image stitching method provided by the present invention performs image stitching in the first and second directions respectively by the seam method, which can make the transition as natural as possible in the process of fusing two adjacent images. This is particularly important for microscopic imaging because, compared with the stitching between multiple images taken by a conventional camera, the pixel processing at the stitching point of microscopic imaging is particularly critical to the quality of the stitched image. The process of stitching images using the seam network method of this embodiment is shown in Figure 4. The five images in the upper left corner of Figure 4 are the unit images to be stitched on one column. According to the above-mentioned seam determination method and stitching method, multiple unit images are stitched in the column direction to obtain intermediate images. Each column obtains an intermediate image, and there are six intermediate images in the figure. The six intermediate images are then stitched in the row direction to obtain the result image. The image on the lower right side of Figure 4 is the seam network of all image units determined according to the above-mentioned image stitching method, and the image on the lower left side of Figure 4 is the result image obtained according to the above-mentioned image stitching method. Comparing the stitched image and seam network obtained by the prior art shown in Figure 3, it can be seen that the image stitching method provided by the present application can effectively improve the image stitching quality.

[0143] In one embodiment of the present invention, an image processing method is provided, as shown in FIG7 , comprising the following steps:

[0144] Scanning and photographing the object to be imaged to obtain multiple sub-images;

[0145] The multiple sub-images are arranged in rows and columns according to the scanning order to obtain a preview image as shown in FIG5 . The upper left area of ​​FIG5 , that is, the area indicated by the rectangular box in FIG5 , has an obvious unreasonable misalignment phenomenon.

[0146] Determining a region of interest based on the preview image, and determining each sub-image corresponding to the region of interest as a unit image to be stitched;

[0147] The unit images to be stitched are stitched using the image stitching method described in any of the above embodiments to obtain a result image as shown in FIG6 . The quality of the stitched image is significantly improved compared to that in FIG5 .

[0148] In one embodiment of the present invention, an image stitching device is provided, as shown in FIG8 , which includes the following modules:

[0149] a scanning module configured to scan and photograph the object to be imaged to obtain a plurality of sub-images;

[0150] an arrangement module configured to arrange the plurality of unit images to be spliced ​​in rows and columns according to a preset arrangement order;

[0151] a preview module configured to arrange the plurality of sub-images in rows and columns according to a scanning order to obtain a preview image;

[0152] a region of interest module configured to determine a region of interest based on the preview image and determine each sub-image corresponding to the region of interest as a unit image to be stitched; the region of interest may be automatically determined by the region of interest module according to preset rules or requirements, or may be manually marked or determined;

[0153] a sequence setting module configured to determine one of the row arrangement direction and the column arrangement direction as a first direction and the other as a second direction. Specifically, the first direction and the second direction can be determined according to the method for determining the row arrangement direction and the column arrangement direction provided in the above-mentioned embodiment of the image stitching method, which will not be repeated here;

[0154] a first sequential stitching module configured to determine seams between adjacent images in a plurality of unit images in a first direction, and stitch the adjacent unit images according to the seams to obtain a plurality of intermediate images;

[0155] The second sequential stitching module is configured to determine seams between adjacent images in the plurality of intermediate images in the second direction, and stitch the adjacent intermediate images according to the seams to obtain a result image.

[0156] Wherein, the first sequential splicing module and the second sequential splicing module both include the following units:

[0157] An overlapping unit configured to determine an overlapping area between two adjacent images. The overlapping unit is configured to determine the overlapping area between two adjacent images according to one of the multiple methods for determining the overlapping area provided in the above-mentioned image stitching method embodiment, which will not be described in detail.

[0158] A seam-finding unit is configured to find a seam composed of a plurality of pixel units in the overlapping area so as to minimize the difference in pixel units between two adjacent images on a path corresponding to the seam. The seam-finding unit can determine the seam according to any one of the multiple methods for finding a seam composed of a plurality of pixel units in the overlapping area provided in the above-mentioned embodiment of the image stitching method, which will not be described in detail.

[0159] In one embodiment of the present invention, referring to FIG9 , a microscopic imaging system is provided, comprising a microscope and the image stitching device as described in any of the above embodiments.

[0160] Some embodiments relate to a microscope including an apparatus for image scanning and image stitching. Optionally, the microscope can be part of or connected to a system that performs one or more method processes such as those in Figures 1 and 7. Figure 9 shows a schematic diagram of a system 100 configured to perform the methods described herein. System 100 includes a microscope 110 and a computer system 120. Microscope 110 is configured to capture images and is connected to computer system 120. Computer system 120 is configured to perform at least a portion of the methods described herein. Computer system 120 can be configured to execute a machine learning algorithm. Computer system 120 and microscope 110 can be separate entities, but can also be integrated into a common housing. Computer system 120 can be part of the central processing system of microscope 110 and / or computer system 120 can be part of a subassembly of microscope 110, such as a sensor, actuator, camera, or lighting unit of microscope 110.

[0161] The computer system 120 can be a local computer device (e.g., a personal computer, laptop, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or it can be a distributed computer system (e.g., having one or more processors and one or more storage devices distributed across various locations, such as a local client and / or one or more remote server locations and / or data centers). The computer system 120 can include any circuit or combination of circuits. In one embodiment, the computer system 120 can include one or more processors that can be of any type. As used herein, a processor can refer to any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA) such as a microscope or microscope component (e.g., a camera), or any other type of processor or processing circuit. Other types of circuits that can be included in the computer system 120 can be custom circuits, application specific integrated circuits (ASICs), and the like, such as one or more circuits (e.g., communication circuits) used in wireless devices such as mobile phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. The computer system 120 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. The computer system 120 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, trackball, touch screen, voice recognition device, or any other device that allows a system user to input information to or receive information from the computer system 120.

[0162] In one embodiment of the present invention, an electronic device is provided, comprising a processor and a memory, wherein the memory is used to store program instructions, and the processor is configured to run the program instructions, and the program instructions are run to perform the steps performed in the above method embodiment.

[0163] In one embodiment of the present invention, a computer-readable storage medium is provided for storing program instructions, wherein the program instructions are configured to be called to execute the steps performed in the above method embodiment.

[0164] In one embodiment of the present invention, a computer program product is provided, including a readable and stored computer program, wherein the computer program includes program instructions. When the program instructions are executed on a computer device, the computer device executes the steps executed in the above method embodiment.

[0165] It should be noted that the above-mentioned image processing method, image stitching device, microscopic imaging system, electronic device, computer-readable storage medium, computer program product embodiments and the image stitching method embodiments belong to the same inventive concept, and the entire content of the image stitching method embodiments are incorporated into the image processing method, image stitching device, microscopic imaging system, electronic device, computer-readable storage medium, and computer program product embodiments by reference.

[0166] Some or all of the method steps may be performed by (or using) a hardware device (e.g., a processor, a microprocessor, a programmable computer or an electronic circuit). In some embodiments, such a device may perform one or more of the most important method steps.

[0167] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. This implementation may be performed using a non-transitory storage medium (such as a digital storage medium, e.g., a floppy disk, DVD, Blu-ray, CD, ROM, PROM, and EPROM, EEPROM, or FLASH) having stored thereon electronically readable control signals that cooperate (or are capable of cooperating) with a programmable computer system to perform the corresponding method. Thus, the digital storage medium may be computer-readable.

[0168] Some embodiments of the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0169] Generally, embodiments of the present invention can be implemented as a computer program product having a program code, which is operable to perform one of the methods when the computer program product runs on a computer. The program code can, for example, be stored on a machine-readable carrier.

[0170] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0171] In other words, an exemplary embodiment of the present invention is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0172] Therefore, a further embodiment of the present invention is a storage medium (or data carrier or computer-readable medium) comprising a computer program stored thereon, the computer program being operable to perform one of the methods described herein when the computer program is executed by a processor. The data carrier, digital storage medium or recorded medium is generally tangible and / or non-transitory. A further embodiment of the present invention is an apparatus as described herein, comprising a processor and a storage medium.

[0173] Therefore, a further embodiment of the present invention is a data stream or a signal sequence representing the computer program for performing one of the methods described herein. The data stream or signal sequence can be configured to be transmitted via a data communication connection, such as the Internet.

[0174] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0175] A further embodiment comprises a computer on which is installed the computer program for performing one of the methods described herein.

[0176] Yet another embodiment of the present invention includes an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, or the like. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0177] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the method described herein. In some embodiments, the field programmable gate array can cooperate with a microprocessor to perform one of the methods described herein. Generally, the method is preferably performed by any hardware device.

[0178] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0179] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0180] The above description is only a specific embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of this application. These improvements and modifications should also be regarded as the scope of protection of this application.

Claims

1. An image stitching method, characterized in that: The following steps are involved: Arranging the multiple unit images to be spliced ​​in rows and columns according to a preset arrangement order; Determine one of the row arrangement direction and the column arrangement direction as a first direction and the other as a second direction; Determine seams between adjacent images in a plurality of unit images in a first direction, and splice the adjacent unit images according to the seams to obtain a plurality of intermediate images; The seams between adjacent images in the plurality of intermediate images in the second direction are determined, and the adjacent intermediate images are spliced ​​according to the seams to obtain a result image.

2. The image stitching method according to claim 1, characterized in that: The steps of determining the seams between adjacent images include: Determine the overlapping area between two adjacent images; A seam consisting of a plurality of pixel units is found in the overlapping area so that the pixel difference between two adjacent images on a path corresponding to the seam is minimized.

3. The image stitching method according to claim 2, characterized in that: The overlapping area is determined by: The image positioning algorithm is used to locate the position of each unit image in the global image, and the overlapping area between two adjacent images is determined according to the located position.

4. The image stitching method according to claim 2 or 3, characterized in that: The unit image is obtained by scanning a scanning device, and the overlapping area is determined by the following method: According to the preset scanning configuration, the position of each unit image in the global image is obtained, and then the overlapping area between two adjacent images is determined.

5. The image stitching method according to any one of claims 2 to 4, characterized in that: The steps of finding the seam in the overlapping area include: For each pixel unit in the overlapping area, the difference of the pixel unit is calculated according to the image data of the two adjacent images in the pixel unit, and a difference table corresponding to the overlapping area is obtained; Planning a path composed of adjacent pixel units based on the difference table, and satisfying that the sum of the differences of the pixel units on the path is minimum; The planned path is mapped to the overlapping area to obtain the seam.

6. The image stitching method according to claim 5, characterized in that: The difference of a pixel unit is calculated as follows: Get the channel data R / G / B value of the adjacent first image in the pixel unit and the channel data R / G / B value of the adjacent another image in the pixel unit Channel data R / G / B value; respectively calculating an R value difference, a G value difference and a B value difference between the first image and another image at the pixel unit; The difference of the pixel unit is calculated according to the R value difference, the G value difference and the B value difference.

7. The image stitching method according to claim 5 or 6, characterized in that: Plan your route by: Starting from the first row or the first column of the difference table, according to the extension direction of the seam, the difference accumulation value corresponding to each unit in the difference table is calculated row by row or column by column; Determine the unit with the smallest cumulative difference value in the last row or last column as the starting point for backtracking; From the backtracking starting point, backtrack along the extension direction of the seam to find its formation path in the difference degree table as the planned path.

8. The image stitching method according to claim 7, characterized in that: The method of backtracking from the backtracking starting point to find the path formed in the difference table is: The backtracking starting point is the unit with the smallest cumulative difference value in the last row; taking the backtracking starting point as the first path unit, the path units of each row are determined row by row from the last row to the first row, wherein the path unit of each row is a unit with the smallest cumulative difference value among two or more units adjacent to the path unit of the previous row; Alternatively, the backtracking starting point is the unit with the smallest cumulative difference value in the last column; taking the backtracking starting point as the first path unit, the path units of each column are determined column by column from the last column to the first column, wherein the path unit of each column is a unit with the smallest cumulative difference value among two or more units adjacent to the path unit of the previous column.

9. The image stitching method according to any one of claims 2 to 8, characterized in that: Using a graph cut algorithm to find the seam in the overlapping area, the steps include: The image of the overlapping area is represented as a graph, where the nodes of the graph represent pixels in the image and the edges of the graph represent the relationship between the pixels; Assign weights to the nodes and edges of the graph; A cut is found on the graph by minimizing a cost function of the cut, and the graph is divided into two parts, the foreground and the background. The cost function is composed of factors including similarity between pixels.

10. The image stitching method according to any one of claims 2 to 9, characterized in that: Finding the seam in the overlapping area using the Voronoi diagram method comprises the following steps: Generate Voronoi diagrams for each unit image to be spliced; The seam positions within the overlapping region are determined by stitching the line network according to the Voronoi diagram.

11. The image stitching method according to any one of claims 1 to 10, characterized in that: The step of splicing adjacent first and second images according to the seam includes: According to the seam, a mask of each of the first image and the second image is generated, wherein the mask located at the seam in the first image The mask corresponding to each pixel unit on the side away from the second image is 1, and the mask corresponding to the remaining pixel units in the first image is 0; the mask corresponding to each pixel unit on the side of the seam away from the first image in the second image is 1, and the mask corresponding to the remaining pixel units in the second image is 0; A bitwise AND logic operation is performed between the image of the pixel unit and the corresponding mask, and the result of the logic operation is arranged according to the positional relationship of each unit image to obtain a spliced ​​image.

12. The image stitching method according to claim 11, characterized in that: The mask of the first image and the second image at the seam is determined by: A pixel unit where the seam is located has a mask of 1 in one of the first image and the second image, and has a mask of 0 in the other image; Alternatively, masks corresponding to pixel units where the seams in the first image and the second image are located are all 1.

13. The image stitching method according to any one of claims 1 to 12, characterized in that: The first direction and the second direction are determined by: The number of columns of the image to be spliced ​​is calculated by multiplying the pixel value of the single image column by the number of columns of the image to be spliced, and the number of rows of the image to be spliced ​​is calculated by multiplying the pixel value of the single image row by the number of rows of the image to be spliced, and the second product is obtained; If the first product is smaller than the second product, determining the column arrangement direction as the first direction, and determining the row arrangement direction as the second direction; If the second product is smaller than the first product, the row arrangement direction is determined to be the first direction, and the column arrangement direction is determined to be the second direction.

14. The image stitching method according to any one of claims 1 to 13, characterized in that: The plurality of unit images to be spliced ​​are obtained by scanning an object by a scanning device in a preset order; The unit images are arranged in rows and columns according to the scanned row and column order, and the multiple unit images to be spliced ​​are arranged in more than two rows and more than two columns.

15. An image processing method, characterized in that: The following steps are involved: Scanning and photographing the object to be imaged to obtain multiple sub-images; Arranging the multiple sub-images in rows and columns according to the scanning order to obtain a preview image; Based on the preview image, determine a region of interest, and determine each sub-image corresponding to the region of interest as a unit image to be stitched; The unit images to be stitched are stitched using the image stitching method according to any one of claims 1 to 14.

16. An image stitching device, characterized in that: The device is configured to: Arranging the multiple unit images to be spliced ​​in rows and columns according to a preset arrangement order; Determine one of the row arrangement direction and the column arrangement direction as a first direction and the other as a second direction; Determine the seams between adjacent images in a plurality of unit images in a first direction, and perform separation of adjacent unit images according to the seams. Perform stitching to obtain multiple intermediate images; The seams between adjacent images in the plurality of intermediate images in the second direction are determined, and the adjacent intermediate images are spliced ​​according to the seams to obtain a result image.

17. The image stitching device according to claim 16, characterized in that: The apparatus is further configured to determine the seam by: Determine the overlapping area between two adjacent images; A seam composed of a plurality of pixel units is found in the overlapping area, so that the difference in pixel units of two adjacent images on a path corresponding to the seam is minimized.

18. The image stitching device according to claim 17, characterized in that: The apparatus is configured to find the seam within the overlap region by: For each pixel unit in the overlapping area, the difference of the pixel unit is calculated according to the image data of the two adjacent images in the pixel unit, and a difference table corresponding to the overlapping area is obtained; Planning a path composed of adjacent pixel units based on the difference table, and satisfying that the sum of the differences of the pixel units on the path is minimum; The planned path is mapped to the overlapping area to obtain the seam.

19. The image stitching device according to any one of claims 16 to 18, characterized in that: The device is also configured to determine the unit images to be spliced ​​by the following steps: Scanning and photographing the object to be imaged to obtain multiple sub-images; Arranging the multiple sub-images in rows and columns according to the scanning order to obtain a preview image; A region of interest is determined according to the preview image, and each sub-image corresponding to the region of interest is determined as a unit image to be spliced.

20. A microscopic imaging system, characterized in that: The invention comprises a microscope and an image stitching device as claimed in any one of claims 16 to 19.

21. An electronic device comprising a processor and a memory, wherein: The memory is used to store program instructions, and the processor is configured to execute the program instructions, wherein the program instructions are executed to perform the steps of the method according to any one of claims 1 to 15.

22. A computer-readable storage medium for storing program instructions, characterized in that: The program instructions are configured to be called to execute the steps of the method according to any one of claims 1 to 15.

23. A computer program product, comprising a readable and stored computer program, the computer program comprising program instructions, characterized in that: When the program instructions are executed on a computer device, the computer device executes the steps of the method according to any one of claims 1 to 15.

Images